应用程序通过对键中的气温进行排序来找出最高气温
2015-09-30 20:11
369 查看
public class MaxTemperatureUsingSecondarySort extends Configured implements Tools{
static class MaxTemperatureMapper extends Mapper<LongWritable, Text, IntPair, NullWritable>{
private NcdcRecordParer parser = new NcdcRecordParser();
@override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{
parser.parse(value)
if(parser.isWalidTemperature()){
context.write(new IntPair(parser.getYearInt(), parser.getAirTemperature()), NullWritable.get());
}
}
}
static class MaxTemperatureReducer extends Reducer<IntPair, NullWritable, IntPair,NullWritable>{
@Ovrride
protected void reduce(IntPair key, Iterable<NullWritable> values, Context coontext) throws IOException ,InterruptedException{
context.write(key, NullWritable.get());
}
}
public static class FirstPartitioner extends Partitioner<IntPair, NullWritableX>{
@Override
public int getPartition(IntPair key, NullWritable value, int numPartitions){
//multiply by 127 to perform some mixing
return Math.abs(key.getFirst() * 127 ) % numPartitions;
}
}
public static class keyComparator extends WritableComparator{
protected KeyComparator(){
supper(IntPair.class, true);
}
@Override
public int compare(WritableComparable w1,Writab;leComparable w2){
IntPair ip1 =(IntPair) w1;
IntPair ip2 =(IntPair) w2;
int cmp = IntPair.compare(ip1.getFirst(), ip2.getFilrst());
if(cmp!=0)
return cmp
return -IntPair.compare(ip1.getSecond(), ip2.getSeconde());
}
}
public static class GroupComparator extends WritableComparator{
protected GroupComparator (){
super(IntPair.class, true);
}
@Ovrride
public int compare(WritableComparable w1, WritableComparable w2 ){
IntPair ip1 =(IntPair) w1;
IntPair ip2 =(IntPair) w2;
int cmp = IntPair.compare(ip1.getFirst(), ip2.getFilrst());
if(cmp!=0)
return cmp
}
}
@Override
public int run(String[] args){
Job job = new Job();
job.setMapperClass(MaxTemperatureMapper.class)
job.setPartitionerClass(FirstPartitioner.class)
job.setSortComparatorClass(KeyComparator.class)
job.setGroupingComparatorClass(GroupComparator.class)
job.setReducerClass(MaxTemp (eratureReducer.class)
job.setOutputKeyClass(IntPair.class);
job.setOutputValueClass(NullWritable.class)
return job.waitForCompletion(true0 ? 0:1;
====================================================================================================
在上述的mapper中我们利用IntPair定义了一个代表年份和气温的组合键, 该类实现了Writable接口
,IntPair于TextPair类相似,后者可以传递Text,由于可以根据各个reduce的组合件获取最高气温
因此无需要在值上附加其他信息, 使用NullWritable即可, 根据辅助排序, reduce输出第一个键就是包含年份和最高气温的信息的IntPair对象。
IntPair的toString()方法返沪iyige以制表符分割的字符串,因而该陈旭输出一组由制表符分割的年份、气温对
我们创建了一个自定义的partitioner以按照组合键的首字段(年份)进行分区,即FirstPartitioner
为了按照年份(升序)和气温(降序)排列键,我们使用setSortComparatorClass 设置了一个自定义键 comparator(即KeyComparator)
以抽取字段并执行比较操作。类似的
为了按照年份对键进行分组,我们使用SetGroupingComparatorClass来自定义一个分组comparator, 只取键的首字段进行比较。
运行这个程序 ,返回各年的最高气温
hadoop jar hadoop-examples.jar MaxTemperatureUsingSecondarySort input/ncdc/all output-secondarysort
hadoop fs -cat output-secondarysort/part-* | sort | head
hadoop权威指南 page= 302
static class MaxTemperatureMapper extends Mapper<LongWritable, Text, IntPair, NullWritable>{
private NcdcRecordParer parser = new NcdcRecordParser();
@override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{
parser.parse(value)
if(parser.isWalidTemperature()){
context.write(new IntPair(parser.getYearInt(), parser.getAirTemperature()), NullWritable.get());
}
}
}
static class MaxTemperatureReducer extends Reducer<IntPair, NullWritable, IntPair,NullWritable>{
@Ovrride
protected void reduce(IntPair key, Iterable<NullWritable> values, Context coontext) throws IOException ,InterruptedException{
context.write(key, NullWritable.get());
}
}
public static class FirstPartitioner extends Partitioner<IntPair, NullWritableX>{
@Override
public int getPartition(IntPair key, NullWritable value, int numPartitions){
//multiply by 127 to perform some mixing
return Math.abs(key.getFirst() * 127 ) % numPartitions;
}
}
public static class keyComparator extends WritableComparator{
protected KeyComparator(){
supper(IntPair.class, true);
}
@Override
public int compare(WritableComparable w1,Writab;leComparable w2){
IntPair ip1 =(IntPair) w1;
IntPair ip2 =(IntPair) w2;
int cmp = IntPair.compare(ip1.getFirst(), ip2.getFilrst());
if(cmp!=0)
return cmp
return -IntPair.compare(ip1.getSecond(), ip2.getSeconde());
}
}
public static class GroupComparator extends WritableComparator{
protected GroupComparator (){
super(IntPair.class, true);
}
@Ovrride
public int compare(WritableComparable w1, WritableComparable w2 ){
IntPair ip1 =(IntPair) w1;
IntPair ip2 =(IntPair) w2;
int cmp = IntPair.compare(ip1.getFirst(), ip2.getFilrst());
if(cmp!=0)
return cmp
}
}
@Override
public int run(String[] args){
Job job = new Job();
job.setMapperClass(MaxTemperatureMapper.class)
job.setPartitionerClass(FirstPartitioner.class)
job.setSortComparatorClass(KeyComparator.class)
job.setGroupingComparatorClass(GroupComparator.class)
job.setReducerClass(MaxTemp (eratureReducer.class)
job.setOutputKeyClass(IntPair.class);
job.setOutputValueClass(NullWritable.class)
return job.waitForCompletion(true0 ? 0:1;
====================================================================================================
在上述的mapper中我们利用IntPair定义了一个代表年份和气温的组合键, 该类实现了Writable接口
,IntPair于TextPair类相似,后者可以传递Text,由于可以根据各个reduce的组合件获取最高气温
因此无需要在值上附加其他信息, 使用NullWritable即可, 根据辅助排序, reduce输出第一个键就是包含年份和最高气温的信息的IntPair对象。
IntPair的toString()方法返沪iyige以制表符分割的字符串,因而该陈旭输出一组由制表符分割的年份、气温对
我们创建了一个自定义的partitioner以按照组合键的首字段(年份)进行分区,即FirstPartitioner
为了按照年份(升序)和气温(降序)排列键,我们使用setSortComparatorClass 设置了一个自定义键 comparator(即KeyComparator)
以抽取字段并执行比较操作。类似的
为了按照年份对键进行分组,我们使用SetGroupingComparatorClass来自定义一个分组comparator, 只取键的首字段进行比较。
运行这个程序 ,返回各年的最高气温
hadoop jar hadoop-examples.jar MaxTemperatureUsingSecondarySort input/ncdc/all output-secondarysort
hadoop fs -cat output-secondarysort/part-* | sort | head
hadoop权威指南 page= 302
相关文章推荐
- 远程推送
- 原 ExpandableListView的用法
- Socket.IO学习之基础入门
- 汇编笔记 CALL(1)
- ocp-9
- ocp-8
- ANR异常产生的原因及其解决办法
- 【错误处理】mysql远程连接ERROR 2003 (HY000): Can't connect to MySQL server on '10.211.55.6' (113)
- 找出数组中第K大的数
- URAL - 1966 - Cycling Roads(并检查集合 + 判刑线相交)
- ocp-7
- 剑指offer—二进制中1的个数
- effective c++ 复制对象时勿忘每一个成分
- 实现只响应第一次按钮单击,第二次不响应
- 当数据类型为long的变量后面没有L加会有什么后果
- 意图(Intent)讲解1:安卓使用意图(Intent)通用方法传递数据
- 个人博客作业2--代码规范和代码复审
- DNS子域授权及view(三)
- 欢迎使用CSDN-markdown编辑器
- ocp-006