mapReduce排序 笔记
2014-09-05 11:28
183 查看
在 mymapper 中加入要排序的类
static class MyMapper extends Mapper<LongWritable, Text, NewK2, LongWritable>{
protected void map(LongWritable key, Text value, org.apache.hadoop.mapreduce.Mapper<LongWritable,Text,NewK2,LongWritable>.Context context) throws java.io.IOException ,InterruptedException {
final String[] splited = value.toString().split("\t");
final NewK2 k2 = new NewK2(Long.parseLong(splited[0]), Long.parseLong(splited[1]));
final LongWritable v2 = new LongWritable(Long.parseLong(splited[1]));
context.write(k2, v2);
};
}
newk2 要继承 WritableComparable
static class NewK2 implements WritableComparable<NewK2>{
Long first;
Long second;
public NewK2(){}
public NewK2(long first, long second){
this.first = first;
this.second = second;
}
@Override
public void readFields(DataInput in) throws IOException {
this.first = in.readLong();
this.second = in.readLong();
}
@Override
public void write(DataOutput out) throws IOException {
out.writeLong(first);
out.writeLong(second);
}
/**
* 当k2进行排序时,会调用该方法.
* 当第一列不同时,升序;当第一列相同时,第二列升序
*/
@Override
public int compareTo(NewK2 o) {
final long minus = o.first - this.first;
if(minus !=0){
return (int)minus;
}
return (int)(this.second - o.second);
}
@Override
public int hashCode() {
return this.first.hashCode()+this.second.hashCode();
}
@Override
public boolean equals(Object obj) {
if(!(obj instanceof NewK2)){
return false;
}
NewK2 oK2 = (NewK2)obj;
return (this.first==oK2.first)&&(this.second==oK2.second);
}
}
问:为什么实现该类?
答:因为原来的v2不能参与排序,把原来的k2和v2封装到一个类中,作为新的k2
static class MyMapper extends Mapper<LongWritable, Text, NewK2, LongWritable>{
protected void map(LongWritable key, Text value, org.apache.hadoop.mapreduce.Mapper<LongWritable,Text,NewK2,LongWritable>.Context context) throws java.io.IOException ,InterruptedException {
final String[] splited = value.toString().split("\t");
final NewK2 k2 = new NewK2(Long.parseLong(splited[0]), Long.parseLong(splited[1]));
final LongWritable v2 = new LongWritable(Long.parseLong(splited[1]));
context.write(k2, v2);
};
}
newk2 要继承 WritableComparable
static class NewK2 implements WritableComparable<NewK2>{
Long first;
Long second;
public NewK2(){}
public NewK2(long first, long second){
this.first = first;
this.second = second;
}
@Override
public void readFields(DataInput in) throws IOException {
this.first = in.readLong();
this.second = in.readLong();
}
@Override
public void write(DataOutput out) throws IOException {
out.writeLong(first);
out.writeLong(second);
}
/**
* 当k2进行排序时,会调用该方法.
* 当第一列不同时,升序;当第一列相同时,第二列升序
*/
@Override
public int compareTo(NewK2 o) {
final long minus = o.first - this.first;
if(minus !=0){
return (int)minus;
}
return (int)(this.second - o.second);
}
@Override
public int hashCode() {
return this.first.hashCode()+this.second.hashCode();
}
@Override
public boolean equals(Object obj) {
if(!(obj instanceof NewK2)){
return false;
}
NewK2 oK2 = (NewK2)obj;
return (this.first==oK2.first)&&(this.second==oK2.second);
}
}
问:为什么实现该类?
答:因为原来的v2不能参与排序,把原来的k2和v2封装到一个类中,作为新的k2
相关文章推荐
- Hadoop阅读笔记(三)——深入MapReduce排序和单表连接
- Hadoop学习笔记: MapReduce二次排序
- Hadoop学习笔记(二):MapReduce的特性-计数器、排序
- Hadoop之MapReduce中的排序和分组(笔记11)
- Hadoop学习笔记(二):MapReduce的特性-计数器、排序
- Hadoop 学习笔记 (十) MapReduce实现排序 全局变量
- SPGridView 研究笔记 Part 2 - 排序与过滤
- c++学习笔记:直接排序
- SPGridView 研究笔记 Part 2 - 排序与过滤
- Asp.net学习笔记----使用GridView+ObjectDataSource进行自定义分页排序
- 数据结构 笔记(一) 快排序,shell排序
- LINQ 学习笔记-字符串数组与整数数组之间的转换,排序
- 数据结构笔记--排序
- PL/SQL 学习笔记(一)----- pl/sql 基本查询与排序
- Silverlight学习笔记二:DataGrid 服务器端分页、排序的实现
- SSH学习笔记之hibernate中的set排序以及struts中的token
- JavaScript学习笔记9--表格排序
- Lucene.net实现自定义排序笔记
- 烙饼排序问题最优次数求解 暑期学习笔记(九)
- 算法学习笔记--排序(简介)