Hadoop 统计文件中某个单词出现的次数
2013-07-11 21:18
513 查看
如文件word.txt内容如下:
what is you name?
my name is zhang san。
要求统计word.txt中出现“is”的次数?
代码如下:
PerWordMapper
PerWordReducer
PerWordCount
what is you name?
my name is zhang san。
要求统计word.txt中出现“is”的次数?
代码如下:
PerWordMapper
package com.hadoop.wordcount; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class PerWordMapper extends Mapper<Object, Text, Text, IntWritable> { public Text keyText = new Text(); public IntWritable intValue = new IntWritable(1); @Override protected void map(Object key, Text value, Context context) throws IOException, InterruptedException { String str = value.toString(); StringTokenizer to = new StringTokenizer(str); while (to.hasMoreTokens()) { String t = to.nextToken(); //此处为判断统计字符串的地方 if(t.equals("is")){ keyText = new Text(t); context.write(keyText, intValue); } } } }
PerWordReducer
package com.hadoop.wordcount; import java.io.IOException; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; public class PerWordReducer extends Reducer<Text, IntWritable, Text, IntWritable> { public IntWritable intValue = new IntWritable(0); @Override protected void reduce(Text key, Iterable<IntWritable> value, Context context) throws IOException, InterruptedException { int sum = 0; while(value.iterator().hasNext()){ sum += value.iterator().next().get(); } intValue.set(sum); context.write(key, intValue); } }
PerWordCount
package com.hadoop.wordcount; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.util.GenericOptionsParser; import com.hadoop.mapreducer.MapperClass; import com.hadoop.mapreducer.ReducerClass; import com.hadoop.mapreducer.WordCount; public class PerWordCount { public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); System.out.println("otherArgs.length:"+otherArgs.length); if (otherArgs.length != 2) { System.err.println("Usage: wordcount <in> <out>"); System.exit(2); } Job job = new Job(conf, "word count"); job.setJarByClass(PerWordCount.class); job.setMapperClass(PerWordMapper.class); job.setCombinerClass(PerWordReducer.class); job.setReducerClass(PerWordReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }
相关文章推荐
- Hadoop 统计文件中某个单词出现的次数
- Hadoop 统计文件中单词出现的次数
- 大数据第8天----------------------文件中统计单词出现的次数
- C++从一个文件中统计所有出现过的单词,并按次数从大到小输出
- 统计文本文件中单词出现次数最多的单词
- sort +awk+uniq 统计文件中出现次数最多的前10个单词
- Java实现读取键盘输入保存到txt文件,再统计并输出每个单词出现次数的方法
- 统计文件中出现次数最多的前10个单词
- Linux命令经典面试题:统计文件中出现次数最多的前10个单词
- 文件夹下有子文件夹又有.txt,子文件夹又有子文件夹又有.txt和子文件夹,把txt文件上面的单词全打出来并统计相同单词出现次数
- 统计英文文件中单词数和各单词出现的频率(次数)
- Spark学习—统计文件单词出现次数
- 10025---sort +awk+uniq 统计文件中出现次数最多的前10个单词
- 统计文件中出现次数最多的前10个单词
- wc:统计一个文件里出现某个单词出现的次数
- 统计文件中某个单词出现的次数
- java 读取键盘输入到txt文件,统计每个单词出现的次数并输出
- sort +awk+uniq 统计文件中出现次数最多的前10个单词
- linux中sort(统计文件中出现次数最多的前10个单词)
- 统计文件中每个单词出现的次数