MapReduce之wordcount
2017-05-27 11:17
134 查看
Step 1:
导入Hadoop中MapReduce的所有jar包
Step 2:WordCount 的Mapper
Step 3:WordCount的Reducer
Step 4:WordCount的Main—RunJob
导入Hadoop中MapReduce的所有jar包
Step 2:WordCount 的Mapper
public class WCMapper extends Mapper<LongWritable, Text, Text, IntWritable>{ IntWritable v = new IntWritable(1); @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String tmp = value.toString(); String[] arr = tmp.split(" "); for(String s : arr){ if(!"".equals(s)){ Text k = new Text(s); context.write(k, v); } } } }
Step 3:WordCount的Reducer
public class WCReducer extends Reducer<Text, IntWritable, Text, IntWritable>{ @Override protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for(IntWritable value : values){ sum += value.get(); } context.write(key, new IntWritable(sum)); } }
Step 4:WordCount的Main—RunJob
public class RunJob { public static void main(String[] args) { try { Configuration conf = new Configuration(); FileSystem fs = FileSystem.newInstance(conf); Job job = Job.getInstance(); job.setJarByClass(RunJob.class); job.setJobName("wordcount"); // job.setInputFormatClass(KeyValueTextInputFormat.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class); job.setMapperClass(WCMapper.class); job.setReducerClass(WCReducer.class); job.setCombinerClass(WCReducer.class); job.setNumReduceTasks(3); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); // job.setCombinerClass(cls); FileInputFormat.addInputPath(job, new Path("/data/")); Path output = new Path("/wc"); if(fs.exists(output)){ fs.delete(output, true); } FileOutputFormat.setOutputPath(job, output); boolean flag = job.waitForCompletion(true); if(flag){ System.out.println("Job finished !"); } } catch (Exception e) { e.printStackTrace(); } } }
相关文章推荐
- 第一次写MapReduce之WordCount实例
- Hadoop MapReduce初窥-wordcount示例
- mapreduce-wordcount例子
- Ubuntu14.04中eclipse下编写mapreduce例子程序WordCount
- AWS EMR运行MAPREDUCE程序-WORDCOUNT
- MapReduce 的简单例子 WordCount的实现
- MapReduce实现WordCount, 及其优化
- Hadoop基础教程-第6章 MapReduce入门(6.3 加速WordCount)(草稿)
- mapreduce程序编写(WordCount)
- MapReduce程序初探 -------------- WordCount
- hadoop之MapReduce WordCount分析
- MapReduce中wordcount详细介绍(包括Hadoop1和Hadoop2版本)
- MapReduce之WordCount
- hadoop基础----hadoop实战(三)-----hadoop运行MapReduce---对单词进行统计--经典的自带例子wordcount
- eclipse 编写mapreduce程序(wordCount)
- Hadoop 用Eclipse来Mapreduce WordCount实战(1)
- MapReduce之WordCount单词计数(上)
- [Mapreduce]eclipse下写wordcount
- MapReduce WordCount 编码实战
- hadoop学习之HDFS(2.1):linux下eclipse中配置hadoop-mapreduce开发环境并运行WordCount.java程序