[Mapreduce]eclipse下写wordcount
2014-11-10 13:28
260 查看
上传两个文件到hdfs上的input文件夹下
代码如下:
注:eclipse下初次运行wordcount可能会有log4j警告,可以在src下建立名为log4j.properties的文件,即可消除警告,内容如下:
代码如下:
import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class wordcount { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "word count"); job.setJarByClass(wordcount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path("hdfs://localhost:9000/input")); //这里直接指定了文件路径.在run configuration下指定也行,但没配置好 FileOutputFormat.setOutputPath(job, new Path("hdfs://localhost:9000/output1")); System.exit(job.waitForCompletion(true) ? 0 : 1); } }
注:eclipse下初次运行wordcount可能会有log4j警告,可以在src下建立名为log4j.properties的文件,即可消除警告,内容如下:
log4j.rootLogger=INFO,Console log4j.appender.Console=org.apache.log4j.ConsoleAppender log4j.appender.Console.Target=System.out log4j.appender.Console.Threshold=DEBUG log4j.appender.Console.layout=org.apache.log4j.PatternLayout log4j.appender.Console.layout.ConversionPattern=[%d]%l%5p:%m%n log4j.appender.DebugFile=org.apache.log4j.RollingFileAppender log4j.appender.DebugFile.File=../log/debugFile.log #log4j.appender.DebugFile.File=debugFile.log log4j.appender.DebugFile.Append=true log4j.appender.DebugFile.Threshold=DEBUG log4j.appender.DebugFile.layout=org.apache.log4j.PatternLayout log4j.appender.DebugFile.layout.ConversionPattern=[%d]%l%5p:%m%n log4j.appender.DebugFile.MaxFileSize=20MB log4j.appender.DebugFile.MaxBackupIndex=10 log4j.logger.com.ibatis=DEBUG log4j.logger.com.ibatis.common.jdbc.SimpleDataSource=DEBUG log4j.logger.com.ibatis.common.jdbc.ScriptRunner=DEBUG log4j.logger.com.ibatis.sqlmap.engine.impl.SqlMapClientDelegate=DEBUG log4j.logger.java.sql=DEBUG log4j.logger.java.sql.Connection = INFO log4j.logger.java.sql.Statement = DEBUG log4j.logger.java.sql.PreparedStatement = DEBUG log4j.logger.java.sql.ResultSet = DEBUG log4j.logger.com.yuetao=DEBUG
相关文章推荐
- Windows 使用Eclipse配置连接hadoop,编译运行MapReduce --本地调试WordCount
- eclipse 编写mapreduce程序(wordCount)
- MapReduce: WordCount的Eclipse实现
- Hadoop 用Eclipse来Mapreduce WordCount实战(1)
- Eclipse下运行hadoop自带的mapreduce程序--wordcount
- [Mapreduce]eclipse下写wordcount
- 在eclipse上搭建mapreduce开发环境及运行wordcount
- Hadoop 用Eclipse来MapReduce WordCount实战 (2)
- 在eclipse上运行MapReduce的wordcount程序所遇到的问题
- Ubuntu14.04中eclipse下编写mapreduce例子程序WordCount
- hadoop学习之HDFS(2.1):linux下eclipse中配置hadoop-mapreduce开发环境并运行WordCount.java程序
- MapReduce的WordCount应用实例
- [hadoop源码阅读][9]-mapreduce-从wordcount开始
- linux下jar命令和eclipse两种方式生成wordcount.jar包和hadoop下wordcount实例的运行
- Eclipse 运行WordCount实例 (连接Linux下的Hadoop集群)
- Mapreduce 测试自带实例 wordcount
- Hadoop eclipse插件安装和在eclipse运行wordcount程序
- Hadoop学习笔记之初识MapReduce以及WordCount实例分析
- hadoop hdfs搭建 mapreduce环境搭建 wordcount程序简单注释
- MapReduce实验之WordCount