[Hadoop] 50030/jobtracker.jsp监控不到eclipse提交的作业
2013-01-14 22:18
465 查看
搭建好eclipse hadoop开发环境后,一般会用hadoop自带的WordCount做个测试,其自带的WordCount.java如下:
如果此时你run java aplication,你的程序只会在eclipse中虚拟的一个云环境中运行,而不会跑上云端去运行,所以无法再master:50070/jobtracker.jsp页面中监控到该作业;需要在main方法中添加几行代码,代码附录如下:
package org.apache.hadoop.examples; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; import org.apache.hadoop.util.GenericOptionsParser; import org.apache.log4j.PropertyConfigurator; public class WordCount { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); //set inpath,outpath Path inpath=new Path("/test/input/file1"); Path outpath=new Path("/test/output/output4"); Job job = new Job(conf, "mango"); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); //job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, inpath); FileOutputFormat.setOutputPath(job,outpath); System.out.println("开始执行"); System.exit(job.waitForCompletion(true) ? 0 : 1); } }
如果此时你run java aplication,你的程序只会在eclipse中虚拟的一个云环境中运行,而不会跑上云端去运行,所以无法再master:50070/jobtracker.jsp页面中监控到该作业;需要在main方法中添加几行代码,代码附录如下:
//在你的文件地址前自动添加:hdfs://master:9000/ conf.set("fs.default.name", "hdfs://master:9000/"); conf.set("hadoop.job.user","mango"); //指定jobtracker的ip和端口号,master在/etc/hosts中可以配置 conf.set("mapred.job.tracker","master:9001");
相关文章推荐
- 50030(jobtracker.jsp)监控不到eclipse提交的作业
- 【Hadoop代码笔记】Hadoop作业提交之JobTracker接收作业提交
- hadoop任务监控页面namenode:50030(在hadoop配置中查找集群jobtracker的ip,访问50030)
- hadoop任务监控页面namenode:50030(在hadoop配置中查找集群jobtracker的ip,访问50030)
- 学习总结八:在eclipse中的写的代码如何提交作业到JobTracker中的?
- 【Hadoop代码笔记】Hadoop作业提交之JobTracker等相关功能模块初始化
- Hadoop JobTracker提交job源码浅析
- Win7下eclipse提交Job到hadoop集群
- linux 系统 eclipse提交job到hadoop集群上的一些坑
- Win7上eclipse无插件提交Hadoop2.2分布式作业
- 【Mapred】jobtracker & tasktracker架构作业是怎么提交的
- 在win7上的eclipse向hadoop提交作业异常-权限/设置调度器
- MapReduce源码解读系列之——作业如何提交到JobTracker
- 【Hadoop代码笔记】Hadoop作业提交之Job初始化
- Yarn客户端能向jobtracker和ResourceManager提交作业
- MapReduce源码分析之Eclipse中的代码如何提交给JobTracker
- hadoop命令行运行wordcount程序 web端监控不到作业状态
- hadoop job -kill 与 yarn application -kii(作业卡了或作业重复提交或MapReduce任务运行到running job卡住)
- window7使用eclipse提交Hadoop作业到Hadoop集群运行方法
- eclipse连接hadoop报错Unknown protocol to job tracker: org.apache.hadoop.hdfs.protocol.ClientProtocol