hadoop jobhistory
2016-03-18 14:20
204 查看
hadoop jobhistory记录下已运行完的MapReduce作业信息并存放在指定的HDFS目录下,默认情况下是没有启动的,需要配置完后手工启动服务。
mapred-site.xml添加如下配置
启动history-server:
停止history-server:
history-server启动之后,可以通过浏览器访问WEBUI: hadoop000:19888
在hdfs上会生成两个目录
mapreduce.jobhistory.done-dir(/history/done): Directory where history files are managed by the MR JobHistory Server(已完成作业信息)
mapreduce.jobhistory.intermediate-done-dir(/history/done_intermediate): Directory where history files are written by MapReduce jobs.(正在运行作业信息)
测试:
通过hive查询city表观察hdfs文件目录和hadoop000:19888
观察hdfs文件目录:
1)历史作业记录是按照年/月/日的形式分别存放在相应的目录(/history/done/2014/10/11/000000);
2)每个作业有2个不同的后缀名的记录:jhist和xml
观察WEBUI: hadoop000:19888
在WEBUI中展现了每个job使用的Map/Reduce的数量、作业提交时间、作业启动时间、作业完成时间、Job ID、提交人User、队列等信息;
点击【job_1413011730351_0002】弹出页面显示类似信息:Aggregation is not enabled. Try the nodemanager at ......
解决方法: yarn-site.xml添加如下配置
重启yarn即可。
参考CDH文档:http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.3.0-cdh5.0.0/hadoop-project-dist/hadoop-common/ClusterSetup.html
mapred-site.xml添加如下配置
<property> <name>mapreduce.jobhistory.address</name> <value>hadoop000:10020</value> <description>MapReduce JobHistory Server IPC host:port</description> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>hadoop000:19888</value> <description>MapReduce JobHistory Server Web UI host:port</description> </property> <property> <name>mapreduce.jobhistory.done-dir</name> <value>/history/done</value> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/history/done_intermediate</value></property>
启动history-server:
$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver
停止history-server:
$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh stop historyserver
history-server启动之后,可以通过浏览器访问WEBUI: hadoop000:19888
在hdfs上会生成两个目录
hadoop fs -ls /history drwxrwx--- - spark supergroup 0 2014-10-11 15:11 /history/done drwxrwxrwt - spark supergroup 0 2014-10-11 15:16 /history/done_intermediate
mapreduce.jobhistory.done-dir(/history/done): Directory where history files are managed by the MR JobHistory Server(已完成作业信息)
mapreduce.jobhistory.intermediate-done-dir(/history/done_intermediate): Directory where history files are written by MapReduce jobs.(正在运行作业信息)
测试:
通过hive查询city表观察hdfs文件目录和hadoop000:19888
hive> select id, name from city;
观察hdfs文件目录:
1)历史作业记录是按照年/月/日的形式分别存放在相应的目录(/history/done/2014/10/11/000000);
2)每个作业有2个不同的后缀名的记录:jhist和xml
hadoop fs -ls /history/done/2014/10/11/000000 -rwxrwx--- 1 spark supergroup 22572 2014-10-11 15:23 /history/done/2014/10/11/000000/job_1413011730351_0002-1413012208648-spark-select+id%2C+name+from+city%28Stage%2D1%29-1413012224777-1-0-SUCCEEDED-root.spark-1413012216261.jhist -rwxrwx--- 1 spark supergroup 160149 2014-10-11 15:23 /history/done/2014/10/11/000000/job_1413011730351_0002_conf.xml
观察WEBUI: hadoop000:19888
在WEBUI中展现了每个job使用的Map/Reduce的数量、作业提交时间、作业启动时间、作业完成时间、Job ID、提交人User、队列等信息;
点击【job_1413011730351_0002】弹出页面显示类似信息:Aggregation is not enabled. Try the nodemanager at ......
解决方法: yarn-site.xml添加如下配置
<property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
重启yarn即可。
参考CDH文档:http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.3.0-cdh5.0.0/hadoop-project-dist/hadoop-common/ClusterSetup.html
相关文章推荐
- Linux下FTP服务器的搭建
- adb和shell和object-c的关联
- Linux 中kernel, shell和bash。
- maven打包 tomcat运行pom配置 或 打成jar包
- for’ loop initial declarations are only allowed in C99 mode
- 诡异的解决了诡异的错误---Win10+Kali linux 2.0 双系统 以及 grub 修复(上)
- Linux软连接和硬链接
- linux学习杂记
- Ganglia监控Hadoop及Hbase集群性能(安装配置)
- VS2013环境下配置OpenGL
- Linux C Programing - Arguments(2)
- LINUX下动态库调用静态库的方法
- Openwrt UCI API的使用
- MySQL高可用浅析
- Linux C Programing - Terminal(1)
- Linux下autoconf
- Linux串口编程(中断方式和select方式)
- ruby的optparse使用小记
- shell和com和jre的关联
- apache和opencv和linq的关联