Hadoop作业优先级控制
2015-05-30 17:42
295 查看
当同时在集群中运行多个作业时,默认情况下,Hadoop将提交的作业放入一个FIFO,一个作业结束后,Hadoop就启动下一个作业。
当一个运行时间长但是优先级较低的作业先于运行时间短而优先级较高的作业提交时,优先级高的作业会长时间排队等待。
为了解决这个问题,Hadoop定义了5种不同的作业优先级,分别是:VERY_HIGH,HIGH,NORMAL,LOW,VERY_LOW,作业的默认优先级是NORMAL,可以通过$hadoop job -set-priority进行修改。
例子:
1.在集群中启动1个运行时间较长的作业
caiyong@caiyong:/opt/hadoop$ bin/hadoop jar hadoop-examples-1.2.1.jar pi 2000 2000
2.查看作业列表
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -list
1 jobs currently running
JobId State StartTime UserName Priority SchedulingInfo
job_201503171201_0003 1 1426565671593 caiyong NORMAL NA
3.查看作业的运行状态
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -status job_201503171201_0003
Job: job_201503171201_0003
file: hdfs://127.0.0.1:8020/home/caiyong/tmp/mapred/staging/caiyong/.staging/job_201503171201_0003/job.xml
tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201503171201_0003
map() completion: 0.012500001
reduce() completion: 0.0
Counters: 19
Job Counters
SLOTS_MILLIS_MAPS=117080
Launched map tasks=26
Data-local map tasks=26
File Input Format Counters
Bytes Read=2832
FileSystemCounters
HDFS_BYTES_READ=5870
FILE_BYTES_WRITTEN=1316654
Map-Reduce Framework
Map output materialized bytes=672
Map input records=24
Spilled Records=48
Map output bytes=432
Total committed heap usage (bytes)=3815768064
CPU time spent (ms)=9530
Map input bytes=576
SPLIT_RAW_BYTES=3038
Combine input records=0
Combine output records=0
Physical memory (bytes) snapshot=4156928000
Virtual memory (bytes) snapshot=9500446720
Map output records=48
4.把作业的优先级提高为VERY_HIGH
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -set-priority job_201503171201_0003 VERY_HIGH
Changed job priority.
5.查看更改后的作业优先级
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -list
1 jobs currently running
JobId State StartTime UserName Priority SchedulingInfo
job_201503171201_0003 1 1426565671593 caiyong VERY_HIGH NA
6.强制结束正在运行的作业
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -kill job_201503171201_0003
Killed job job_201503171201_0003
当一个运行时间长但是优先级较低的作业先于运行时间短而优先级较高的作业提交时,优先级高的作业会长时间排队等待。
为了解决这个问题,Hadoop定义了5种不同的作业优先级,分别是:VERY_HIGH,HIGH,NORMAL,LOW,VERY_LOW,作业的默认优先级是NORMAL,可以通过$hadoop job -set-priority进行修改。
例子:
1.在集群中启动1个运行时间较长的作业
caiyong@caiyong:/opt/hadoop$ bin/hadoop jar hadoop-examples-1.2.1.jar pi 2000 2000
2.查看作业列表
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -list
1 jobs currently running
JobId State StartTime UserName Priority SchedulingInfo
job_201503171201_0003 1 1426565671593 caiyong NORMAL NA
3.查看作业的运行状态
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -status job_201503171201_0003
Job: job_201503171201_0003
file: hdfs://127.0.0.1:8020/home/caiyong/tmp/mapred/staging/caiyong/.staging/job_201503171201_0003/job.xml
tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201503171201_0003
map() completion: 0.012500001
reduce() completion: 0.0
Counters: 19
Job Counters
SLOTS_MILLIS_MAPS=117080
Launched map tasks=26
Data-local map tasks=26
File Input Format Counters
Bytes Read=2832
FileSystemCounters
HDFS_BYTES_READ=5870
FILE_BYTES_WRITTEN=1316654
Map-Reduce Framework
Map output materialized bytes=672
Map input records=24
Spilled Records=48
Map output bytes=432
Total committed heap usage (bytes)=3815768064
CPU time spent (ms)=9530
Map input bytes=576
SPLIT_RAW_BYTES=3038
Combine input records=0
Combine output records=0
Physical memory (bytes) snapshot=4156928000
Virtual memory (bytes) snapshot=9500446720
Map output records=48
4.把作业的优先级提高为VERY_HIGH
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -set-priority job_201503171201_0003 VERY_HIGH
Changed job priority.
5.查看更改后的作业优先级
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -list
1 jobs currently running
JobId State StartTime UserName Priority SchedulingInfo
job_201503171201_0003 1 1426565671593 caiyong VERY_HIGH NA
6.强制结束正在运行的作业
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -kill job_201503171201_0003
Killed job job_201503171201_0003
相关文章推荐
- [shell] Bash编程总结
- shell <<EOF
- Linux批量部署无密钥脚本
- Caused by: Action class [cn.eos.action.TestAction] not found - action - file:/G:/xampp/tomcat/webapp
- linux学习笔记—别名与快捷键
- Linux 的 Crontab 命令运用(转)
- Linux修改/etc/sysconfig/i18n后桌面无法显示
- linux配置java环境变量(详细)
- OpenGL--骨骼动画
- OpenGL--骨骼动画
- centos安装Python2.7记录
- linux上从文件的最后向前查看
- Linux系统编程——多任务的同步与互斥
- linux学习笔记—shell脚本执行方式
- opencv实现人像优化
- Linux动态共享库
- 读毛星云(浅墨)opencv(五)博客总结
- Linux下安装scikit-learn numpy scipy
- linux下驱动模块化编译,动态加载以及卸载
- linux下驱动模块化编译,动态加载以及卸载