hadoop1 & hadoop2 fair-schduler 配置和使用
2015-08-28 11:03
549 查看
hadoop1
配置 mapred-site.xml,增加如下内容<property> <name>mapred.jobtracker.taskScheduler</name> <value>org.apache.hadoop.mapred.FairScheduler</value> </property> <property> <name>mapred.fairscheduler.allocation.file</name> <value>/etc/hadoop/conf/pools.xml</value> </property>
配置 pools.xml,增加如下内容
<queue name="default”> <minResources>1024 mb,1vcores</minResources> <maxResources>61440 mb,20vcores</maxResources> <maxRunningApps>10</maxRunningApps> <weight>2.0</weight> <schedulingPolicy>fair</schedulingPolicy> </queue> <queue name=“hadoop”> <minResources>1024 mb,10vcores</minResources> <maxResources>3072000 mb,960vcores</maxResources> <maxRunningApps>60</maxRunningApps> <weight>5.0</weight> <schedulingPolicy>fair</schedulingPolicy> <aclSubmitApps>hadoop,yarn,spark</aclSubmitApps> </queue> <queue name="spark"> <minResources>1024 mb,10vcores</minResources> <maxResources>61440 mb,20vcores</maxResources> <maxRunningApps>10</maxRunningApps> <weight>4.0</weight> <schedulingPolicy>fair</schedulingPolicy> <aclSubmitApps>yarn,spark</aclSubmitApps> </queue> <userMaxAppsDefault>20</userMaxAppsDefault>
提交作业指定队列方式
-Dmapred.job.queue.name=hadoop
hadoop2
配置 yarn-site.xml,增加如下内容<property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value> </property> <property> <name>yarn.scheduler.fair.allocation.file</name> <value>/home/cluster/conf/hadoop/fair-scheduler.xml</value> </property> <property> <name>yarn.scheduler.fair.user-as-default-queue</name> //如果希望以用户名作为队列,可以将该属性配置为true,默认为true,所以如果不想以用户名为队列的,必须显式的设置成false <value>false</value> </property>
配置 fair-scheduler.xml,增加如下内容
<queue name="default”> <minResources>1024 mb,1vcores</minResources> <maxResources>61440 mb,20vcores</maxResources> <maxRunningApps>10</maxRunningApps> <weight>2.0</weight> <schedulingPolicy>fair</schedulingPolicy> </queue> <queue name=“hadoop”> <minResources>1024 mb,10vcores</minResources> <maxResources>3072000 mb,960vcores</maxResources> <maxRunningApps>60</maxRunningApps> <weight>5.0</weight> <schedulingPolicy>fair</schedulingPolicy> <aclSubmitApps>hadoop,yarn,spark</aclSubmitApps> </queue> <queue name="spark"> <minResources>1024 mb,10vcores</minResources> <maxResources>61440 mb,20vcores</maxResources> <maxRunningApps>10</maxRunningApps> <weight>4.0</weight> <schedulingPolicy>fair</schedulingPolicy> <aclSubmitApps>yarn,spark</aclSubmitApps> </queue> <userMaxAppsDefault>20</userMaxAppsDefault>
提交作业指定队列方式
-Dmapreduce.job.queuename=root.hadoop
spark
提交作业指定队列方式--queue=root.spark
相关文章推荐
- hadoop1 & hadoop2 fair-schduler 配置和使用
- LeetCode题解:Container With Most Water
- 对select()参数fdset的完整理解 http://blog.csdn.net/maray/article/details/8285775
- LeetCode题解:Climbing Stairs
- OGG-01705 input trail file './dirdat/fc075295' is greater than the size of the file
- Windows Server 2008 如何部署Domain Controller
- MediaInfo参数大全
- http://blog.csdn.net/woshiyjk/article/details/7895888
- linux中waitpid系统调用
- Caused by: libcore.io.GaiException: getaddrinfo failed: EAI_NODATA (No address associated with hostn
- UVA10976 Fractions Again?!
- [转]do_select()函数分析,理解select(),poll(),poll_wait()函数的关系
- hdu4393 Throw nails(只用模拟前面500来次,后面根据速度、位置、id值排序即可)
- UVA 12238 Anti-Rhyme Pairs(hash+二分)
- OC:copy 与 retain 的区别
- hdu 4630 No Pain No Game (区间gcd相关x线段树or树状数组)
- odoo(Openerp)如何在字段的on_change方法中动态返回domain
- 浅析cookie-free domain这么回事
- [leetcode] 11.Container With Most Water
- 什么会在main函数之前执行