hadoop2.*能力调度器 4000 capacity-scheduler

capacity-scheduler能力调度器

1.  添加用户组dwgroup（集群中的namenode主备机器都要做以下操作）

groupadd dwgroup           //添加用户组

useradd -G  dwgroup 用户//将用户加到dwgroup用户组里

 

2.  yarn-site.xml

新增配置

<property>

<name>yarn.resourcemanager.scheduler.class</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>

</property>

 

去掉的配置

<!--llama set -->

<property>

<name>yarn.scheduler.minimum-allocation-mb</name>

<value>0</value>

</property>

<property>

<name>yarn.scheduler.minimum-allocation-vcores</name>

<value>0</value>

</property>

 

 

3.  capacity-scheduler.xml

更改的配置文件

<property>

   <name>yarn.scheduler.capacity.root.queues</name>

    <value>default,dwgroup</value>

    <description>

      The queues at the this level (root is theroot queue).

    </description>

  </property>

 

更改的配置

#--------------------default-----------------------------

<property>

   <name>yarn.scheduler.capacity.root.default.capacity</name>

    <value>30</value>

    <description>Defaultqueue target capacity.</description>

  </property>

 

  <property>

   <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
    <value>1</value>

    <description>

      Default queue user limita percentage from 0.0 to 1.0.

    </description>

  </property>

 

  <property>

 
  <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
   <value>100</value>

    <description>

      The maximum capacity ofthe default queue.

    </description>

  </property>

 

  <property>

   <name>yarn.scheduler.capacity.root.default.state</name>

   <value>RUNNING</value>

    <description>

      The state of the defaultqueue. State can be one of RUNNING or STOPPED.

    </description>

  </property>

 

  <property>

   <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>

   <value>*</value>

    <description>

      The ACL of who can submitjobs to the default queue.

    </description>

  </property>

 

  <property>

   <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>

   <value>*</value>

    <description>

      The ACL of who canadminister jobs on the default queue.

    </description>

  </property>

 

#------------------------dwgroup -----------------------------------

  <property>

 <name>yarn.scheduler.capacity.root.dwgroup.capacity</name>

  <value>70</value>

  <description>Defaultqueue target capacity.</description>

  </property>

 

  <property>

  <name>yarn.scheduler.capacity.root.dwgroup.user-limit-factor</name>

  <value>1</value>

  <description>

  Default queue user limit apercentage from 0.0 to 1.0.

  </description>

  </property>

 

  <property>

  <name>yarn.scheduler.capacity.root.dwgroup.maximum-capacity</name>

 <value>100</value>

  <description>

  The maximum capacity of thedefault queue.

  </description>

  </property>

 

  <property>

  <name>yarn.scheduler.capacity.root.dwgroup.state</name>

 <value>RUNNING</value>

  <description>

  The state of the defaultqueue. State can be one of RUNNING or STOPPED.

  </description>

  </property>

 

  <property>

 <name>yarn.scheduler.capacity.root.dwgroup.acl_submit_applications</name>

  <value>*</value>

  <description>

  The ACL of who can submitjobs to the default queue.

  </description>

  </property>

 

  <property>

 <name>yarn.scheduler.capacity.root.dwgroup.acl_administer_queue</name>

  <value>* </value>

  <description>

  The ACL of who canadminister jobs on the default queue.

  </description>

  </property>

 

更改的配置

<property>

   <name>yarn.scheduler.capacity.queue-mappings</name>

    <value>g:dwgroup:dwgroup,g:%:default</value>

    <description>

      A list of mappings that will be used toassign jobs to queues

      The syntax for this list is[u|g]:[name]:[queue_name][,next mapping]*

      Typically this list will be used to mapusers to queues,

      for example, u:%user:%user maps all usersto queues with the same name

      as the user.

    </description>

  </property>

 

更改完以上3个配置文件让配置文件生效

/data/hadoop/bin下执行以下命令

 

yarnrmadmin –refreshQueues

 

或者重启yarn执行以下命令

/data/hadoop/sbin/stop-yarn.sh

/data/hadoop/sbin/start-yarn.sh

 

 

 

 

注意：后续有新增用户，集群中的每台机器都要创建该用户，并且指定对应的dwgroup用户组