Hadoop 2 配置优化

hdfs-default.xml

<property>
<name>dfs.blocksize</name>
<value>134217728</value>
<description>
The default block size for new files, in bytes.You can use the following suffix (caseinsensitive)k(kilo), m(mega), g(giga), t(tera),p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.),Or provide complete size in bytes (such as 134217728 for 128 MB).
</description>
</property>

这个就是hdfs里一个文件块的大小了，默认128M。在不对太大的话会有较少map同时计算，太小的话也浪费可用map个数资源，而且文件太小namenode就浪费内存多。根据需要进行设置

<property>
<name>dfs.datanode.handler.count</name>
<value>10</value>
<description>The number of server threads for the datanode.</description>
</property>

设定 namenode server threads 的数量，这些 threads 會用 RPC 跟其他的 datanodes 沟通。当 datanodes 数量太多时会发現很容易出現 RPC timeout，解決方法是提升网络速度或提高这个值，但要注意的是thread数量多也表示namenode消耗的内存也随着增加。

mapreduce

<property>
<name>mapreduce.map.cpu.vcores</name>
<value>1</value>
<description>The number of virtual cores to request from the scheduler for
each map task.
</description>
</property>
<property>
<name>mapreduce.reduce.cpu.vcores</name>
<value>1</value>
<description>The number of virtual cores to request from the scheduler for
each reduce task.
</description>
</property>

map和reduce使用的CPU数目默认是一个可根据机器的性能调大以充分发挥机器的性能

<property>
<name>mapreduce.map.memory.mb</name>
<value>1536</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>3072</value>
</property>

map和reduce使用的内存目默认是1024。

yarn

<description>Amount of physical memory, in MB, that can be allocated
for containers.</description>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>8192</value>
</property>

<property>
<description>Number of vcores that can be allocated for containers. This is used by the RM scheduler when allocating resources for containers. This is not used to limit the number of physical cores used by YARN containers.
</description>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>8</value>
</property>

根据机器的物理内存调节大小，yarn.nodemanager.resource.memory-mb决定了每个节点所启动的最大container的数量。每个container中用一个map或者是reduce任务。map和reduce任务默认需要1024M内存。若yarn.nodemanager.resource.memory-mb 为8192M则每个节点最多可以执行8个任务。yarn.nodemanager.resource.cpu-vcores决定了每个节点所使用最大cpu的数量

可参考以下公式确定每个节点的container数量

公式从下面的网址获得> http://www.dataguru.cn/thread-480519-1-1.html

yarn-site.xml yarn.nodemanager.resource.memory-mb = Containers个数* 每个Container内存

yarn-site.xml yarn.scheduler.minimum-allocation-mb = 每个Container内存

yarn-site.xml yarn.scheduler.maximum-allocation-mb = Containers个数* 每个Container内存

mapred-site.xml mapreduce.map.memory.mb = 每个Container内存

mapred-site.xml mapreduce.reduce.memory.mb = 2 * 每个Container内存

mapred-site.xml mapreduce.map.java.opts = 0.8 * 每个Container内存

mapred-site.xml mapreduce.reduce.java.opts = 0.8 * 2 * 每个Container内存