is running beyond virtual memory limits

 Current usage: 314.6 MB of 2.9 GB physical memory used; 8.7 GB of 6.2 GB virtual memory used. Killing container.
http://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits
2.9G可能会是mapreduce.map.memory.mb，设置的，但是会大于=1G，即使设置了小于1G；

6.2G=2.9*2.1得出的

8.7G是map堆积出来的，让hadoop认为内存泄漏。个人认为是申请给map的资源一般设置为512M或者1024M（默认的），多个map并没有释放。

（RM）单个容器申请最大最小值；容器是yarn的资源抽象，包括内存与cpu

yarn.scheduler.minimum-allocation-mb  1G

yarn.scheduler.maximum-allocation-mb  8G

（NM）每个节点的内存使用最大值与物理内存与虚拟内存的内存对比率，

yarn.nodemanager.resource.memory-mb  8G

yarn.nodemanager.vmem-pmem-ratio     2.1

（AM）mapreduce的内存设置

mapreduce.map.memory.mb       1024M

mapreduce.reduce.memory.mb    1024M   值在Container之间

mapreduce.map.java.opts

mapreduce.reduce.java.opts

说明：这两个参主要是为需要运行JVM程序（java、scala等）准备的，通过这两个设置可以向JVM中传递参数的，与内存有关的是，-Xmx，-Xms等选项。此数值大小，应该在AM中的<map.mb和<reduce.mb.大概少四分一。

解决方法一：

setting yarn.nodemanager.vmem-check-enabled to false in yarn-site.xml,

Note : This is happening on Centos/RHEL 6 due to its aggressive allocation of virtual memory.

It can be resolved either by :

Disable virtual memory usage check by setting yarn.nodemanager.vmem-check-enabled tofalse;

Increase VM:PM ratio by setting yarn.nodemanager.vmem-pmem-ratio to some higher value.

解决方法二（合理方法）：

For our example cluster, we have the minimum RAM for a Container (yarn.scheduler.minimum-allocation-mb) = 2 GB. We’ll thus assign 4 GB for Map task Containers, and 8 GB for Reduce tasks Containers.

In mapred-site.xml:

mapreduce.map.memory.mb: 4096

mapreduce.reduce.memory.mb: 8192

Each Container will run JVMs for the Map and Reduce tasks. The JVM heap size should be set to lower than the Map and Reduce memory defined above, so that they are within the bounds of the Container memory allocated by YARN.

In mapred-site.xml:

mapreduce.map.java.opts: -Xmx3072m

mapreduce.reduce.java.opts: -Xmx6144m

The above settings configure the upper limit of the physical RAM that Map and Reduce tasks will use.

For our example cluster, we have the minimum RAM for a Container (yarn.scheduler.minimum-allocation-mb) = 2 GB. We’ll thus assign 4 GB for Map task Containers, and 8 GB for Reduce tasks Containers.

In mapred-site.xml:

mapreduce.map.memory.mb

:
4096

mapreduce.reduce.memory.mb

:
8192

Each Container will run JVMs for the Map and Reduce tasks. The JVM heap size should be set to lower than the Map and Reduce memory defined above, so that they are within the bounds of the Container memory allocated by YARN.

In mapred-site.xml:

mapreduce.map.java.opts

: 

-Xmx3072m

mapreduce.reduce.java.opts

: 

-Xmx6144m

The above settings configure the upper limit of the physical RAM that Map and Reduce tasks will use.

To sum it up:
In YARN, you should use the 

mapreduce

 configs,
not the 

mapred

 ones. EDIT: This
comment is not applicable anymore now that you've edited your question.
What you are configuring is actually how much you want to request, not what is the max to allocate.
The max limits are configured with the 

java.opts

 settings
listed above.

Finally, you may want to check this other SO
question that describes a similar problem (and solution).

个人结论：

在确定内存不会泄漏的情况下可以设置虚拟内存率To False；

按照第二条的建议去配置

Spark好像有自己的内存机制，不使用java虚拟机内存机制

Spark是结合依赖Yarn，与mapreduce的运行机制是两码事。Spark只是用了yarn资源调配器。而yarn是以container去抽象资源。

至于hadoop，

yarn.nodemanager.resource.memory-mb  是限定节点使用机器最大内存。