hadoop virtual memory limits
2016-09-07 14:43
483 查看
Container [pid=32800,containerID=container_1470655434720_0894_01_000002] is running beyond virtual memory limits. Current usage: 224.3 MB of 8 GB physical memory used; 34.0 GB of 16.8 GB virtual memory used. Killing container. Dump of the process-tree for container_1470655434720_0894_01_000002 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 32808 32800 32800 32800 (java) 850 52 36405874688 57074 /soft/java/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx32768m -Djava.io.tmpdir=/export/grid/10/hadoop/yarn/local/usercache/hdp/appcache/application_1470655434720_0894/container_1470655434720_0894_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/export/grid/06/hadoop/yarn/log/application_1470655434720_0894/container_1470655434720_0894_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 172.23.197.72 59297 attempt_1470655434720_0894_m_000000_0 2 |- 32800 32798 32800 32800 (bash) 1 0 108617728 335 /bin/bash -c /soft/java/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx32768m -Djava.io.tmpdir=/export/grid/10/hadoop/yarn/local/usercache/hdp/appcache/application_1470655434720_0894/container_1470655434720_0894_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/export/grid/06/hadoop/yarn/log/application_1470655434720_0894/container_1470655434720_0894_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 172.23.197.72 59297 attempt_1470655434720_0894_m_000000_0 2 1>/export/grid/06/hadoop/yarn/log/application_1470655434720_0894/container_1470655434720_0894_01_000002/stdout 2>/export/grid/06/hadoop/yarn/log/application_1470655434720_0894/container_1470655434720_0894_01_000002/stderr Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143
根据报错提示可以看出是虚拟内存超过限制导致的,yarn上面的内存设置主要有以下几个参数:
yarn.nodemanager.resource.memory-mb 表示该节点上YARN可使用的物理内存总量,默认是8192(MB),注意,如果你的节点内存资源不够8GB,则需要调减小这个值,而YARN不会智能的探测节点的物理内存总量。 yarn.nodemanager.vmem-pmem-ratio 任务每使用1MB物理内存,最多可使用虚拟内存量,默认是2.1。 yarn.nodemanager.pmem-check-enabled 是否启动一个线程检查每个任务正使用的物理内存量,如果任务超出分配值,则直接将其杀掉,默认是true。 yarn.nodemanager.vmem-check-enabled 是否启动一个线程检查每个任务正使用的虚拟内存量,如果任务超出分配值,则直接将其杀掉,默认是true。 yarn.scheduler.minimum-allocation-mb 单个任务可申请的最少物理内存量,默认是1024(MB),如果一个任务申请的物理内存量少于该值,则该对应的值改为这个数。 yarn.scheduler.maximum-allocation-mb 单个任务可申请的最多物理内存量,默认是8192(MB)。
修改yarn.nodemanager.vmem-check-enabled为false可解决问题
相关文章推荐
- [hadoop] - Container [xxxx] is running beyond physical/virtual memory limits.
- hadoop yarn:beyond virtual memory limits
- Yarn运行Job报错:container is running beyond virtual memory limits
- hadoop集群跑mr程序报错beyond physical memory limits
- 解决running beyond virtual memory limits. Current usage: 35.5 MB of 1 GB physical memory used; 16.8 G
- running beyond virtual/physical memory limits问题解决
- Container is running beyond virtual memory limits. Current usage: 611.1 MB of 1 GB physical memory u
- beyond virtual memory limits. Current usage: 142.3 MB of 1 GB physical memory used;
- 【hadoop】 running beyond virtual memory错误原因及解决办法
- is running beyond virtual memory limits
- is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 2.6 GB of 40 GB virtual memory used
- Container [pid=6263,containerID=container_1494900155967_0001_02_000001] is running beyond virtual memory limits
- yarn Container beyond physical/virtual memory limits
- hadoop job报错is running beyond physical memory limits
- 【hadoop】 running beyond virtual memory错误原因及解决办法
- 关于VMware出现 Not enough physical memory is available to power on this virtual machine
- 滴滴云服务器安装php的时候报错virtual memory exhausted: Cannot allocate memory make: *** [ext/fileinfo/libmagic/ap
- 编译时:virtual memory exhausted: Cannot allocate memory
- WinCE 5.0 virtual memory space
- error C3859: virtual memory range for PCH exceeded。 原因及解决