Hadoop yarn OutOfMemoryError: unable to create new native thread
2015-08-23 19:00
417 查看
Bug
2015-08-23 18:00:12,084 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:713) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371) at java.lang.UNIXProcess.initStreams(UNIXProcess.java:172) at java.lang.UNIXProcess$2.run(UNIXProcess.java:145) at java.lang.UNIXProcess$2.run(UNIXProcess.java:143) at java.security.AccessController.doPrivileged(Native Method) at java.lang.UNIXProcess.(UNIXProcess.java:143) at java.lang.ProcessImpl.start(ProcessImpl.java:130) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022) at org.apache.hadoop.util.Shell.runCommand(Shell.java:485) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.containerIsAlive(DefaultContainerExecutor.java:430) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.signalContainer(DefaultContainerExecutor.java:401) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.cleanupContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher.handle(ContainersLauncher.java:139) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher.handle(ContainersLauncher.java:55) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106) at java.lang.Thread.run(Thread.java:744) 2015-08-23 18:00:12,086 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye.. 2015-08-23 18:13:35,544 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: STARTUP_MSG:
Solution
At first,I thought my mapreduce program may need more memorys and there may be a limit on nproc .So,I changed the conf in linux and hadoop/etc/security/limits.conf * soft nofile 65536 * hard nofile 65536 /etc/security/limits.d/90-nproc.conf * soft nproc unlimited * hard nproc unlimited mapred-site.xml mapreduce.map.memory.mb 4096 mapreduce.reduce.memory.mb 8192 mapreduce.map.java.opts -Xmx3072m mapreduce.reduce.java.opts -Xmx7168m
But,It didn’t work.The truth is that I have not enough memory to allocate to each maps and reduces.In fact, they don’t need a lot of memory and I over allocate to them.The solution is
mapred-site.xml mapreduce.map.memory.mb 1024 mapreduce.reduce.memory.mb 2048 mapreduce.map.java.opts -Xmx800m mapreduce.reduce.java.opts -Xmx1600m
Environment
Centos 6.4 core 3.10.80 Hadoop 2.6
相关文章推荐
- 详解HDFS Short Circuit Local Reads
- Hadoop_2.1.0 MapReduce序列图
- 使用Hadoop搭建现代电信企业架构
- 单机版搭建Hadoop环境图文教程详解
- hadoop常见错误以及处理方法详解
- hadoop 单机安装配置教程
- hadoop的hdfs文件操作实现上传文件到hdfs
- hadoop实现grep示例分享
- Apache Hadoop版本详解
- linux下搭建hadoop环境步骤分享
- hadoop client与datanode的通信协议分析
- Hadoop2.X/YARN环境搭建--CentOS7.0 JDK配置
- hadoop中一些常用的命令介绍
- Hadoop单机版和全分布式(集群)安装
- 用PHP和Shell写Hadoop的MapReduce程序
- hadoop map-reduce中的文件并发操作
- Hadoop1.2中配置伪分布式的实例
- Hadoop2.X/YARN环境搭建--CentOS7.0系统配置
- java结合HADOOP集群文件上传下载
- 用python + hadoop streaming 分布式编程(一) -- 原理介绍,样例程序与本地调试