hive on tez Caused by: java.lang.OutOfMemoryError: Java heap space
2017-05-25 09:21
543 查看
昨天进行两个hive表关联导出数据,不幸的是爆出如下错误:
Status: Failed
Vertex failed, vertexName=Map 5, vertexId=vertex_1443634917922_0008_1_05, diagnostics=[Task failed, taskId=task_1443634917922_0008_1_05_000006, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
仔细观看日志发现上面错误信息,主要是因为内存不足,有意思的是我整个服务器都在给它跑,怎么会内存不足呢?后来查阅资料和咨询我一个同事吴哥发现,这个内存不足是值java堆内存,好把,既然内存不足,那我就看看hive给的默认内存是多少
hive>SET
hive.tez.container.size;
hive.tez.container.size=6144;
hive>SET
hive.tez.java.opts;
hive.tez.java.opts=-server
-Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps
尴尬了,确实有点小,那么容器我给它20G,java.opt给80%容器试试,反正服务器内存大
SET hive.tez.container.size=20480;
SET hive.tez.java.opts=-Xmx16384m;
解决了
,真是天天踩坑。
Status: Failed
Vertex failed, vertexName=Map 5, vertexId=vertex_1443634917922_0008_1_05, diagnostics=[Task failed, taskId=task_1443634917922_0008_1_05_000006, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
仔细观看日志发现上面错误信息,主要是因为内存不足,有意思的是我整个服务器都在给它跑,怎么会内存不足呢?后来查阅资料和咨询我一个同事吴哥发现,这个内存不足是值java堆内存,好把,既然内存不足,那我就看看hive给的默认内存是多少
hive>SET
hive.tez.container.size;
hive.tez.container.size=6144;
hive>SET
hive.tez.java.opts;
hive.tez.java.opts=-server
-Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps
尴尬了,确实有点小,那么容器我给它20G,java.opt给80%容器试试,反正服务器内存大
SET hive.tez.container.size=20480;
SET hive.tez.java.opts=-Xmx16384m;
解决了
,真是天天踩坑。
相关文章推荐
- Caused by: java.lang.OutOfMemoryError: Java heap space 异常
- Caused by: java.lang.OutOfMemoryError: Java heap space
- Caused by: java.lang.OutOfMemoryError: Java heap space
- Caused by: java.lang.OutOfMemoryError: Java heap space
- Caused by: java.lang.OutOfMemoryError: Java heap space解决办法
- spark Caused by: java.lang.OutOfMemoryError: Java heap space 问题
- Tomcat启动项目Caused by: java.lang.OutOfMemoryError: Java heap space
- Eclipse Tomcat 内存不足:Caused by: java.lang.OutOfMemoryError: Java heap space
- Caused by: java.lang.OutOfMemoryError: Java heap space
- MyEclipse 8.6 出现"Caused by: java.lang.OutOfMemoryError: PermGen space"解决
- MyEclipse 9 出现"Caused by: java.lang.OutOfMemoryError: PermGen space"解决
- hiveserver2服务异常停止,报错:java.lang.OutOfMemoryError: Java heap space
- hive dynamic partitions insert java.lang.OutOfMemoryError: Java heap space
- how-to: resolve "java.lang.OutOfMemoryError: Java heap space" during using beeline && hiveserver2
- java.lang.OutOfMemoryError: Java heap space
- java.lang.OutOfMemoryError: Java heap space
- java.lang.OutOfMemoryError: Java heap space解决方法
- java.lang.OutOfMemoryError: Java heap space + myeclipse中分配tomcat启动时所占内存大小
- Caused by: java.lang.OutOfMemoryError: Java heap space
- Exception in thread "main" java.lang.OutOfMemoryError: Java heap space