hiveserver2服务异常停止java.lang.OutOfMemoryError: Java heap space

2017-01-02 15:34:04,461 WARN  [IPC Client (188363230) connection to nn01/192.168.1.4:8020 from hive]: ipc.Client (Client.java:run(955)) - Unexpected error reading responses on connection Thread[IPC Client (188363230) connection to nn01.100.cloud/172.16.8.4:8020
from hive,5,main]

java.lang.OutOfMemoryError: Java heap space

    at com.google.protobuf.CodedInputStream.<init>(CodedInputStream.java:573)

    at com.google.protobuf.CodedInputStream.newInstance(CodedInputStream.java:55)

    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:199)

    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)

    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)

    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)

    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)

    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)

    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)

    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:949)

2017-01-02 15:34:03,365 FATAL [Thread-51]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(91)) - Error starting HiveServer2: could not start ThriftBinaryCLIService

java.lang.OutOfMemoryError: Java heap space

    at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76)

    at org.apache.thrift.transport.TSocket.<init>(TSocket.java:78)

    at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:114)

    at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)

    at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)

    at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:121)

    at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:86)

    at java.lang.Thread.run(Thread.java:745)

2017-01-02 15:34:29,186 INFO  [Thread-50]: server.HiveServer2 (HiveServer2.java:stop(266)) - Shutting down HiveServer2

2017-01-02 15:34:29,186 INFO  [Thread-50]: thrift.ThriftCLIService (ThriftCLIService.java:stop(137)) - Thrift server has stopped

2017-01-02 15:34:29,187 INFO  [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.

2017-01-02 15:34:29,187 INFO  [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.

2017-01-02 15:34:29,187 INFO  [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.

2017-01-02 15:34:29,188 INFO  [Thread-3]: server.HiveServer2 (HiveStringUtils.java:run(679)) - SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down HiveServer2 at nn02.100.cloud/192.168.1.5

分析及解决：

java.lang.OutOfMemoryError: Java heap space内存溢出报错

This Error happened during connect hiveserver2 via beeline, both happened from hiveserver node and remote node.

heapsize太小了，可以适当的调大些

Root cause, the heapsize of hadoop opts is too small, need to increase the size in hive-env.sh as bold:

if [ "$SERVICE" = "cli" ]; then
if [ -z "$DEBUG" ]; then
export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -Xmx12288m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
else
export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -Xmx12288m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"
fi
fi

# The heap size of the jvm stared by hive shell script can be controlled via:
#
export HADOOP_HEAPSIZE=2048

继续分析查看下进程：
监控语句如下：

jstat -gcutil [进程号] 5000 --每5秒打印一次jvm各个内存区的状态

另外找了一个监控系统内存的语句：

vmstat | sed -n '/[0-9]/p'  

观察2天后发现新生代和老年代都已经爆满，永久代几乎爆满，jvm一直在做full GC，但是没有效果，各个内存区都被占用了。

就日志来看，老年代和持久代的内存只要上去了，就再没有下降过，而当这两块区域饱和后，新生代也出现了这种现象。
是有什么资源没释放吧？一看代码，果然，有的连接没有关闭！

如果还是有问题，hive0.14版本，自身也存在着bug，可以考虑进行定期重启hiveserver2服务，别笑实际工作中很多大的公司都是这样做的。

参考：
http://blog.csdn.net/gklifg/article/details/50418109 http://blog.csdn.net/lulynn/article/details/46455097