jvm启动一段时间后无法使用的原因
2016-03-04 17:10
555 查看
在生产环境中,有时候会遇到程序从开发到测试一切正常,但是将程序部署到线上后,一段时间之后程序无法向外提供服务,但是端口却正常暴露出来的情况。
最近在搞分布式服务时遇到了这个问题,服务框架选用dubbo,服务通过spring容器部署使用,底层默认用netty进行消息通信。服务由开发完成到测试完成一切正常,但是部署到线上一段时间后,服务服务提供服务,注册中心显示服务依然在在线,但是服务使用者总是报超时异常。
无奈,到线上进行排查之后,发现了如下的问题:
从线程dump数据看,有200多个线程正在同时waiting 这个<0x00000000b24213f8> 东西,同时还有类似logback的waiting信息,因为我们所有的日志都是发送到kafka进行存储查看,而且发送日志都是异步的过程,所以怀疑kafka消息队列不通,导致线程阻塞,有了怀疑后,迅速到线上验证,发现kafka服务器版本和客户端版本不一致,升级kafka服务器版本后,问题不在出现。
总结:异步操作确实会提高程序的响应能力,但是上线的过程中,一定要确认异步操作可以正确执行,防止异步线程堆积,拖死正常业务,同时异步线程也要考虑做堆积线程的清理工作。
最近在搞分布式服务时遇到了这个问题,服务框架选用dubbo,服务通过spring容器部署使用,底层默认用netty进行消息通信。服务由开发完成到测试完成一切正常,但是部署到线上一段时间后,服务服务提供服务,注册中心显示服务依然在在线,但是服务使用者总是报超时异常。
无奈,到线上进行排查之后,发现了如下的问题:
"New I/O server worker #1-2" #1414 daemon prio=5 os_prio=0 tid=0x00007f334c03d000 nid=0x3749 waiting on condition [0x00007f3324b3f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b24f8350> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353) at ch.qos.logback.core.AsyncAppenderBase.put(AsyncAppenderBase.java:156) at ch.qos.logback.core.AsyncAppenderBase.append(AsyncAppenderBase.java:147) at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88) at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48) at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:273) at ch.qos.logback.classic.Logger.callAppenders(Logger.java:260) at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:442) at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396) at ch.qos.logback.classic.Logger.debug(Logger.java:503) at com.alibaba.dubbo.common.logger.slf4j.Slf4jLogger.debug(Slf4jLogger.java:30) at com.alibaba.dubbo.common.logger.support.FailsafeLogger.debug(FailsafeLogger.java:79) at com.alibaba.dubbo.remoting.exchange.support.header.HeartbeatHandler.received(HeartbeatHandler.java:82) at com.alibaba.dubbo.remoting.transport.MultiMessageHandler.received(MultiMessageHandler.java:28) at com.alibaba.dubbo.remoting.transport.AbstractPeer.received(AbstractPeer.java:123) at com.alibaba.dubbo.remoting.transport.netty.NettyHandler.messageReceived(NettyHandler.java:91) at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302) at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.messageReceived(NettyCodecAdapter.java:148) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "DubboServerHandler-121.196.245.7:20880-thread-200" #243 daemon prio=5 os_prio=0 tid=0x00007f333c117800 nid=0x1bca waiting on condition [0x00007f3324d81000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b24213f8> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "DubboServerHandler-121.196.245.7:20880-thread-199" #242 daemon prio=5 os_prio=0 tid=0x00007f333c115800 nid=0x1bc9 waiting on condition [0x00007f3324dc2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b24213f8> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "DubboServerHandler-121.196.245.7:20880-thread-198" #241 daemon prio=5 os_prio=0 tid=0x00007f333c114000 nid=0x1bc8 waiting on condition [0x00007f3324e03000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b24213f8> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) |
总结:异步操作确实会提高程序的响应能力,但是上线的过程中,一定要确认异步操作可以正确执行,防止异步线程堆积,拖死正常业务,同时异步线程也要考虑做堆积线程的清理工作。
相关文章推荐
- Nt函数原型头文件
- Java定位CPU使用高问题--转载
- java string与byte互转
- java集合类——Stack类
- tomcat 日志切割shell脚本
- Android平台arm64 ptrace hook bridge_code debug
- swift 学习资料网站
- SpringMVC使用@ResponseBody时返回json的日期格式
- 2dx中文乱码问题
- DNS查询
- QT5.4.2静态编译(包含QtWebKit)及配置方法
- does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), o
- Jstorm 在Debian上的安装以及错误排查
- c:forEach
- 硬盘管理命令smartctl
- 【poj2396】Budget 有源汇上下界可行流
- huhx的android封神之路-------->ContentProvider的使用
- SQLServer数据库备份
- Java中通过jsch来连接远程服务器执行linux命令
- 【错误总结】makefile的编写问题:一定要注意不能有随随便便的tab、空格等