OutOfMemoryError: unable to create new native thread
2015-05-29 11:06
387 查看
1. Problem Description
In our production and staging environment, we occasionally encountered this error:
This hurt our business heavily, dev team and sys team got nervous.
Environment:
Linux 64bit, JDK 7, 2 tomcat instances hosting 2 cometD powered web app, we observed that once the 2 tomcats' total threads get greater than 1000, the error would probably reproduce.
2. Analysis and Solution
Reference [2] seems a great methodology to analyse this issue, but sadly just doesn't work for our case, anyway, I learned a lot from this awesome material.
When test team conducts load test, I monitored the JVM memory/threads, and OS memory/IO, didn't find anything abnormal.
Our heap size is 2048MB-2048MB, perm size is 128MB-256MB, at peak time, heap usage is 1250MB, perm usage is 75MB, tomcats' total threads number is around 1000. Seems not heavy load at all, weird.
Then I went to reference [1], and found our server's
open files limit is 1024, after change to 8000, the error seems gone.
Nevertheless, we did another optimization, enabled
Tomcat's NIO connector, so our cometD longpolling threads can be released quickly.
Reference:
[1] http://www.devgrok.com/2012/03/resolving-outofmemoryerror-unable-to.html
[2] http://www.javacodegeeks.com/2012/09/outofmemoryerror-unable-to-create-new.html
In our production and staging environment, we occasionally encountered this error:
Exception in thread "default-workqueue-7" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:691) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) Exception in thread "default-workqueue-20" Exception in thread "default-workqueue-10" at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$AWQThreadFactory$1.run(AutomaticWorkQueueImpl.java:353)
This hurt our business heavily, dev team and sys team got nervous.
Environment:
Linux 64bit, JDK 7, 2 tomcat instances hosting 2 cometD powered web app, we observed that once the 2 tomcats' total threads get greater than 1000, the error would probably reproduce.
2. Analysis and Solution
Reference [2] seems a great methodology to analyse this issue, but sadly just doesn't work for our case, anyway, I learned a lot from this awesome material.
When test team conducts load test, I monitored the JVM memory/threads, and OS memory/IO, didn't find anything abnormal.
Our heap size is 2048MB-2048MB, perm size is 128MB-256MB, at peak time, heap usage is 1250MB, perm usage is 75MB, tomcats' total threads number is around 1000. Seems not heavy load at all, weird.
Then I went to reference [1], and found our server's
open files limit is 1024, after change to 8000, the error seems gone.
Nevertheless, we did another optimization, enabled
Tomcat's NIO connector, so our cometD longpolling threads can be released quickly.
Reference:
[1] http://www.devgrok.com/2012/03/resolving-outofmemoryerror-unable-to.html
[2] http://www.javacodegeeks.com/2012/09/outofmemoryerror-unable-to-create-new.html
相关文章推荐
- 下载什么图标自己选
- Hadoop Wordcount 程序 详解
- java文件读写操作大全
- 124页第44题第三章
- Android导入AS工程
- salt-run源码分析
- iOS 在添加了UITapGestureRecognizer的UIView上响应按钮的点击事件
- Android Studio v1.0安装教程(Windows)
- maven org.springframework.web.filter.CharacterEncodingFilter cannot be cast to javax.servlet.Filter
- 熟悉GRE作文考试的流程步骤
- 如何做好网络营销
- 手机维护小常识
- 自己总结的一些三大框架面试题
- linux 查找
- ACM-奋斗的小蜗牛
- p124第三章43题作业
- [LeetCode] Combination Sum III 组合之和之三
- java常用简单正则表达式写法
- lamp setup (centos)
- 【转】MySQL索引背后的数据结构及算法原理