java 代码问题定位方法

通过centreon发现 知道，建议平台服务器负载偏高，达到load达到2以上
登陆服务器，使用top命令查看，发现java进程占用cpu高达200%左右
使用top -p "pid" -H 查看占用cpu较高java线程

[align=left] PIDUSER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND [/align][align=left] 10483 root 15 0 1659m 1.2g 12m S 21.6 30.5 0:04.61 java [/align][align=left] 10360 root 15 0 1659m 1.2g 12m S 19.0 30.5 0:17.04 java [/align][align=left] 10417 root 15 0 1659m 1.2g 12m S 17.0 30.5 0:11.83 java [/align][align=left] 10391 root 15 0 1659m 1.2g 12m S 15.6 30.5 0:12.54 java [/align][align=left] 10169 root 15 0 1659m 1.2g 12m S 14.3 30.5 0:56.24 java [/align][align=left] 10320 root 15 0 1659m 1.2g 12m S 14.0 30.5 0:17.67 java [/align][align=left] 10358 root 15 0 1659m 1.2g 12m S 13.3 30.5 0:24.33 java [/align][align=left] 10479 root 15 0 1659m 1.2g 12m R 12.3 30.5 0:04.05 java[/align]通过jstack -l "pid" > pid.log 打印出java进程各个线程的具体信息
在通过瑞刚提供的脚本显示出16进制的java线程号nid=0x28f9
通过16进制的线程号找到对应jstack打印出的 pid.log中的字段 如下：
通过下面红色解释，可以看出这是一段有关登陆的过程"resin-tcp-connection-*:8080-2547" daemon prio=10 tid=0x93854000 nid=0x292f runnable [0x91afd000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Unknown Source)
at oracle.net.ns.DataPacket.receive(Unknown Source)
at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1099) #连接oracle DB 获取信息
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1070)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:478)
......................................
- locked <0xe8c7a048> (a oracle.jdbc.driver.T4CPreparedStatement)
- locked <0xbc46bf98> (a oracle.jdbc.driver.T4CConnection)
......................................
at com.cyou.zhidao.customers.dao.CustomersDao.findCustomerByAccount(CustomersDao.java:171)
at com.cyou.zhidao.customers.service.LoginService.findCustomerByAccount(LoginService.java:61)
at com.cyou.zhidao.customers.service.LoginService$$FastClassByCGLIB$$aa9346d0.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:617)
at com.cyou.zhidao.customers.service.LoginService$$EnhancerByCGLIB$$69db3f91.findCustomerByAccount(<generated>)
at com.cyou.zhidao.customers.action.LoginAction.login(LoginAction.java:119) #知道登陆报文
...........................
at com.caucho.server.dispatch.ServletFilterChain.doFilter(ServletFilterChain.java:106)
at com.cyou.web.servlet.CyUrlRewriteFilter.doFilter(CyUrlRewriteFilter.java:51)
at com.caucho.server.dispatch.ServletFilterChain.doFilter(ServletFilterChain.java:106)
at com.cyou.web.servlet.CyUrlRewriteFilter.doFilter(CyUrlRewriteFilter.java:51)
at com.caucho.server.dispatch.FilterFilterChain.doFilter(FilterFilterChain.java:70)
at com.cyou.web.servlet.LoginFilter.doFilter(LoginFilter.java:77) #表示登陆相关的信息
at com.caucho.server.dispatch.FilterFilterChain.doFilter(FilterFilterChain.java:70)
atcom.cyou.web.servlet.RequestFilter.doFilter(RequestFilter.java:61) #表示公司自行开发的代码报文
........................
- locked <0xb85467c0> (a java.lang.Object)
at com.caucho.util.ThreadPool.runTasks(ThreadPool.java:520)
at com.caucho.util.ThreadPool.run(ThreadPool.java:442)
at java.lang.Thread.run(Thread.java:619)

Locked ownable synchronizers:
- None
之后又用同样的方法找到了其他几个线程对应的字段如下：
从标红部分可以看出 此连接为阻塞状态，这种状态的连接有很多
"resin-tcp-connection-*:8080-2681" daemon prio=10 tid=0x96902c00 nid=0x292d waiting for monitor entry [0x91c9a000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:388)
- waiting to lock <0xb7a0e2c0> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at com.caucho.vfs.QServerSocketWrapper.accept(QServerSocketWrapper.java:97)
at com.caucho.server.port.Port.accept(Port.java:813)
at com.caucho.server.port.TcpConnection.run(TcpConnection.java:495)
at com.caucho.util.ThreadPool.runTasks(ThreadPool.java:520)
at com.caucho.util.ThreadPool.run(ThreadPool.java:442)
at java.lang.Thread.run(Thread.java:619)

Locked ownable synchronizers:
- None
由于发现阻塞的连接很多 至此我们可以初步判断造成cpu压力过高是由于连接过高造成
然后通过netstat -na |grep ":8080" |awk '{++SS[$6]}END{for(a in SS) print a,SS[a]}'查看resin个连接端口的状态数量
发现TIME_WAIT状态连接高达2700条左右，ESTABLISHED才6条左右；
考虑如何减少TIME_WAIT状态连接，1、查看resin keepalived已开启，2、系统内核参数已初始化完毕
考虑为何为造成如此高的TIME_WAIT连接状态
通过查看访问日志，发现没一秒钟有大量访问zhidao平台登陆页的连接创建，查看nginx配置文件 发现并未对此login.do页面进行防刷策略。
添加防刷策略后，cpu下降到30%

大家可以借鉴下 此次问题排查的思路；