Weblogic报错 <BEA-000337> <BEA-101215> Oracle后台大量等待

作者 阿九【转载时请务必以超链接形式标明文章原始出处和作者信息】

今天应用的人反映个税系统无法登陆，查看weblogic后台日志，发现有两个比较多的报错 <BEA-000337> <BEA-101215>

错误日志：

####<2012-11-23 上午09时58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '3' for queue: 'weblogic.kernel.Default' has been busy for "909" seconds working on the request "Http Request: /bimsweb/slpt/accept/showDeclareInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> ####<2012-11-23 上午09时58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '8' for queue: 'weblogic.kernel.Default' has been busy for "907" seconds working on the request "Http Request: /bimsweb/slpt/accept/showDeclareInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> ####<2012-11-23 上午09时58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '15' for queue: 'weblogic.kernel.Default' has been busy for "908" seconds working on the request "Http Request: /bimsweb/slpt/accept/showDeclareInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> ####<2012-11-23 上午09时58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '23' for queue: 'weblogic.kernel.Default' has been busy for "873" seconds working on the request "Http Request: /bimsweb/slpt/accept/queryDeclareAcceptInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> ####<2012-11-23 上午09时58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '28' for queue: 'weblogic.kernel.Default' has been busy for "926" seconds working on the request "Http Request: /bimsweb/slpt/accept/showDeclareInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> ####<2012-11-23 上午09时58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '39' for queue: 'weblogic.kernel.Default' has been busy for "909" seconds working on the request "Http Request: /bimsweb/slpt/accept/showDeclareInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> ####<2012-11-23 上午09时58分43秒 CST> <Warning> <WebLogicServer> <sxdsrjk> <myserver> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '46' for queue: 'weblogic.kernel.Default' has been busy for "929" seconds working on the request "Http Request: /bimsweb/slpt/accept/queryDeclareAcceptInfo.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> ####<2012-11-23 上午10时00分59秒 CST> <Error> <HTTP> <sxdsrjk> <myserver> <ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <BEA-101215> <Malformed Request "null". Request parsing failed, Code: -10>

日志显示，新连接的会话由于系统繁忙导致等待。

检查后台数据库情况，在告警日志中没有记录任何异常报错，查看等待会话，发现有大量的

“global cache cr request”、“db file scattered read”、“db file sequential read”、“buffer busy waits”等待事件。

通过操作系统topas命令也看到磁盘使用率100%，CPU使用率很高，52%在使用、36%在等待、CPU没有空闲。

通过查看后台视图发现大量等待会话执行的SQL语句均为“SELECT”语句，将这些会话杀死后，系统恢复正常。

初步判断是这些SQL语句大量的、频繁的执行导致数据库出现过多的“global cache cr request”、“db file scattered read”、“db file sequential read” 等待事件，导致后续连接的会话不可用。

相关SQL语句，还要在跟踪分析。