weblogic oracle.jdbc.driver.PhysicalConnection.rollback
2017-10-24 15:54
489 查看
先是凌晨的时候中间件连数据库出了问题:
org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (IO 错误: The Network Adapter could not establish the connection)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1244) ~[com.springsource.org.apache.commons.dbcp-1.2.2.osgi.jar:na]
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:882) ~[com.springsource.org.apache.commons.dbcp-1.2.2.osgi.jar:na]
at com.ic.core.util.IcDataSourceManager.getConnection(IcDataSourceManager.java:81) ~[eic-core-1.0.6.jar:na]
然后出现连接不够用的情况:
####<2017-10-24 上午12时25分28秒 CST> <Info> <Common> <zc-sc-app04> <sys_app04_8001> <[STUCK] ExecuteThread: '61' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1508775928273> <BEA-000628> <Created "1" resources for pool "CTDataSource",
out of which "1" are available and "0" are unavailable.>
####<2017-10-24 上午12时25分28秒 CST> <Info> <Common> <zc-sc-app04> <sys_app04_8001> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1508775928273> <BEA-000628> <Created "1" resources for pool "CTDataSource",
out of which "1" are available and "0" are unavailable.>
weblogic状态开始告警。
上午10点钟的时候用jstack看到有事务在回滚,按理说事务回滚结束之后连接池会回收连接的,但等了两个小时看还是这个状态。
"[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f1ad1c53000 nid=0x516e waiting for monitor entry [0x00007f1a9d6d5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:3896)
- waiting to lock <0x000000070d83a480> (a oracle.jdbc.driver.T4CConnection)
at weblogic.jdbc.wrapper.Connection.forcedCleanup(Connection.java:153)
at weblogic.common.resourcepool.ResourcePoolImpl.timeoutInactiveResources(ResourcePoolImpl.java:1953)
at weblogic.common.resourcepool.ResourcePoolImpl.access$1500(ResourcePoolImpl.java:41)
at weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2675)
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
检查了一下gv$session,并未发现有活动的连接,且当前的Weblogic连接都没有执行的SQL语句,我认为这是僵尸的线程,只能重启。
后来确定这是一个bug.
Many Stuck Threads on JDBC Connection Rollback (文档 ID 1584478.1)
There are lots of messages like the following in the server log, and the server response is slow:
<WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "611" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@1f12dec2", which is more than the configured
time (StuckThreadMaxTime) of "600" seconds. Stack trace:
oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:3896)
weblogic.jdbc.wrapper.Connection.forcedCleanup(Connection.java:153)
weblogic.common.resourcepool.ResourcePoolImpl.timeoutInactiveResources(ResourcePoolImpl.java:1953)
weblogic.common.resourcepool.ResourcePoolImpl.access$1500(ResourcePoolImpl.java:41)
weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2675)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
>
<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "631" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@20cb3eea", which is more than
the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:3896)
weblogic.jdbc.wrapper.Connection.forcedCleanup(Connection.java:153)
weblogic.common.resourcepool.ResourcePoolImpl.timeoutInactiveResources(ResourcePoolImpl.java:1953)
weblogic.common.resourcepool.ResourcePoolImpl.access$1500(ResourcePoolImpl.java:41)
weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2675)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
>
The issue depends on an uncontrollable timing sequence to occur, and also may involve a requirement that the datasource has been disabled or flushed (whether internally or manually) previously. The basic symptom
is of a hanging thread, trying to do a rollback on a connection whose socket is in an externally-induced condition (maybe due to a firewall) that will relegate any standard JDBC call to hang indefinitely in a socket read. This issue is addressed by Bug
10169939.
org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (IO 错误: The Network Adapter could not establish the connection)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1244) ~[com.springsource.org.apache.commons.dbcp-1.2.2.osgi.jar:na]
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:882) ~[com.springsource.org.apache.commons.dbcp-1.2.2.osgi.jar:na]
at com.ic.core.util.IcDataSourceManager.getConnection(IcDataSourceManager.java:81) ~[eic-core-1.0.6.jar:na]
然后出现连接不够用的情况:
####<2017-10-24 上午12时25分28秒 CST> <Info> <Common> <zc-sc-app04> <sys_app04_8001> <[STUCK] ExecuteThread: '61' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1508775928273> <BEA-000628> <Created "1" resources for pool "CTDataSource",
out of which "1" are available and "0" are unavailable.>
####<2017-10-24 上午12时25分28秒 CST> <Info> <Common> <zc-sc-app04> <sys_app04_8001> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1508775928273> <BEA-000628> <Created "1" resources for pool "CTDataSource",
out of which "1" are available and "0" are unavailable.>
weblogic状态开始告警。
上午10点钟的时候用jstack看到有事务在回滚,按理说事务回滚结束之后连接池会回收连接的,但等了两个小时看还是这个状态。
"[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f1ad1c53000 nid=0x516e waiting for monitor entry [0x00007f1a9d6d5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:3896)
- waiting to lock <0x000000070d83a480> (a oracle.jdbc.driver.T4CConnection)
at weblogic.jdbc.wrapper.Connection.forcedCleanup(Connection.java:153)
at weblogic.common.resourcepool.ResourcePoolImpl.timeoutInactiveResources(ResourcePoolImpl.java:1953)
at weblogic.common.resourcepool.ResourcePoolImpl.access$1500(ResourcePoolImpl.java:41)
at weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2675)
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
检查了一下gv$session,并未发现有活动的连接,且当前的Weblogic连接都没有执行的SQL语句,我认为这是僵尸的线程,只能重启。
后来确定这是一个bug.
Many Stuck Threads on JDBC Connection Rollback (文档 ID 1584478.1)
SYMPTOMS
There are lots of messages like the following in the server log, and the server response is slow:<WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "611" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@1f12dec2", which is more than the configured
time (StuckThreadMaxTime) of "600" seconds. Stack trace:
oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:3896)
weblogic.jdbc.wrapper.Connection.forcedCleanup(Connection.java:153)
weblogic.common.resourcepool.ResourcePoolImpl.timeoutInactiveResources(ResourcePoolImpl.java:1953)
weblogic.common.resourcepool.ResourcePoolImpl.access$1500(ResourcePoolImpl.java:41)
weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2675)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
>
<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "631" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@20cb3eea", which is more than
the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:3896)
weblogic.jdbc.wrapper.Connection.forcedCleanup(Connection.java:153)
weblogic.common.resourcepool.ResourcePoolImpl.timeoutInactiveResources(ResourcePoolImpl.java:1953)
weblogic.common.resourcepool.ResourcePoolImpl.access$1500(ResourcePoolImpl.java:41)
weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2675)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
>
CAUSE
The issue depends on an uncontrollable timing sequence to occur, and also may involve a requirement that the datasource has been disabled or flushed (whether internally or manually) previously. The basic symptomis of a hanging thread, trying to do a rollback on a connection whose socket is in an externally-induced condition (maybe due to a firewall) that will relegate any standard JDBC call to hang indefinitely in a socket read. This issue is addressed by Bug
10169939.
相关文章推荐
- 一个未解决的问题:java.lang.AbstractMethodError: oracle.jdbc.driver.OracleConnection.getHoldability()
- oracle.jdbc.OracleConnection.physicalConnectionWithin问题
- 换成atomikos后报java.lang.AbstractMethodError: oracle.jdbc.driver.LogicalConnection.createClob()Ljava/sq
- jdbc实现篇-源码(Oracle、MySQL)-第一部分-Dirver注册和获取及Connection获取原理
- Connection-通过OracleDriver的基本方法建立Connection连接
- java.sql.SQLException: No suitable driver found for jdbc:oracle:thin:@localhost:1521:ORCL
- sqoop:Failed to download file from http://hdp01:8080/resources//oracle-jdbc-driver.jar due to HTTP error: HTTP Error 404: Not Found
- java.lang.ClassCastException: weblogic.jdbc.wrapper.Blob_oracle_sql_BLOB
- 在linux上通过JDBC连接ORACLE 时总是出现 java.sql.SQLRecoverableException: IO Error: Connection reset 的问题
- sqoop could not load db driver class:oracle.jdbc.OracleDriver
- Cannot create PoolableConnectionFactory ([Microsoft][SQLServer 2000 Driver for JDBC]Error establishing socket.)错误
- java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver解决办法
- 1.通过原始的Driver接口获取JDBC数据库连接Connection
- JSP中配置Oracle以及oracle.jdbc.driver.OracleDriver No suitable driver解决办法
- Class org.apache.struts2.json.JSONWriter can not access a member of class oracle.jdbc.driver.Physica
- Oracle JDBC Driver 深藏的 Bug
- How to add Oracle JDBC driver in your Maven local repository
- Java JDBC Thin Driver 连接 Oracle有三种方式
- jdbc实现篇-源码(Oracle、MySQL)之Connection创建过程
- 关于oracle jdbc driver OracleDriver添加maven仓储问题