proxool 连接池警告分析：appears to have started a thread named [HouseKeeper] but has failed to stop it

1. 问题：日志中出现下面的警告：

警告: The web application [ROOT] appears to have started a thread named [HouseKeeper] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Thread.sleep(Native Method) org.logicalcobwebs.proxool.HouseKeeperThread.run(HouseKeeperThread.java:46)
2. 原因调查：

使用的是 proxool 连接池，查看源码，分析如下：

在项目某种情况下，ProxoolFacade.shutdown(); 被调用，他会调用：

protected static void shutdown(String finalizer, int delay) {

ConnectionPool[] cps = ConnectionPoolManager.getInstance().getConnectionPools();
for (int i = 0; i < cps.length; i++) {
removeConnectionPool(finalizer, cps[i], delay);
}

// If a shutdown hook was registered then remove it
try {
if (shutdownHook != null) {
ShutdownHook.remove(shutdownHook);
}
} catch (Throwable t) {
if (LOG.isDebugEnabled()) {
LOG.debug("Unanticipated error during removal of ShutdownHook. Ignoring it.", t);
}
}

// Stop threads
PrototyperController.shutdown();
HouseKeeperController.shutdown();

}

这里 首先 关闭连接池，然后在最后面关闭了一个 housekeeper 的 demaon线程：

HouseKeeperController.shutdown();

/**
* Stop all house keeper threads.
*/
protected static void shutdown() {
synchronized(LOCK) {
Iterator i = houseKeeperThreads.iterator();
while (i.hasNext()) {
HouseKeeperThread hkt = (HouseKeeperThread) i.next();
LOG.info("Stopping " + hkt.getName() + " thread");
hkt.cancel();
}
houseKeeperThreads.clear();
}
}

对每一个 housekeeper 线程调用 cancel() 方法，一般只有一个 housekeeper线程：

hkt.cancel();

public class HouseKeeperThread extends Thread {
private static final Log LOG = LogFactory.getLog(HouseKeeperThread.class);
private boolean stop;
public HouseKeeperThread(String name) {
setDaemon(true);
setName(name);
}
public void run() {
while (!stop) {
HouseKeeper hk = HouseKeeperController.getHouseKeeperToRun();
while (hk != null && !stop) {
try {
//                    if (LOG.isDebugEnabled()) {
//                        LOG.debug("About to sweep " + hk.getAlias());
//                    }
hk.sweep();
} catch (ProxoolException e) {
LOG.error("Couldn't sweep " + hk.getAlias(), e);
}
hk = HouseKeeperController.getHouseKeeperToRun();
}
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
LOG.error("Interrupted", e);
}
}
}
protected void cancel() {
stop = true;
}
}

尼玛 stop 尽然是一个 普通 的 boolean 变量，感觉应该使用 volatile 修饰！

保错的原因是，housekeeper 线程因为每隔5秒执行一次，而系统 想要 stop 它的时候，它正在 sleep ，所以抛出了异常：

警告: The web application [ROOT] appears to have started a thread named [HouseKeeper] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Thread.sleep(Native Method)
org.logicalcobwebs.proxool.HouseKeeperThread.run(HouseKeeperThread.java:46)

但是，因为 设置了 stop = true，所以，当它 sleep 完成之后，也可能会 判断 stop == true, 也可能判断 stop == false，因为这里没有使用 volatile 修饰！

所以最坏的结果，就是 有一个名叫 "HouseKeeper" 的 deamon jvm 线程没有被成功关闭。

3. 日志显示，数据库连接池，确实被正确关闭了：

ContainerBackgroundProcessor[StandardEngine[Catalina]] - Proxool shoutdown
[INFO ] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:484):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Shutting down 'autumn' pool immediately [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/05/00) - #0006 removed because of shutdown.
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #6 closed
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/04/00) - #0005 removed because of shutdown.
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #5 closed
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/03/00) - #0004 removed because of shutdown.
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #4 closed
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/02/00) - #0003 removed because of shutdown.
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #3 closed
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/01/00) - #0002 removed because of shutdown.
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #2 closed
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/00/00) - #0001 removed because of shutdown.
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #1 closed
[INFO ] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:564):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 'autumn' pool has been closed down by ContainerBackgroundProcessor[StandardEngine[Catalina]] in 6 milliseconds.
[DEBUG] org.logicalcobwebs.proxool.ShutdownHook.remove(ShutdownHook.java:41):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Removed shutdownHook
[INFO ] org.logicalcobwebs.proxool.PrototyperController.shutdown(PrototyperController.java:100):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Stopping Prototyper thread
[INFO ] org.logicalcobwebs.proxool.HouseKeeperController.shutdown(HouseKeeperController.java:107):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Stopping HouseKeeper thread

排除了 poxool 导致 数据库连接泄露的问题。