storm提交topology时supervisor无法创建worker

问题描述

在本地eclipse运行是好好的，可是提交到集群上去之后什么都没反应。提交topology之后，在storm UI上看topology的worker数与在程序里设置的值是一样的，但是到对应的机器去查看的时候，发现那个worker并没有存在，在日志目录下也没有找到worker的日志，提交topology也并没有报什么错误提示，所以很茫然，此时倒是很希望它弄出一个错误出来好去定位问题，可惜现实中什么错误日志都没有。

个我理解：本地运行和在集群模式上运行是不一样的，集群模式运行是由supervisor去创建workder（JVM），此时在创建时环境

排查问题：

1、jps 看是否有worker存在（其实不存在，所以这步没用）

2、看supervisor.log，刚才提交的topology时把相关的信息输出到此文件中去了

tail -200 supervisor.log |grep "Launching worker with command"

3、在第2步中找到了刚才supervisor创建worker的命令：

java -server -Xmx768m -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=worker-6703.log -Dstorm.home=/root/install/storm-0.9.1 -Dlogback.configurationFile=/root/install/storm-0.9.1/logback/cluster.xml -Dstorm.id=sso-17-1427954461
-Dworker.id=34f986e4-e2b6-4933-a2bf-0a9950660e02 -Dworker.port=6703 -cp /root/install/storm-0.9.1/lib/reflectasm-1.07-shaded.jar:/root/install/storm-0.9.1/lib/tools.cli-0.2.2.jar:/root/install/storm-0.9.1/lib/carbonite-1.3.2.jar:/root/install/storm-0.9.1/lib/ring-core-1.1.5.jar:/root/install/storm-0.9.1/lib/joda-time-2.0.jar:/root/install/storm-0.9.1/lib/httpclient-4.1.1.jar:/root/install/storm-0.9.1/lib/commons-fileupload-1.2.1.jar:/root/install/storm-0.9.1/lib/slf4j-api-1.6.5.jar:/root/install/storm-0.9.1/lib/objenesis-1.2.jar:/root/install/storm-0.9.1/lib/servlet-api-2.5-20081211.jar:/root/install/storm-0.9.1/lib/disruptor-2.10.1.jar:/root/install/storm-0.9.1/lib/curator-client-1.0.1.jar:/root/install/storm-0.9.1/lib/snakeyaml-1.11.jar:/root/install/storm-0.9.1/lib/compojure-1.1.3.jar:/root/install/storm-0.9.1/lib/logback-core-1.0.6.jar:/root/install/storm-0.9.1/lib/clojure-1.4.0.jar:/root/install/storm-0.9.1/lib/jetty-6.1.26.jar:/root/install/storm-0.9.1/lib/commons-codec-1.4.jar:/root/install/storm-0.9.1/lib/hiccup-0.3.6.jar:/root/install/storm-0.9.1/lib/ring-devel-0.3.11.jar:/root/install/storm-0.9.1/lib/logback-classic-1.0.6.jar:/root/install/storm-0.9.1/lib/ring-jetty-adapter-0.3.11.jar:/root/install/storm-0.9.1/lib/netty-3.6.3.Final.jar:/root/install/storm-0.9.1/lib/jline-2.11.jar:/root/install/storm-0.9.1/lib/storm-core-0.9.1-incubating.jar:/root/install/storm-0.9.1/lib/commons-lang-2.5.jar:/root/install/storm-0.9.1/lib/servlet-api-2.5.jar:/root/install/storm-0.9.1/lib/guava-13.0.jar:/root/install/storm-0.9.1/lib/curator-framework-1.0.1.jar:/root/install/storm-0.9.1/lib/httpcore-4.1.jar:/root/install/storm-0.9.1/lib/clout-1.0.1.jar:/root/install/storm-0.9.1/lib/minlog-1.2.jar:/root/install/storm-0.9.1/lib/zookeeper-3.3.3.jar:/root/install/storm-0.9.1/lib/asm-4.0.jar:/root/install/storm-0.9.1/lib/json-simple-1.1.jar:/root/install/storm-0.9.1/lib/clj-time-0.4.1.jar:/root/install/storm-0.9.1/lib/clj-stacktrace-0.2.4.jar:/root/install/storm-0.9.1/lib/commons-logging-1.1.1.jar:/root/install/storm-0.9.1/lib/tools.macro-0.1.0.jar:/root/install/storm-0.9.1/lib/log4j-over-slf4j-1.6.6.jar:/root/install/storm-0.9.1/lib/tools.logging-0.2.3.jar:/root/install/storm-0.9.1/lib/commons-exec-1.1.jar:/root/install/storm-0.9.1/lib/kryo-2.17.jar:/root/install/storm-0.9.1/lib/jgrapht-core-0.9.0.jar:/root/install/storm-0.9.1/lib/math.numeric-tower-0.0.1.jar:/root/install/storm-0.9.1/lib/meat-locker-0.3.1.jar:/root/install/storm-0.9.1/lib/ring-servlet-0.3.11.jar:/root/install/storm-0.9.1/lib/commons-io-1.4.jar:/root/install/storm-0.9.1/lib/junit-3.8.1.jar:/root/install/storm-0.9.1/lib/jetty-util-6.1.26.jar:/root/install/storm-0.9.1/lib/core.incubator-0.1.0.jar:/root/install/storm-0.9.1/conf:/app/storm/data/supervisor/stormdist/sso-17-1427954461/stormjar.jar
backtype.storm.daemon.worker sso-17-1427954461 0328962b-1692-4683-8c22-727c9cbb17ca 6703 34f986e4-e2b6-4933-a2bf-0a9950660e02

4、在命令行下执行第3步的那个命令，此时终于报错了

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/install/storm-0.9.1/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/storm/data/supervisor/stormdist/sso-17-1427954461/stormjar.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Detected both log4j-over-slf4j.jar AND slf4j-log4j12.jar on the class path, preempting StackOverflowError. 
SLF4J: See also http://www.slf4j.org/codes.html#log4jDelegationLoop for more details.
Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.apache.log4j.Logger.getLogger(Logger.java:39)
        at org.apache.log4j.Logger.getLogger(Logger.java:43)
        at org.apache.zookeeper.ZooKeeper.<clinit>(ZooKeeper.java:108)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:190)
        at backtype.storm.zookeeper$eval1825$loading__4784__auto____1826.invoke(zookeeper.clj:16)
        at backtype.storm.zookeeper$eval1825.invoke(zookeeper.clj:16)
        at clojure.lang.Compiler.eval(Compiler.java:6511)
        at clojure.lang.Compiler.eval(Compiler.java:6501)
        at clojure.lang.Compiler.load(Compiler.java:6952)
        at clojure.lang.RT.loadResourceScript(RT.java:359)
        at clojure.lang.RT.loadResourceScript(RT.java:350)
        at clojure.lang.RT.load(RT.java:429)
        at clojure.lang.RT.load(RT.java:400)
        at clojure.core$load$fn__4890.invoke(core.clj:5415)
        at clojure.core$load.doInvoke(core.clj:5414)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invoke(core.clj:5227)
        at clojure.core$load_lib.doInvoke(core.clj:5264)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invoke(core.clj:603)
        at clojure.core$load_libs.doInvoke(core.clj:5302)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invoke(core.clj:603)
        at clojure.core$require.doInvoke(core.clj:5381)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at backtype.storm.cluster$eval1819$loading__4784__auto____1820.invoke(cluster.clj:16)
        at backtype.storm.cluster$eval1819.invoke(cluster.clj:16)
        at clojure.lang.Compiler.eval(Compiler.java:6511)
        at clojure.lang.Compiler.eval(Compiler.java:6501)
        at clojure.lang.Compiler.load(Compiler.java:6952)
        at clojure.lang.RT.loadResourceScript(RT.java:359)
        at clojure.lang.RT.loadResourceScript(RT.java:350)
        at clojure.lang.RT.load(RT.java:429)
        at clojure.lang.RT.load(RT.java:400)
        at clojure.core$load$fn__4890.invoke(core.clj:5415)
        at clojure.core$load.doInvoke(core.clj:5414)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invoke(core.clj:5227)
        at clojure.core$load_lib.doInvoke(core.clj:5264)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invoke(core.clj:603)
        at clojure.core$load_libs.doInvoke(core.clj:5302)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invoke(core.clj:603)
        at clojure.core$require.doInvoke(core.clj:5381)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at backtype.storm.tuple$eval1815.invoke(tuple.clj:41)
        at clojure.lang.Compiler.eval(Compiler.java:6511)
        at clojure.lang.Compiler.eval(Compiler.java:6500)
        at clojure.lang.Compiler.load(Compiler.java:6952)
        at clojure.lang.RT.loadResourceScript(RT.java:359)
        at clojure.lang.RT.loadResourceScript(RT.java:350)
        at clojure.lang.RT.load(RT.java:429)
        at clojure.lang.RT.load(RT.java:400)
        at clojure.core$load$fn__4890.invoke(core.clj:5415)
        at clojure.core$load.doInvoke(core.clj:5414)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invoke(core.clj:5227)
        at clojure.core$load_lib.doInvoke(core.clj:5264)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invoke(core.clj:603)
        at clojure.core$load_libs.doInvoke(core.clj:5302)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invoke(core.clj:603)
        at clojure.core$require.doInvoke(core.clj:5381)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at backtype.storm.daemon.executor$eval1416$loading__4784__auto____1417.invoke(executor.clj:16)
        at backtype.storm.daemon.executor$eval1416.invoke(executor.clj:16)
        at clojure.lang.Compiler.eval(Compiler.java:6511)
        at clojure.lang.Compiler.eval(Compiler.java:6501)
        at clojure.lang.Compiler.load(Compiler.java:6952)
        at clojure.lang.RT.loadResourceScript(RT.java:359)
        at clojure.lang.RT.loadResourceScript(RT.java:350)
        at clojure.lang.RT.load(RT.java:429)
        at clojure.lang.RT.load(RT.java:400)
        at clojure.core$load$fn__4890.invoke(core.clj:5415)
        at clojure.core$load.doInvoke(core.clj:5414)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invoke(core.clj:5227)
        at clojure.core$load_lib.doInvoke(core.clj:5264)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invoke(core.clj:603)
        at clojure.core$load_libs.doInvoke(core.clj:5302)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invoke(core.clj:603)
        at clojure.core$require.doInvoke(core.clj:5381)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at backtype.storm.daemon.worker$eval3$loading__4784__auto____4.invoke(worker.clj:16)
        at backtype.storm.daemon.worker$eval3.invoke(worker.clj:16)
        at clojure.lang.Compiler.eval(Compiler.java:6511)
        at clojure.lang.Compiler.eval(Compiler.java:6501)
        at clojure.lang.Compiler.load(Compiler.java:6952)
        at clojure.lang.RT.loadResourceScript(RT.java:359)
        at clojure.lang.RT.loadResourceScript(RT.java:350)
        at clojure.lang.RT.load(RT.java:429)
        at clojure.lang.RT.load(RT.java:400)
        at clojure.core$load$fn__4890.invoke(core.clj:5415)
        at clojure.core$load.doInvoke(core.clj:5414)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.lang.Var.invoke(Var.java:415)
        at backtype.storm.daemon.worker.<clinit>(Unknown Source)
Caused by: java.lang.IllegalStateException: Detected both log4j-over-slf4j.jar AND slf4j-log4j12.jar on the class path, preempting StackOverflowError. See also http://www.slf4j.org/codes.html#log4jDelegationLoop for more details.
        at org.apache.log4j.Log4jLoggerFactory.<clinit>(Log4jLoggerFactory.java:49)
        ... 102 more

5、在我的另一个topology是在另一工程，是用maven来管理依赖的，该topology提交上去没有问题，能正常运行。可是我在另一工程里没有用maven管理，有一些依赖不太一样，现在还在忙别的事，该笔记回头完善。。。