Kafka错误1_kafka中 FailedToSendMessageException解决_修改host.name

注：在启动Kafka的过程中，同样遇到这个问题，在将

host.name=Slave1

host.name=Slave2

host.name=Slave3

后面改为机器的IP地址后，解决问题；

原文来自：http://my.oschina.net/u/1475616/blog/343875

这两天在学kafka,把官网上的demo部署到自己的虚拟机上运行,正常;

然后部署到公司的研发线主机上,发现producer一直无法发送消息;

部分错误日志如下:

[2014-11-13 09:58:09,660] WARN Error while fetching metadata [{TopicMetadata for topic mor ->

No partition metadata for topic mor due to kafka.common.LeaderNotAvailableException}] for topic [mor]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)

[2014-11-13 09:58:09,660] ERROR Failed to send requests for topics mor with correlation ids in [17,24] (kafka.producer.async.DefaultEventHandler)

[2014-11-13 09:58:09,660] ERROR Error in handling batch of 17 events (kafka.producer.async.ProducerSendThread)

kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.

at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)

at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)

at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)

at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)

at scala.collection.immutable.Stream.foreach(Stream.scala:526)

at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)

at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)

操作步骤完全一样,只是在研发线上修改了zookeeper的监听端口,输入命令时也是按照对应的端口输入的,所以不是参数有问题导致的;

对比了研发线和虚拟机上的配置,发现除了这个端口之外,没有区别;

鼓捣了一天,没找到问题出在哪里;

第二天来上班,把server.properties文件里

#host.name=localhost

的注释去掉,然后运行,发现问题解决了;

(但是在虚拟机里明明没有修改这个东西也是可以正常运行的;

怀疑是发行版不同,一些内部的配置有区别导致的;

虚拟机:

Distributor ID: Ubuntu

Description: Ubuntu 14.04.1 LTS

Release: 14.04

Codename: trusty

研发线:

LSB Version: :

Distributor ID: RedHatEnterpriseServer

Description: Red Hat Enterprise Linux Server release 5.4 (Tikanga)

Release: 5.4

Codename: Tikanga

)

在解决问题的过程中,还发现其他的问题,但是目前没发现影响运行;

就是zookeeper和broker跑起来之后,创建producer,创建话题,创建Consumer,的时候,zookeeper都会报异常,部分记录如下:

[2014-11-13 09:12:13,486] INFO Got user-level KeeperException when processing sessionid:0x149a6b3e36c0001 type:setData cxid:0x3 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/config/topics/morning Error:KeeperErrorCode = NoNode for /config/topics/morning
(org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 09:12:13,506] INFO Got user-level KeeperException when processing sessionid:0x149a6b3e36c0001 type:create cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/config/topics Error:KeeperErrorCode = NodeExists for /config/topics
(org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 09:12:13,535] INFO Processed session termination for sessionid: 0x149a6b3e36c0001 (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 09:19:50,958] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:create cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 09:19:50,982] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/config Error:KeeperErrorCode = NoNode for /config (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 09:19:50,998] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/admin Error:KeeperErrorCode = NoNode for /admin (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 09:19:51,295] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:setData cxid:0x19 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/controller_epoch Error:KeeperErrorCode = NoNode for /controller_epoch
(org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 09:19:51,374] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:delete cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for
/admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 10:31:50,651] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf5001a type:setData cxid:0x19 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/consumers/test-consumer-group/offsets/mor/0 Error:KeeperErrorCode = NoNode for /consumers/test-consumer-group/offsets/mor/0 (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 10:31:50,661] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf5001a type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/consumers/test-consumer-group/offsets Error:KeeperErrorCode = NoNode
for /consumers/test-consumer-group/offsets (org.apache.zookeeper.server.PrepRequestProcessor)

一开始producer无法发送消息,以为是和这些异常有关,但是在虚拟机里运行的时候,同样出现这些异常,但是也没有影响producer发送消息;

在网上查了半天,有人说是不正确关闭zookeeper和server的原因,有人说是没有删除/tmp下的zookeeper日志和kafka日志的原因,

总之,我试了上面提到的方法,还是报这些异常;

如果有哪位仁兄知道这些异常是什么原因引起的,请告诉我,十分感谢;

另外,从本机运行java代码的Consumer连接研发线的时候,连接迅速被关闭了,收不到producer发来的消息;

原因是配置的超时时间太短,zookeeper没有读完Consumer的数据,连接就被Consumer断开了,部分日志如下:

[2014-11-13 10:28:47,989] INFO Accepted socket connection from /192.168.50.33:2676 (org.apache.zookeeper.server.NIOServerCnxn)

[2014-11-13 10:28:47,989] WARN EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)

解决办法,就是把配置里的超时时间配置的长点儿,如下:

props.put("zookeeper.session.timeout.ms", "400000");

对了,还有个问题,kafka官网上关于java版producer部分的代码,有个地方写的不是很明确,如下:

broker1和broker2代表broker对应的hostname,而不是broker的id.正确示例如下:

?