Kafka分布式消息队列（二）：环境搭建&测试

本文基于Kafka 0.8

在一台机器上构建一个3个节点的kafka集群，并测试producer、consumer在正常情况下的行为，以及在lead broker/follow broker失效情况下的行为

1.下载并解压kafka 0.8.0 release

$ mkdir kafka

$ wget http://apache.dataguru.cn/kafka/0.8.0/kafka_2.8.0-0.8.0.tar.gz
$ tar -zxvf kafka_2.8.0-0.8.0.tar.gz

$ cd kafka_2.8.0-0.8.0

$ ll

total 2560

drwxr-xr-x 6 root root 4096 Dec 17 17:44 ./

drwxr-xr-x 4 root root 4096 Dec 17 18:20 ../

drwxr-xr-x 3 root root 4096 Dec 17 18:16 bin/

drwxr-xr-x 2 root root 4096 Dec 17 17:43 config/

-rw-r--r-- 1 root root 2520145 Nov 27 06:21 kafka_2.8.0-0.8.0.jar

drwxr-xr-x 2 root root 4096 Nov 27 06:21 libs/

-rw-r--r-- 1 root root 12932 Nov 27 06:21 LICENSE

drwxr-xr-x 2 root root 4096 Dec 17 18:00 logs/

-rw------- 1 root root 47165 Dec 17 18:10 nohup.out

-rw-r--r-- 1 root root 162 Nov 27 06:21 NOTICE

2.启动一个单节点的zookeeper

$ nohup bin/zookeeper-server-start.sh config/zookeeper.properties &

3. 准备启动一个3个broker节点的kafka集群，因此做如下配置

$ cp config/server.properties config/server-1.properties

$ cp config/server.properties config/server-2.properties

并做如下修改：

config/server-1.properties:

broker.id=1

port=9093

log.dir=/tmp/kafka-logs-1

config/server-2.properties:

broker.id=2

port=9094

log.dir=/tmp/kafka-logs-2

说明：

broker.id: broker节点的唯一标识

port: broker节点使用端口号

log.dir: 消息目录位置

4. 启动3个broker节点

$ JMX_PORT=9997 bin/kafka-server-start.sh config/server-1.properties &

$ JMX_PORT=9998 bin/kafka-server-start.sh config/server-2.properties &

$ JMX_PORT=9999 bin/kafka-server-start.sh config/server.properties &

5. 创建topic并查看

$ bin/kafka-create-topic.sh --zookeeper localhost:2181 --replica 3 --partition 1 --topic 3test

creation succeeded!

$ bin/kafka-list-topic.sh --zookeeper localhost:2181

topic: 3test partition: 0 leader: 2 replicas: 2,1,0 isr: 2,1,0

topic: test partition: 0 leader: 0 replicas: 0 isr: 0

topic: test_topic partition: 0 leader: 1 replicas: 0,1,2 isr: 1,2,0

说明：

partiton： partion id，由于此处只有一个partition，因此partition id 为0

leader：当前负责读写的lead broker id

relicas：当前partition的所有replication broker list

isr：relicas的子集，只包含出于活动状态的broker

6.启动consumer & producer，并在producer启动后的console输入一些信息

$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic 3test

message1

message3

message2

$ bin/kafka-console-producer.sh --broker-list localhost:9092,localhost:9093,localhost:9094 --topic 3test

message1

message3

message2

producer发送的数据consumer都能正常消费

7. 干掉follow broker

杀掉一个非lead broker（lead broker id为2）

$ pkill -9 -f server-1.properties

查看topic：

$ bin/kafka-list-topic.sh --zookeeper localhost:2181

topic: 3test partition: 0 leader: 2 replicas: 2,1,0 isr: 2,0

topic: test partition: 0 leader: 0 replicas: 0 isr: 0

topic: test_topic partition: 0 leader: 2 replicas: 0,1,2 isr: 2,0

此时，存活的broker只有2,0

测试：produce发送消息，consumer能正常接收到

8. 继续干掉leader broker
干掉leader broker后，连续查看topic状态

$ pkill -9 -f server-2.properties

$ bin/kafka-list-topic.sh --zookeeper localhost:2181

topic: 3test partition: 0 leader: 2 replicas: 2,1,0 isr: 2,0

topic: test partition: 0 leader: 0 replicas: 0 isr: 0

topic: test_topic partition: 0 leader: 2 replicas: 0,1,2 isr: 2,0

$ bin/kafka-list-topic.sh --zookeeper localhost:2181

topic: 3test partition: 0 leader: 2 replicas: 2,1,0 isr: 2,0

topic: test partition: 0 leader: 0 replicas: 0 isr: 0

topic: test_topic partition: 0 leader: 2 replicas: 0,1,2 isr: 2,0

$ bin/kafka-list-topic.sh --zookeeper localhost:2181

topic: 3test partition: 0 leader: 0 replicas: 2,1,0 isr: 0

topic: test partition: 0 leader: 0 replicas: 0 isr: 0

topic: test_topic partition: 0 leader: 0 replicas: 0,1,2 isr: 0

$ bin/kafka-list-topic.sh --zookeeper localhost:2181

topic: 3test partition: 0 leader: 0 replicas: 2,1,0 isr: 0

topic: test partition: 0 leader: 0 replicas: 0 isr: 0

topic: test_topic partition: 0 leader: 0 replicas: 0,1,2 isr: 0

杀掉leader broker过了一会，broker 0成为新的leader broker

测试：produce发送消息，consumer能正常接收到

参考：Kafka
QuickStart