kafka windows单机安装测试

版本 kafka_2.10-0.8.2.1

初步了解一下一下kafka的基本用法。

win7 64 bit

jdk1.7

1)下载kafka_2.10-0.8.2.1.tgz并解压（整合scala）

2)修改配置文件(可选)

config/zookeeper.properties

  dataDir=/zookeeper

这么写建到d盘根目录

  dataDir=zookeeper
这么写建到bin/windows目录

3)启动zookeeper（窗口1)

cd bin\windows

zookeeper-server-start ..\..\config\zookeeper.properties  

或

zookeeper-server-start ../../config/zookeeper.properties  

4)zookeeper测试（窗口2)

echo stat | nc 192.168.25.42 2181

D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>zookeeper-shell.bat 192.168.25.42:2181

注： 这里没有提示符

D:\fk\tech\cloud\zookeeper\zookeeper-3.4.6\bin>zkcli.cmd 192.168.25.42:2181Connecting to localhost:2181
ls /测试ok
5)启动kafka（窗口3)
kafka-server-start.bat ..\..\config\server.properties

6)创建topic（窗口4)

kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic mytopic

D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>kafka-topics.

bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --

topic mytopic

Created topic "mytopic".

7)查看topic（窗口4)

kafka-topics.bat --describe --zookeeper localhost:2181 --topic mytopic

D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>kafka-topics

bat --describe --zookeeper localhost:2181 --topic mytopic

Topic:mytopic   PartitionCount:3        ReplicationFactor:1     Configs:

Topic: mytopic  Partition: 0    Leader: 0       Replicas: 0     Isr: 0

Topic: mytopic  Partition: 1    Leader: 0       Replicas: 0     Isr: 0

Topic: mytopic  Partition: 2    Leader: 0       Replicas: 0     Isr: 0

8）发布/生产消息窗口（窗口5)

kafka-console-producer.bat --broker-list localhost:9092 --topic mytopic

有一个错误提示，但是不影响(kafa的bugKAFKA-1711，KAFKA-1824,预计0.9修复)

D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>kafka-consol

-producer.bat --broker-list localhost:9092 --topic mytopic

[2015-09-30 10:06:11,420] WARN Property topic is not valid (kafka.utils.Verifia

leProperties)

9）消费消息窗口（窗口6)

kafka-console-consumer.bat --zookeeper localhost:2181 --from-beginning --topic mytopic

10）producer窗口输入，可以在consumer窗口输出

D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>kafka-console

-producer.bat --broker-list localhost:9092 --topic mytopic

[2015-09-30 10:06:11,420] WARN Property topic is not valid (kafka.utils.Verifiab

leProperties)

hello

kafka

error

this is a test msg from producer

中文

D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>kafka-console

-consumer.bat --zookeeper localhost:2181 --from-beginning --topic mytopic

hello

kafka

error

this is a test msg from producer

中文

10)znode查看

ls /

[consumers, config, controller, admin, brokers, zookeeper, controller_epoch

ls /consumers

[console-consumer-21728]

ls /config

[topics, changes]

ls /controller

[]

ls /admin

[delete_topics]

ls /brokers

[topics, ids]

ls /zookeeper

[quota]

ls /controller_epoch

[]

11)检查offset(窗口4)

 注意：group为consumerid,需要在zk里面查ls /consumer

D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>kafka-consume

r-offset-checker.bat --topic mytopic  --zookeeper localhost:2181 --broker-info 1

27.0.0.1:9092 --group console-consumer-64061

Group           Topic                          Pid Offset          logSize

Lag             Owner

console-consumer-64061 mytopic                        0   4               4

0               console-consumer-64061_zynb0411-1443580144073-95285fc3

-0

console-consumer-64061 mytopic                        1   1               1

0               console-consumer-64061_zynb0411-1443580144073-95285fc3

-0

console-consumer-64061 mytopic                        2   1               1

0               console-consumer-64061_zynb0411-1443580144073-95285fc3

-0

BROKER INFO

0 -> zynb0411:9092

D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>

12)simple-consumer(底层操作) 

指定offset

D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>kafka-simple-

consumer-shell.bat  --topic mytopic --offset 1 --broker-list 127.0.0.1:9092 --pa

rtition 0

error

this is a test msg from producer

中文

分别查看不同的partition

D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>kafka-simple-

consumer-shell.bat  --topic mytopic --offset 0 --broker-list 127.0.0.1:9092 --pa

rtition 0

kafka

error

this is a test msg from producer

中文

终止批处理操作吗(Y/N)? y

D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>kafka-simple-

consumer-shell.bat  --topic mytopic --offset 0 --broker-list 127.0.0.1:9092 --pa

rtition 1

go

终止批处理操作吗(Y/N)? y

D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>kafka-simple-

consumer-shell.bat  --topic mytopic --offset 0 --broker-list 127.0.0.1:9092 --pa

rtition 2

hello

终止批处理操作吗(Y/N)? y

D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>

==================================性能测试==================================

1. 测试前准备：创建topic

1.1 查看参数描述：./bin/kafka-topics.sh

1.2 创建topic，根据实际应用场景选择replication-factor和partitions 

./bin/kafka-topics.sh --create --zookeeper zookeeper_1,zookeeper_2,zookeeper_3 --replication-factor 1 --partitions 1 --topic topic_name

1.3 查看创建topic的详细信息

./bin/kafka-topics.sh --describe --zookeeper zookeeper_1,zookeeper_2,zookeeper_3 --topic topic_name

2. producer测试： 主要不同kafka 8部分  

2.1 首先看看参数：./bin/kafka-producer-perf-test.sh

usage: producer-performance [-h] --topic TOPIC --num-records NUM-RECORDS --record-size RECORD-SIZE --throughput THROUGHPUT --producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...]    脚本规范

optional arguments:

  -h, --help             show this help message and exit

  --topic TOPIC          produce messages to this topic

  --num-records   NUM-RECORDS

                            number of messages to produce

  --record-size     RECORD-SIZE

                           message size in bytes

  --throughput     THROUGHPUT

                           throttle maximum message throughput to *approximately* THROUGHPUT messages/sec

  --producer-props   PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...]

                                 kafka producer related configuaration properties like bootstrap.servers,client.id etc..

参数producer-props可以并列填写多个参数，这里给个案例：

--producer-props bootstrap.servers=broker_1,broker_2,broker_3 acks=all

这里设定两个参数bootstrap.servers和acks，其他的都是默认，具体代表的意义，可以参看apache kafaka官网的documents>>producer config；

最后列个完整的producer测试脚本

 ./bin/kafka-producer-perf-test.sh --topic topic_name --throughput 100000 --num-records 100000 --producer-props --producer-props bootstrap.servers=broker_1,broker_2,broker_3 acks=all --record-size 1000

3. consumer测试

需要特别注意一点，就是在测试不同大小的消息队列时，最好使用不同的topic。完整流程最好这样：（1）确定测试的record-size = A1；（2）新建topic；（3）使用producer脚本和（2）的topic，脚本参数设定record-size = A1，产生足量的数据；（4）consumer测试，对应message-size = A1

按照和producer类似的方法，这里只列个脚本案例：

./bin/kafka-consumer-perf-test.sh --messages 100000 --topic topic_name --zookeeper zookeeper_1,zookeeper_2,zookeeper_3 --broker_1,broker_2,broker_3 --message-size A1