在ArchLinux上安装单机版本kafka
2016-04-29 16:51
651 查看
在ArchLinux上安装单机版本kafka
Table of Contents
1. 下载2. 启动zookeeper
3. 测试zookeeper
4. 启动kafka服务
5. 测试kafka服务5.1. 创建topic test
5.2. 发送测试消息
5.3. 接收测试消息
1 下载
wget https://archive.apache.org/dist/kafka/0.8.1/kafka_2.10-0.8.1.tgz然后解压tar zxvf kafka_2.10-0.8.1.tgz
2 启动zookeeper
kafka安装包自带了zookeeper程序,很方便。进入bin目录,运行下面的脚本[dean@dell_xps_13 bin]$ ./zookeeper-server-start.sh ../config/zookeeper.properties [2016-04-29 16:03:41,484] INFO Reading configuration from: ../config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2016-04-29 16:03:41,484] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain) [2016-04-29 16:03:41,498] INFO Reading configuration from: ../config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2016-04-29 16:03:41,499] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain) [2016-04-29 16:03:41,503] INFO Server environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:host.name=localhost.localdomain (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:java.version=1.8.0_77 (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:java.vendor=Oracle Corporation (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:java.home=/usr/lib/jvm/java-8-openjdk/jre (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:java.class.path=:/home/dean/work/software/kafka_2.10-0.8.1/bin/../core/build/dependant-libs-2.8.0/*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../perf/build/libs//kafka-perf_2.8.0*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../clients/build/libs//kafka-clients*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../examples/build/libs//kafka-examples*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/jopt-simple-3.2.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/kafka_2.10-0.8.1.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/log4j-1.2.15.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/metrics-annotation-2.2.0.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/metrics-core-2.2.0.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/scala-library-2.10.1.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/slf4j-api-1.7.2.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/snappy-java-1.0.5.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/zkclient-0.3.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/zookeeper-3.3.4.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../core/build/libs/kafka_2.8.0*.jar (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:os.version=4.5.1-1-ARCH (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:user.name=dean (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:user.home=/home/dean (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,503] INFO Server environment:user.dir=/home/dean/work/software/kafka_2.10-0.8.1/bin (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,507] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,507] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,507] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer) [2016-04-29 16:03:41,520] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxn) [2016-04-29 16:03:41,535] INFO Snapshotting: 0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2016-04-29 16:10:44,072] INFO Accepted socket connection from /0:0:0:0:0:0:0:1:34216 (org.apache.zookeeper.server.NIOServerCnxn) [2016-04-29 16:10:44,111] INFO Client attempting to establish new session at /0:0:0:0:0:0:0:1:34216 (org.apache.zookeeper.server.NIOServerCnxn) [2016-04-29 16:10:44,113] INFO Creating new log file: log.1 (org.apache.zookeeper.server.persistence.FileTxnLog) [2016-04-29 16:10:44,118] INFO Established session 0x154610b75630000 with negotiated timeout 30000 for client /0:0:0:0:0:0:0:1:34216 (org.apache.zookeeper.server.NIOServerCnxn) [2016-04-29 16:24:37,145] INFO Processed session termination for sessionid: 0x154610b75630000 (org.apache.zookeeper.server.PrepRequestProcessor) [2016-04-29 16:24:37,151] INFO Closed socket connection for client /0:0:0:0:0:0:0:1:34216 which had sessionid 0x154610b75630000 (org.apache.zookeeper.server.NIOServerCnxn)
3 测试zookeeper
现在需要连接zookeeper server, 确人服务正常工作仍然在bin目录里,运行下面的脚本[dean@dell_xps_13 bin]$ ./zookeeper-shell.sh localhost:2181 Connecting to localhost:2181 Welcome to ZooKeeper! JLine support is disabled WATCHER:: WatchedEvent state:SyncConnected type:None path:null help ZooKeeper -server host:port cmd args stat path [watch] set path data [version] ls path [watch] delquota [-n|-b] path ls2 path [watch] setAcl path acl setquota -n|-b val path history redo cmdno printwatches on|off delete path [version] sync path listquota path get path [watch] create [-s] [-e] path data acl addauth scheme auth quit getAcl path close connect host:port ls / [zookeeper] exit这里连接了本地运行的zookeeper, 通过端口2181, 进入shell后,运行了两个命令:help 和 ls.
4 启动kafka服务
还是在bin目录下,运行命令[dean@dell_xps_13 bin]$ ./kafka-server-start.sh ../config/server.properties [2016-04-29 16:33:51,877] INFO Verifying properties (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,896] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,896] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,897] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,897] INFO Property log.retention.check.interval.ms is overridden to 60000 (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,897] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,897] INFO Property log.segment.bytes is overridden to 536870912 (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,898] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,898] INFO Property num.network.threads is overridden to 2 (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,898] INFO Property num.partitions is overridden to 2 (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,898] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,898] INFO Property socket.receive.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,898] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,898] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,898] INFO Property zookeeper.connect is overridden to localhost:2181 (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,898] INFO Property zookeeper.connection.timeout.ms is overridden to 1000000 (kafka.utils.VerifiableProperties) [2016-04-29 16:33:51,907] INFO [Kafka Server 0], starting (kafka.server.KafkaServer) [2016-04-29 16:33:51,908] INFO [Kafka Server 0], Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer) [2016-04-29 16:33:51,914] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread) [2016-04-29 16:33:51,918] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:host.name=localhost.localdomain (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:java.version=1.8.0_77 (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:java.home=/usr/lib/jvm/java-8-openjdk/jre (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:java.class.path=:/home/dean/work/software/kafka_2.10-0.8.1/bin/../core/build/dependant-libs-2.8.0/*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../perf/build/libs//kafka-perf_2.8.0*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../clients/build/libs//kafka-clients*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../examples/build/libs//kafka-examples*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/jopt-simple-3.2.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/kafka_2.10-0.8.1.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/log4j-1.2.15.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/metrics-annotation-2.2.0.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/metrics-core-2.2.0.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/scala-library-2.10.1.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/slf4j-api-1.7.2.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/snappy-java-1.0.5.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/zkclient-0.3.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/zookeeper-3.3.4.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../core/build/libs/kafka_2.8.0*.jar (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:os.version=4.5.1-1-ARCH (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:user.name=dean (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:user.home=/home/dean (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,918] INFO Client environment:user.dir=/home/dean/work/software/kafka_2.10-0.8.1/bin (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,919] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@40f08448 (org.apache.zookeeper.ZooKeeper) [2016-04-29 16:33:51,925] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181 (org.apache.zookeeper.ClientCnxn) [2016-04-29 16:33:51,968] INFO Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2016-04-29 16:33:51,972] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x154610b75630001, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn) [2016-04-29 16:33:51,976] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) [2016-04-29 16:33:52,022] INFO Log directory '/tmp/kafka-logs' not found, creating it. (kafka.log.LogManager) [2016-04-29 16:33:52,028] INFO Starting log cleanup with a period of 60000 ms. (kafka.log.LogManager) [2016-04-29 16:33:52,030] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager) SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [2016-04-29 16:33:52,047] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor) [2016-04-29 16:33:52,047] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer) [2016-04-29 16:33:52,083] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$) [2016-04-29 16:33:52,100] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector) [2016-04-29 16:33:52,152] INFO Registered broker 0 at path /brokers/ids/0 with address localhost.localdomain:9092. (kafka.utils.ZkUtils$) [2016-04-29 16:33:52,162] INFO [Kafka Server 0], started (kafka.server.KafkaServer) [2016-04-29 16:33:52,195] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
5 测试kafka服务
5.1 创建topic test
[dean@dell_xps_13 bin]$ ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test Created topic "test".
5.2 发送测试消息
[dean@dell_xps_13 bin]$ [dean@dell_xps_13 bin]$ ./kafka-console-producer.sh --broker-list localhost:9092 --topic test SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. This is a test This is another message
5.3 接收测试消息
运行下面的命令[dean@dell_xps_13 bin]$ ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. This is a test This is another message现在ArchLinux本地的开发环境已经设置好,后续可以开始编写、测试使用librdkafka++的应用程序了。
Created: 2016-04-29 Fri 16:48Validate
相关文章推荐
- Kafka 之 中级
- 滚动更新与固定版本Linux之争
- Linux下Kafka单机安装配置方法(图文)
- Kafka使用入门教程第1/2页
- Logstash 与Elasticsearch整合使用示例
- kafka-manager 的编译和使用(附安装包)
- Kafka+Log4j实现日志集中管理
- Kafka深度解析
- Kafka设计解析(三)- Kafka High Availability (下)
- kafka+storm初探
- storm集群 + kafka单机性能测试
- flume、kafka、storm常用命令
- kafka集群的安装
- kafka 一些基本知识
- Kafka入门经典教程
- Kafka初步学习总结
- 自研轻量级分布式实时计算框架light_drtc
- note of kafka learning (first part, before replication)
- archlinux 静态网络配置