Install Clustered (4 nodes) Apache Kafka and Zookeeper on Ubuntu 10.04

http://www.jonzobrist.com/2012/04/17/install-apache-kafka-and-zookeeper-on-ubuntu-10-04/

Roughly followed quickstart guide
http://incubator.apache.org/kafka/quickstart.html

Need Java? Pissed that Ubuntu dropped official Sun JDK from their Partner Repository?

Use
https://launchpad.net/~ferramroberto/+archive/java

Need this to add-apt-repository

apt-get install -y python-software-properties

sudo add-apt-repository ppa:ferramroberto/java

apt-get update

apt-get -y install sun-java6-jdk

java -version

#Should return something like

java version “1.6.0_26″

Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

Note that repository is yet to have a JDK for Ubuntu 12.04

I worked around this with a binary install of the latest 1.6 JDK to /usr/java

#Setup the Kafka user, install Kafka

I’m using 4 nodes, their IP’s are 192.168.1.50,51,52,53, all run Kafka and Zookeeper. The Zookeeper Admin guide recommends an odd number for the ensemble so majority rule is less likely to be split evenly/split brain style. although
I did get an error from one of them urging me to run either Kafka or Zookeeper in an odd number of nodes, which I’m not sure the reasoning behind.

groupadd -g 3320 kafka

useradd -m -d /usr/local/kafka -s /bin/bash -u 3320 -g 3320 kafka

wget http://people.apache.org/~nehanarkhede/kafka-0.7.0-incubating/kafka-0.7.0-incubating-src.tar.gz
tar -zxvf kafka-0.7.0-incubating-src.tar.gz

cd kafka-0.7.0-incubating-src

mv * ~kafka/

chown -R kafka:kafka ~kafka

chown -R kafka:kafka ~kafka/*

su kafka

cd

./sbt update

./sbt package

#Now edit ~kafka/.bashrc and add these lines, you may have to change the quotes, since web clipboards will likely paste in non working versions of fancy quotes

declare -x JAVA_OPTS=”-Xmx3600M -Xms256M”

declare -x PATH=”${HOME}/bin:${PATH}”

mkdir -p ~/zookeeper ~/logs ~/run/logs

#Configure Kafka

#setup myid file, myid has to match the server’s setting in ~kafka/config/zookeeper.properties

#This will be different on each server (1,2,3,4)

echo “2” > /usr/local/kafka/zookeeper/myid

#Setup Kafka’s config file, server.properties, it needs direct pointers to all servers in the Kafka cluster on the zk.connect setting

brokerid=2
port=9092
num.threads=8
socket.send.buffer=1048576
socket.receive.buffer=1048576
max.socket.request.bytes=104857600
log.dir=/usr/local/kafka/logs
num.partitions=1
log.flush.interval=10000
log.default.flush.interval.ms=1000
log.default.flush.scheduler.interval.ms=1000
log.retention.hours=168
log.file.size=536870912
log.cleanup.interval.mins=1
enable.zookeeper=true
zk.connect=192.168.1.50:2181,192.168.1.51:2181,192.168.1.52:2181,192.168.1.53:2181
zk.connectiontimeout.ms=1000000

#Setup Zookeeper’s config file, needs server.X for each node along with IP:port:port

vi config/zookeeper.properties

I ran this bash one liner on each node to get the server.X lines

echo "server.$(grep brokerid server.properties  | sed 's/=/ /' | awk '{ print $2 }')=$(hostname -I):2888:3888"

Then removed the spaces and put them into zookeeper.properties

dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
tickTime=2000
dataDir=/usr/local/kafka/zookeeper/
initLimit=5
syncLimit=2
server.1=192.168.1.50:2888:3888
server.2=192.168.1.51:2888:3888
server.3=192.168.1.52:2888:3888
server.4=192.168.1.53:2888:3888

#Setup Kafka’s producer.properties

broker.list=1:192.168.1.50:9092,2:192.168.1.51:9092,3:192.168.1.52:9092,4:192.168.1.53:9092
zk.connect=192.168.1.50:2181,192.168.1.51:2181,192.168.1.52:2181,192.168.1.53:2181
producer.type=sync
compression.codec=0
serializer.class=kafka.serializer.StringEncoder

#Setup Kafka’s consumer.properties

zk.connect=192.168.1.50:2181,192.168.1.51:2181,192.168.1.52:2181,192.168.1.53:2181
zk.connectiontimeout.ms=1000000
groupid=test-consumer-group

#Start it all, the docs say to use something like djb’s daemon tools, but for now I just run these in a screen, as user kafka

screen -R zookeeper
cd
~/bin/zookeeper-server-start.sh ~/config/zookeeper.properties
CTRL+A D

screen -R kafka
cd
~/bin/kafka-server-start.sh ~/config/server.properties
CTRL+A D

Check zookeeper

telnet localhost 2181
ruok

#expect back imok

#Send some Kafka messages

#Start Producer

bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic test
This is a test
This is more testing

#Shutdown producer

CTRL+D

#Start consumer

#Receive the messages

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

#Expect to see

This is a test

This is more testing

References
http://incubator.apache.org/kafka/quickstart.html
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html
http://zookeeper.apache.org/doc/r3.4.3/recipes.html
http://zookeeper.apache.org/doc/trunk/zookeeperJMX.html

Download Kafka from
http://incubator.apache.org/kafka/downloads.html