Hadoop集群(一) Zookeeper搭建
2017-10-10 17:48
387 查看
作为Hadoop初学者,自然要从安装入手。而hadoop的优势就是分布式,所以,也一定要安装分布式的系统。
整体安装步骤,包括Zookeeper+HDFS+Hbase,为了文章简洁,我会分三篇blog记录我的安装步骤。
本文记录的是集群环境准备,zookeeper的安装步骤。
~~~~~~~~~~~环境准备部分~~~~~~~~~~~~~~~~~
IP部署规划(准备了三个虚拟机,操作系统是Centos6.7)
Zookeeper主机分配
创建的用户有:
整体安装步骤,包括Zookeeper+HDFS+Hbase,为了文章简洁,我会分三篇blog记录我的安装步骤。
本文记录的是集群环境准备,zookeeper的安装步骤。
~~~~~~~~~~~环境准备部分~~~~~~~~~~~~~~~~~
IP部署规划(准备了三个虚拟机,操作系统是Centos6.7)
Zookeeper主机分配
192.168.67.101 c6701 192.168.67.102 c6702 192.168.67.103 c6703HDFS主机分配
192.168.67.101 c6701 --Namenode+datanode 192.168.67.102 c6702 --datanode 192.168.67.103 c6703 --datanodeHbase主机分配
192.168.67.101 c6701 --Master + region 192.168.67.102 c6702 --region 192.168.67.103 c6703 --region1. 为了分别管理各个服务,我为每个服务,都单独创建了用户.
创建的用户有:
zk hdfs hbase2. 为每一个用户创建ssh免密,并分发到全部节点,这样,我就只在C6701上执行命令,通过远程的方式安装C6702和C6703
ssh-keygen -t rsa -f ~/.ssh/id_rsa拷贝密钥到全部节点即可(只有c6701访问02和03不需要密码,反向访问是需要密码的)
ssh-copy-id c6701 ssh-copy-id c6702 ssh-copy-id c67033. 强调一下,/etc/hosts 需要设置
[root@c6701 ~]# more /etc/hosts 192.168.67.101 c6701.org c6701 192.168.67.102 c6702.org c6702 192.168.67.103 c6703.org c67034. 下面在C6701执行,中间会调用ssh远程执行一些命令,验证ssh免密是否生效
ssh c6702 "cat /proc/cpuinfo" ssh c6702 "hostname"5. 下载安装软件(内网地址)
cd /tmp/software wget http://192.21.104.48/deploy/jdk-8u144-linux-x64.tar.gz wget http://192.21.104.48/deploy/zookeeper-3.4.6.tar.gz wget http://192.21.104.48/deploy/hbase-1.1.3.tar.gz wget http://192.21.104.48/deploy/hadoop-2.6.0-EDH-0u2.tar.gz wget http://192.21.104.48/deploy/hadoop-2.7.1.tar.gz[/code]6. 安装JDK,全部节点都要安装#tar -xzvf jdk-8u144-linux-x64.tar.gz -C /usr/local7. 添加下面信息到.bash_profileexport JAVA_HOME=/usr/local/jdk1.8.0_144 export JRE_HOME=/usr/local/jdk1.8.0_144/jre export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$PATH source /etc/profile8. 检查版本信息# java -version java version "1.8.0_144" Java(TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
~~~~~~~~~~~zookeeper的安装部分~~~~~~~~~~~~~~~~~
1. 在C6701安装Zookeeperuseradd zk echo "zk:zk" | chpasswd su - zk mkdir zk tar -zxvf /tmp/software/zookeeper-3.4.6.tar.gz -C /home/zk/zk~~~~~~~~~~~~~zoo.cfg的配置~~~~~~~~~~~~~~~ $ pwd /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg $ cat /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg |grep -v '^#' tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper/data dataLogDir=/data/zookeeper/log clientPort=2181 autopurge.snapRetainCount=3 autopurge.purgeInterval=6 server.1=c6701:2888:3888 server.2=c6702:2888:3888 server.3=c6703:2888:3888 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~2. 根据zoo.cfg创建data和log两个文件夹#mkdir -p /data/zookeeper/data #mkdir -p /data/zookeeper/log #chown -R zk:zk /data/zookeeper #chown -R zk:zk /data/zookeeper/data #chown -R zk:zk /data/zookeeper/log3.在zookeeper的目录中,创建上述两个文件夹。进入zkdata文件夹,创建文件myid,填入1。这里写入的1,是在zoo.cfg文本中的server.1中的1。当我们把所有文件都配置完毕,我们把hadoop1中yarn目录复制到其它机器中,我们在修改每台机器中对应的myid文本,hadoop2中的myid写入2。其余节点,安照上面配置,依此写入相应的数字。Zkdatalog文件夹,是为了指定zookeeper产生日志指定相应的路径。# su - zk -c "echo 1 > /data/zookeeper/data/myid"4. 添加环境变量,在/etc/profile目录中,将ZOOKEEPER_HOME/bin在原有的PATH后面加入":$ZOOKEEPER_HOME/bin"
关于环境变量修改/etc目录下的profile文件,也可以在根目录下的.bashrc目录下添加环境变量。这两者有什么区别:.bashrc是对当前目录用户的环境变量,profile文件是对所有用户都开放的目录。当系统加载文件中,先从profile找相应的路劲,如果没有会在.bashrc文件中找对应的环境变量路径。这两者大家稍至了解。
然后 source /etc/profile
5. 安装c6702的zookeeper#ssh c6702 "useradd zk" #ssh c6702 "echo "zk:zk" | chpasswd"为zk用户ssh免密#ssh-copy-id zk@c6702拷贝软件#scp -r /tmp/software/hadoop-* root@c6702:/tmp/software #ssh c6702 "chmod 777 /tmp/software/*"创建目录,解压软件#ssh zk@c6702 "mkdir zk" #ssh zk@c6702 "tar -zxvf /tmp/software/zookeeper-3.4.6.tar.gz -C /home/zk/zk" #ssh zk@c6702 "ls -al zk" #ssh zk@c6702 "ls -al zk/zookeeper*" #ssh zk@c6702 "rm /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg" #scp -r /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg zk@c6702:/home/zk/zk/zookeeper-3.4.6/conf/. #ssh zk@c6702 "cat /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg |grep -v '^#'" 根据zoo.cfg创建目录 #ssh c6702 "mkdir -p /data/zookeeper/data" #ssh c6702 "chown -R zk:zk /data/zookeeper" #ssh c6702 "chown -R zk:zk /data/zookeeper/data" #ssh c6702 "mkdir -p /data/zookeeper/log" #ssh c6702 "chown -R zk:zk /data/zookeeper/log" 创建文件myid,填入2 ssh zk@c6702 "echo 2 > /data/zookeeper/data/myid"6. 安装c6703的zookeeper#ssh c6703 "useradd zk" #ssh c6703 "echo "zk:zk" | chpasswd"为zk用户ssh免密ssh-copy-id zk@c6703拷贝软件#scp -r /tmp/software/hadoop-* root@c6703:/tmp/software #ssh c6703 "chmod 777 /tmp/software/*"创建目录,解压软件#ssh zk@c6703 "mkdir zk" #ssh zk@c6703 "tar -zxvf /tmp/software/zookeeper-3.4.6.tar.gz -C /home/zk/zk" #ssh zk@c6703 "ls -al zk" #ssh zk@c6703 "ls -al zk/zookeeper*" #ssh zk@c6703 "rm /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg" #scp -r /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg zk@c6703:/home/zk/zk/zookeeper-3.4.6/conf/. #ssh zk@c6703 "cat /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg |grep -v '^#'"根据zoo.cfg创建目录#ssh c6703 "mkdir -p /data/zookeeper/data" #ssh c6703 "chown -R zk:zk /data/zookeeper" #ssh c6703 "chown -R zk:zk /data/zookeeper/data" #ssh c6703 "mkdir -p /data/zookeeper/log" #ssh c6703 "chown -R zk:zk /data/zookeeper/log" 创建文件myid,填入3 ssh zk@c6703 "echo 3 > /data/zookeeper/data/myid"7. 启动ZK/home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh start /home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh status远程启动命令ssh zk@c6702 "/home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh\ start" ssh zk@c6703 "/home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh\ start" ssh zk@c6702 "/home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh\ status" ssh zk@c6703 "/home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh\ status"
8. 没有全部启动三个ZK前的状态[vagrant@c7003 bin]$ ./zkServer.sh status ZooKeeper JMX enabled by default Using config: /home/vagrant/zookeeper-3.4.10/bin/../conf/zoo.cfg Error contacting service. It is probably not running.9. 需要将三个ZK全启动之后,才能选举出leader$ ./zkServer.sh status ZooKeeper JMX enabled by default Using config: /home/vagrant/zookeeper-3.4.10/bin/../conf/zoo.cfg Mode: follower [vagrant@c7002 bin]$ ./zkServer.sh status ZooKeeper JMX enabled by default Using config: /home/vagrant/zookeeper-3.4.10/bin/../conf/zoo.cfg Mode: leader=======2018.1.13更新==========
客户端连接方式
客户端是随机顺序的方式连接server的,无法指定,或者优先读哪个一个zookeeper server。
./zkCli.sh -server c6701:2181,c6702:2181,c6703:2181
输出如下:
第一次连接,连接到c6701上[zookeeper@c6702 bin]$ /usr/local/hadoop/zookeeper-3.4.6/bin/zkCli.sh -server c6701:2181,c6702:2181,c6703:2181 Connecting to c6701:2181,c6702:2181,c6703:2181 ...... 2018-01-11 21:07:30,797 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=c6701:2181,c6702:2181,c6703:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@4b4bc1e Welcome to ZooKeeper! 2018-01-11 21:07:30,830 [myid:] - INFO [main-SendThread(c6701.python279.org:2181):ClientCnxn$SendThread@975] - Opening socket connection to server c6701.python279.org/192.168.67.101:2181. Will not attempt to authenticate using SASL (unknown error) JLine support is enabled 2018-01-11 21:07:30,873 [myid:] - INFO [main-SendThread(c6701.python279.org:2181):ClientCnxn$SendThread@852] - Socket connection established to c6701.python279.org/192.168.67.101:2181, initiating session 2018-01-11 21:07:30,916 [myid:] - INFO [main-SendThread(c6701.python279.org:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server c6701.python279.org/192.168.67.101:2181, sessionid = 0x160e70285f70001, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null第二次连接,连接到c6702上Connecting to c6701:2181,c6702:2181,c6703:2181 ...... 2018-01-11 21:10:18,442 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=c6701:2181,c6702:2181,c6703:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@4b4bc1e Welcome to ZooKeeper! 2018-01-11 21:10:18,489 [myid:] - INFO [main-SendThread(c6702.python279.org:2181):ClientCnxn$SendThread@975] - Opening socket connection to server c6702.python279.org/192.168.67.102:2181. Will not attempt to authenticate using SASL (unknown error) JLine support is enabled 2018-01-11 21:10:18,508 [myid:] - INFO [main-SendThread(c6702.python279.org:2181):ClientCnxn$SendThread@852] - Socket connection established to c6702.python279.org/192.168.67.102:2181, initiating session 2018-01-11 21:10:18,561 [myid:] - INFO [main-SendThread(c6702.python279.org:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server c6702.python279.org/192.168.67.102:2181, sessionid = 0x260e70284650002, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null
至此,基础环境部署完。并且zookeeper也安装完成。下面我们会在下一篇文章中,继续安装HDFS。
系列文章,请参考
Hadoop集群(一) Zookeeper搭建
Hadoop集群(二) HDFS搭建
Hadoop集群(三) Hbase搭建
Hadoop集群(四) Hadoop升级
相关文章推荐
- Hadoop 2.4.0+zookeeper3.4.6+hbase0.98.3分布式集群搭建
- 第八章 搭建hadoop2.2.0集群,Zookeeper集群和hbase-0.98.0-hadoop2-bin.tar.gz集群
- hadoop2.5.2HA高可靠性集群搭建(zookeeper3.4.6+hbase0.98.11)
- 伪分布式集群环境hadoop、hbase、zookeeper搭建
- 分布式集群环境hadoop、hbase、zookeeper搭建(全)
- hadoop集群搭建二: hadoop+zookeeper 配置namenode与resourcemanager的高可用
- hadoop2.x配合ZooKeeper集群环境搭建
- Hadoop2.0集群、Hbase集群、Zookeeper集群、Hive工具、Sqoop工具、Flume工具搭建总结
- Hadoop+Hbase+Zookeeper集群环境搭建
- Hadoop-2.6.0+Zookeeper-3.4.6+Spark-1.5.0+Hbase-1.1.2+Hive-1.2.0集群搭建
- hadoop-2.7.4+hbase-1.3.1+zookeeper-3.4.9搭建分布式集群环境
- Hadoop2.5.2 HA高可靠性集群搭建(Hadoop+Zookeeper)
- Ubuntu16.04 下 Spark2.0.2+Hadoop2.7.3+Zookeeper3.4.9+HBase1.1.7集群环境搭建--5
- 配置密码分布式集群环境hadoop、hbase、zookeeper搭建(全)
- hadoop、spark、zookeeper、hive集群搭建脚本
- 集群RedHat6.5+JDK1.8+Hadoop2.7.3+Spark2.1.1+zookeeper3.4.6+kafka2.11+flume1.6环境搭建步骤
- Hadoop加zookeeper搭建高可靠集群
- hadoop2.6.5+zookeeper3.4.10+hbase1.3.1分布式集群搭建
- Hadoop-2.6.0+Zookeeper-3.4.6+Spark-1.5.0+Hbase-1.1.2+Hive-1.2.0集群搭建
- Hadoop+Flume+Kafka+Zookeeper集群环境搭建(一)