Hadoop自动化安装脚本
2016-09-29 14:51
232 查看
参考资料和说明
参考资料: hadoop-2.3.0详细安装过程码云GIT: hadoop-2.4.0家族自动化安装脚本 》》》 获取邀请码
版本信息: hadoop-2.4.0-x64、hive-1.2.1、Zookeeper-3.4.5、Hbase-0.98.21
你没看错,这就是2.4.0版本的安装脚本,虽然参考的是2.3.0的。
安装环境
虚拟机软件: VirtualBox-4.3.8-92456-Win.exe虚拟机系统: CentOS-6.5-x86_64-bin-DVD1.iso,64位,分配1CPU,2G内存
Hadoop版本: hadoop-2.4.0-64bit.tar.gz 下载
Nginx文件服务器: nginx-1.10.0.zip Windows版本,解压即用,主要用来提供文件下载。
主要说一下这些脚本都是干什么的,具体安装过程请到git中查看脚本内容。
执行脚本
sloth/linux/hadoop/sln-2.4.0.shhadoop自动安装的主要入口,以root用户执行,操作示例如下:
Last login: Wed Sep 28 15:49:27 2016 from 172.16.4.102 [root@name00 ~]# cd /home/install/hadoop/ [root@name00 hadoop]# ./sln-2.4.0.sh ./sln-2.4.0.sh 错误:请传入参数,(1、类型:name/node;2、机器名:name00/node01;3、机器IP)! 例如:./sln-2.4.0.sh name name00 127.0.0.1 [root@name00 hadoop]# ./sln-2.4.0.sh node node04 172.16.117.4 参数确认: 类型:node 机器名: node04 机器IP:172.16.117.4 继续/跳过(Y/N):y 开始安装…… SSH链接慢?(Y/N): 跳过…… 授权公钥……继续/跳过(Y/N): 跳过…… 下载脚本……继续/跳过(Y/N): 跳过…… 删除已rpm安装的软件……继续/跳过(Y/N): 跳过…… 安装java……继续/跳过(Y/N): 跳过…… 安装mysql……继续/跳过(Y/N): 跳过…… 安装hadoop……继续/跳过(Y/N): 跳过…… 安装hive……继续/跳过(Y/N): 跳过…… 安装zookeeper……继续/跳过(Y/N): 跳过…… 安装hbase……继续/跳过(Y/N): 跳过…… 设置环境变量……继续/跳过(Y/N): 跳过…… 立即加入Hadoop集群……继续/跳过(Y/N): 跳过…… [root@name00 hadoop]#
允许用户交互,根据需要选择安装。
测试和监控
# 切换用户 su hadoop # 启动所有 cd /home/hadoop/src/hadoop-2.4.0/sbin/;./start-all.sh # 停止所有 cd /home/hadoop/src/hadoop-2.4.0/sbin/;./stop-all.sh
验证
hadoop fs -ls hdfs://name00:9000 / hadoop fs -mkdir hdfs://name00:9000 /testfolder
mapreduce例子
cd /home/hadoop/src/hadoop-2.4.0/share/hadoop/mapreduce/ hadoop fs -copyFromLocal /home/logs/hive/hive.log /input/hive.log hadoop fs -ls /input hadoop jar hadoop-mapreduce-examples-2.4.0.jar wordcount /input/hive.log /wordout hadoop fs -ls /wordout hadoop fs -tail /wordout/part-r-00000
web监控
服务器状态
http://name00:50070/dfshealth.html
集群信息
http://name00:8088/cluster
查看所有日志信息
http://name00:50070/logs/
问题或期望:
1. Hadoop所有节点两两间无秘访问(已解决)hadoop需要在各个节点创建一个用户,并保证用户之间能够无秘访问,互联互通,虽然现在能够用脚本自动生成公钥,并将公钥分发给各个节点,但是两两节点间第一次登录访问还是需要确认和输入密码,节点少当然没问题,但是节点多了,一个个去确认也麻烦,求解决方案!!!
如下,新增节点,启动节点的时候需要输入确认
[hadoop@node11704 ~]$ cd /home/hadoop/src/hadoop-2.4.0/sbin/;./start-yarn.sh starting yarn daemons starting resourcemanager, logging to /home/hadoop/src/hadoop-2.4.0/logs/yarn-hadoop-resourcemanager-node11904.out The authenticity of host 'node11901 (172.16.119.1)' can't be established. RSA key fingerprint is db:e7:f0:db:d7:f7:f7:98:c2:b6:73:ec:a9:6e:0e:9c. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'node11902 (172.16.119.2)' can't be established. RSA key fingerprint is db:e7:f0:db:d7:f7:f7:98:c2:b6:73:ec:a9:6e:0e:9c. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'node11703 (172.16.117.3)' can't be established. RSA key fingerprint is db:e7:f0:db:d7:f7:f7:98:c2:b6:73:ec:a9:6e:0e:9c. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'node11704 (172.16.117.4)' can't be established. RSA key fingerprint is db:e7:f0:db:d7:f7:f7:98:c2:b6:73:ec:a9:6e:0e:9c. Are you sure you want to continue connecting (yes/no)? yes node11901: Warning: Permanently added 'node11901,172.16.119.1' (RSA) to the list of known hosts. node11901: nodemanager running as process 1809. Stop it first. yes node11902: Warning: Permanently added 'node11902,172.16.119.2' (RSA) to the list of known hosts. node11902: nodemanager running as process 1755. Stop it first. yes node11703: Warning: Permanently added 'node11703,172.16.117.3' (RSA) to the list of known hosts. node11703: nodemanager running as process 2169. Stop it first. yes node11704: Warning: Permanently added 'node11704,172.16.117.4' (RSA) to the list of known hosts. node11704: starting nodemanager, logging to /home/hadoop/src/hadoop-2.4.0/logs/yarn-hadoop-nodemanager-node11704.out [hadoop@node11704 sbin]$ jps 2767 NodeManager 2824 Jps
[b]解决方法:[/b]
ssh加速,自动添加到know_hosts
[root@xytr1 ~]# ssh 192.168.5.56 The authenticity of host '192.168.5.56 (192.168.5.56)' can't be established. RSA key fingerprint is a1:7f:35:ec:1a:5e:33:df:e8:82:56:cf:b6:99:b6:b9. Are you sure you want to continue connecting (yes/no)? yes ß 这里一定要确认才能通过 Warning: Permanently added '192.168.5.56' (RSA) to the list of known hosts. Last login: Tue Jul 13 16:29:52 2010 from linux1 解决方案是,把 本地 的ssh客户端配置文件 ssh_config 做修改 [root@xytr1 ~]# vi /etc/ssh/ssh_config 找到: # StrictHostKeyChecking ask 修改为 StrictHostKeyChecking no 这样下次就不会出现需要确认的提示了。
2. 下载更新脚本
目前的实现方式如下
read -p "下载脚本……继续/跳过(Y/N):" isY if [ "${isY}" == "y" ] || [ "${isY}" == "Y" ];then ##下载脚本,并删除已rpm安装的软件 ssh root@$3 "mkdir -p /home/install" ## 下载脚本 #ssh root@$3 "cd /home/install;wget -N -r -nH -np -k -L -p $URL_SHELL" #层级下载有点问题 ## 下载脚本包 ssh root@$3 "cd /home/install;rm -f linux.zip;wget $URL_SHELL/linux.zip" ## 解压并提权 ssh root@$3 "cd /home/install;unzip -o linux.zip;chmod a+x /home/install -R" else echo ' 跳过……' fi
需要先在windows下打包一个所有脚本的zip包,再通过wget命令下载,我希望直接从文件服务器遍历下载更新过的脚本,并覆盖,这样就不用每次更新一个脚本都要手动打包一次。
原来我尝试过一次
## 下载脚本 ssh root@$3 "cd /home/install;wget -N -r -nH -np -k -L -p $URL_SHELL"
但是这个命令本地执行没问题,但远端执行就返回错误,会中断整个安装脚本,所以才改成下载压缩包的。
相关文章推荐
- Hadoop自动化安装shell脚本
- hadoop集群安装与配置--hadoop自动化安装脚本
- 自动化脚本腾讯云配置集群(四)自动化安装从节点的JDK+Hadoop
- LAMP自动化安装脚本 推荐
- 自动化安装admuser的脚本
- linux(centos6)下 hadoop安装之ssh配置脚本 permission denied
- 用SHELL脚本自动化安装Nagios服务器端和客户端的
- CentOS 或 Fedora 下自动化安装mono和jexus的脚本(Linux下的Asp.net运行环境配置)
- Apache,tomcat,nginx,apache+tomcat,nginx+tomcat自动化安装脚本
- Ubuntu13.04-amd64安装完后的Chrome及VIMIDE自动化配置脚本
- 用SHELL脚本自动化安装Nagios服务器端和客户端的
- 用SHELL脚本自动化安装Nagios服务器端和客户端的 推荐
- linux(centos6)下 hadoop安装之ssh配置脚本 permission denied
- CentOS使用yum配置安装LAMP环境自动化脚本
- php自动化安装脚本--还是接上篇
- 用SHELL脚本自动化安装Nagios服务器端和客户端的
- LAMP自动化安装脚本更新(添加了mcrypt)
- MySQL 自动化编译安装脚本
- Ubuntu 12.04 Hadoop自动安装脚本
- 用SHELL脚本自动化安装Nagios服务器端和客户端的