Hadoop2.7.2集群的部署
2016-04-11 11:07
411 查看
原文地址:http://happyshome.cn/blog/deploy/centos/hadoop2.7.2/cluster.html
![](http://ww4.sinaimg.cn/large/006pQ25sgw1f2pihzxdw3j30dw04qq3o.jpg)
本文使用了4台虚拟机成功的部署了hadoop2.7.2的集群开发环境。
假设你的环境已经具备了运行Hadoop的必要条件,如果没有你可以参考Centos上源码安装Hadoop2.7.2。
这里我准备了4台Centos7虚拟机,下面我们马上来进行配置。
[/code]
[/code]
[/code]
[/code]
[/code]
你也可以通过查看 http://hmaster:50070进行查看
![](http://ww4.sinaimg.cn/large/006pQ25sgw1f2pihzxdw3j30dw04qq3o.jpg)
本文使用了4台虚拟机成功的部署了hadoop2.7.2的集群开发环境。
1. 前言
假设你的环境已经具备了运行Hadoop的必要条件,如果没有你可以参考Centos上源码安装Hadoop2.7.2。
2. 准备工作
这里我准备了4台Centos7虚拟机,下面我们马上来进行配置。# 编辑host vi /etc/hosts # 每个节点添加IP、主机名称 192.168.10.162 hmaster 192.168.10.163 hslave1 192.168.10.164 hslave2 192.168.10.166 hslave3 # 创建hadoop用户 useradd hadoop # 设置密码 passwd hadoop # 设置权限 chown hadoop /app/hadoop-2.7.2/ -R chgrp hadoop /app/hadoop-2.7.2/ -R # ssh自动登陆 su hadoop # 这里一路回车即可 ssh-keygen -t rsa ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hmaster ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hslave1 ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hslave2 ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hslave3 chmod 0600 ~/.ssh/authorized_keys # 关闭防火墙 systemctl stop firewalld # 也可以编辑sysctl.conf vi /etc/sysctl.conf # 添加以下内容 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1
[/code]
3. 部署
3.1. 设置hadoop环境变量
# 切换到hadoop用户,添加hadoop环境变量 vi ~/.bashrc # 添加以下内容 export HADOOP_PREFIX=/app/hadoop-2.7.2 export HADOOP_HOME=$HADOOP_PREFIX export HADOOP_COMMON_HOME=$HADOOP_PREFIX export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop export HADOOP_HDFS_HOME=$HADOOP_PREFIX export HADOOP_MAPRED_HOME=$HADOOP_PREFIX export HADOOP_YARN_HOME=$HADOOP_PREFIX export PATH=$PATH:$HADOOP_PREFIX/sbin:$HADOOP_PREFIX/bin # 退出,使之生效 source ~/.bashrc
[/code]
3.2. 设置master
# 1. 编辑core-site.xml vi /app/hadoop-2.7.2/etc/hadoop/core-site.xml # 添加以下内容 <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hmaster:9000/</value> </property> </configuration> # 2. 创建NameNode目录 mkdir /home/hadoop/namenode chown hadoop /home/hadoop/namenode/ chgrp hadoop /home/hadoop/namenode/ # 3. 编辑hdfs-site.xml vi /app/hadoop-2.7.2/etc/hadoop/hdfs-site.xml # 添加以下内容 <configuration> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/home/hadoop/datanode</value> </property> <property> <name>dfs.namenode.data.dir</name> <value>/home/hadoop/namenode</value> </property> </configuration> # 4. 编辑mapred-site.xml vi /app/hadoop-2.7.2/etc/hadoop/mapred-site.xml # 添加以下内容 <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> # 5. 编辑yarn-site.xml设置ResourceManager和NodeManagers <configuration> <property> <name>yarn.resourcemanager.hostname</name> <value>hmaster</value> </property> <property> <name>yarn.nodemanager.hostname</name> <value>hmaster</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration> # 6. 编辑slaves添加DataNode vi /app/hadoop-2.7.2/etc/hadoop/slaves # 添加以下内容 hmaster hslave1 hslave2 hslave3
[/code]
3.3. 设置DataNode
# 1. 编辑core-site.xml vi /app/hadoop-2.7.2/etc/hadoop/core-site.xml # 添加以下内容 <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hmaster:9000/</value> </property> </configuration> # 2. 创建DataNode目录 mkdir /home/hadoop/datanode chown hadoop /home/hadoop/datanode/ chgrp hadoop /home/hadoop/datanode/ # 3. 编辑hdfs-site.xml vi /app/hadoop-2.7.2/etc/hadoop/hdfs-site.xml # 添加以下内容 <configuration> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/home/hadoop/datanode</value> </property> </configuration>
[/code]
3.4. 启动
# 在master上切换到hadoop用户 su hadoop hdfs namenode -format # 启动命令后, 使用jps查看任务是否启动, Slave节点:DataNode, Master节点: NameNode, DataNode, SecondaryNameNode cd /app/hadoop-2.7.2/etc/hadoop sbin/start-dfs.sh # 启动YARN, 查看Master节点上的ResourceManager和NodeManager是否启动 sbin/start-yarn.sh # 查看集群状态 hdfs dfsadmin -report Configured Capacity: 74985766912 (69.84 GB) Present Capacity: 21625159680 (20.14 GB) DFS Remaining: 21625114624 (20.14 GB) DFS Used: 45056 (44 KB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ------------------------------------------------- Live datanodes (4): Name: 192.168.10.163:50010 (hslave1) Hostname: hslave1 Decommission Status : Normal Configured Capacity: 18746441728 (17.46 GB) DFS Used: 12288 (12 KB) Non DFS Used: 13337149440 (12.42 GB) DFS Remaining: 5409280000 (5.04 GB) DFS Used%: 0.00% DFS Remaining%: 28.85% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Sat Apr 09 02:18:49 CST 2016 Name: 192.168.10.164:50010 (hslave2) Hostname: hslave2 Decommission Status : Normal Configured Capacity: 18746441728 (17.46 GB) DFS Used: 12288 (12 KB) Non DFS Used: 13339152384 (12.42 GB) DFS Remaining: 5407277056 (5.04 GB) DFS Used%: 0.00% DFS Remaining%: 28.84% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Sat Apr 09 02:18:50 CST 2016 Name: 192.168.10.162:50010 (hmaster) Hostname: hmaster Decommission Status : Normal Configured Capacity: 18746441728 (17.46 GB) DFS Used: 12288 (12 KB) Non DFS Used: 13345161216 (12.43 GB) DFS Remaining: 5401268224 (5.03 GB) DFS Used%: 0.00% DFS Remaining%: 28.81% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Sat Apr 09 02:18:50 CST 2016 Name: 192.168.10.166:50010 (hslave3) Hostname: hslave3 Decommission Status : Normal Configured Capacity: 18746441728 (17.46 GB) DFS Used: 8192 (8 KB) Non DFS Used: 13339144192 (12.42 GB) DFS Remaining: 5407289344 (5.04 GB) DFS Used%: 0.00% DFS Remaining%: 28.84% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Sat Apr 09 02:18:51 CST 2016
[/code]
你也可以通过查看 http://hmaster:50070进行查看
![](http://ww2.sinaimg.cn/large/006pQ25sgw1f2ph3f1o42j30dw05pmxt.jpg)
相关文章推荐
- linux 环境配置文
- 【OpenCV】OpenCV配置—VS2012+win8版
- shell下一层输入yes
- Android——FileOutputStream与openFileOutput()的区别分析【第一个用于文件存储,第二个用于SD卡存储】【转】
- 技术笔记:XMPP之openfire+spark+smack
- Linux 防火墙开放特定端口 (iptables)
- Linux 关闭iptable防火墙
- CentOS7安装Mono和MonoDevelop
- Linux 下ffmpeg视频转码实战三——ffprobe
- windows下使用命令动态输出tomcat日志
- linux awk命令详解
- linux 复制文件时,报cp: omitting directory `XXX'
- linux_tomcat7服务器日志爆满导致java崩溃的问题
- Linux 下ffmpeg视频转码实战二——参数宝典
- linux下解压命令执行出错 Not found in archive
- centos 安装svn
- stopPropagation, preventDefault 和 return false 的区别
- git在Linux上的安装和使用简介
- Linux之获取配置文件行字段
- linux安装软件:/lib/ld-linux.so.2: bad ELF interpreter: 没有那个文件或目录