centos6.5 hadoop2.6.3分布式集群安装

1.环境准备：centos6.5 jdk.1.8

192.168.118.154	master	NameNode,SecondaryNameNode,ResourceManager
192.168.118.153	slave1	DataNode.NodeManager
192.168.118.152	slave2	DataNode.NodeManager

2.添加操作系统用户

adduser Hadoop
pwd hadoop

3.修改host文件

vi /etc/hosts

192.168.118.154 master
192.168.118.153 slave1
192.168.118.152 slave2

4.配置秘钥登录
CentOS 6.5配置SSH免密码登录
提示：可以在3台机器分别生成id_rsa.pub，使用scp命令将153，152面的文件拷贝到154，使用cat追加到authorized_keys文件，这样authorized_keys中就有3个公钥。再将authorized_keys scp至其他两天机器，这样配置比较快。

5.下载并解压hadoop2.6.3

wget http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.6.3/hadoop-2.6.3.tar.gz

tar xvf hadoop-2.6.3.tar.gz

6.修改hadoop配置文件 
提前建好目录  ./tmp,./dfs/data,/dfs/name。
将整个hadoop文件目录所有着赋予hadoop

chown -R hadoop ./

目录： ./etc/hadoop

   core-site.xml

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/opt/hadoop/hadoop-2.6.3/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>4096</value>
</property>
</configuration>

 hdfs-site.xml

<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/opt/hadoop/hadoop-2.6.3/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/opt/hadoop/hadoop-2.6.3/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9001</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>

yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>

7. 修改hadoop-env.sh, yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_45

8. 格式化NameNode(注意：必须使用hadoop账号格式话)

./bin/Hadoop namenode -format

9.启动Hadoop

./sbin/start-all.sh

10.验证结果 : jps
master：

17609 SecondaryNameNode
17753 ResourceManager
17434 NameNode

slave1，slave2

1840 DataNode
1926 NodeManager