Hadoop2.7.3的安装详细过程(伪分布式)

1.安装java

2.设置ssh免密码登入

3.安装Hadoop

1.安装java：

rpm -ivh jdk-8u101-linux-i586.rpm

然后设置环境变量
vi /etc/profile
在最末端添加：

#set java env
JAVA_HOME=/usr/java/jdk1.8.0_101
PATH=$JAVA_HOME/bin:$PATH
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME
export PATH
export CLASSPATH

最后，再在终端输入java -version测试一下，到此，java便安装结束了。

2.设置ssh免密码登入

ssh-keygen -t rsa

这将会产生一个公钥和私钥

ssh-copy-id -i slaver1

连续按几次enter直至结束

再来一次

ssh-copy-id -i slaver2

3.安装hadoop

tar -xzvf hadoop-2.7.3.tar.gz

cd进入解压后的文件夹

首先，先修改hadoop下面etc/hadoop里面的5个文件：

cd /usr/hadoop/hadoop-2.7.3/etc/hadoop

1.core-site.xml

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/usr/hadoop/hadoop-2.7.3/tmp</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

2.yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<!-- reducer获取数据的方式 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<value>org.apache.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>

</configuration>

3.mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>master:9001</value>
</property>

</configuration>

4.hdfs-site.xml

<configuration>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/hadoop/hadoop-2.7.3/hdf/data</value>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/hadoop/hadoop-2.7.3/hdf/name</value>
<final>true</final>
</property>

</configuration>

5.hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_101
export HADOOP_HOME=/usr/hadoop/hadoop-2.7.3
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export  HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

6.yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_101
export HADOOP_HOME=/usr/hadoop/hadoop-2.7.3
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export  HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

hadoop.env.sh和yarn-env.sh这里的后三行网上很多人都没有写，听说64位的机器才需要加上，但是我的是32位的，没有加上去仍然不行。。

然后把Hadoop文件夹发到其他主机上

scp -r /usr/hadoop slaver1:/usr/hadoop

scp -r /usr/hadoop slaver1:/usr/hadoop

在最后加上环境变量，所有的主机都要设置。

vi /etc/profile

#set hadoop env
export HADOOP_HOME=/usr/hadoop/hadoop-2.7.3
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export HADOOP_LOG_DIR=/usr/hadoop/hadoop-2.7.3/logs
export YARN_LOG_DIR=$HADOOP_LOG_DIR

最后，启动hadoop集群

./sbin/start-all.sh

在终端输入

[root@master ~]# jps

28597 NameNode

28934 ResourceManager

29800 Jps

28792 SecondaryNameNode

[root@master hadoop-2.7.3]# ./bin/hdfs dfsadmin -report

16/10/29 10:02:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

Configured Capacity: 97575411712 (90.87 GB)

Present Capacity: 79984197632 (74.49 GB)

DFS Remaining: 79984140288 (74.49 GB)

DFS Used: 57344 (56 KB)

DFS Used%: 0.00%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0

Missing blocks (with replication factor 1): 0

Live datanodes (2):

Name: 192.168.183.84:50010 (slaver1)

Hostname: localhost.localdomain

Decommission Status : Normal

Configured Capacity: 49795620864 (46.38 GB)

DFS Used: 28672 (28 KB)

Non DFS Used: 8847638528 (8.24 GB)

DFS Remaining: 40947953664 (38.14 GB)

DFS Used%: 0.00%

DFS Remaining%: 82.23%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Sat Oct 29 10:02:28 EDT 2016

Name: 192.168.183.51:50010 (slaver2)

Hostname: localhost.localdomain

Decommission Status : Normal

Configured Capacity: 47779790848 (44.50 GB)

DFS Used: 28672 (28 KB)

Non DFS Used: 8743575552 (8.14 GB)

DFS Remaining: 39036186624 (36.36 GB)

DFS Used%: 0.00%

DFS Remaining%: 81.70%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Sat Oct 29 10:02:29 EDT 2016

进入http://192.168.183.70:50070/

可以看到



到此，Hadoop安装成功。