install cluster hadoop 安装集群版hadoop

前期准备

1，每台主机均已上传并解压相关软件，并安装jdk 1.6版本以上，具体方法可以参照上篇文章。

2，配置ssh互通，本质就是把本机的.ssh/id_rsa.pub文件传输到本机和远程主机.ssh/authorized_keys中

2.1 配置从master到其它主机无密码登录，理论上只设置此步骤即可

[hadoop@linux1 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):

/home/hadoop/.ssh/id_rsa already exists.

Overwrite (y/n)? y

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/hadoop/.ssh/id_rsa.

Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

24:be:da:90:5e:e3:ff:be:d1:4a:ce:f0:3c:55:01:3b hadoop@linux1

[hadoop@linux1 ~]$ cd .ssh

[hadoop@linux1 .ssh]$ ls

authorized_keys id_dsa id_dsa.pub id_rsa id_rsa.pub known_hosts

[hadoop@linux1 .ssh]$ cp id_rsa.pub authorized_keys

[hadoop@linux1 .ssh]$ ssh linux1

The authenticity of host 'linux1 (172.16.251.11)' can't be established.

RSA key fingerprint is ed:1a:0b:46:f2:08:75:c6:e5:05:25:d0:7b:25:c6:61.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'linux1,172.16.251.11' (RSA) to the list of known hosts.

Last login: Mon Dec 17 09:21:37 2012 from dtydb6

scp authorized_keys linux2:/home/hadoop/.ssh/

通过以上配置，从linux1 ssh登录linux2、linux3就不再提示输入密码了

2.2 配置从其它主机登录，以下命令分别在linux2和linux3执行

2.2.1 从ssh-keygen -t rsa，生成id_rsa.pub文件

2.2.2 scp id_rsa.pub linux1:/home/hadoop/.ssh/id_rsa.pub_linux2

2.2.3 cat id_rsa.pub_linux2 >> authorized_keys

cat id_rsa.pub_linux3 >> authorized_keys

2.2.4 scp到其它所有主机

scp authorized_keys linux2:/home/hadoop/.ssh/

scp authorized_keys linux3:/home/hadoop/.ssh/

2.3 验证ssh互通是否配置完成

ssh linux2 date

3，安装hadoop

tar -zxvf hadoop-1.0.4.tar.gz

设置环境变量

export JAVA_HOME=/usr/java/jdk1.7.0_07

PATH=$PATH:$HOME/bin:/monitor/apache-flume-1.2.0/bin:/hadoop/hadoop-1.0.4/bin

默认参数设置在src/core/core-default.xml, src/hdfs/hdfs-default.xml and src/mapred/mapred-default.xml.等相关目录，个性化的配置在conf目录下的相关文件

3.1 conf/hadoop-env.sh 配置hadoop相关进程的运行参数

设置JAVA_HOME

export JAVA_HOME=/usr/java/jdk1.7.0_07

3.2 conf/core-site.xml 设置namenode的URI访问地址

[hadoop@linux1 hadoop-1.0.4]$ vi conf/core-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://linux1:9000</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/home/hadoop/hadoop-1.0.4/var</value>

</property>

</configuration>

3.3 JobTracker相关配置信息

vi conf/mapred-site.xml

[hadoop@linux1 hadoop-1.0.4]$ vi conf/mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>mapred.job.tracker</name>

<value>linux1:9001</value>

</property>

</configuration>

3.4 hdfs配置信息，主要配置name和data的存放路径

[hadoop@linux1 hadoop-1.0.4]$ vi conf/hdfs-site.xml

<configuration>

<property>

<name>dfs.name.dir</name>

<value>/home/hadoop/name1, /home/hadoop/name2</value>

<description> </description>

</property>

<property>

<name>dfs.data.dir</name>

<value>/home/hadoop/data1, /home/hadoop/data2</value>

<description> </description>

</property>

<property>

<name>dfs.replication</name>

<value>2</value>

</property>

</configuration>

3.4 配置master和slaves文件

[hadoop@linux1 conf]$ vi masters

linux1

[hadoop@linux1 conf]$ vi slaves

linux2

linux3

3.5 把配置好的配置文件，软件分发到其他主机

[hadoop@linux1 conf]$ scp * linux3:/home/hadoop/hadoop-1.0.4/conf

或者hadoop整体打包到其它主机

4，运行hadoop

4.1 format

"conf/hdfs-site.xml" 21L, 566C written

[hadoop@linux1 hadoop-1.0.4]$ hadoop namenode -format

12/12/17 10:52:19 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG: host = linux1/172.16.251.11

STARTUP_MSG: args = [-format]

STARTUP_MSG: version = 1.0.4

STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012

************************************************************/

12/12/17 10:52:19 INFO util.GSet: VM type = 64-bit

12/12/17 10:52:19 INFO util.GSet: 2% max memory = 19.33375 MB

12/12/17 10:52:19 INFO util.GSet: capacity = 2^21 = 2097152 entries

12/12/17 10:52:19 INFO util.GSet: recommended=2097152, actual=2097152

12/12/17 10:52:19 INFO namenode.FSNamesystem: fsOwner=hadoop

12/12/17 10:52:20 INFO namenode.FSNamesystem: supergroup=supergroup

12/12/17 10:52:20 INFO namenode.FSNamesystem: isPermissionEnabled=true

12/12/17 10:52:20 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100

12/12/17 10:52:20 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)

12/12/17 10:52:20 INFO namenode.NameNode: Caching file names occuring more than 10 times

12/12/17 10:52:20 INFO common.Storage: Image file of size 112 saved in 0 seconds.

12/12/17 10:52:20 INFO common.Storage: Storage directory /home/hadoop/name1 has been successfully formatted.

12/12/17 10:52:20 INFO common.Storage: Image file of size 112 saved in 0 seconds.

12/12/17 10:52:20 INFO common.Storage: Storage directory /home/hadoop/name2 has been successfully formatted.

12/12/17 10:52:20 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at linux1/172.16.251.11

************************************************************/

[hadoop@linux1 hadoop-1.0.4]$ cd /home/hadoop/name1

[hadoop@linux1 name1]$ ls -lrt

total 8

drwxrwxr-x 2 hadoop hadoop 4096 Dec 17 10:52 image

drwxrwxr-x 2 hadoop hadoop 4096 Dec 17 10:52 current

4.2 启动hadoop

[hadoop@linux1 hadoop-1.0.4]$ bin/start-all.sh

[hadoop@linux1 hadoop-1.0.4]$ bin/start-all.sh

starting namenode, logging to /home/hadoop/hadoop-1.0.4/libexec/../logs/hadoop-hadoop-namenode-linux1.out

linux3: starting datanode, logging to /home/hadoop/hadoop-1.0.4/libexec/../logs/hadoop-hadoop-datanode-linux3.out

linux2: starting datanode, logging to /home/hadoop/hadoop-1.0.4/libexec/../logs/hadoop-hadoop-datanode-linux2.out

linux1: starting secondarynamenode, logging to /home/hadoop/hadoop-1.0.4/libexec/../logs/hadoop-hadoop-secondarynamenode-linux1.out

starting jobtracker, logging to /home/hadoop/hadoop-1.0.4/libexec/../logs/hadoop-hadoop-jobtracker-linux1.out

linux2: starting tasktracker, logging to /home/hadoop/hadoop-1.0.4/libexec/../logs/hadoop-hadoop-tasktracker-linux2.out

linux3: starting tasktracker, logging to /home/hadoop/hadoop-1.0.4/libexec/../logs/hadoop-hadoop-tasktracker-linux3.out

也可分开启动，分别执行start-dfs.sh，和start-mapred.sh

4.3 启动后，使用jps命令查看进程情况

[hadoop@linux1 logs]$ /usr/java/jdk1.7.0_07/bin/jps

17152 NameNode

17772 Jps

17315 SecondaryNameNode

17408 JobTracker

[hadoop@linux2 hadoop-1.0.4]$ /usr/java/jdk1.7.0_07/bin/jps

14355 DataNode

14867 Jps

14446 TaskTracker

也可使用web浏览器查看

参考资料
http://blog.csdn.net/hguisu/article/details/7237395 http://hadoop.apache.org/docs/r1.0.4/cluster_setup.html