集群安装：HA与Federation（测试）

准备工作：

修改网卡和主机名（在三个节点同时操作）

1）、修改网卡1：具体过程不介绍了

注意：此处只用一个网卡就可以了，本地IP（10.10.10.1）可以不用配置

Vi/etc/sysconfig/network-scripts/ifcfg-eth0

修改网卡2：

设置好后重启网络服务：

看网卡的配置信息：

2）、修改主机名（修改/etc/hosts文件）

（在三个节点同时操作）
[root@hadoop2 ~]# vi /etc/hosts

# Do not remove the following line, orvarious programs
# that require network functionality willfail.
127.0.0.1 localhost
192.168.6.52 hadoop2
10.10.10.2 hadoop2priv

192.168.6.51 hadoop1
10.10.10.1 hadoop1priv

192.168.6.53 hadoop3
10.10.10.3 hadoop3priv

注意：可以把此文件拷贝到节点2和节点3

修改完文件后，最后记得在相应的机器上执行hostname master(你修改后的名字) ，hostname slave1等；（此步可以省略）
[root@hadoop2 ~]# vi /etc/hostname
hadoop2
[root@hadoop2 ~]# hostname hadoop2

3）、关闭防火墙

需要关闭SELINX，执行：/usr/sbin/setenforce0

注意：最好是手动关闭。
还有：要把各个服务器的防火墙给关闭了，不然，后面运行时会报错。

Linux关闭防火墙命令：1） 永久性生效，重启后不会复原 开启：chkconfigiptables
on 关闭：chkconfig iptables off 2）
即时生效，重启后复原 开启：serviceiptables start
关闭：service iptables stop

2、创建用户和组，并配置信任关系来实现免密码登陆

1）、创建用户和组（在三个节点同时操作）

[root@hadoop ~]# groupadd -g 200 hadoop
[root@hadoop ~]# useradd -u 200 -g hadoophadoop
[root@hadoop ~]# passwd hadoop
Changing password for user hadoop.
New UNIX password:
BAD PASSWORD: it is based on a dictionaryword
Retype new UNIX password:
passwd: all authentication tokens updatedsuccessfully.
[root@hadoop ~]# su - hadoop

2）、配置信任关系

在节点1
在hadoop用户下生成密钥：rsa格式的密钥都选择默认格式
[hadoop@hadoop ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key(/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for nopassphrase):
Enter same passphrase again:
Your identification has been saved in/home/hadoop/.ssh/id_rsa.
Your public key has been saved in/home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
1a:d9:48:f8:de:5b:be:e7:1f:5b:fd:48:df:59:59:94hadoop@hadoop
[hadoop@hadoop ~]$ cd .ssh
[hadoop@hadoop .ssh]$ ls
id_rsa id_rsa.pub

把公钥添加到密钥中去
[hadoop@hadoop ~]$ cat .ssh/id_rsa.pub >>.ssh/authorized_keys

#修改master
密钥权限,非常容易错误的地方。
chmod go-rwx /home/hadoop/.ssh/authorized_keys

[hadoop@h1 .ssh]$ ll
total 24
-rw------- 1 hadoop hadoop 391 Jun 7 17:07 authorized_keys
注意：权限为600
-rw------- 1 hadoop hadoop 1675 Jun 7 17:06 id_rsa
-rw-r--r-- 1 hadoop hadoop 391 Jun 7 17:06 id_rsa.pub

在节点2：（注意：在hadoop用户下）
[root@hadoop2 ~]# su - hadoop
[hadoop@hadoop2 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key(/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for nopassphrase):
Enter same passphrase again:
Your identification has been saved in/home/hadoop/.ssh/id_rsa.
Your public key has been saved in/home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
43:c8:05:b9:6d:44:2c:b9:f3:c9:da:2d:64:b7:e9:83hadoop@hadoop2
[hadoop@hadoop2 ~]$ cd .ssh
[hadoop@hadoop2 .ssh]$ ls
id_rsa id_rsa.pub
[hadoop@hadoop2 .ssh]$

在节点3：（注意：在hadoop用户下）
[root@hadoop3 ~]# su - hadoop
[hadoop@hadoop3 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key(/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for nopassphrase):
Enter same passphrase again:
Your identification has been saved in/home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
be:2b:89:28:99:e2:46:1d:86:3c:cf:01:78:eb:5d:f3hadoop@hadoop3
[hadoop@hadoop3 ~]$ cd .ssh
[hadoop@hadoop3 .ssh]$ ls
id_rsa id_rsa.pub
[hadoop@hadoop3 .ssh]$

在节点1上，添加节点2和节点3的公钥到节点1的authorized_keys中：
[hadoop@hadoop1 ~]$ ssh hadoop2 cat.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'hadoop2(192.168.6.52)' can't be established.
RSA key fingerprint isbe:ac:97:91:50:9c:63:b6:4d:35:3f:60:be:e1:ab:3d.
Are you sure you want to continueconnecting (yes/no)? yes
Warning: Permanently added'hadoop2,192.168.6.52' (RSA) to the list of known hosts.
hadoop@hadoop2's password:

[hadoop@hadoop1 ~]$ ssh hadoop3 cat.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The authenticity of host 'hadoop3(192.168.6.53)' can't be established.
RSA key fingerprint isbe:ac:97:91:50:9c:63:b6:4d:35:3f:60:be:e1:ab:3d.
Are you sure you want to continueconnecting (yes/no)? yes
Warning: Permanently added'hadoop3,192.168.6.53' (RSA) to the list of known hosts.
hadoop@hadoop3's password:

把节点上的authorized_keys 文件拷贝到节点2和节点3上：
[hadoop@hadoop1 ~]$ scp.ssh/authorized_keys hadoop2:~/.ssh
hadoop@hadoop2's password:
authorized_keys 100% 792 0.8KB/s 00:00

[hadoop@hadoop1 ~]$ scp.ssh/authorized_keys hadoop3:~/.ssh
hadoop@hadoop3's password:
authorized_keys 100% 1188 1.2KB/s 00:00
注意：拷贝到节点2和节点3上的authorized_keys文件的权限和节点1修改后的一样，所以，在节点2和节点3上就不用修改了
[hadoop@hadoop2 .ssh]$ ll
total 32
-rw------- 1 hadoop hadoop 1188 Dec 1914:20 authorized_keys

验证节点1能否连通节点1、节点2和节点3
[hadoop@hadoop1 ~]$ ssh hadoop1
Last login: Thu Dec 19 14:14:24 2013 fromhadoop1
[hadoop@hadoop1 ~]$ ssh hadoop2 date
Thu Dec 19 14:16:09 CST 2013
[hadoop@hadoop1 ~]$ ssh hadoop3 date
Thu Dec 19 14:22:31 CST 2013

验证节点2能否连通节点1、节点2和节点3
[hadoop@hadoop2 ~]$ ssh hadoop2
Last login: Thu Dec 19 14:15:42 2013 fromhadoop2
[hadoop@hadoop2 ~]$ ssh hadoop1 date
Thu Dec 19 14:23:12 CST 2013
[hadoop@hadoop2 ~]$ ssh hadoop3 date
The authenticity of host 'hadoop3(192.168.6.53)' can't be established.
RSA key fingerprint isbe:ac:97:91:50:9c:63:b6:4d:35:3f:60:be:e1:ab:3d.
Are you sure you want to continueconnecting (yes/no)? yes
Warning: Permanently added'hadoop3,192.168.6.53' (RSA) to the list of known hosts.
Thu Dec 19 14:23:18 CST 2013

验证节点3能否连通节点1、节点2和节点3
hadoop@hadoop3 .ssh]$ ssh hadoop3
The authenticity of host 'hadoop3(192.168.6.53)' can't be established.
RSA key fingerprint isbe:ac:97:91:50:9c:63:b6:4d:35:3f:60:be:e1:ab:3d.
Are you sure you want to continueconnecting (yes/no)? yes
Warning: Permanently added'hadoop3,192.168.6.53' (RSA) to the list of known hosts.
Last login: Thu Dec 19 14:22:03 2013 fromhadoop1
[hadoop@hadoop3 ~]$ ssh hadoop2 date
The authenticity of host 'hadoop2(192.168.6.52)' can't be established.
RSA key fingerprint isbe:ac:97:91:50:9c:63:b6:4d:35:3f:60:be:e1:ab:3d.
Are you sure you want to continueconnecting (yes/no)? yes
Warning: Permanently added'hadoop2,192.168.6.52' (RSA) to the list of known hosts.
Thu Dec 19 14:24:08 CST 2013
[hadoop@hadoop3 ~]$ ssh hadoop2 date
Thu Dec 19 14:24:16 CST 2013

3、下载并安装 JAVA JDK系统软件

（注意：在root用户下安装，不然会报错）
请参考linux中安装jdk（在三个节点同时操作）
#下载jdk
wget http://60.28.110.228/source/package/jdk-6u21-linux-i586-rpm.bin #安装jdk
chmod +x jdk-6u21-linux-i586-rpm.bin
别忘了赋权限
./jdk-6u21-linux-i586-rpm.bin

安装,执行命令
[root@hn ~]# rpm -ivh jdk-6u17-linux-i586.rpm
(jdk的默认路径为/usr/java/jdk1.6.0_17)

#配置环境变量
注意：此处可以修改.bash_profile，也可以修改/etc/profile，也可以修改
/etc/profile.d/java.sh。通常修改.bash_profile，下面以此为例。（此处最好修改/etc/profile文件）
[root@linux64 ~]# vi .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startupprograms

PATH=$PATH:$HOME/bin

export PATH
unset USERNAME

添加java的参数
exportJAVA_HOME=/home/hadoop/java/jdk1.7.0_45
exportHADOOP_HOME=/home/hadoop/hadoop/hadoop-1.1.2
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
exportPATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH

vi /etc/profile
#复制粘贴一下内容 到 vi 中。
exportJAVA_HOME=/home/hadoop/java/jdk1.8.0_25
export HADOOP_HOME=/home/hadoop/hadoop/hadoop-2.2.0
exportPATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
exportHADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
exportHADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
exportHADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

#手动立即生效
source /etc/profile

#测试 ：下面两个命令都可能报下面的错误
[hadoop@ha2 ~]$ java
[hadoop@ha2 ~]$ jps
Error: dl failure on line 863
Error: failed/home/hadoop/java/jdk1.7.0_45/jre/lib/i386/client/libjvm.so, because/home/hadoop/java/jdk1.7.0_45/jre/lib/i386/client/libjvm.so: cannot restoresegment prot after reloc: Permission denied

这是因为SELINUX的问题，需要关闭SELINX，执行：/usr/sbin/setenforce 0

注意：最好是手动关闭。
还有：要把各个服务器的防火墙给关闭了，不然，后面运行时会报错。

Linux关闭防火墙命令：1） 永久性生效，重启后不会复原 开启：chkconfigiptables
on 关闭：chkconfig iptables off 2）
即时生效，重启后复原 开启：serviceiptables start
关闭：service iptables stop

4、检查基础环境

/sbin/ifconfig

[hadoop@master root]$ /sbin/ifconfig eth0 Link encap:Ethernet HWaddr 00:0C:29:7A:DE:12 inet addr:192.168.1.100 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe7a:de12/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:821 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1591 (1.5 KiB) TX bytes:81925 (80.0 KiB) Interrupt:67 Base address:0x2024

ping master
ssh master
jps
echo $JAVA_HOME
echo $HADOOP_HOME
Hadoop

5、在节点1解压安装hadoop

创建hadoop的安装目录（在三个节点同时操作）
[root@hadoop1 hadoop]# mkdir -p/opt/hadoop
[root@hadoop1 hadoop]# chown -Rhadoop:hadoop /opt/hadoop/
[root@hadoop1 hadoop]# chmod 755 *
[root@hadoop1 hadoop]# ll
total 61036
-rwxr-xr-x 1 hadoop hadoop 62428860 Dec 1722:48 hadoop-1.0.3.tar.gz

解压hadoop软件 （只需要在节点1上安装，配置好后拷贝到其他节点即可）
tar -xzvf hadoop-1.0.3.tar.gz

6、安装Hadoop2.2，搭建集群

在hadoopMaster上安装hadoop
首先到Apache官网上下载hadoop2.2的压缩文件，将其解压到当前用户的根文件夹中（home/fyzwjd/），将解压出的文件夹改名为hadoop。

$ sudo mv hadoop-2.2.0 hadoop
配置之前，先在本地文件系统创建以下文件夹：~/hadoop/tmp、~/dfs/data、~/dfs/name。 主要涉及的配置文件有7个：都在/hadoop/etc/hadoop文件夹下，可以用gedit命令对其进行编辑。

~/hadoop/etc/hadoop/hadoop-env.sh
~/hadoop/etc/hadoop/yarn-env.sh
~/hadoop/etc/hadoop/slaves
~/hadoop/etc/hadoop/core-site.xml
~/hadoop/etc/hadoop/hdfs-site.xml
~/hadoop/etc/hadoop/mapred-site.xml
~/hadoop/etc/hadoop/yarn-site.xml

(1) 配置文件1：hadoop-env.sh

修改JAVA_HOME值（export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_51）

(2) 配置文件2：yarn-env.sh

注意：这个可以不用配置
修改JAVA_HOME值（exportJAVA_HOME=/usr/lib/jvm/jdk1.7.0_51）

(3) 配置文件3：slaves，指定dataNode的列表。

SY-0226
SY-0225
SY-0355

(4) 配置文件4：core-site.xml

<configuration>

<property>
<name>fs.defaultFS</name>
<value>hdfs://SY-0217:8020</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>

</configuration>

(5) 配置文件5：hdfs-site.xml

<configuration>

<property>
<name>dfs.nameservices</name>
<value>hadoop-cluster1,hadoop-cluster2</value>
<description>
Comma-separated list of nameservices.
</description>
</property>
配置HDFS的命名空间：hadoop-cluster1,hadoop-cluster2这两个名字可以自己起

<!-- hadoop cluster1-->
<property>
<name>dfs.ha.namenodes.hadoop-cluster1</name>
<value>nn1,nn2</value>
<description>
The prefix for a given nameservice, contains a comma-separated
list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).
</description>
</property>
这两个名字是可以自己起的

<property>
<name>dfs.namenode.rpc-address.hadoop-cluster1.nn1</name>
<value>SY-0217:8020</value>
<description>
RPC address for nomenode1 of hadoop-cluster1
</description>
</property>

<property>
<name>dfs.namenode.rpc-address.hadoop-cluster1.nn2</name>
<value>SY-0355:8020</value>
<description>
RPC address for nomenode2 of hadoop-test
</description>
</property>

<property>
<name>dfs.namenode.http-address.hadoop-cluster1.nn1</name>
<value>SY-0217:50070</value>
<description>
The address and the base port where the dfs namenode1 web ui will listenon.
</description>
</property>

<property>
<name>dfs.namenode.http-address.hadoop-cluster1.nn2</name>
<value>SY-0355:50070</value>
<description>
The address and the base port where the dfs namenode2 web ui will listenon.
</description>
</property>

<!-- hadoop cluster2-->
<property>
<name>dfs.ha.namenodes.hadoop-cluster2</name>
<value>nn3,nn4</value>
<description>
The prefix for a given nameservice, contains a comma-separated
list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).
</description>
</property>

<property>
<name>dfs.namenode.rpc-address.hadoop-cluster2.nn3</name>
<value>SY-0226:8020</value>
<description>
RPC address for nomenode1 of hadoop-cluster1
</description>
</property>

<property>
<name>dfs.namenode.rpc-address.hadoop-cluster2.nn4</name>
<value>SY-0225:8020</value>
<description>
RPC address for nomenode2 of hadoop-test
</description>
</property>

<property>
<name>dfs.namenode.http-address.hadoop-cluster2.nn3</name>
<value>SY-0226:50070</value>
<description>
The address and the base port where the dfs namenode1 web ui will listenon.
</description>
</property>

<property>
<name>dfs.namenode.http-address.hadoop-cluster2.nn4</name>
<value>SY-0225:50070</value>
<description>
The address and the base port where the dfs namenode2 web ui will listenon.
</description>
</property>

<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/dongxicheng/hadoop/hdfs/name</value>
<description>Determines where on the local filesystem the DFS namenode
should store the name table(fsimage). If this is a comma-delimited list
of directories then the name table is replicated in all of the
directories, for redundancy. </description>
</property>
namenode元数据信息存放地址，配置一个本地地址；可以配置多个，用逗号分隔

<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://SY-0355:8485;SY-0225:8485;SY-0226:8485/hadoop-cluster1</value>
<description>A directory on shared storage between the multiplenamenodes
inan HA cluster. This directory will be written by the active and read
bythe standby in order to keep the namespaces synchronized. This directory
does not need to be listed in dfs.namenode.edits.dir above. It should be
left empty in a non-HA cluster.
</description>
</property>
注意：配置每个命名空间的journalNode，JournalNode应配置为奇数个，用分号隔开；每个NameNode的命名空间有active和standby两个NameNode节点，并且这两个节点共用相同的JournalNode；千万要注意，在不同的NameNode配置文件中，NameNode命名空间需要指定不同的名字，active和standby两个NameNode节点的命名空间名字相同。例如，NameNode节点nn1和nn2为上面的hadoop-cluster1。

<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/dongxicheng/hadoop/hdfs/data</value>
<description>Determines where on the local filesystem an DFS datanode
should store its blocks. If thisis a comma-delimited
list of directories, then data will be stored in all named
directories, typically on different devices.
Directories that do not exist are ignored.
</description>
</property>
dateNode的本地目录：建议也用多个，可以提高并发。

<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>false</value>
<description>
Whether automatic failover is enabled. See the HDFS High
Availability documentation for details on automatic HA
configuration.
</description>
</property>

<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/dongxicheng/hadoop/hdfs/journal/</value>
</property>

</configuration>
注意：最好在/home/dongxicheng/hadoop/hdfs下创建好以下目录[hadoop@h11 hdfs]$ mkdir data name journal

(6) 配置文件6：mapred-site.xml

<configuration>

<!-- MR YARN Application properties-->

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>The runtime framework for executing MapReduce jobs.
Can be one of local, classic or yarn.
</description>
</property>

<!-- jobhistory properties -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>SY-0355:10020</value>
<description>MapReduce JobHistory Server IPChost:port</description>
</property>

<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>SY-0355:19888</value>
<description>MapReduce JobHistory Server Web UIhost:port</description>
</property>

</configuration>

(7) 配置文件7：yarn-site.xml

<configuration>

<!-- Resource Manager Configs -->
<property>
<description>The hostname of the RM.</description>
<name>yarn.resourcemanager.hostname</name>

<value>SY-0217</value>
</property>
配置.resourcemanager的节点，修改为相应的主机名

<property>
<description>The address of the applications manager interface inthe RM.</description>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>

<property>
<description>The address of the schedulerinterface.</description>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>

<property>
<description>The http address of the RM webapplication.</description>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>

<property>
<description>The https adddress of the RM webapplication.</description>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>${yarn.resourcemanager.hostname}:8090</value>
</property>

<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>

<property>
<description>The address of the RM admininterface.</description>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8033</value>
</property>

<property>
<description>The class to use as the resourcescheduler.</description>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>

</property>
此时用的调度器是FairScheduler，默认CapacitySchedule；

<property>
<description>fair-scheduler conf location</description>
<name>yarn.scheduler.fair.allocation.file</name>
<value>${yarn.home.dir}/etc/hadoop/fairscheduler.xml</value>
</property>
需要创建fairscheduler.xml，并拷贝到${yarn.home.dir}目录下，fairscheduler.xml的配置在下面

<property>
<description>List of directories to store localized files in. An

application's localized file directory will be found in:
${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}.
Individual containers' work directories, called container_${contid},will
be subdirectories of this.
</description>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/dongxicheng/hadoop/yarn/local</value>
</property>
yarn.nodemanager的本地目录，需要创建；最好是配置多个目录，用逗号分隔

<property>
<description>Whether to enable log aggregation</description>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>

<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/tmp/logs</value>
</property>

<property>
<description>Amount of physical memory, in MB, that can beallocated

for containers.</description>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>1024</value>
</property>
注意：我的NM虚拟机是1G内存，1核CPU，当该值配置小于1024时，NM是无法启动的！会报错：
NodeManager from slavenode2 doesn't satisfy minimum allocations, Sending SHUTDOWN signalto the NodeManager.
所以，在配置此参数时，应按照NameManager虚拟机来配置。

<property>
<description>Number of CPU cores that can be allocated
for containers.</description>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>1</value>
</property>

<property>
<description>the valid service name should only contain a-zA-Z0-9_and can not start with numbers</description>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

</configuration>

（8）、配置文件8：fairscheduler.xml的配置

<?xml version="1.0"?>
<allocations>

<queue name="infrastructure">
<minResources>102400 mb, 50 vcores </minResources>
指定使用的内存数和cpu核数
<maxResources>153600 mb, 100 vcores </maxResources>
<maxRunningApps>200</maxRunningApps>
<minSharePreemptionTimeout>300</minSharePreemptionTimeout>
<weight>1.0</weight>
<aclSubmitApps>root,yarn,search,hdfs</aclSubmitApps>
允许访问的用户
</queue>

<queue name="tool">
<minResources>102400 mb, 30 vcores</minResources>
<maxResources>153600 mb, 50 vcores</maxResources>
</queue>

<queue name="sentiment">
<minResources>102400 mb, 30 vcores</minResources>
<maxResources>153600 mb, 50 vcores</maxResources>
</queue>

</allocations>

7、将hadoop文件夹拷贝到其他节点上。

注意：修改不同NameNode命名空间下active NameNode、standby NameNode的hdfs-site.xml配置文件的下列参数，需要指定不同的命名空间。如，hadoop-cluster1需要两两一样
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://SY-0355:8485;SY-0225:8485;SY-0226:8485/hadoop-cluster1</value>

scp –r/home/fyzwjd/hadoopfyzwjd@hadoopSlave1:~/
scp –r/home/fyzwjd/hadoopfyzwjd@hadoopSlave2:~/

8、启动与停止集群

注意：所有操作均在Hadoop部署目录下进行。

启动Hadoop集群：
-------------------------------------------------------------------
(1) 启动nn1与nn2
Step1 :
在各个JournalNode节点上，输入以下命令启动journalnode服务：
sbin/hadoop-daemon.sh start journalnode
注意：此操作在其中的三个节点上都要操作。

Step2:
在[nn1]上，对其进行格式化，并启动：
bin/hdfs namenode -format -clusterIdhadoop-cluster
sbin/hadoop-daemon.sh start namenode

Step3:
在[nn2]上，同步nn1的元数据信息：
bin/hdfs namenode -bootstrapStandby

Step4:
启动[nn2]：
sbin/hadoop-daemon.sh start namenode

经过以上四步操作，nn1和nn2均处理standby状态

Step5:
将[nn1]切换为Active
bin/hdfs haadmin -ns hadoop-cluster1-transitionToActive nn1

-------------------------------------------------------------------
(2) 启动nn3与nn4
Step1:
在[nn3]上，对其进行格式化，并启动：
bin/hdfs namenode -format -clusterIdhadoop-cluster
sbin/hadoop-daemon.sh start namenode

Step2:
在[nn4]上，同步nn3的元数据信息：
bin/hdfs namenode -bootstrapStandby

Step3:
启动[nn4]：
sbin/hadoop-daemon.sh start namenode

经过以上三步操作，nn3和nn4均处理standby状态

Step4:
将[nn3]切换为Active
bin/hdfs haadmin -ns hadoop-cluster2-transitionToActive nn3

-------------------------------------------------------------------
（3）启动所有datanode
Step6:
在[nn1]上，启动所有datanode
sbin/hadoop-daemons.sh start datanode

然后，启动yarn：
Sbin/start-yarn.sh

-------------------------------------------------------------------
（4）关闭Hadoop集群：
在[nn1]上，输入以下命令
sbin/stop-dfs.sh

在运行MRJS的slave1上执行以下命令启动MR JobHistory Server：
sbin/mr-jobhistory-daemon.sh starthistoryserver

停止集群
在RM和NN所在节点master执行：
先停止yarn：
sbin/stop-yarn.sh
再停止hdfs：
sbin/stop-dfs.sh
在运行JobHistoryServer的slave1上执行：
停止JobHistoryServer：
sbin/mr-jobhistory-daemon.sh stophistoryserver
第六步：再次启动
注意，再次启动时，所有的格式化命令都不用执行了！！！