为Linux集群创建新账户，并配置hadoop集群

转自:http://blog.csdn.net/bluesky8640/article/details/6945776

之前装python、jdk、hadoop都是用的root账户，这是一个绝对的失策，linux对用户访问权限很严格，新创的hod账户根本无法启动hadoop，而root在hod模式下使用torque是不被建议的，所以只能用hod账户再重新装一遍.

创建用户、设置密码、修改用户、删除用户：

useradd testuser 创建用户testuser

passwd testuser 给已创建的用户testuser设置密码

说明：新创建的用户会在/home下创建一个用户目录testuser

usermod --help 修改用户这个命令的相关参数

userdel testuser 删除用户testuser

rm -rf testuser 删除用户testuser所在目录

上面的几个命令只有root账号才可以使用，如果你不知道自己的系统上面的命令在什么位置可以使用如下命令查找其路径：locate useradd

在root账户下为服务器集群(x101~x156)创建新账户hod.

addhoduser.exp

#!/usr/bin/expect -f
set password 123456
for {set i 1} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}

puts "$ip begins!"

spawn ssh $ip
expect "#"
send "useradd hod\r"
expect "#"
send "passwd hod\r"
expect "*New UNIX password*"
send "$password\r"
expect "*Retype new UNIX password*"
send "$password\r"
expect "#"
send "exit\r"
expect eof

puts "$ip end!"
}
close

在hod账户下进行如下的设置：

在每个服务器的hod目录下创建.ssh目录：（错！这部操作被证明是极大的错误，不用自己创建.ssh目录，使用“ssh-keygen -t rsa”会自动创建.ssh，自己创建.ssh目录会由于文件夹权限问题使得ssh无密码访问失效）

mkdirssh.exp(不需要使用)

#!/usr/bin/expect -f
set password 123456
for {set i 1} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}

spawn ssh $ip
expect "*password:*"
send "$password\r"
expect eof

expect "#"
send "mkdir /home/hod/.ssh\r"
expect "#"
send "exit\r"
expect eof
}
close

在x101服务器的/home/hod/.ssh目录下创建authorized_keys文件，并设置为644权限，这很关键.(这部操作也就不必了，后面ssh-copy-id -i ~/.ssh/id_rsa.pub hod@x101会自动创建authorized_keys文件，且mod为600，这个权限也是可以的)

SSH无密码登录steps:

为了让在A上可以无密码SSH到B上

首先在A上执行ssh-keygen -t rsa

一路回车

将公钥导入到authorized_keys文件中

cat ~/.ssh/rsa_id.pub >> authorized_keys

修改文件和文件夹权限

chmod 600 authorized_keys

chmod 700 ~/.ssh

将authorized_keys分发到远程主机

scp ~/.ssh/authorized_keys meteor@192.168.70.93:/home/meteor/.ssh


现在就可以无密码登录了。

ssh meteor@192.168.70.93

为x101~x156配置ssh无密码登录：

wumimadenglu.exp

#!/usr/bin/expect -f
set password 123456
for {set i 1} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}

spawn ssh $ip
expect "*password:*"
send "$password\r"
expect eof

expect "#"
send "ssh-keygen -t rsa\r"
expect "*save the key*"
send "\r"
expect "*Enter passphrase*"
send "\r"
expect "*Enter same passphrase again*"
send "\r"
expect eof

expect "#"
send "ssh-copy-id -i ~/.ssh/id_rsa.pub hod@x101\r"
expect "*password:*"
send "$password\r"
expect "#"
send "exit\r"
expect eof
}
close

最后只要将x101下的authorized_keys文件拷贝到x102~x156的.ssh目录即可.这方法会比之前为root账户配置ssh无密码登录快很多很多.

copyauthorized_keystoall.exp

#!/usr/bin/expect -f
set password 123456
for {set i 2} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}

spawn scp /home/hod/.ssh/authorized_keys hod@$ip:/home/hod/.ssh/
expect "*password:*"
send "$password\r"
expect eof
}
close

这里需要说明一点，在进行scp的时候尽量使用/home/hod/.ssh/authorized_keys这样的绝对目录，使用~/.ssh/authorized_keys可能会由于找不到目录而出错

后续操作：

配置好ssh无密码登录之后，我在hod账户目录下安装了python和jdk，方法都是先拷贝安装包或压缩包再解压安装

copypythontoall.sh

for((i=2;i<57;i++))
do
if [ $i -lt 10 ]
then
scp /home/hod/Python-2.5.1.tgz hod@x10$i:/home/hod/
echo "it is the first if: x10$i"
else
scp /home/hod/Python-2.5.1.tgz hod@x1$i:/home/hod/
echo "it is the first else: x1$i"
fi
done

installpythontoall.exp

#!/usr/bin/expect -f
for {set i 2} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}

spawn ssh $ip
expect "#"
send "tar zxvf /home/hod/Python-2.5.1.tgz\r"
expect "#"
send "cd /home/hod/Python-2.5.1\r"
expect "#"
send "./configure\r"
expect "#"
send "make\r"
expect "#"
send "sudo make install\r"
expect "*Password:*"
send "123456\r"
expect "#"
send "exit\r"
expect eof

puts "$ip completed"
sleep 3
}
close

copyjdktoall.sh

for((i=2;i<57;i++))
do
if [ $i -lt 10 ]
then
scp /home/hod/jdk-7u1-linux-i586.tar.gz hod@x10$i:/home/hod/
echo "it is the first if: x10$i"
else
scp /home/hod/jdk-7u1-linux-i586.tar.gz hod@x1$i:/home/hod/
echo "it is the first else: x1$i"
fi
done

installjdktoall.exp

#!/usr/bin/expect -f
for {set i 2} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}

spawn ssh $ip
expect "#"
send "tar zxvf /home/hod/jdk-7u1-linux-i586.tar.gz\r"
expect "#"
send "exit\r"
expect eof

puts "$ip completed"
sleep 3
}
close

安装hadoop：

installhadooptoall.exp

#!/usr/bin/expect -f
set password 123456
for {set i 2} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}

spawn scp /home/hod/hadoop-0.20.2.tar.gz hod@$ip:/home/hod/

spawn ssh $ip
expect "#"
send "mkdir /home/hod/hadoop\r"
expect "#"
send "cp /home/hod/hadoop-0.20.2.tar.gz /home/hod/hadoop/\r"
expect "#"
send "tar zxvf /home/hod/hadoop/hadoop-0.20.2.tar.gz\r"
expect "#"
send "exit\r"
expect eof

puts "$ip completed"
sleep 3
}
close

tar zxvf /home/hod/hadoop/hadoop-0.20.2.tar.gz会有问题，该操作会把hadoop压缩包解压在/home/hod目录下，应该在这之前先执行cd /home/hod/hadoop操作

在x101按之前root的操作配置好参数：

copypeizhitoall.sh

for((i=2;i<57;i++))
do
if [ $i -lt 10 ]
then
scp -r /home/hod/hadoop/hadoop-0.20.2/conf hod@x10$i:/home/hod/hadoop/hadoop-0.20.2/
scp /home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf/hodrc hod@x10$i:/home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf/
echo "it is the first if: x10$i"
else
scp -r /home/hod/hadoop/hadoop-0.20.2/conf hod@x1$i:/home/hod/hadoop/hadoop-0.20.2/
scp /home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf/hodrc hod@x1$i:/home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf/
echo "it is the first else: x1$i"
fi
done

最后用root用户在/etc/profile文件末尾加上下面语句，并把/etc/profile拷贝覆盖x102~x156

export JAVA_HOME=/home/hod/jdk1.7.0_01
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib:$CLASSPATH
export HADOOP_HOME=/home/hod/hadoop/hadoop-0.20.2
export PATH=$HADOOP_HOME/bin:$PATH
export CLUSTER_NAME=clus
export RM_QUEUE=batch
export RM_HOME=/usr/local
export PYTHON_HOME=/home/hod/Python-2.5.1
export PATH=$PYTHON_HOME:$PATH
export HOD_PYTHON_HOME=/home/hod/Python-2.5.1/python
export HOD_CONF_DIR=/home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf
export HOD_HOME=/home/hod/hadoop/hadoop-0.20.2/contrib/hod
export PATH=$HOD_HOME/bin:$PATH