ubuntu执行远程脚本，分布式系统搭建

多数执行远程脚本都是基于ssh的,

所以

1.配置ssh免密登陆（多数分布式系统，如hadoop）

2.执行远程脚本

一. 配置ssh免密登陆

首先生成 Master 的公匙，在 Master 节点终端中执行：

$ cd ~/.ssh            # 如果没有该目录，先执行一次ssh localhost
$ ssh-keygen -t rsa    # 一直按回车就可以，生成的密钥保存为.ssh/id_rsa

Master 节点需能无密码 ssh 本机，这一步还是在 Master 节点上执行：

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

完成后可以使用 ssh Master 验证一下。接着将公匙传输到 Slave1 节点：

$ scp ~/.ssh/id_rsa.pub hadoop@Slave1:/home/hadoop/

scp时会要求输入Slave1上hadoop用户的密码(hadoop)，输入完成后会提示传输完毕。

接着在 Slave1节点 上将ssh公匙保存到相应位置，执行

$ cat ~/id_rsa.pub >> ~/.ssh/authorized_keys

如果有其他 Slave 节点，也要执行 将公匙传输到 Slave 节点、在 Slave 节点上加入授权 这两步。

最后在 Master 节点上就可以无密码SSH到Slave1节点了。

$ ssh Slave1

二. 执行远程脚本

1.python+paramiko

网上的评价都是paramiko很强大

使用方式（一般的测试脚本都能跑的很顺）：

import paramiko
import select
client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect('host.example.com')
transport = client.get_transport()
channel = transport.open_session()
channel.exec_command("tail -f /var/log/everything/current")
while True:
rl, wl, xl = select.select([channel],[],[],0.0)
if len(rl) > 0:
# Must be stdout
print channel.recv(1024)

其实我们的需求很简单，分布式执行JOB，分散cpu和内存的压力，但是用paramiko的时候，总是在远程代码执行到20~50min后，远程的脚本调用的进程被挂起，一直sleep（我们的远程脚本有调用slaver本地的），不知道怎么回事（现在也不明白，描述起来比较玄奥，有可能是这个模块的BUG）

2.python+parallel

Linux系统映像规模当超过30时，小小的变更也会浪费很多时间，Linux系统下有很多可用的集中管理方案，例如著名的puppet，这个是用ruby开发，号称可以管理以万计的服务器。不过对于大多数SA来说，这个不怎么轻量，设置有点复杂。

这里推荐PSSH，比DSH更强大的批量管理工具，先来看下它的命令列表。

pssh 包安装 5 个实用程序：

parallel-ssh （pssh） 在多个主机上并行地运行命令。

parallel-scp （pscp）把文件并行地复制到多个主机上。

parallel-rsync （psync）通过 rsync 协议把文件高效地并行复制到多个主机上。

parallel-slurp （pslurp）把文件并行地从多个远程主机复制到中心主机上。

parallel-nuke （pnuke）并行地在多个远程主机上杀死进程。

parallel-ssh [-vAiIP] [-h hosts_file] [-H [user@]host[:port]] [-l user] [-p par] [-o outdir] [-e errdir] [-t timeout] [-O options] [-x args] [-X arg] command ...

具体参数的意义，我从官网抠下来：

-h host_file

–hosts host_file

Read hosts from the given host_file. Lines in the host file are of the

form [user@]host[:port] and can include blank lines and comments

(lines beginning with “#”). If multiple host files are given (the -h

option is used more than once), then pssh behaves as though these

files were concatenated together. If a host is specified multiple

times, then pssh will connect the given number of times.

-H

[user@]host[:port]

–host

[user@]host[:port]

-H

“[user@]host[:port] [ [user@]host[:port ] … ]”

–host

“[user@]host[:port] [ [user@]host[:port ] … ]”

Add the given host strings to the list of hosts. This option may be given multiple times, and may be used in conjunction with the -h

option.

-l user

–user user

Use the given username as the default for any host entries that don’t specifically specify a user.

-p parallelism

–par parallelism

Use the given number as the maximum number of concurrent connections.

-t timeout

–timeout timeout

Make connections time out after the given number of seconds. With a value of 0, pssh will not timeout any connections.

-o outdir

–outdir outdir

Save standard output to files in the given directory. Filenames are of

the form [user@]host[:port][.num] where the user and port are only

included for hosts that explicitly specify them. The number is a

counter that is incremented each time for hosts that are specified

more than once.

-e errdir

–errdir errdir

Save standard error to files in the given directory. Filenames are of

the same form as with the -o option.

-x args

–extra-args args

Passes extra SSH command-line arguments (see the ssh(1) man page for

more information about SSH arguments). This option may be specified

multiple times. The arguments are processed to split on whitespace,

protect text within quotes, and escape with backslashes. To pass

arguments without such processing, use the -X option instead.

-X arg

–extra-arg arg

Passes a single SSH command-line argument (see the ssh(1) man page for

more information about SSH arguments). Unlike the -x option, no

processing is performed on the argument, including word splitting. To

pass multiple command-line arguments, use the option once for each

argument.

-O options

–options options

SSH options in the format used in the SSH configuration file (see the

ssh_config(5) man page for more information). This option may be

specified multiple times.

-A

–askpass

Prompt for a password and pass it to ssh. The password may be used for

either to unlock a key or for password authentication. The password is

transferred in a fairly secure manner (e.g., it will not show up in

argument lists). However, be aware that a root user on your system

could potentially intercept the password.

-i

–inline

Display standard output and standard error as each host completes.

–inline-stdout

Display standard output (but not standard error) as each host

completes.

-v

–verbose

Include error messages from ssh with the -i and \ options.

-I

–send-input

Read input and send to each ssh process. Since ssh allows a command

script to be sent on standard input, the -I option may be used in lieu

of the command argument.

-P

–print

Display output as it arrives. This option is of limited usefulness

because output from different hosts are interleaved

终于跑起来了