install ceph by ceph-deploy

使用阿里云源安装ceph Luminous
https://liuxu.co/2017/09/19/install-ceph-Luminous-on-centos7-with-ceph-deploy/
0、如果曾经在节点上安装过ceph，那么需要先清理一下环境

ceph-deploy uninstall ceph-node1 ceph-node2 ceph-node3

ceph-deploy purge ceph-node1 ceph-node2 ceph-node3

ceph-deploy purgedata ceph-node1 ceph-node2 ceph-node3

ceph-deploy forgetkeys

1、在每台ceph节点上都生成一个ceph-deploy用户，密码也为ceph-deply

sudo useradd -d /home/ceph-deploy -m ceph-deploy

passwd ceph-deploy

(ceph-deploy)

echo "ceph-deploy ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph-deploy

sudo chmod 0440 /etc/sudoers.d/ceph-deploy

sudo setenforce 0

2、在其中一台节点上生成ssh key，并将ssh key 拷贝至其余ceph节点

su ceph-deploy

ssh-keygen 一路回车

ssh-copy-id ceph-deploy@ceph-node1

ssh-copy-id ceph-deploy@ceph-node2

ssh-copy-id ceph-deploy@ceph-node3

安装ceph-deploy

sudo yum install ceph-deploy -y

3、安装ceph

cd /home/ceph-deploy

ceph-deploy install ceph-node1 ceph-node2 ceph-node3 --repo-url=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/ --gpg-url=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7

4、新建ceph集群

ceph-deploy new ceph-node1 ceph-node2 ceph-node3

vim ceph.conf

[global]
fsid = 69e24e55-e1f2-4451-8ad9-5125500daac5
mon_initial_members = ceph-node1,ceph-node2,ceph-node3
mon_host = 192.168.56.131,192.168.56.132,192.168.56.133
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public network = 192.168.56.130/27

5、新建并初始化monitor

ceph-deploy --overwrite-conf mon create-initial

6、查看各节点上的disk

ceph-deploy disk list ceph-node1

ceph-deploy disk list ceph-node2

ceph-deploy disk list ceph-node3

vim cleanOsd.sh

#!/bin/sh

hosts="ceph-node1 ceph-node2 ceph-node3"
dev="b c"
for hostN in $hosts
do
for i in $dev
do
ceph-deploy disk zap ${hostN}:sd${i}
done
done

sh cleanOsd.sh

vim createOsd.sh

#!/bin/sh

hosts="ceph-node1 ceph-node2 ceph-node3"
dev="b c"
for hostN in $hosts
do
for i in $dev
do
ceph-deploy osd create ${hostN}:sd${i}
done
done

sh createOsd.sh

7、设置admin key

ceph-deploy admin ceph-node1 ceph-node2 ceph-node3

sudo chmod +r /etc/ceph/ceph.client.admin.keyring

8、检验ceph安装是否成功

ceph -s

Q&A:

1. ceph create mon时出现ip地址为0.0.0.0的情况

是因为使用了hostname xxx修改过主机名导致的。重启一下节点。

2. nohup python /usr/bin/ceph-rest-api -n client.admin > /var/log/ceph/ceph-rest-api.log &

3. curl -H "Accept: application/json" '192.168.56.131:5000/api/v0.1/status'| python -m json.tool

4. minikube start --vm-driver=none --iso-url https://raw.githubusercontent.com/cilium/minikube-iso/master/minikube.iso
5. ceph osd lspools

ceph osd pool create testpool 256

6. centos build nbd
http://www.zphj1987.com/2016/05/19/ceph%E7%9A%84jewel%E6%96%B0%E6%94%AF%E6%8C%81%E7%9A%84rbd-nbd/ http://www.hl10502.com/2017/09/14/centos-build-nbd/
7. set ceph replica size to 1

ceph osd pool set pool-name size 1