CentOS7上Glusterfs的安装及使用(gluster/heketi)
2017-11-08 13:16
405 查看
1.glusterfs安装
安装并设置自启动:yum -y install centos-release-gluster yum -y install glusterfs-server systemctl enable glusterd systemctl start glusterd
配置每台机器hosts:
vim /etc/hosts 10.132.47.78 lk-glusterfs-47-78 10.132.47.79 lk-glusterfs-47-79 10.132.47.80 lk-glusterfs-47-80
2.glusterfs使用
2.1 法一:通过gluster命令(gluster-client)使用
2.1.1 安装glusterfs client客户端命令:
yum安装:yum -y install centos-release-gluster yum -y install glusterfs-client
/etc/hosts中添加glusterfs-server信息:
vim /etc/hosts 10.132.47.78 lk-glusterfs-47-78 10.132.47.79 lk-glusterfs-47-79 10.132.47.80 lk-glusterfs-47-80
2.1.2 gluster命令使用
为存储池添加节点Node:gluster peer probe lk-glusterfs-47-78 gluster peer probe lk-glusterfs-47-79 gluster peer probe lk-glusterfs-47-80
创建GlusterFS卷并使用(以复本卷为例):
每台机器上以/gluster/gv0目录各创建一个brick:
mkdir -p /gluster/gv0
以每台机器的brick创建一个有3复本的逻辑卷gv0:
gluster volume create gv0 replica 3 lk-glusterfs-47-78:/gluster/gv0 lk-glusterfs-47-79:/gluster/gv0 lk-glusterfs-47-80:/gluster/gv0 force
启用volume:
gluster volume start gv0
client挂载gv0卷到/mnt/glusterfs目录并使用:
mkdir /mnt/glusterfs mount -t glusterfs lk-glusterfs-47-79:/gv0 /mnt/glusterfs
附:其他相关操作
从GlusterFS卷gv0移除某一brick: gluster volume remove-brick gv0 replica 2 lk-glusterfs-47-79:/gluster/gv0 force 删除GlusterFS卷gv0: 需要先stop卷: gluster volume stop gv0 再删: gluster volume delete gv0
2.2 法二:通过Heketi提供的restapi使用
2.2.1 Heketi安装(方式一:yum部署)
安装(装在某台节点上,如10.132.47.79):yum -y install heketi heketi-client
配置秘钥对:
Heketi使用SSH来配置GlusterFS的所有节点。创建SSH密钥对:
ssh-keygen -f /etc/heketi/heketi_key -t rsa -N '' chown heketi:heketi /etc/heketi/heketi_key*
制作完成后会在当前目录下生成heketi_key、heketi_key.pub,将公钥heketi_key.pub拷贝到所有glusterfs节点上/etc/heketi/keketi_key.pub(包括你登陆的第一个节点),/etc/heketi/heketi.json 中 的 keyfile 指向 生成的 key(包含路径)
ssh-copy-id -i /etc/heketi/heketi_key.pub root@10.132.47.78 ssh-copy-id -i /etc/heketi/heketi_key.pub root@10.132.47.79 ssh-copy-id -i /etc/heketi/heketi_key.pub root@10.132.47.80
备注,以上ssh-copy-id一句等效于以下三句:
scp /etc/heketi/heketi_key.pub root@10.132.47.78:/tmp ssh root@10.132.47.78 cat /tmp/id_rsa.pub >> /root/.ssh/authorized_keys
创建存储db的文件夹:
mkdir /dcos/heketi chown -R heketi:heketi /dcos/heketi
配置 heketi.json:
这里需要注意只是测试的话用mock 授权,standalone模式就 ssh 授权,k8s下就 kubernetes授权
vim /etc/heketi/heketi.json { "_port_comment": "Heketi Server Port Number", "port": "8088", "_use_auth": "Enable JWT authorization. Please enable for deployment", "use_auth": false, "_jwt": "Private keys for access", "jwt": { "_admin": "Admin has access to all APIs", "admin": { "key": "123456" }, "_user": "User only has access to /volumes endpoint", "user": { "key": "123456" } }, "_glusterfs_comment": "GlusterFS Configuration", "glusterfs": { "_executor_comment": [ "Execute plugin. Possible choices: mock, ssh", "mock: This setting is used for testing and development.", " It will not send commands to any node.", "ssh: This setting will notify Heketi to ssh to the nodes.", " It will need the values in sshexec to be configured." ], "executor": "ssh", "_sshexec_comment": "SSH username and private key file information", "sshexec": { "keyfile": "/etc/heketi/heketi_key", "user": "root" }, "_db_comment": "Database file name", "db": "/dcos/heketi/heketi.db" } }
重启heketi服务:
systemctl enable heketi systemctl restart heketi
测试heketi是否好用:
curl http://localhost:8088/hello[/code]2.2.2 Heketi安装(方式二:容器化部署)
docker run启动:
You will need to create a directory which has a directory containing configuraiton and any private key if necessary, and an empty directory used for storing the database. Directory and files must be read/write by user with id 1000 and if an ssh private key is used, it must also have a mod of 0600.mkdir -p /dcos/heketi-docker/config mkdir -p /dcos/heketi-docker/db cp /etc/heketi/heketi.json /dcos/heketi-docker/config/ ---复用2.2.1的heketi.json cp /etc/heketi/heketi_key /dcos/heketi-docker/config/ ---复用2.2.1的heketi_key chmod 600 /dcos/heketi-docker/config/heketi_key chown 1000:1000 -R /dcos/heketi-docker To run: # docker run --name=heketi -d -p 8089:8088 \ -v /dcos/heketi-docker/config:/etc/heketi \ -v /dcos/heketi-docker/db:/dcos/heketi \ -v /etc/hosts:/etc/hosts \ -v /etc/localtime:/etc/localtime \ heketi/heketi:4
更进一步,可以将heketi部署到k8s上:chmod 600 /dcos/heketi-docker/config/heketi_key chown 1000:1000 -R /dcos/heketi-docker
heketi.yaml:apiVersion: extensions/v1beta1 kind: Deployment metadata: name: heketi namespace: kube-system labels: app: heketi spec: replicas: 1 selector: matchLabels: app: heketi template: metadata: labels: app: heketi spec: containers: - name: heketi image: heketi/heketi:4 ----这个镜像包含heketi和heketi-cli imagePullPolicy: IfNotPresent volumeMounts: - name: config mountPath: /etc/heketi - name: db mountPath: /dcos/heketi - name: time mountPath: /etc/localtime - name: hosts mountPath: /etc/hosts ports: - containerPort: 8088 name: heketi-api volumes: - name: config hostPath: path: /dcos/heketi-docker/config - name: db hostPath: path: /dcos/heketi-docker/db - name: time hostPath: path: /etc/localtime - name: hosts hostPath: path: /etc/hosts nodeSelector: kubernetes.io/hostname: k8smaster01 --- apiVersion: v1 kind: Service metadata: labels: app: heketi name: heketi namespace: kube-system spec: type: NodePort ports: - name: heketi port: 8089 protocol: TCP targetPort: 8088 nodePort: 30088 selector: app: heketi2.2.3 heketi-cli使用
Heketi集群初始化
创建cluster:heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true cluster create {"id":"353fe5b92fa2a4d8671b6efdac8a5fd5","nodes":[],"volumes":[]}
将3个节点作为node添加到cluster:heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true node add --cluster="353fe5b92fa2a4d8671b6efdac8a5fd5" --management-host-name=10.132.47.78 --storage-host-name=10.132.47.78 --zone=1 {"zone":1,"hostnames":{"manage":["10.132.47.78"],"storage":["10.132.47.78"]},"cluster":"353fe5b92fa2a4d8671b6efdac8a5fd5","id":"a117cd328d609acc15e88dc0b6ab4889","devices":[]}) heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true node add --cluster="353fe5b92fa2a4d8671b6efdac8a5fd5" --management-host-name=10.132.47.79 --storage-host-name=10.132.47.79 --zone=1 {"zone":1,"hostnames":{"manage":["10.132.47.79"],"storage":["10.132.47.79"]},"cluster":"353fe5b92fa2a4d8671b6efdac8a5fd5","id":"5d3fbc8d0ac62d9c950d81df2f274dc6","devices":[]} heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true node add --cluster="353fe5b92fa2a4d8671b6efdac8a5fd5" --management-host-name=10.132.47.80 --storage-host-name=10.132.47.80 --zone=1 {"zone":1,"hostnames":{"manage":["10.132.47.80"],"storage":["10.132.47.80"]},"cluster":"353fe5b92fa2a4d8671b6efdac8a5fd5","id":"a302a8367f18524ab5236e99f09072fe","devices":[]}
注意:
对接k8s的话,上边这个必须management-host-name要用ip地址,不可以用域名,否则从controller-manager中可以看到报错:glusterfs: failed to create endpoint Endpoints "glusterfs-dynamic-gluster-pvc1" is invalid: [subsets[0].addresses[0].ip: Invalid value: "lk-glusterfs-47-80": must be a valid IP address, (e.g. 10.9.8.7), subsets[0].addresses[1].ip: Invalid value: "lk-glusterfs-47-79": must be a valid IP address, (e.g. 10.9.8.7), subsets[0].addresses[2].ip: Invalid value: "lk-glusterfs-47-78": must be a valid IP address, (e.g. 10.9.8.7)]
每台设备node上各添加一块裸硬盘/dev/sdb(没创建过任何分区),创建device:heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true device add --name="/dev/sdb" --node="a117cd328d609acc15e88dc0b6ab4889" heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true device add --name="/dev/sdb" --node="5d3fbc8d0ac62d9c950d81df2f274dc6" heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true device add --name="/dev/sdb" --node="a302a8367f18524ab5236e99f09072fe"
其实以上三步可以简化成通过topology文件创建的方式:
创建topology.json文件:vim /etc/heketi/topology.json { "clusters": [ { "nodes": [ { "node": { "hostnames": { "manage": [ "lk-glusterfs-47-78" ], "storage": [ "10.132.47.78" ] }, "zone": 1 }, "devices": [ "/dev/sdb" ] }, { "node": { "hostnames": { "manage": [ "lk-glusterfs-47-79" ], "storage": [ "10.132.47.79" ] }, "zone": 1 }, "devices": [ "/dev/sdb" ] }, { "node": { "hostnames": { "manage": [ "lk-glusterfs-47-80" ], "storage": [ "10.132.47.80" ] }, "zone": 1 }, "devices": [ "/dev/sdb" ] } ] } ] }
该文件格式比较简单,基本上是告诉heketi要创建一个3节点的集群,其中每个节点包含的配置有FQDN,IP地址以及至少一个将用作GlusterFS块的备用块设备。
将该文件发送给heketi创建:以systemd起heketi: heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 topology load --json=/etc/heketi/topology.json 以容器起heketi&heketi-cli: docker exec 容器ID heketi-cli --server http://10.142.21.21:30088 --user admin --secret 123456 topology load --json=/etc/heketi/topology.json 以k8s方式起heketi&heketi-cli: kubectl exec -it heketi-5ff9bb8c89-dzsc9 -n kube-system heketi-cli topology load -- --json=/etc/heketi/topology.json --server http://10.142.21.21:30088 --user admin --secret 123456
结果:
创建成功后,heketi会在每个gluster节点上创建一个逻辑卷组(vg_2d8771e16bfe0b267fe2b7133584af43),通过vgscan或vgdisplay可以看到
创建volume:以systemd起heketi: heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 volume create --size=100 --replica=3 --clusters=d691b29a06374c4da7e94bba71d027bf 以容器起heketi: docker exec 容器ID heketi-cli --server http://10.142.21.23:30088 --user admin --secret 123456 volume create --size=100 --replica=3 --clusters=d691b29a06374c4da7e94bba71d027bf 以k8s方式起heketi&heketi-cli: kubectl exec -it heketi-5ff9bb8c89-dzsc9 -n kube-system heketi-cli topology load -- --server http://10.142.21.21:30088 --user admin --secret 123456 volume create --size=100 --replica=3 --clusters=d691b29a06374c4da7e94bba71d027bf
结果:
结果是在相应节点的逻辑卷组(vg_2d8771e16bfe0b267fe2b7133584af43)下会创建逻辑卷(/var/lib/heketi/mounts/vg_7157a2d1d7899269823997ad62e6debd/brick_1185ef33c9719d9063ccc2ccd0df96e7/brick)作为glusterfs的brickgluster volume info Volume Name: vol_50b5237866378d655af326a74fc7d68c Type: Distributed-Replicate Volume ID: b82dc1b3-b59c-48cd-ace1-155a0c05c42f Status: Started Snapshot Count: 0 Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: 10.142.21.22:/var/lib/heketi/mounts/vg_7157a2d1d7899269823997ad62e6debd/brick_1185ef33c9719d9063ccc2ccd0df96e7/brick Brick2: 10.142.21.27:/var/lib/heketi/mounts/vg_a850891bc0f47849bfdbdb115ee19656/brick_75894b35ae36dd5253bd79df1539b970/brick Brick3: 10.142.21.21:/var/lib/heketi/mounts/vg_78a29014ac0df3d31b1a357e096e8917/brick_3423914cb4d9873c670f5fab0ffe44e5/brick Brick4: 10.142.21.25:/var/lib/heketi/mounts/vg_25cc714589cec30e3550297cced4ce44/brick_6ca46dd4a13f7ac0b598d83fbf43c2fc/brick Brick5: 10.142.21.26:/var/lib/heketi/mounts/vg_12e5dbb7b9372e0d72b8ec2166b68048/brick_87e969be58d93ff7deba65587d2e5637/brick Brick6: 10.142.21.24:/var/lib/heketi/mounts/vg_54b5dc50bfe00644a39e289d9abf7c99/brick_5a21762ac6c5ab35472648455310af56/brick Options Reconfigured: transport.address-family: inet nfs.disable: on heketi-cli --server http://10.142.21.21:30088 --user admin --secret 123456 topology info
参考:
1.https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart
2.https://jimmysong.io/kubernetes-handbook/practice/storage-for-containers-using-glusterfs-with-openshift.html
相关文章推荐
- centos7下的glusterfs的安装与使用
- gluster快速入门安装使用(centos-rhel环境)
- 在Centos中yum安装和卸载软件的使用方法
- 使用云计算~Eucalyptus tools安装到centos5.3记录日志
- centos上安装使用和升级subversion&Apache
- 弹性计算云Eucalyptus安装及使用(centos5.3版)
- 在CentOs 5.1中使用rpm安装NGINX+php+mysql(二)
- CentOS 中安装使用 xmms 音乐播放器
- 使用CentOS光盘镜像作为YUM源安装rpm软件包
- CentOS使用yum安装软件提示需要KEY的解决方法
- 在centos 下安装和使用MySQL
- 在centos 下安装和使用MySQL
- 使用yum来安装CentOS图形界面包
- CentOS 5下使用yum安装mplayer
- 在Centos中yum安装和卸载软件的使用方法
- Linux基础知识--(04)使用虚拟机安装CentOS 5.3
- 在CentOs 5.1中使用rpm安装NGINX+php+mysql(一)
- 在CentOs 5.1中使用rpm安装NGINX+php+mysql
- CentOS中安装subversion,并使用svn+ssh访问
- 在CentOs 5.1中使用rpm安装NGINX+php+mysql