初探glusterfs-使用小结FAQ

注：建议使用3副本（1x3），1x2副本危险，2x2也危险。

一、配置相关
1、快速建立一个卷来提供服务的流程示例
【数据盘分区】
1）ext4格式
~]# fdisk /dev/sdb <<_EOF
n
p
1

p
w
_EOF
~]# mkfs.ext4 /dev/sdb1

~]# mkdir /data
~]# cat <<_EOF >>/etc/fstab
UUID=$(blkid /dev/sdb1 |cut -d'"' -f2) /data                    ext4    defaults        0 0
_EOF

~]# mount -a

2）xfs格式
如果分区所在设备已经挂载，要先卸载并删掉现有系统。
yum install lvm2 xfsprogs -y
pvcreate /dev/sdb
vgcreate vg0 /dev/sdb
lvcreate -l 100%FREE -n lv01 vg0
mkfs.xfs -f -i size=512 /dev/vg0/lv01
mkdir /data
cat <<_EOF >>/etc/fstab
UUID=$(blkid /dev/vg0/lv01 |cut -d'"' -f2) /data                   xfs     defaults        0 0
_EOF

mount -a
# df -h |grep data
/dev/mapper/vg0-lv01  16T   33M  16T   1% /data

【配置服务】以在 10.60.200.11 上配置为例
yum install glusterfs-server
service glusterd start
chkconfig glusterd on

【配置集群】
gluster peer probe 10.60.200.12
gluster peer probe 10.60.200.13
每台集群节点上建立目录
mkdir /data/gv1/brick1 -p

【提供data域】
创建卷gv1作为主数据域：
# gluster volume create gv1 replica 3 transport tcp \
10.60.200.11:/data/gv1/brick1 \
10.60.200.12:/data/gv1/brick1 \
10.60.200.13:/data/gv1/brick1

【启动】
# gluster volume start gv1

【查看现状】
# gluster volume info

Volume Name: gv1
Type: Replicate
Volume ID: 32b1866c-1743-4dd9-9429-6ecfdfa168a2
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.60.200.11:/data/gv1/brick1
Brick2: 10.60.200.12:/data/gv1/brick1
Brick2: 10.60.200.13:/data/gv1/brick1

挂载测试：
mount -t glusterfs 10.60.200.11:/gv1 /mnt/test
~]# df -h /mnt/test/
Filesystem                      Size  Used Avail Use% Mounted on
10.60.200.11:/gv1               5.4T  1.8T  3.4T  34% /mnt/test

写入fstab：
10.60.200.11:/gv1    /mnt/test    glusterfs    defaults,_netdev,backup-volfile-servers=10.60.200.12:10.60.200.13  0 0

2、挂载时，可以使用参数来提供备用节点
3.6以后兼容旧版本的方法：
mount -t glusterfs -o backupvolfile-server=10.60.200.12 10.60.200.11:/gv1 /mnt/test

3.6以后推荐的方法：
mount -t glusterfs -o backup-volfile-servers=10.60.200.12 10.60.200.11:/gv1 /mnt/test

多个backup的方法：
mount -t glusterfs -o backup-volfile-servers=10.60.200.12:10.60.200.13 10.60.200.11:/gv1 /mnt/test

请特别注意：
mount -t glusterfs 和 mount.glusterfs 的用法是不一样的。

3、参数配置
实例：
ovirt的优化做了如下工作：
---
优化后，配置将做如下调整：
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
storage.owner-gid: 36
storage.owner-uid: 36
cluster.server-quorum-type: server
cluster.quorum-type: auto
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
auth.allow: *
user.cifs: enable
nfs.disable: off
performance.readdir-ahead: on
---

---配置卷，以gv1为例：
gluster volume set gv1 diagnostics.count-fop-hits on
gluster volume set gv1 diagnostics.latency-measurement on
gluster volume set gv1 storage.owner-gid 36
gluster volume set gv1 storage.owner-uid 36
gluster volume set gv1 cluster.server-quorum-type server
gluster volume set gv1 cluster.quorum-type auto
gluster volume set gv1 network.remote-dio enable
gluster volume set gv1 cluster.eager-lock enable
gluster volume set gv1 performance.stat-prefetch off
gluster volume set gv1 performance.io-cache off
gluster volume set gv1 performance.read-ahead off
gluster volume set gv1 performance.quick-read off
gluster volume set gv1 auth.allow \*
gluster volume set gv1 user.cifs enable
gluster volume set gv1 nfs.disable off
---配置卷

二、调整相关
1、扩容 distributed-replicate 类型的节点组
1）初始状态
root@n11 ~]# gluster volume info

Volume Name: gv1
Type: Distributed-Replicate
Volume ID: 32b1866c-1743-4dd9-9429-6ecfdfa168a2
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.60.200.11:/data/gv1/brick1
Brick2: 10.60.200.12:/data/gv1/brick1

2）增加1对节点，注意，因为 replica=2，需要成对的添加
[root@n11 ~]# volume add-brick gv_test1 replica 2 10.60.200.21:/data/gv1/brick1 10.60.200.22:/data/gv1/brick1
[root@n11 ~]# gluster volume info

Volume Name: gv1
Type: Distributed-Replicate
Volume ID: 32b1866c-1743-4dd9-9429-6ecfdfa168a2
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.60.200.11:/data/gv1/brick1
Brick2: 10.60.200.12:/data/gv1/brick1
Brick3: 10.60.200.21:/data/gv1/brick1
Brick4: 10.60.200.22:/data/gv1/brick1

2、扩容 replica 类型的节点
1）初始状态
[root@n11 ~]# gluster volume info

Volume Name: ovirt-data
Type: Replicate
Volume ID: 6a2f6489-bb8f-43d9-b46b-6258d95c7571
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.60.200.31:/data/ovirt/restore
Brick2: 10.60.200.12:/data/ovirt/data

2）增加一个副本，从2个副本增加到3个
[root@n11 ~]# gluster volume add-brick [卷的名称] replica [新的副本数量] [新的节点路径]
[root@n11 ~]# gluster volume add-brick ovirt-data replica 3 10.60.200.11:/data/ovirt/restore
volume add-brick: failed: One or more nodes do not support the required op-version. Cluster op-version must atleast be 30600.

上述操作遇到异常，下面是解决办法：
[root@n11 ~]# gluster volume set ovirt-data op-version 30600
volume set: failed: Option "cluster.op-version" is not valid for a single volume
说明这个参数是一个全局的参数，不能只针对单个卷有效。
[root@n11 ~]# gluster volume set all op-version 30600
volume set: success

再次操作：
[root@n11 ~]# gluster volume add-brick ovirt-data replica 3 10.60.200.11:/data/ovirt/restore
volume add-brick: success

[root@n11 ~]# gluster volume info ovirt-data

Volume Name: ovirt-data
Type: Replicate
Volume ID: 6a2f6489-bb8f-43d9-b46b-6258d95c7571
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.60.200.31:/data/ovirt/restore
Brick2: 10.60.200.12:/data/ovirt/data
Brick3: 10.60.200.11:/data/ovirt/restore

三、故障相关
1、修复实例：2个副本的复制卷，其中一个挂掉且不能上线后，怎么处理？
1）现状
gluster> volume info
Volume Name: gv1
Type: Replicate
Volume ID: 32b1866c-1743-4dd9-9429-6ecfdfa168a2
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.60.200.11:/data/gv1/brick1
Brick2: 10.60.200.12:/data/gv1/brick1

2）假设
10.60.200.11 挂掉

准备替代的brick：
10.60.200.31:/data/gv1/brick1

3）操作
在 10.60.200.12 上执行 peer probe 来将新的节点加入集群：
gluster> peer probe 10.60.200.31

执行 replace-brick 操作：
gluster> volume replace-brick gv1 10.60.200.11:/data/gv1/brick1 10.60.200.31:/data/gv1/brick1 commit force
volume replace-brick: success: replace-brick commit successful

查看状态：
gluster> volume status
Status of volume: gv1
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick 10.60.200.31:/data/gv1/brick1                     49153   Y       9188
Brick 10.60.200.12:/data/gv1/brick1                     49152   Y       23944
NFS Server on localhost                                 N/A     N       N/A
Self-heal Daemon on localhost                           N/A     Y       14839
NFS Server on 10.60.200.31                              N/A     N       N/A
Self-heal Daemon on 10.60.200.31                        N/A     N       9199

Task Status of Volume ovirt-data
------------------------------------------------------------------------------
There are no active volume tasks

gluster> volume info

Volume Name: gv1
Type: Replicate
Volume ID: 6a2f6489-bb8f-43d9-b46b-6258d95c7571
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.60.200.31:/data/gv1/brick1
Brick2: 10.60.200.12:/data/gv1/brick1

查看 heal 的状态：
gluster> volume heal gv1 info
Brick 10.60.200.31:/data/gv1/brick1
Number of entries: 0

Brick 10.60.200.12:/data/gv1/brick1
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/1d04adc8-9921-4e1c-b192-84fba64224db/bd3c9ecf-8100-42db-9db5-038dd95a4d54 - Possibly undergoing heal

/09cb8372-a68d-47dc-962e-70b5225be6bc/images/1a719266-2b0e-4f6a-9586-825e5998c67b/74b85a40-847e-43ca-a74e-a09d5274339f
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/2d7f2fee-c796-4458-a7c2-86cb2e227834/2ef4b3fa-1656-4d15-8bd4-3356eab2d37e - Possibly undergoing heal

/09cb8372-a68d-47dc-962e-70b5225be6bc/dom_md/outbox
/09cb8372-a68d-47dc-962e-70b5225be6bc/master/vms
/09cb8372-a68d-47dc-962e-70b5225be6bc/master/tasks
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/48722b60-7ce6-4714-942a-4cace1076c84/cba01bf2-5011-43d2-b724-d6bca8d49477
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/6b4c4ca2-8fab-476c-9321-f23872618837
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/b89e79cf-98bb-44a5-b9a0-33bbff2e4e22
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/560a7b8c-fbfe-4495-b315-e46593b4523e
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/e4c55db2-b33a-4c0d-8bc0-dc0c421ca302
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/e18d032d-c70b-4104-98b5-a3f7fb8d2e27
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/f5e408f3-eded-4bd2-93fd-205cbcd41714
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/2a629e0c-0029-480e-87f4-ac5f417a4448
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/48722b60-7ce6-4714-942a-4cace1076c84/cba01bf2-5011-43d2-b724-d6bca8d49477.meta
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/48722b60-7ce6-4714-942a-4cace1076c84/cba01bf2-5011-43d2-b724-d6bca8d49477.lease
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/5f11ddd0-9f74-4845-80ed-d3e5dc081d80/6c090128-6a01-4c0e-a3e2-df4a95e30894
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/2d7f2fee-c796-4458-a7c2-86cb2e227834/2ef4b3fa-1656-4d15-8bd4-3356eab2d37e.meta
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/2d7f2fee-c796-4458-a7c2-86cb2e227834/2ef4b3fa-1656-4d15-8bd4-3356eab2d37e.lease
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/5f11ddd0-9f74-4845-80ed-d3e5dc081d80/bc95b575-99fe-4cdf-a341-aeeed053e2b3
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/5f11ddd0-9f74-4845-80ed-d3e5dc081d80/bc95b575-99fe-4cdf-a341-aeeed053e2b3.meta
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/5f11ddd0-9f74-4845-80ed-d3e5dc081d80/bc95b575-99fe-4cdf-a341-aeeed053e2b3.lease
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/5f11ddd0-9f74-4845-80ed-d3e5dc081d80/6c090128-6a01-4c0e-a3e2-df4a95e30894.meta
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/5f11ddd0-9f74-4845-80ed-d3e5dc081d80/6c090128-6a01-4c0e-a3e2-df4a95e30894.lease
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/1d04adc8-9921-4e1c-b192-84fba64224db/bd3c9ecf-8100-42db-9db5-038dd95a4d54.meta
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/1d04adc8-9921-4e1c-b192-84fba64224db/bd3c9ecf-8100-42db-9db5-038dd95a4d54.lease
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/1a719266-2b0e-4f6a-9586-825e5998c67b/74b85a40-847e-43ca-a74e-a09d5274339f.meta
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/1a719266-2b0e-4f6a-9586-825e5998c67b/74b85a40-847e-43ca-a74e-a09d5274339f.lease
Number of entries: 28

等待完成后：
gluster> volume heal gv1 info
Brick 10.60.200.31:/data/gv1/brick1
Number of entries: 0

Brick 10.60.200.12:/data/gv1/brick1
Number of entries: 0

2、在增加或者删除一个集群中的节点时，要注意的问题
现状：
a+b 组成： replica 1x2 的集群。
-----------------------------
Number of Bricks: 1 x 2 = 2
Bricks:
Brick1: a.test.com:/data/test
Brick2: b.test.com:/data/test

增加c和d，变成：replica 1x4 的集群。
gluster volume add-brick gv1 replica 4 c.test.com:/data/test d.test.com:/data/test
-----------------------------
Number of Bricks: 1 x 4 = 4
Bricks:
Brick1: a.test.com:/data/test
Brick2: b.test.com:/data/test
Brick3: c.test.com:/data/test
Brick4: d.test.com:/data/test

移除c，变成：replica 1x3 的集群
gluster volume remove-brick gv1 replica 3 c.test.com:/data/test force
-----------------------------
Number of Bricks: 1 x 3 = 3
Bricks:
Brick1: a.test.com:/data/test
Brick2: b.test.com:/data/test
Brick3: d.test.com:/data/test

这一步，可能导致数据错误，因为a+b+d，将重新heal数据。

此时的解决方法是：
移除d，变成：replica 1x2 的集群，因为a和b的数据最接近一致，heal的时间最短。
gluster volume remove-brick gv1 replica 2 d.test.com:/data/test force
-----------------------------
Number of Bricks: 1 x 2 = 2
Bricks:
Brick1: a.test.com:/data/test
Brick2: b.test.com:/data/test

ZYXW、参考
1、[Gluster-users] Replacing a failed brick https://www.gluster.org/pipermail/gluster-users/2013-August/014000.html 2、expand brick http://avid.force.com/pkb/articles/en_US/how_to/Adding-a-node-to-a-cluster-using-gluster https://bugzilla.redhat.com/show_bug.cgi?id=1168897 http://comments.gmane.org/gmane.comp.file-systems.gluster.user/20225 3、Resolving Peer Rejected http://gluster.readthedocs.io/en/latest/Administrator%20Guide/Resolving%20Peer%20Rejected/ 4、KVM虚拟化开源高可用方案(七)GLUSTERFS搭建及常见故障处理 http://xiaoli110.blog.51cto.com/1724/1071106?utm_source=tuicool&utm_medium=referral 5、Accessing Data - Setting Up GlusterFS Client http://gluster.readthedocs.io/en/latest/Administrator%20Guide/Setting%20Up%20Clients/[/code]