linux下高可用集群之RHCS ISCSI和GFS2 详解

linux下高可用集群之RHCS ISCSI和GFS2 详解1.RHCS: RedHat Cluster Suite,红帽集群套件
RHCS必备基础包:cman rgmanager system-cluster-config
2.RHCS集群部署基本前提：
2.1.时间同步；建议使用ntp服务
2.2.跳板机与各节点名称解析且每个主机的主机名与其'uname -n'保持一致；
2.3.跳板机与各节点之间ssh实现密钥认证
2.4.配置好每个节点的Yum;
3.本实验使用3个节点主机实现RHCS集群,GW主机作为跳板主机，IP分布如下:
1.1.1.18 node1.wilow.com node1
1.1.1.19 node2.wilow.com node2
1.1.1.20 node2.wilow.com node3
1.1.1.88 GW.wilow.com GW #作为跳板主机,管理node1,node2,node3节点主机
4.在GW.willow.com跳板主机上向3个节点主机分别安装cman,rgmanager,system-cluster-config
[root@GW ~]# for I in {1..3}; do ssh node$I 'yum install -y cman rgmanager system-config-cluster'; done
[root@GW ~]#
5.RHCS集群服务启动前提：
5.1.每个集群都有唯一集群名称；
5.2.至少有一个fence设备；
5.3.至少应该有三个节点；两个节点的场景中要使用qdisk仲裁磁盘；
6.创建集群名称，节点以及fence设备

[root@node1 cluster]# system-config-cluster &
6.1.创建一个新的集群名称:






6.2.增加3个节点主机：






6.3.增加一个fence设备：



6.4.保存配置:File->Save,默认集群配置文件保存至/etc/cluster/cluster.conf



6.5.分别在各节点启动cman服务，否则fence设备卡住，无法启动
并且会自动启动ccsd服务，从而将cluster.conf文件传播给其他节点，
保证所有节点cluster.conf文件同步一致
[root@node1 cluster]# service cman start
Starting cluster:
Loading modules... done
Mounting configfs... done
Starting ccsd... done
Starting cman... done
Starting daemons... done
Starting fencing... done
[ OK ]
[root@node2 cluster]# service cman start
[root@node3 cluster]# service cman start
6.6.启动rgmanager服务来管理资源等信息
[root@GW ~]# for I in {1..3}; do ssh node$I 'service rgmanager start'; done
6.7.安装一个apache服务来做集群测试：

[root@GW ~]# for I in {1..3}; do ssh node$I 'yum install -y httpd'; done[root@GW ~]#ssh node1 'echo node1.willow.com > /var/www/html/index.html'[root@GW ~]#ssh node2 'echo node2.willow.com > /var/www/html/index.html'[root@GW ~]#ssh node3 'echo node3.willow.com > /var/www/html/index.html'[root@GW ~]# for I in {1..3}; do ssh node$I 'chkconfig httpd off'; done6.8.查看集群的基本信息:[root@node1 ~]# cman_tool status
Version: 6.2.0
Config Version: 2
Cluster Name: tcluster
Cluster Id: 28212
Cluster Member: Yes
Cluster Generation: 12
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 8
Flags: Dirty
Ports Bound: 0 177
Node name: node1.willow.com
Node ID: 1
Multicast addresses: 239.192.110.162
Node addresses: 1.1.1.18
6.9.查看集群各节点状态:
[root@node1 ~]# clustat
Cluster Status for tcluster @ Thu Aug 18 10:37:31 2016
Member Status: Quorate

Member Name ID Status
------ ---- ---- ------
node1.willow.com 1 Online, Local
node2.willow.com 2 Online
node3.willow.com 3 Online
6.10.增加一个VIP和一个httpd集群资源:
[root@node1 cluster]# system-config-cluster &







6.11.由于在cman中资源无法启动，必须创建服务:如下创建一个webservice服务









6.12.将刚才配置的cluster.conf文件信息再次传播至其他节点:



6.13.查看webservcie服务运行在哪个节点上
[root@node1 ~]# clustat
Cluster Status for tcluster @ Thu Aug 18 10:55:17 2016
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
node1.willow.com 1 Online, Local, rgmanager
node2.willow.com 2 Online, rgmanager
node3.willow.com 3 Online, rgmanager

Service Name Owner (Last) State
------- ---- ----- ------ -----
service:webservice node1.willow.com started
6.14.查看VIP启动状况:ifconfig命令无法查看
[root@node1 ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:10:47:b6 brd ff:ff:ff:ff:ff:ff
inet 1.1.1.18/24 brd 1.1.1.255 scope global eth0
inet 1.1.1.100/24 scope global secondary eth0
这时，通过web测试访问http://1.1.1.100 会显示node1节点的web信息

6.15.查看clusvcadm帮助:[root@node2 cluster]# clusvcadm -h6.16.将当前node1.willow.com运行的webservice服务组迁移至node2.willow.com

[root@node1 ~]# clusvcadm -r webservice -m node2.willow.com
Trying to relocate service:webservice to node2.willow.com...Success
service:webservice is now running on node2.willow.com
[root@node1 ~]# clustat



这时，通过web测试访问http://1.1.1.100 会显示node2节点的web信息
7.通过nfs让3个节点共享同一页面，实现共享存储
7.1.创建nfs共享并增加nfs资源
[root@GW ~]# mkdir /web/ha/
[root@GW ~]# vim /etc/exports
/web/ha 1.1.1.0/24(ro)
[root@GW ~]# service nfs start
[root@GW ~]# chkconfig nfs on
[root@node1 cluster]# system-config-cluster &







7.2.重启一下webservice服务组
[root@node1 ~]# clusvcadm -R webservice
这时，通过web测试访问http://1.1.1.100 会显示共享的web信息
8.以命令方式配置并启动基本集群,但资源和服务无法通过命令方式配置
8.1.停止各集群服务并清空集群配置文件

[root@node1 ~]# clusvcadm -s webservice
[root@GW ~]# for I in {1..3};do ssh node$I 'service rgmanager stop'; done
[root@GW ~]# for I in {1..3};do ssh node$I 'service cman stop'; done
[root@GW ~]# for I in {1..3};do ssh node$I 'rm -rf /etc/cluster/*'; done
8.2.创建一个集群名称
[root@node1 cluster]# ccs_tool create tcluster
8.3.增加一个fence设备
[root@node1 cluster]# ccs_tool addfence meatware fence_manual
8.3.增加节点
[root@node1 cluster]# ccs_tool addnode -h #查看帮助
[root@node1 cluster]# ccs_tool addnode -n 1 -v 1 -f meatware node1.willow.com
[root@node1 cluster]# ccs_tool addnode -n 2 -v 1 -f meatware node2.willow.com
[root@node1 cluster]# ccs_tool addnode -n 3 -v 1 -f meatware node3.willow.com
[root@node1 cluster]# ccs_tool lsnode #查看刚加入的节点主机
Cluster name: tcluster, config_version: 5

Nodename Votes Nodeid Fencetype
node1.willow.com 1 1 meatware
node2.willow.com 1 2 meatware
node3.willow.com 1 3 meatware
8.4.启动cman和rgmanager集群服务
[root@node1 cluster]# service cman start
[root@node2 cluster]# service cman start
[root@node3 cluster]# service cman start
[root@GW ~]# for I in {1..3}; do ssh node$I 'service rgmanager start'; done
[root@node1 cluster]#
8.5.如需配置额外资源和服务,只能使用图形化工具system-config-cluster
9.iSCSI安装及配置
9.1.命令及概念解释:
服务器端iSCSI Target:安装包是scsi-target-utils
工作在3260端口
客户端认证方式：基于IP 及 基于用户的CHAP认证
客户端iSCSI Initiator:安装包是 iscsi-initiator-utils
9.2.tgtadm模式化命令,服务器端命令

--mode常用模式：target、logicalunit、account
target --op new、delete、show、update、bind、unbind
logicalunit --op new、delete
account --op new、delete、bind、unbind
--lld,短选项是:-L
--tid, 短选项是:-t
--lun, 短选项是:-l
--backing-store <path>, 短选项是:-b
--initiator-address <address>,短选项是: -I
--targetname <targetname>，短选项是:-T,
9.3.targetname: target命令规则:
iqn.yyyy-mm.<reversed domain name>[:identifier]
例如:iqn.2016-08.com.willow:tstore.disk1
9.4.iscsiadm模式化命令，客户端命令
-m {discovery|node|session|iface}
discovery: 发现某服务器是否有target输出，以及输出了哪些target；
node: 管理跟某target的关联关系；
session: 会话管理
iface: 接口管理
iscsiadm -m discovery [ -d debug_level ] [ -P printlevel ] [ -I iface -t type -p ip:port [ -l ] ]
-d: 0-8
-I:
-t type: SendTargets(st), SLP, and iSNS
-p: IP:port
iscsiadm -m discovery -d 2 -t st -p 172.16.100.100
iscsiadm -m node [ -d debug_level ] [ -L all,manual,automatic ] | [ -U all,manual,automatic ]
iscsiadm -m node [ -d debug_level ] [ [ -T targetname -p ip:port -I ifaceN ] [ -l | -u ] ] [ [ -o operation ] [ -n name ] [ -v value ] ]
iscsi-initiator-utils:
不支持discovery认证；
如果使用基于用户的认证，必须首先开放基于IP的认证；

9.5.安装iscsi服务器端:iSCSI Target,安装包是scsi-target-utils

首先新增一块硬盘或分区或raid作测试，这里我新增两块分区,/dev/sda5,/dev/sda6(省略)
[root@GW ~]# yum install -y scsi-target-utils
[root@GW ~]# service tgtd start
[root@GW ~]# chkconfig tgtd on
9.6.新增一个target并取名:
[root@GW ~]# tgtadm --lld iscsi --mode target --op new --targetname iqn.2016-08.com.willow:tstore.disk1 --tid 1
9.7.显示target:
[root@GW ~]# tgtadm --lld iscsi --mode target --op show
Target 1: iqn.2016-08.com.willow:tstore.disk1
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
Account information:
ACL information:
9.8.新增一个lun:
[root@GW ~]# tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1 --backing-store /dev/sda6
显示target:[root@GW ~]# tgtadm --lld iscsi --mode target --op showTarget 1: iqn.2016-08.com.willow:tstore.disk1
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1012 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sda6
Backing store flags:
Account information:
ACL information:
9.9.基于IP认证:
[root@GW ~]# tgtadm --lld iscsi --mode target --op bind --tid 1 --initiator-address 1.1.1.0/24
显示target:
[root@GW ~]# tgtadm --lld iscsi --mode target --op show
Target 1: iqn.2016-08.com.willow:tstore.disk1
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1012 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sda6
Backing store flags:
Account information:
ACL information:
1.1.1.0/24
9.10.安装iscsi客户端:iSCSI Initiator,安装包是iscsi-initiators-utils
[root@node1 cluster]# yum install -y iscsi-initiator-utils
客户端命名:
[root@node1 cluster]# iscsi-iname -p iqn.2016-08.com.willow
iqn.2016-08.com.willow:b66a29864
[root@node1 cluster]# echo "InitiatorName=`iscsi-iname -p iqn.2016-08.com.willow`" > /etc/iscsi/initiatorname.iscsi
[root@node1 cluster]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2016-08.com.willow:2be4ce331532
9.11.启动iscsi服务并发现targets

[root@node1 cluster]# service iscsi start
iscsid (pid 3168) is running...
Setting up iSCSI targets: iscsiadm: No records found
[ OK ]
[root@node1 ~]# iscsiadm -m discovery -t sendtargets -p 1.1.1.88
1.1.1.88:3260,1 iqn.2016-08.com.willow:tstore.disk1
[root@node1 cluster]# ls /var/lib/iscsi/send_targets/
1.1.1.88,3260
9.12.登入连接targets:

[root@node1 ~]# iscsiadm -m node -T iqn.2016-08.com.willow:tstore.disk1 -p 1.1.1.88 -l
Logging in to [iface: default, target: iqn.2016-08.com.willow:tstore.disk1, portal: 1.1.1.88,3260] (multiple)
Login to [iface: default, target: iqn.2016-08.com.willow:tstore.disk1, portal: 1.1.1.88,3260] successful.
[root@node1 ~]# fdisk -l
Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 25 200781 83 Linux
/dev/sda2 26 1299 10233405 83 Linux
/dev/sda3 1300 1560 2096482+ 82 Linux swap / Solaris

Disk /dev/sdb: 1011 MB, 1011677184 bytes
32 heads, 61 sectors/track, 1012 cylinders
Units = cylinders of 1952 * 512 = 999424 bytes

Disk /dev/sdb doesn't contain a valid partition table
[root@node1 ~]#
9.13.登出target
[root@node1 ~]# iscsiadm -m node -T iqn.2016-08.com.willow:tstore.disk1 -p 1.1.1.88 -u

Logging out of session [sid: 1, target: iqn.2016-08.com.willow:tstore.disk1, portal: 1.1.1.88,3260]
Logout of [sid: 1, target: iqn.2016-08.com.willow:tstore.disk1, portal: 1.1.1.88,3260] successful.
9.14.徹底刪除登入target數據庫信息
[root@node1 ~]# chkconfig iscsi on
[root@node1 ~]# ls /var/lib/iscsi/send_targets/1.1.1.88,3260/
iqn.2016-08.com.willow:tstore.disk1,1.1.1.88,3260,1,default st_config
[root@node1 ~]# iscsiadm -m node -T iqn.2016-08.com.willow:tstore.disk1 -p 1.1.1.88 -o delete
[root@node1 ~]# ls /var/lib/iscsi/send_targets/1.1.1.88,3260/
st_config
[root@node1 ~]# rm -rf /var/lib/iscsi/send_targets/1.1.1.88,3260/
9.10.target上解除ACL綁定
[root@GW ~]# tgtadm --lld iscsi --mode target --op unbind --tid 1 --initiator-address 1.1.1.0/24
[root@GW ~]# tgtadm --lld iscsi --mode target --op show

9.15.基于帐号认证:

[root@GW ~]# tgtadm --lld iscsi --mode account --op new --user willow --password willow
[root@GW ~]# tgtadm --lld iscsi --mode account --op bind --tid 1 --user willow
[root@GW ~]# tgtadm --lld iscsi --mode target --op show
Target 1: iqn.2016-08.com.willow:tstore.disk1
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1012 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sda6
Backing store flags:
Account information:
willow
ACL information:
[root@GW ~]#
此时发现连接target无法发现，报错信息显示登入失败：
[root@node1 ~]# iscsiadm -m discovery -t st -p 1.1.1.88
iscsiadm: Login failed to authenticate with target
iscsiadm: discovery login to 1.1.1.88 rejected: initiator failed authorization

iscsiadm: Could not perform SendTargets discovery: iSCSI login failed due to authorization failure
[root@node1 ~]#
[root@node1 ~]# iscsiadm -m discovery -d 2 -t st -p 1.1.1.88 #此命令会显示更多debug报错信息
如果使用基于用户的认证，必须首先开放基于IP的认证；
[root@GW ~]# tgtadm --lld iscsi --mode target --op bind --tid 1 --initiator-address 1.1.1.10/24
[root@node1 ~]# vim /etc/iscsi/iscsid.conf #启用chap认证并修改对应帐号密码
node.session.auth.authmethod = CHAP
node.session.auth.username = willow
node.session.auth.password = willow
[root@node1 ~]# service iscsi restart
iscsiadm: No matching sessions found
Stopping iSCSI daemon:
iscsid is stopped [ OK ]
Starting iSCSI daemon: [ OK ]
[ OK ]
Setting up iSCSI targets: iscsiadm: No records found
[ OK ]
[root@node1 ~]# iscsiadm -m discovery -t st -p 1.1.1.88
1.1.1.88:3260,1 iqn.2016-08.com.willow:tstore.disk1
[root@node1 ~]# iscsiadm -m node -T iqn.2016-08.com.willow:tstore.disk1 -p 1.1.1.88 -l
Logging in to [iface: default, target: iqn.2016-08.com.willow:tstore.disk1, portal: 1.1.1.88,3260] (multiple)
Login to [iface: default, target: iqn.2016-08.com.willow:tstore.disk1, portal: 1.1.1.88,3260] successful.
[root@node1 ~]# fdisk -l

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 25 200781 83 Linux
/dev/sda2 26 1299 10233405 83 Linux
/dev/sda3 1300 1560 2096482+ 82 Linux swap / Solaris

Disk /dev/sdb: 1011 MB, 1011677184 bytes
32 heads, 61 sectors/track, 1012 cylinders
Units = cylinders of 1952 * 512 = 999424 bytes

Disk /dev/sdb doesn't contain a valid partition table
[root@node1 ~]#
10.对于服务器端target来说，使用命令创建的target，重启服务器，配置丢失，下面介绍将所有命令配置到targets.conf文件中，重启自动生效
[root@GW ~]# vim /etc/tgt/targets.conf

<target iqn.2016-08.com.willow:tstore.disk1>
backing-store /dev/sda6
backing-store /dev/sda7
incominguser willow willow
initiator-address 1.1.1.0/24
</target>

[root@GW ~]# service tgtd restart
Stopping SCSI target daemon: Stopping target framework daemon
[ OK ]
Starting SCSI target daemon: Starting target framework daemon

[root@GW ~]# tgtadm --lld iscsi --mode target --op show
Target 1: iqn.2016-08.com.willow:tstore.disk1
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1012 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sda6
Backing store flags:
LUN: 2
Type: disk
SCSI ID: IET 00010002
SCSI SN: beaf12
Size: 1012 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sda7
Backing store flags:
Account information:
willow
ACL information:
1.1.1.0/24
[root@GW ~]#
上面发现sda6,sda7按顺序分配lun号码
11.重新配置targets.conf文件，手动分配lun号码
[root@GW ~]# vim /etc/tgt/targets.conf
<target iqn.2016-08.com.willow:tstore.disk1>
<backing-store /dev/sda6>
vendor_id willow
lun 6
</backing-store>
<backing-store /dev/sda7>
vendor_id willow
lun 7
</backing-store>
incominguser willow willow
initiator-address 1.1.1.0/24
</target>
[root@GW ~]# service tgtd restart
Stopping SCSI target daemon: Stopping target framework daemon
[ OK ]
Starting SCSI target daemon: Starting target framework daemon

[root@GW ~]# tgtadm --lld iscsi --mode target --op show
Target 1: iqn.2016-08.com.willow:tstore.disk1
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 6
Type: disk
SCSI ID: IET 00010006
SCSI SN: beaf16
Size: 1012 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sda6
Backing store flags:
LUN: 7
Type: disk
SCSI ID: IET 00010007
SCSI SN: beaf17
Size: 1012 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sda7
Backing store flags:
Account information:
willow
ACL information:
1.1.1.0/24
[root@GW ~]#

12.配置集群文件系统gfs2,目的是让共享存储文件系统让多个节点挂载同时使用，立即通知锁信息
mkfs.gfs2命令:
-j #: 指定日志区域的个数，有几个就能够被几个节点所挂载；
-J #: 指定日志区域的大小，默认为128MB;
-p {lock_dlm|lock_nolock}:
-t <name>: 锁表的名称，格式为clustername:locktablename,
lustername为当前节点所在的集群的名称，locktablename要在当前集群唯一；
12.1.安装cman, rgmanger, gfs2-utils 软件包
[root@node1 ~]# yum install -y cman rgmanger gfs2-utils
[root@node1 ~]# mkfs.gfs2 -j 2 -p lock_dlm -t tcluster:sdb1 /dev/sdb1
This will destroy any data on /dev/sdb1.

Are you sure you want to proceed? [y/n] y

Device: /dev/sdb1
Blocksize: 4096
Device Size 0.47 GB (122236 blocks)
Filesystem Size: 0.47 GB (122235 blocks)
Journals: 2
Resource Groups: 2
Locking Protocol: "lock_dlm"
Lock Table: "tcluster:sdb1"
UUID: 6ACB5ED9-1808-879B-F5BF-A1DB5E253FB4

[root@node1 ~]#
注意:-j 2:代表只能挂载两个节点主机，挂载第3个节点将会提示报错信息，如下
[root@GW ~]# for I in {1..3};do ssh node$I 'mkdir /mnt/iscsi'; done
[root@node1 ~]# mount -t gfs2 /dev/sdb1 /mnt/iscsi/
[root@node2 iscsi]# mount /dev/sdb1 /mnt/iscsi/
[root@node3 ~]# mount /dev/sdb1 /mnt/iscsi/
/sbin/mount.gfs2: Too many nodes mounting filesystem, no free journals

[root@node1 iscsi]# gfs2_tool journals /dev/sdb1 #查看挂载节点的个数
journal1 - 128MB
journal0 - 128MB
2 journal(s) found.
[root@node1 iscsi]#
[root@node1 iscsi]# gfs2_jadd -j 1 /dev/sdb1 #新增1个挂载节点
Filesystem: /mnt/iscsi
Old Journals 2
New Journals 3
[root@node1 iscsi]#
[root@node3 ~]# mount /dev/sdb1 /mnt/iscsi/ #再次挂载正常
13.Clvm集群逻辑卷 创建及使用:
[root@GW ~]# vim /etc/tgt/targets.conf
<target iqn.2016-09.com.willow:tsan.disk1>
<backing-store /dev/sdb>
vendor_id Willow
lun 1
</backing-store>
initiator-address 1.1.1.0/24
incominguser sanuser sanpass
</target>
[root@GW ~]# service tgtd start
[root@GW ~]# chkconfig tgtd on
[root@GW ~]# for I in {1..3}; do ssh node$I 'echo "InitiatorName=`iscsi-iname -p iqn.2016-09.com.willow`" > /etc/iscsi/initiatorname.iscsi';done
[root@GW ~]# vim iscsi.sed
s@#node.session.auth.authmethod = CHAP@node.session.auth.authmethod = CHAP@
s@#node.session.auth.username = username@node.session.auth.username = sanuser@
s@#node.session.auth.password = password@node.session.auth.password = sanpass@
[root@GW ~]# for I in {1..3}; do scp iscsi.sed node$I:/etc/iscsi/iscsi.sed;ssh node$I 'sed -i -f /etc/iscsi/iscsi.sed /etc/iscsi/iscsid.conf';done
[root@GW ~]# for I in {1..3}; do ssh node$I 'service iscsi start;chkconfig iscsi on';done
[root@GW ~]# for I in {1..3}; do ssh node$I 'iscsiadm -m discovery -t st -p 1.1.1.88';done
1.1.1.88:3260,1 iqn.2016-09.com.willow:tsan.disk1
1.1.1.88:3260,1 iqn.2016-09.com.willow:tsan.disk1
1.1.1.88:3260,1 iqn.2016-09.com.willow:tsan.disk1
[root@GW ~]# for I in {1..3}; do ssh node$I 'iscsiadm -m node -T iqn.2016-09.com.willow:tsan.disk1 -p 1.1.1.88 -l';done
Logging in to [iface: default, target: iqn.2016-09.com.willow:tsan.disk1, portal: 1.1.1.88,3260] (multiple)
Login to [iface: default, target: iqn.2016-09.com.willow:tsan.disk1, portal: 1.1.1.88,3260] successful.
Logging in to [iface: default, target: iqn.2016-09.com.willow:tsan.disk1, portal: 1.1.1.88,3260] (multiple)
Login to [iface: default, target: iqn.2016-09.com.willow:tsan.disk1, portal: 1.1.1.88,3260] successful.
Logging in to [iface: default, target: iqn.2016-09.com.willow:tsan.disk1, portal: 1.1.1.88,3260] (multiple)
Login to [iface: default, target: iqn.2016-09.com.willow:tsan.disk1, portal: 1.1.1.88,3260] successful.
[root@GW ~]# for I in {1..3}; do ssh node$I 'yum install -y cman rgmanager gfs2-utils lvm2-cluster';done

[root@node1 ~]# ccs_tool create tcluster
[root@node1 ~]# ccs_tool addfence meatware fence_manual
running ccs_tool update...
[root@node1 ~]# ccs_tool lsfence
Name Agent
meatware fence_manual
[root@node1 ~]# ccs_tool addnode -v 1 -n 1 -f meatware node1.willow.com
running ccs_tool update...
Segmentation fault
[root@node1 ~]# ccs_tool addnode -v 1 -n 2 -f meatware node2.willow.com
running ccs_tool update...
Segmentation fault
[root@node1 ~]# ccs_tool addnode -v 1 -n 3 -f meatware node3.willow.com
running ccs_tool update...
Segmentation fault
[root@node1 ~]# ccs_tool lsnode

Cluster name: tcluster, config_version: 5

Nodename Votes Nodeid Fencetype
node1.willow.com 1 1 meatware
node2.willow.com 1 2 meatware
node3.willow.com 1 3 meatware
[root@node1 ~]#
[root@node1 ~]# service cman start
[root@node2 ~]# service cman start
[root@node3 ~]# service cman start
[root@GW ~]# for I in {1..3}; do ssh node$I 'chkconfig cman on ';done
[root@GW ~]# for I in {1..3}; do ssh node$I 'service rgmanager start;chkconfig rgmanager on'; done
locking_type = 1 改成 locking_type = 3
[root@GW ~]# for I in {1..3}; do ssh node$I 'servcie clvmd start;chkconfig clvmd on';done
[root@node1 ~]# pvcreate /dev/sdb
[root@node1 ~]# vgcreate clustervg /dev/sdb
[root@node1 ~]# lvcreate -L 500M -n clusterlv clustervg
[root@node1 ~]# mkfs.gfs2 -j 2 -p lock_dlm -t tcluster:lktb1 /dev/clustervg/clusterlv
This will destroy any data on /dev/clustervg/clusterlv.

Are you sure you want to proceed? [y/n] y

Device: /dev/clustervg/clusterlv
Blocksize: 4096
Device Size 0.49 GB (128000 blocks)
Filesystem Size: 0.49 GB (127997 blocks)
Journals: 2
Resource Groups: 2
Locking Protocol: "lock_dlm"
Lock Table: "tcluster:lktb1"
UUID: 0B1A05E3-DEA2-E505-D2F2-F2183C6C51D3
[root@node1 ~]# mkdir /mydata
[root@node1 ~]# mount -t gfs2 /dev/clustervg/clusterlv /mydata
[root@node1 ~]# gfs2_tool df /mydata/
[root@node2 ~]# mkdir /mydata
[root@node2 ~]# mount -t gfs2 /dev/clustervg/clusterlv /mydata/
[root@node2 ~]# cd /mydata/
[root@node2 mydata]# ls
[root@node2 mydata]# touch 1.txt
[root@node1 ~]# gfs2_tool gettune /mydata #查看参数
[root@node1 ~]# gfs2_tool settune /mydata new_files_directio 1
[root@node1 ~]# gfs2_tool journals /mydata
[root@node1 ~]# gfs2_jadd -j 1 /dev/clustervg/clusterlv
Filesystem: /mydata
Old Journals 2
New Journals 3
扩展集群逻辑卷:
[root@node1 ~]# lvextend -L +500M /dev/clustervg/clusterlv
Extending logical volume clusterlv to 1000.00 MB
Logical volume clusterlv successfully resized
[root@node1 ~]# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
clusterlv clustervg -wi-ao 1000.00M
[root@node1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 9.5G 2.5G 6.6G 28% /
/dev/sda1 190M 12M 169M 7% /boot
tmpfs 252M 0 252M 0% /dev/shm
/dev/sr0 3.3G 3.3G 0 100% /mnt/cdrom
/dev/mapper/clustervg-clusterlv
500M 388M 113M 78% /mydata
[root@node1 ~]# gfs2_grow /dev/clustervg/clusterlv

将会持续更新博客.................