Heartbeat + Drbd +Mysql 构建高可用的MYSQL数据库服务

1，什么是DRBD

是由内核模块和相关脚本而构成，用以构建高可用性的集群。其实现方式是通过网络来镜像整个设备。您可以把它看作是一种网络RAID1

Drbd 负责接收数据，把数据写到本地磁盘，然后发送给另一个主机。另一个主机再将数据存到自己的磁盘中。其他所需的组件有集群成员服务，如TurboHA 或 心跳连接，以及一些能在块设备上运行的

应用程序

环境

CentOS 5.4

Eth0：192.168.8.58

Eth1：192.168.10.1

CentOS 5.2

Eth0：192.168.8.59

Eth1：192.168.10.2

2. /etc/hosts 文件中新增加的内容，两边一样

mater:192.168.10.1

slave:192.168.10.2

OS:CentOS 5.4

软件：drbd8.3 kmod-8.3

安装drbd 及drbd内核模块

#

yum install -y drbd83.x86_64 kmod-drbd83.x86_64

3.修改配置文件

#mv /etc/drbd.conf /etc/drbd.conf.bak 备份原配置文件

#vim /etc/drbd.conf

global { usage-count yes; }

common { syncer { rate 100M; } }

resource r0 {

protocol C;

net {

cram-hmac-alg sha1;

shared-secret "FooFunFactory";

}

on master {

device /dev/drbd0;

disk /dev/sda4;

address 192.168.10.1:7898;

meta-disk internal;

}

on slave {

device /dev/drbd0;

disk /dev/sda4;

address 192.168.10.2:7898;

meta-disk internal;

}

}

我的是很简单的，为了测试，还需要大家根据自己的需要自己修改~

好了两台机器都写完配置文件后，两边的drbd.conf 配置一模一样

在ha1和ha2上分别输入命令

#

drbdadm create-md r0

“r0"是我们在drbd.conf里定义

弹出的信息，敲回车，如果drbd.conf配置好的话，有的信息是不会弹出的。

HA1上

[root@ha1 etc]#

drbdadm create-md r0

v08 Magic number not found

v07 Magic number not found

v07 Magic number not found

v08 Magic number not found

Writing meta data...

initialising activity log

NOT initialized bitmap

New drbd meta data block sucessfully created.

HA2 上

[root@ha2 etc]#

drbdadm create-md r0

v08 Magic number not found

v07 Magic number not found

v07 Magic number not found

v08 Magic number not found

Writing meta data...

initialising activity log

NOT initialized bitmap

New drbd meta data block sucessfully created.

现在我们可以启动DRBD了,分别在两台主机上执行:

[root@ha1 etc]# /etc/init.d/drbd start

Starting DRBD resources: [ d(r0) s(r0) n(r0) ].

..........

***************************************************************

DRBD's startup script waits for the peer node(s) to appear.

- In case this node was already a degraded cluster before the

reboot the timeout is 0 seconds. [degr-wfc-timeout]

- If the peer was available before the reboot the timeout will

expire after 0 seconds. [wfc-timeout]

(These values are for resource 'r0'; 0 sec -> wait forever)

To abort waiting enter 'yes' [ 23]: //在这个时候你在HA2 上启动服务，就会马上成功启动服务，因为他在等待另一个节点

[root@ha2 etc]# /etc/init.d/drbd start

Starting DRBD resources: [ d(r0) s(r0) n(r0) ].

[root@ha1 etc]# cat /proc/drbd

version: 8.2.6 (api:88/proto:86-88)

GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32

0: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent C r---

ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:8385604

[root@ha2 etc]# cat /proc/drbd

version: 8.2.6 (api:88/proto:86-88)

GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32

0: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent C r---

ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:8385604

"/proc/drbd"中显示了drbd当前的状态.第一行的st表示两台主机的状态,都是"备机"状态.

ds是磁盘状态,都是"不一致"状态.

这是由于,DRBD无法判断哪一方为主机,以哪一方的磁盘数据作为标准数据.所以,我们需要初始化

一个主机.在drbd1上执行:

[root@drbd1 local]#

drbdsetup /dev/drbd0 primary -o

主备机状态分别是"主/备",主机磁盘状态是"实时",备机状态是"不一致".

在第3行,可以看到数据正在同步中,即主机正在将磁盘上的数据,传递到备机上.现在的进度是[>...................] sync'ed: 0.4% (1040316/1040316)K

稍等一会,在数据同步完后,再查看一下ha1的DRBD状态:

磁盘状态都是"实时",表示数据同步完成了.

你现在可以把主机上的DRBD设备挂载到一个目录上进行使用.备机的DRBD设备无法被挂载,因为它是

用来接收主机数据的,由DRBD负责操作.

在drbd1执行

root@ha1:~#

mkfs.ext3 /dev/drbd0

//网上这里有很多不同的版本，具体drbd深入我还没有研究，我现在安装的版本是只要在第一台上面格式化文件系统，他会自动传过去。如果你在ha2上格式化，他会提示出错，而且格式化要在上面相应

工作做完以后才可以到这一步。

root@ha1:~#

mount /dev/drbd0 /mnt

root@ha1:~#

cd /mnt

root@ha1:/mnt#

touch huzi

root@dha1:/mnt# ls

huzi lost+found

可以看到,在主机drbd1上产生的文件huzi,也完整的保存在备机drbd2的DRBD分区上.

这就是DRBD的网络RAID-1功能. 在主机上的任何操作,都会被同步到备机的相应磁盘分区上,达到数据备份的效果.

DRBD的主备机切换有时,你需要将DRBD的主备机互换一下.可以执行下面的操作:

在主机上,先要卸载掉DRBD设备

root@drbd1~#

umount /mnt

将主机降级为"备机"

root@ha1~#

drbdadm secondary r0

[root@ha1 ~]#

cat /proc/drbd

version: 8.2.6 (api:88/proto:86-88)

GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32

0: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---

ns:8650688 nr:24 dw:265108 dr:8385749 al:105 bm:584 lo:0 pe:0 ua:0 ap:0 oos:0

现在,两台主机都是"备机".

在备机drbd2上,将它升级为"主机".

[root@ha2 mnt]#

drbdadm primary r0

[root@ha2 etc]#

cat /proc/drbd

version: 8.2.6 (api:88/proto:86-88)

GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32

0: cs:Connected strimary/Secondary ds:UpToDate/UpToDate C r---

ns:0 nr:32 dw:32 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:0

[root@ha2 etc]#

mount /dev/drbd0 /mnt

[root@ha2 etc]#

cd /mnt

[root@ha2 mnt]#

ls

huzi lost+found

[root@ha2 mnt]#

ll

total 16

-rw-r--r-- 1 root root 0 Mar 19 11:02 huzi

drwx------ 2 root root 16384 Mar 19 11:01 lost+found

现在drbd2成为主机了。

DRBD相关切换命令

DRBD切换

drbdadm secondary r0 //把主机切换成备机

drbdadm primary r0 //把备机切换成主机

注意，在我测试的情况。DRBD只有主机可以读写，备机不能够挂载，也就是说不能够读。可能是我的drbd.conf文件配置方面的原因 ，我会继续找一些文章来读，这样对于数据库是可以满足的，如果想

做WEB。可以有些台只是用来读取的，不过我们这里只讨论数据库

------------------------------------------------------------------

上面的DRBD已经成功配置完成，可以正常切换，现在我们让两台成为高可用，所谓高可用，就是双机热备，互备。这里我不解释太多，相应搞过HA的人都清楚，不清楚的上www.linux-ha.org 上面看看文

档。

主机ha1和ha2上安装heartbeat

yum –y install heartbeat && yum –y install heartbeat

//这里我也不清楚为什么要运行两次他才给我安装heartbeat ，嘿嘿，哪位知道告诉我为什么，我是每次都要晕两次才可以我安装

在主机ha1和ha2 上安装mysql mysql-server

yum -y install mysql mysql-server

以下只在ha1上进行操作

在ha1上启动mysql

service mysqld start

//初始化数据库数据文件

再停止数据库

service mysqld stop

mount /dev/drbd0 /mnt

cp /var/lib/mysql/* /mnt -ar

umount /mnt

配置heartbeat

cp /usr/share/doc/heartbeat-2.1.3/haresources /etc/ha.d

cp /usr/share/doc/heartbeat-2.1.3/ha.cf /etc/ha.d

cp /usr/share/doc/heartbeat-2.1.3/authkeys /etc/ha.d

cd /etc/ha.d

chmod 600 authkeys

authkeys文件中下面两行前面的#号去掉

vi authkeys

auth 1

1 crc

配置ha.cf 内容如下，我就不一一解释了，有英语注释，注意的地方我会加注释

debugfile /var/log/ha-debug

logfile /var/log/ha-log

logfacility local0

keepalive 2

deadtime 10

warntime 5

initdead 120

ucast eth1 192.168.10.2 //这里是对方的ip 主要用于HA的相互检测 网络接口要注意了，我用的是专用接口，ha2主机改为192.168.10.1

auto_failback on

watchdog /dev/watchdog

node ha1

node ha2

ping 192.168.10.1

respawn hacluster /usr/lib/heartbeat/ipfail

apiauth ipfail gid=haclient uid=hacluster

添加自动加载drbd 及持载文件系统的脚本，放在/etc/ha.d/resource.d 目录下面

vi mysqld_umount

#!/bin/sh

#

# High-Availability Pre-Startup/Shutdown Script

#

# Description: Runs on Startup or shutdown of heartbeat (not resource based).

# Also runs before start, after start or before stop,

# and after stop.

#

# Author: Matthew Soffen

#

# Support: linux-ha@lists.linux-ha.org

#

# License: GNU Lesser General Public License (LGPL)

#

# Copyright: (C) 2002 Matthew Soffen

#

#

unset LC_ALL; export LC_ALL

unset LANGUAGE; export LANGUAGE

prefix=/usr

exec_prefix=/usr

. /etc/ha.d/shellfuncs

case "$1" in

'start')

#/sbin/drbdadm -- --do-what-I-say primary all

/sbin/drbdadm primary all

#drbdsetup /dev/drbd1 primary -o

/bin/mount /dev/drbd0 /var/lib/mysql

;;

'pre-start')

;;

'post-start')

;;

'stop')

/bin/umount /var/lib/mysql

/sbin/drbdadm secondary all

;;

'pre-stop')

;;

'post-stop')

;;

*)

echo "Usage: $0 { start | pre-start | post-start | stop | pre-stop | post-stop }"

;;

esac

exit 0

配置haresource

添加如下一行

master IPaddr::192.168.8.60/24/eth0:1 mysqld_umount mysqld

工作完成,把/etc/ha.d目录全部覆盖到ha2上面去

scp -r /etc/ha.d 192.168.10.2:/etc

提示输入对方root用户的密码，全部转输完以后，注意修改/etc/ha.d/ha.cf 中的ucast eth1 192.168.10.2 为对方的IP也就是HA1主机的ETH1的IP 改为ucast eth1 192.168.10.1

两边同时启动heartbeat 观察日志

HA1上 HA的日志

heartbeat[4039]: 2009/03/19_15:11:25 info: heartbeat: version 2.1.3

heartbeat[4039]: 2009/03/19_15:11:25 info: Heartbeat generation: 1237438269

heartbeat[4039]: 2009/03/19_15:11:25 info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth1

heartbeat[4039]: 2009/03/19_15:11:25 info: glib: ucast: bound send socket to device: eth1

heartbeat[4039]: 2009/03/19_15:11:25 info: glib: ucast: bound receive socket to device: eth1

heartbeat[4039]: 2009/03/19_15:11:25 info: glib: ucast: started on port 694 interface eth1 to 192.168.10.2

heartbeat[4039]: 2009/03/19_15:11:25 info: glib: ping heartbeat started.

heartbeat[4039]: 2009/03/19_15:11:25 info: G_main_add_TriggerHandler: Added signal manual handler

heartbeat[4039]: 2009/03/19_15:11:25 info: G_main_add_TriggerHandler: Added signal manual handler

heartbeat[4039]: 2009/03/19_15:11:25 notice: Using watchdog device: /dev/watchdog

heartbeat[4039]: 2009/03/19_15:11:25 info: G_main_add_SignalHandler: Added signal handler for signal 17

heartbeat[4039]: 2009/03/19_15:11:25 info: Local status now set to: 'up'

heartbeat[4039]: 2009/03/19_15:11:26 info: Link ha2:eth1 up.

heartbeat[4039]: 2009/03/19_15:11:26 info: Status update for node ha2: status active

heartbeat[4039]: 2009/03/19_15:11:26 info: Link 192.168.254.1:192.168.254.1 up.

heartbeat[4039]: 2009/03/19_15:11:26 info: Status update for node 192.168.254.1: status ping

harc[4048]: 2009/03/19_15:11:26 info: Running /etc/ha.d/rc.d/status status

heartbeat[4039]: 2009/03/19_15:11:27 info: Comm_now_up(): updating status to active

heartbeat[4039]: 2009/03/19_15:11:27 info: Local status now set to: 'active'

heartbeat[4039]: 2009/03/19_15:11:27 info: Starting child client "/usr/lib/heartbeat/ipfail" (498,496)

heartbeat[4065]: 2009/03/19_15:11:27 info: Starting "/usr/lib/heartbeat/ipfail" as uid 498 gid 496 (pid 4065)

heartbeat[4039]: 2009/03/19_15:11:27 info: remote resource transition completed.

heartbeat[4039]: 2009/03/19_15:11:27 info: remote resource transition completed.

heartbeat[4039]: 2009/03/19_15:11:27 info: Local Resource acquisition completed. (none)

heartbeat[4039]: 2009/03/19_15:11:28 info: ha2 wants to go standby [foreign]

heartbeat[4039]: 2009/03/19_15:11:30 info: standby: acquire [foreign] resources from ha2

heartbeat[4068]: 2009/03/19_15:11:30 info: acquire local HA resources (standby).

ResourceManager[4081]: 2009/03/19_15:11:30 info: Acquiring resource group: ha1 IPaddr::192.168.0.253/24/eth0:1 mysqld_umount mysqld

IPaddr[4108]: 2009/03/19_15:11:31 INFO: Resource is stopped

ResourceManager[4081]: 2009/03/19_15:11:31 info: Running /etc/ha.d/resource.d/IPaddr 192.168.0.253/24/eth0:1 start

IPaddr[4206]: 2009/03/19_15:11:31 INFO: Using calculated netmask for 192.168.0.253: 255.255.255.0

IPaddr[4206]: 2009/03/19_15:11:32 INFO: eval ifconfig eth0:0 192.168.0.253 netmask 255.255.255.0 broadcast 192.168.0.255

IPaddr[4177]: 2009/03/19_15:11:32 INFO: Success

ResourceManager[4081]: 2009/03/19_15:11:32 info: Running /etc/ha.d/resource.d/mysqld_umount start

ResourceManager[4081]: 2009/03/19_15:11:33 info: Running /etc/ha.d/resource.d/mysqld start

ipfail[4065]: 2009/03/19_15:11:34 info: Ping node count is balanced.

heartbeat[4068]: 2009/03/19_15:11:35 info: local HA resource acquisition completed (standby).

heartbeat[4039]: 2009/03/19_15:11:35 info: Standby resource acquisition done [foreign].

heartbeat[4039]: 2009/03/19_15:11:35 info: Initial resource acquisition complete (auto_failback)

heartbeat[4039]: 2009/03/19_15:11:35 info: remote resource transition completed.

通过以上观察，资源全部加载成功，我们来验证一下

网络加载

eth0:0 Link encap:Ethernet HWaddr 00:0C:29:39:A8:2B

inet addr:192.168.0.253 Bcast:192.168.0.255 Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

Interrupt:169 Base address:0x2000

文件系统加载

[root@ha1 resource.d]#

mount

/dev/sda3 on / type ext3 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

/dev/sda5 on /data type ext3 (rw)

/dev/sda1 on /boot type ext3 (rw)

tmpfs on /dev/shm type tmpfs (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

/dev/drbd0 on /var/lib/mysql type ext3 (rw)

[root@ha1 resource.d]

version: 8.2.6 (api:88/proto:86-88)

GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32

0: cs:Connectedst:Primary/Secondaryds:UpToDate/UpToDate C r---

ns:512 nr:1008 dw:1520 dr:5950 al:9 bm:8 lo:0 pe:0 ua:0 ap:0 oos:0

DRBD是在主机状态 ，MYSQL我就不贴出来，验证MYSQL就没有必要了，大家自己去验证吧，我已经验证过了，其实大家都知道，文件系统都过来了，还验证个啥子

资源的状态，HA处在备机状态

heartbeat[4412]: 2009/03/19_13:30:43 info: foreign HA resource release completed (standby).

heartbeat[3795]: 2009/03/19_13:30:43 info: Local standby process completed [foreign].

ipfail[3820]: 2009/03/19_13:30:44 info: No giveup timer to abort.

heartbeat[3795]: 2009/03/19_13:30:45 WARN: 1 lost packet(s) for [ha1] [16:18]

heartbeat[3795]: 2009/03/19_13:30:45 info: remote resource transition completed.

heartbeat[3795]: 2009/03/19_13:30:45 info: No pkts missing from ha1!

heartbeat[3795]: 2009/03/19_13:30:45 info: Other node completed standby takeover of foreign resources.

[root@ha2 ha.d]#

[root@ha2 ha.d]#

cat /proc/drbd

version: 8.2.6 (api:88/proto:86-88)

GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32

0: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---

ns:392 nr:832 dw:1224 dr:2989 al:9 bm:8 lo:0 pe:0 ua:0 ap:0 oos:0

切换测试

HA1上停止HA服务

[root@ha1 resource.d]#

service heartbeat stop

Stopping High-Availability services:

[ OK ]

[root@ha1 resource.d]#

在HA2上观察

[root@ha2 ha.d]# tail /var/log/ha-log -f

heartbeat[3795]: 2009/03/19_13:55:21 info: Standby resource acquisition done [foreign].

harc[4636]: 2009/03/19_13:55:21 info: Running /etc/ha.d/rc.d/status status

mach_down[4652]: 2009/03/19_13:55:21 info: Taking over resource group IPaddr::192.168.0.253/24/eth0:1

ResourceManager[4678]: 2009/03/19_13:55:21 info: Acquiring resource group: ha1 IPaddr::192.168.0.253/24/eth0:1 mysqld_umount mysqld

IPaddr[4705]: 2009/03/19_13:55:22 INFO: Resource is stopped

ResourceManager[4678]: 2009/03/19_13:55:22 info: Running /etc/ha.d/resource.d/IPaddr 192.168.0.253/24/eth0:1 start

IPaddr[4803]: 2009/03/19_13:55:22 INFO: Using calculated netmask for 192.168.0.253: 255.255.255.0

IPaddr[4803]: 2009/03/19_13:55:22 INFO: eval ifconfig eth0:0 192.168.0.253 netmask 255.255.255.0 broadcast 192.168.0.255

IPaddr[4774]: 2009/03/19_13:55:23 INFO: Success

ResourceManager[4678]: 2009/03/19_13:55:23 info: Running /etc/ha.d/resource.d/mysqld_umount start

ResourceManager[4678]: 2009/03/19_13:55:23 info: Running /etc/ha.d/resource.d/mysqld start

mach_down[4652]: 2009/03/19_13:55:25 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired

mach_down[4652]: 2009/03/19_13:55:25 info: mach_down takeover complete for node ha1.

heartbeat[3795]: 2009/03/19_13:55:25 info: mach_down takeover complete.

验证不贴出来了，通过上面已经正常切换了，在10秒内

服务启动的顺序，drbd让系统自动加载，heartbeat 放在/etc/rc.local里面，不然会出错，为什么自己去想，因为heartbeat 在drbd前面启动，你可以调整他们两的启动顺序。MYSQLD不要开机自动加载

启动，这里我把注意的地方写一下，自己快半年没弄HA啦。今天做起来有点问题，HA通信的IP是相互对方的IP地址，fence ip 一定要存在，因为我是虚拟机测试的，开始没注意，花了点时间来查找问题

这个对比AB复制，备机不可以用来读，但保证了高可用性。DRBD在生产上已经有人使用，如果想达到AB复制效果，再加一台做复制就行了。这个相信大家可以加一台做复制啰。有什么问题请跟帖哈

参考

/article/4226928.html

http://blog.chinaunix.net/u2/66227/showart_1018701.html