LVS NAT + Keepalived HOWTO

By Adam Fletcher (C) 2002, released under GPL

Install, testing and running of a Keepalived HA based LVS/NAT

1. keepalived - what is it?

From Alexandre Cassen, author of keepalived:

"The main goal of the keepalived project is to add a strong & robust keepalive facility to the Linux
Virtual Server project. This project is written in C with multilayer TCP/IP stack checks. Keepalived implements a framework based on three family checks : Layer3, Layer4 & Layer5. This framework
gives the daemon the ability of checking a LVS server pool states. When one of the server of the LVS server pool is down, keepalived informs the
linux kernel via a setsockopt call to remove this server entrie from the LVS topology. In addition keepalived implements a VRRPv2 stack to handle director failover. So in short keepalived is a userspace daemon for LVS cluster nodes healthchecks and LVS directors
failover.

"keepalived is a project started to create a full-featured virtual router for Linux, which includes load balancing through Linux Virtual Server, failover via VRRP and health checks to monitor real servers. Essentially,
it is a single package for doing what is typically done in Linux via lvs+mon+fake+hearbeat. With keepalived an administrator can quickly build a redundant load balancing solution without the hassle of using numerous packages and custom scripts.

keepalived will:

setup the IP addresses (i.e., no need to create the IP's with ifconfig, etc)
handle failover between load balancers (i.e., no need to run heartbeat to watch for failures, or write scripts to handle the failure)
do health checks on services, bringing them in and out of pools (no need to run mon+custom scripts)
send smtp notifications on transition from master to backup send smtp notifications on health check failures
do LVS NAT (no need to run lvs-configure, or do it by hand with ipvsadm)

keepalived is available from www.keepalived.org

Software used in this example:

Linux kernel 2.4.19 (from kernel.org)
LVS kernel patch, 1.0.6 for Linux kernel 2.4.19 (From www.linuxvirtualserver.org)
keepalived 0.7.1 (from www.keepalived.org)

Optional software:

ipvsadm 1.21 (from www.linuxvirtualserver.org)
OpenSSL (latest from openssl.org, I would recommend at least 0.9.6e or above due to security problems in any earlier version

2. Plan your network!

Draw out a logical diagram of your network, either by hand or with a tool like xfig or Visio. Planning your network saves hassle and time later! Make a list of the IP addresses you are going to use, any external
router IPs you may need, the IP addresses of the machine you are going to load balancer and other related information.

3. Configuring your kernel

Configuring your kernel for LVS should be done according to the directions on www.linuxvirtualserver.org.
Be sure you enable full NAT, and IP Forwarding.

After patching your kernel for the latest LVS and installing the new kernel and rebooting, you should turn on IP forwarding. Many Linux distributions allow you to do this through the system configuration editor
(YaST2 on SuSE, linuxconf on Red Hat, for example), or you can do this in your keepalived startup scripts (we'll give some examples of this later). For now, just

echo "1" > /proc/sys/net/ipv4/ip_forwarding

as root.

4. Building ipvsadm (optional)

ipvsadm is a tool available from www.linuxvirtualserver.org that
allows you to setup virtual servers by hand. It is also a useful debugging/status tool, so I recommend building this small tool.

5. Building keepalived

The quick version:

example-01:~ # tar xzvf keepalived-0.7.1.tar.gz; cd keepalived-0.7.1; ./configure; make; make install

Keepalived is very simple to build - grab the latest package from www.keepalived.org,
untar, the run ./configure, then make, and make install.

For more information, read the INSTALL file shipped with keepalived.

If you have any trouble, or keepalived says it is not installing support for something you expected (such as SSL health checks), be sure to verify that you have the missing library or header file in the location
keepalived expects it to be - for instance, the location of LVS's header file has changed in recent releases, so keepalived may not find the header in older versions of LVS.

6. Setting up keepalived: a simple network: 1 load balancer/virtual router, 1 real server on port 22 (ssh).

Now that we have keepalived built and installed, let's set up this network:

Client (on the internet somewhere) --> load balancer --> realserver

Load balancer IPs:

IP of load balancer's external interface(eth0): 192.168.1.9
external VIP of our realserver: 192.168.1.11
IP of load balancer's interface(eth1): 10.20.40.2
internal VIP our realserver will use as a default gateway: 10.20.40.1

Realserver:

IP: 10.20.40.10
be sure to set the default gateway to 10.20.40.1

Our first step is to configure keepalived. The typical location for

this file is /etc/keepalived/keepalived.conf

Note that keepalived, as of this writing, does not report errors in the

configuration file! This means if something is not right in the config file it may be difficult to notice. Try starting keepalived with the -d option, which will dump a config to syslog.

-- cut here --

! This is a comment

! Configuration File for keepalived

global_defs {

! this is who emails will go to on alerts

notification_email {

admins@example.com

fakepager@example.com

! add a few more email addresses here if you would like

}

notification_email_from admins@example.com

! I use the local machine to relay mail

smtp_server 127.0.0.1

smtp_connect_timeout 30

! each load balancer should have a different ID

! this will be used in SMTP alerts, so you should make

! each router easily identifiable

lvs_id LVS_EXAMPLE_01

}

! vrrp_sync_groups make sure that several router instances

! stay together on a failure - a good example of this is

! that the external interface on one router fails and the backup server

! takes over, you want the internal interface on the failed server

! to failover as well, otherwise nothing will work.

! you can have as many vrrp_sync_group blocks as you want.

vrrp_sync_group VG1 {

group {

VI_1

VI_GATEWAY

}

}

! each interface needs at least one vrrp_instance

! each vrrp_instance is a group of VIPs that are logically grouped

! together

! you can have as many vrrp_instaces as you want

vrrp_instance VI_1 {

state MASTER

interface eth0

lvs_sync_daemon_inteface eth0

! each virtual router id must be unique per instance name!

virtual_router_id 51

! MASTER and BACKUP state are determined by the priority

! even if you specify MASTER as the state, the state will

! be voted on by priority (so if your state is MASTER but your

! priority is lower than the router with BACKUP, you will lose

! the MASTER state)

! I make it a habit to set priorities at least 50 points apart

! note that a lower number is lesser priority - lower gets less vote

priority 150

! how often should we vote, in seconds?

advert_int 1

! send an alert when this instance changes state from MASTER to BACKUP

smtp_alert

! this authentication is for syncing between failover servers

! keepalived supports PASS, which is simple password

! authentication

! or AH, which is the IPSec authentication header.

! I don't use AH

! yet as many people have reported problems with it

authentication {

auth_type PASS

auth_pass example

}

! these are the IP addresses that keepalived will setup on this

! machine. Later in the config we will specify which real

! servers are behind these IPs

! without this block, keepalived will not setup and takedown the

! any IP addresses

virtual_ipaddress {

192.168.1.11

! and more if you want them

}

}

! now I setup the instance that the real servers will use as a default

! gateway

! most of the config is the same as above, but on a different interface

vrrp_instance VI_GATEWAY {

state MASTER

interface eth1

lvs_sync_daemon_inteface eth1

virtual_router_id 52

priority 150

advert_int 1

smtp_alert

authentication {

auth_type PASS

auth_pass example

}

virtual_ipaddress {

10.20.40.1

}

}

! now we setup more information about are virtual server

! we are just setting up one for now, listening on port 22 for ssh

! requests.

! notice we do not setup a virtual_server block for the 10.20.40.1

! address in the VI_GATEWAY instance. That's because we are doing NAT

! on that IP, and nothing else.

virtual_server 192.168.1.11 22 {

delay_loop 6

! use round-robin as a load balancing algorithm

lb_algo rr

! we are doing NAT

lb_kind NAT

nat_mask 255.255.255.0

protocol TCP

! there can be as many real_server blocks as you need

real_server 10.20.40.10 22 {

! if we used weighted round-robin or a similar lb algo,

! we include the weight of this server

weight 1

! here is a health checker for this server.

! we could use a custom script here (see the keepalived docs)

! but we will just make sure we can do a vanilla tcp connect()

! on port 22

! if it fails, we will pull this realserver out of the pool

! and send email about the removal

TCP_CHECK {

connect_timeout 3

connect_port 22

}

}

}

! that's all

-- cut here --

When you start keepalived with the -d flag, you should see this in /var/log/message (or equivalent):

Sep 12 14:13:11 example-01 Keepalived: ------< Global definitions >------

Sep 12 14:13:11 example-01 Keepalived: LVS ID = LVS_EXAMPLE_01

Sep 12 14:13:11 example-01 Keepalived: Smtp server = 127.0.0.1

Sep 12 14:13:11 example-01 Keepalived: Smtp server connection timeout = 100

Sep 12 14:13:11 example-01 Keepalived: Email notification from = admins@example.com, fakepager@example.com

Sep 12 14:13:11 example-01 Keepalived: Email notification = admins@example.com

Sep 12 14:13:11 example-01 Keepalived: ------< SSL definitions >------

Sep 12 14:13:11 example-01 Keepalived: Using autogen SSL context

Sep 12 14:13:11 example-01 Keepalived: ------< VRRP Topology >------

Sep 12 14:13:11 example-01 Keepalived: VRRP Instance = VI_1

Sep 12 14:13:11 example-01 Keepalived: Want State = MASTER

Sep 12 14:13:11 example-01 Keepalived: Runing on device = eth0

Sep 12 14:13:11 example-01 Keepalived: Virtual Router ID = 51

Sep 12 14:13:11 example-01 Keepalived: Priority = 150

Sep 12 14:13:11 example-01 Keepalived: Advert interval = 1sec

Sep 12 14:13:11 example-01 Keepalived: Preempt Active

Sep 12 14:13:11 example-01 Keepalived: Authentication type = SIMPLE_PASSWORD

Sep 12 14:13:11 example-01 Keepalived: Password = example

Sep 12 14:13:11 example-01 Keepalived: VIP count = 1

Sep 12 14:13:11 example-01 Keepalived: VIP1 = 192.168.1.11/32

Sep 12 14:13:11 example-01 Keepalived: VRRP Instance = VI_GATEWAY

Sep 12 14:13:11 example-01 Keepalived: Want State = MASTER

Sep 12 14:13:11 example-01 Keepalived: Runing on device = eth1

Sep 12 14:13:11 example-01 Keepalived: Virtual Router ID = 52

Sep 12 14:13:11 example-01 Keepalived: Priority = 150

Sep 12 14:13:11 example-01 Keepalived: Advert interval = 1sec

Sep 12 14:13:11 example-01 Keepalived: Preempt Active

Sep 12 14:13:11 example-01 Keepalived: Authentication type = SIMPLE_PASSWORD

Sep 12 14:13:11 example-01 Keepalived: Password = example

Sep 12 14:13:11 example-01 Keepalived: VIP count = 1

Sep 12 14:13:11 example-01 Keepalived: VIP1 = 10.20.40.1/32

Sep 12 14:13:11 example-01 Keepalived: ------< VRRP Sync groups >------

Sep 12 14:13:11 example-01 Keepalived: VRRP Sync Group = VG1, MASTER

Sep 12 14:13:11 example-01 Keepalived: monitor = VI_1

Sep 12 14:13:11 example-01 Keepalived: monitor = VI_GATEWAY

Sep 12 14:13:11 example-01 Keepalived: ------< LVS Topology >------

Sep 12 14:13:11 example-01 Keepalived: System is compiled with LVS v1.0.4

Sep 12 14:13:11 example-01 Keepalived: VIP = 192.168.1.11, VPORT = 22

Sep 12 14:13:11 example-01 Keepalived: delay_loop = 10, lb_algo = rr

Sep 12 14:13:11 example-01 Keepalived: protocol = TCP

Sep 12 14:13:11 example-01 Keepalived: lb_kind = NAT

Sep 12 14:13:11 example-01 Keepalived: RIP = 10.20.40.11, RPORT = 22, WEIGHT = 1

Sep 12 14:13:11 example-01 Keepalived: ------< Health checkers >------

Sep 12 14:13:11 example-01 Keepalived: 10.20.40.11:22

Sep 12 14:13:11 example-01 Keepalived: Keepalive method = TCP_CHECK

Sep 12 14:13:11 example-01 Keepalived: Connection timeout = 10

Sep 12 14:13:11 example-01 Keepalived: Connection port = 22

Let's see what ipvsadm has to say about this, after keepalived starts up:

example-01:~ # ipvsadm

IP Virtual Server version 1.0.4 (size=65536)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.1.11:ssh rr

-> 10.20.40.10:ssh Masq 1 0 0

example-01:~ #

And finally, we should see the new IP addresses in our IP address list:

example-01:~ # ip addr list

1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 brd 127.255.255.255 scope host lo

2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100

link/ether 00:e0:81:21:bb:1c brd ff:ff:ff:ff:ff:ff

inet 192.168.1.9/24 brd 192.168.1.254 scope global eth0

inet 192.168.1.11/32 scope global eth0

3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100

link/ether 00:e0:81:21:bb:1d brd ff:ff:ff:ff:ff:ff

inet 10.20.40.2/24 brd 10.20.40.255 scope global eth1

inet 10.20.40.1/32 scope global eth1

example-01:~ #

ipvsadm, ip addr list, and starting keepalived with the -d option are good ways to verify your config is working.

7. Failover

With our basic config from above, we can easily move to a failover situation. All you have to do is setup keepalived on another box, copy over the keepalived.conf, change the lvs_id, change any priorities down
50 points, states to BACKUP, and run keepalived. You'll see in the logs on the backup server that the server accepts it's BACKUP state, and if you unplug the network cable(s) from the MASTER server, the BACKUP server takes over the MASTER state.

For the example, use the config file from the simple example above on the MASTER machine. On the BACKUP machine, use this config file:

-- cut here --

! This is a comment

! Configuration File for keepalived

global_defs {

! this is who emails will go to on alerts

notification_email {

admins@example.com

fakepager@example.com

! add a few more email addresses here if you would like

}

notification_email_from admins@example.com

! I use the local machine to relay mail

smtp_server 127.0.0.1

smtp_connect_timeout 30

! each load balancer should have a different ID

! this will be used in SMTP alerts, so you should make

! each router easily identifiable

! this is router 2

lvs_id LVS_EXAMPLE_02

}

! vrrp_sync_groups make sure that several router instances

! stay together on a failure - a good example of this is

! that the external interface on one router fails and the backup server

! takes over, you want the internal interface on the failed server

! to failover as well, otherwise nothing will work.

! you can have as many vrrp_sync_group blocks as you want.

vrrp_sync_group VG1 {

group {

VI_1

VI_GATEWAY

}

}

! each interface needs at least one vrrp_instance

! each vrrp_instance is a group of VIPs that are logically grouped

! together

! you can have as many vrrp_instaces as you want

vrrp_instance VI_1 {

! we are the failover

state BACKUP

interface eth0

lvs_sync_daemon_inteface eth0

! each virtual router id must be unique per instance name!

! instance names are the same on MASTER and BACKUP, so the

! virtual router_id is the same as VI_1 on the MASTER

virtual_router_id 51

! MASTER and BACKUP state are determined by the priority

! even if you specify MASTER as the state, the state will

! be voted on by priority (so if your state is MASTER but your

! priority is lower than the router with BACKUP, you will lose

! the MASTER state)

! I make it a habit to set priorities at least 50 points apart

! note that a lower number is lesser priority -

! lower gets less vote

priority 100

! how often should we vote, in seconds?

advert_int 1

! send an alert when this instance changes state from

! MASTER to BACKUP

smtp_alert

! this authentication is for syncing between failover servers

! keepalived supports PASS, which is simple

! password authentication

! or AH, which is the ipsec authentication header.

! I don't use AH

! yet as many people have reported problems with it

authentication {

auth_type PASS

auth_pass example

}

virtual_ipaddress {

192.168.1.11

! and more if you want them

}

}

! now I setup the instance that the real servers will use as a default

! gateway

! most of the config is the same as above, but on a different interface

vrrp_instance VI_GATEWAY {

state BACKUP

interface eth1

lvs_sync_daemon_inteface eth1

virtual_router_id 52

priority 100

advert_int 1

smtp_alert

authentication {

auth_type PASS

auth_pass example

}

virtual_ipaddress {

10.20.40.1

}

}

! now we setup more information about are virtual server

! we are just setting up one for now, listening on port 22 for ssh

! requests.

! notice we do not setup a virtual_server block for the 10.20.40.1

! address in the VI_GATEWAY instance. That's because we are doing NAT

! on that IP, and nothing else.

virtual_server 192.168.1.11 22 {

delay_loop 6

! use round-robin as a load balancing alogorithm

lb_algo rr

! we are doing NAT

lb_kind NAT

nat_mask 255.255.255.0

protocol TCP

! there can be as many real_server blocks as you need

real_server 10.20.40.10 22 {

! if we used weighted round-robin or a similar lb algo,

! we include the weight of this server

weight 1

! here is a health checker for this server.

! we could use a custom script here (see the keepalived docs)

! but we will just make sure we can do a vanilla tcp connect()

! on port 22

! if it fails, we will pull this realserver out of the pool

! and send email about the removal

TCP_CHECK {

connect_timeout 3

connect_port 22

}

}

}

! that's all

-- cut here --

Notice how little is different between the MASTER and BACKUP config file - just the lvs_id directive, the priorities, and the state directive. That's it, that's all. Make sure these are different but everything
else is the same.

Once you startup keepalived on the MASTER and the BACKUP, you should be able to kill keepalived on the MASTER server and watch the BACKUP take over in the logs on the BACKUP server.

If you did an ip addr list on the backup server, you won't see the VIPs until the backup server takes over the MASTER state.

8. Setting up keepalived: a more complicated network: 2 VIPs (1 http/https, 1 ssh) with 2 real servers in each.

Load balancer IPs:

IP of load balancers's external interface(eth0): 192.168.1.9
external VIP of our http/https realservers: 192.168.1.11
external VIP of our ssh realservers: 192.168.1.12
IP of load balancer's interface(eth1): 10.20.40.2
internal VIP our realserver will use as a default gateway: 10.20.40.1

Realserver 1 (http, https):

IP: 10.20.40.10
be sure to set the default gateway to 10.20.40.1

Realserver 2 (http, https):

IP: 10.20.40.11
be sure to set the default gateway to 10.20.40.1

Realserver 3 (ssh):

IP: 10.20.40.20
be sure to set the default gateway to 10.20.40.1

Realserver 4 (ssh):

IP: 10.20.40.21
be sure to set the default gateway to 10.20.40.1

A few oddities occur with this setup. In particular, you'll want to learn to use the "genhash" command that comes with keepalived to generate MD5 sums for the HTTP_GET and the SSL_GET service checks. Also, you'll
want to setup persistence on the https - persistence will allow your clients to always connect to the same realserver, in case you have something like a shopping cart that's state is maintained on the realserver.

genhash is simple to use. Let's say we have a test.html on our web servers, and use that for service checks.

example-01:~ # genhash -s 192.168.1.11 -p 80 -u /test.html

-----------------------[ HTTP Header Buffer ]-----------------------

0000 48 54 54 50 2f 31 2e 31 - 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.

0010 0a 44 61 74 65 3a 20 54 - 68 75 2c 20 31 32 20 53 .Date: Thu, 12 S

0020 65 70 20 32 30 30 32 20 - 31 39 3a 34 31 3a 35 39 ep 2002 19:41:59

0030 20 47 4d 54 0d 0a 53 65 - 72 76 65 72 3a 20 41 70 GMT..Server: Ap

0040 61 63 68 65 2f 32 2e 30 - 2e 33 39 20 28 55 6e 69 ache/2.0.39 (Uni

0050 78 29 20 6d 6f 64 5f 73 - 73 6c 2f 32 2e 30 2e 33 x) mod_ssl/2.0.3

0060 39 20 4f 70 65 6e 53 53 - 4c 2f 30 2e 39 2e 36 20 9 OpenSSL/0.9.6

0070 50 48 50 2f 34 2e 32 2e - 31 0d 0a 4c 61 73 74 2d PHP/4.2.1..Last-

0080 4d 6f 64 69 66 69 65 64 - 3a 20 54 75 65 2c 20 30 Modified: Tue, 0

0090 33 20 53 65 70 20 32 30 - 30 32 20 31 37 3a 34 31 3 Sep 2002 17:41

00a0 3a 31 31 20 47 4d 54 0d - 0a 45 54 61 67 3a 20 22 :11 GMT..ETag: "

00b0 31 65 35 35 66 2d 34 32 - 2d 64 33 36 63 33 62 63 1e55f-42-d36c3bc

00c0 30 22 0d 0a 41 63 63 65 - 70 74 2d 52 61 6e 67 65 0"..Accept-Range

00d0 73 3a 20 62 79 74 65 73 - 0d 0a 43 6f 6e 74 65 6e s: bytes..Conten

00e0 74 2d 4c 65 6e 67 74 68 - 3a 20 36 36 0d 0a 43 6f t-Length: 66..Co

00f0 6e 6e 65 63 74 69 6f 6e - 3a 20 63 6c 6f 73 65 0d nnection: close.

0100 0a 43 6f 6e 74 65 6e 74 - 2d 54 79 70 65 3a 20 74 .Content-Type: t

0110 65 78 74 2f 68 74 6d 6c - 3b 20 63 68 61 72 73 65 ext/html; charse

0120 74 3d 49 53 4f 2d 38 38 - 35 39 2d 31 0d 0a 0d 0a t=ISO-8859-1....

-----------------------[ HTTP Header Ascii Buffer ]-----------------------

HTTP/1.1 200 OK

Date: Thu, 12 Sep 2002 19:41:59 GMT

Server: Apache/2.0.39 (Unix) mod_ssl/2.0.39 OpenSSL/0.9.6 PHP/4.2.1

Last-Modified: Tue, 03 Sep 2002 17:41:11 GMT

ETag: "1e55f-42-d36c3bc0"

Accept-Ranges: bytes

Content-Length: 66

Connection: close

Content-Type: text/html; charset=ISO-8859-1

-----------------------[ HTML Buffer ]-----------------------

0000 3c 48 54 4d 4c 3e 0a 3c - 42 4f 44 59 3e 0a 54 68 <HTML>.<BODY>.Th

0010 69 73 20 69 73 20 61 20 - 74 65 73 74 20 70 61 67 is is a test pag

0020 65 20 66 6f 72 20 6d 6f - 6e 69 74 6f 72 69 6e 67 e for monitoring

0030 2e 0a 3c 2f 42 4f 44 59 - 3e 0a 3c 2f 48 54 4d 4c ..</BODY>.</HTML

0040 3e 0a - >.

-----------------------[ HTML MD5 resulting ]-----------------------

0000 42 28 34 d1 d2 b9 72 ee - e9 e5 b8 75 e4 bd 8c 33 B(4...r....u...3

-----------------------[ HTML MD5 final resulting ]-----------------------

422834d1d2b972eee9e5b875e4bd8c33

example-01:~#

That very last string is what you need to keep track, as you will use this in your service check setup below.

Now for the config file:

-- cut here --

! This is a comment

! Configuration File for keepalived

global_defs {

! this is who emails will go to on alerts

notification_email {

admins@example.com

fakepager@example.com

! add a few more email addresses here if you would like

}

notification_email_from admins@example.com

! I use the local machine to relay mail

smtp_server 127.0.0.1

smtp_connect_timeout 30

! each load balancer should have a different ID

! this will be used in SMTP alerts, so you should make

! each router easily identifiable

lvs_id LVS_EXAMPLE_01

}

! takes over, you want the internal interface on the failed server

! to failover as well, otherwise nothing will work.

! you can have as many vrrp_sync_group blocks as you want.

vrrp_sync_group VG1 {

group {

VI_1

VI_GATEWAY

}

}

! now we setup more information about are virtual server

! we are just setting up one for now, listening on port 22 for ssh

! requests.

! each interface needs at least one vrrp_instance

! each vrrp_instance is a group of VIPs that are logically grouped

! together

! you can have as many vrrp_instaces as you want

vrrp_instance VI_1 {

state MASTER

interface eth0

lvs_sync_daemon_inteface eth0

! each virtual router id must be unique per instance name!

virtual_router_id 51

priority 150

! how often should we vote, in seconds?

advert_int 1

smtp_alert

authentication {

auth_type PASS

auth_pass example

}

virtual_ipaddress {

192.168.1.11

192.168.1.12

! and more if you want them

}

}

! now I setup the instance that the real servers will use as a default

! gateway

! most of the config is the same as above, but on a different interface

vrrp_instance VI_GATEWAY {

state MASTER

interface eth1

lvs_sync_daemon_inteface eth1

virtual_router_id 52

priority 150

advert_int 1

smtp_alert

authentication {

auth_type PASS

auth_pass example

}

virtual_ipaddress {

10.20.40.1

}

}

! vrrp_sync_groups make sure that several router instances

! stay together on a failure - a good example of this is

! that the external interface on one router fails and the backup server

! notice we do not setup a virtual_server block for the 10.20.40.1

! address in the VI_GATEWAY instance. That's because we are doing NAT

! on that IP, and nothing else.

virtual_server 192.168.1.12 22 {

delay_loop 6

! use round-robin as a load balancing algorithm

lb_algo rr

! we are doing NAT

lb_kind NAT

nat_mask 255.255.255.0

protocol TCP

! there can be as many real_server blocks as you need

real_server 10.20.40.20 22 {

! if we used weighted round-robin or a similar lb algo,

! we include the weight of this server

weight 1

! here is a health checker for this server.

! we could use a custom script here (see the keepalived docs)

! but we will just make sure we can do a vanilla tcp connect()

! on port 22

! if it fails, we will pull this realserver out of the pool

! and send email about the removal

TCP_CHECK {

connect_timeout 3

connect_port 22

}

}

real_server 10.20.40.21 22 {

! if we used weighted round-robin or a similar lb algo,

! we include the weight of this server

weight 1

TCP_CHECK {

connect_timeout 3

connect_port 22

}

}

}

virtual_server 192.168.1.11 80 {

delay_loop 10

lb_algo rr

lb_kind NAT

nat_mask 255.255.255.0

protocol TCP

! use this to specify which host keepalived asks for during an HTTP GET

virtualhost www.example.com

real_server 10.20.40.10 80 {

weight 1

HTTP_GET {

! for the path, don't include the host if you use

! a virtualhost

url {

path /test.html

! the results from genhash go here

digest 422834d1d2b972eee9e5b875e4bd8c33

}

connect_timeout 10

connect_port 80

! keepalived will retry this many times before a failure

! is marked

nb_get_retry 3

! each retry will occur after this delay

delay_before_retry 10

}

}

real_server 10.20.40.11 80 {

weight 1

HTTP_GET {

url {

path /test.html

digest 422834d1d2b972eee9e5b875e4bd8c33

}

connect_timeout 10

nb_get_retry 3

delay_before_retry 10

connect_port 80

}

}

}

virtual_server 192.168.1.11 443 {

delay_loop 10

lb_algo rr

lb_kind NAT

nat_mask 255.255.255.0

protocol TCP

virtualhost www.example.com

real_server 10.20.40.10 443 {

weight 1

SSL_GET {

url {

path /test.html

digest 422834d1d2b972eee9e5b875e4bd8c33

}

connect_timeout 10

connect_port 80

nb_get_retry 3

delay_before_retry 10

}

}

real_server 10.20.40.11 443 {

weight 1

SSL_GET {

url {

path /test.html

digest 422834d1d2b972eee9e5b875e4bd8c33

}

connect_timeout 10

nb_get_retry 3

delay_before_retry 10

connect_port 80

}

}

}

! that's all

-- cut here --

Okay let's see what keepalived -d shows us:

Sep 12 14:13:11 example-01 Keepalived: ------< Global definitions >------

Sep 12 14:13:11 example-01 Keepalived: LVS ID = LVS_EXAMPLE_01

Sep 12 14:13:11 example-01 Keepalived: Smtp server = 127.0.0.1

Sep 12 14:13:11 example-01 Keepalived: Smtp server connection timeout = 100

Sep 12 14:13:11 example-01 Keepalived: Email notification from = admins@example.com, fakepager@example.com

Sep 12 14:13:11 example-01 Keepalived: Email notification = admins@example.com

Sep 12 14:13:11 example-01 Keepalived: ------< SSL definitions >------

Sep 12 14:13:11 example-01 Keepalived: Using autogen SSL context

Sep 12 14:13:11 example-01 Keepalived: ------< VRRP Topology >------

Sep 12 14:13:11 example-01 Keepalived: VRRP Instance = VI_1

Sep 12 14:13:11 example-01 Keepalived: Want State = MASTER

Sep 12 14:13:11 example-01 Keepalived: Runing on device = eth0

Sep 12 14:13:11 example-01 Keepalived: Virtual Router ID = 51

Sep 12 14:13:11 example-01 Keepalived: Priority = 150

Sep 12 14:13:11 example-01 Keepalived: Advert interval = 1sec

Sep 12 14:13:11 example-01 Keepalived: Preempt Active

Sep 12 14:13:11 example-01 Keepalived: Authentication type = SIMPLE_PASSWORD

Sep 12 14:13:11 example-01 Keepalived: Password = example

Sep 12 14:13:11 example-01 Keepalived: VIP count = 2

Sep 12 14:13:11 example-01 Keepalived: VIP1 = 192.168.1.11/32

Sep 12 14:13:11 example-01 Keepalived: VIP1 = 192.168.1.12/32

Sep 12 14:13:11 example-01 Keepalived: VRRP Instance = VI_GATEWAY

Sep 12 14:13:11 example-01 Keepalived: Want State = MASTER

Sep 12 14:13:11 example-01 Keepalived: Runing on device = eth1

Sep 12 14:13:11 example-01 Keepalived: Virtual Router ID = 52

Sep 12 14:13:11 example-01 Keepalived: Priority = 150

Sep 12 14:13:11 example-01 Keepalived: Advert interval = 1sec

Sep 12 14:13:11 example-01 Keepalived: Preempt Active

Sep 12 14:13:11 example-01 Keepalived: Authentication type = SIMPLE_PASSWORD

Sep 12 14:13:11 example-01 Keepalived: Password = example

Sep 12 14:13:11 example-01 Keepalived: VIP count = 1

Sep 12 14:13:11 example-01 Keepalived: VIP1 = 10.20.40.1/32

Sep 12 14:13:11 example-01 Keepalived: ------< VRRP Sync groups >------

Sep 12 14:13:11 example-01 Keepalived: VRRP Sync Group = VG1, MASTER

Sep 12 14:13:11 example-01 Keepalived: monitor = VI_1

Sep 12 14:13:11 example-01 Keepalived: monitor = VI_GATEWAY

Sep 12 14:13:11 example-01 Keepalived: ------< LVS Topology >------

Sep 12 14:13:11 example-01 Keepalived: System is compiled with LVS v1.0.4

Sep 12 14:13:11 example-01 Keepalived: VIP = 192.168.1.11, VPORT = 22

Sep 12 14:13:11 example-01 Keepalived: delay_loop = 10, lb_algo = rr

Sep 12 14:13:11 example-01 Keepalived: protocol = TCP

Sep 12 14:13:11 example-01 Keepalived: lb_kind = NAT

Sep 12 14:13:11 example-01 Keepalived: RIP = 10.20.40.20, RPORT = 22, WEIGHT = 1

Sep 12 14:13:11 example-01 Keepalived: RIP = 10.20.40.21, RPORT = 22, WEIGHT = 1

Sep 12 14:13:11 example-01 Keepalived: VIP = 192.168.1.12, VPORT = 80

Sep 12 14:13:11 example-01 Keepalived: VirtualHost = www.example.com

Sep 12 14:13:11 example-01 Keepalived: delay_loop = 10, lb_algo = rr

Sep 12 14:13:11 example-01 Keepalived: protocol = TCP

Sep 12 14:13:11 example-01 Keepalived: lb_kind = NAT

Sep 12 14:13:11 example-01 Keepalived: RIP = 10.20.40.10, RPORT = 80, WEIGHT = 1

Sep 12 14:13:11 example-01 Keepalived: RIP = 10.20.40.11, RPORT = 80, WEIGHT = 1

Sep 12 14:13:11 example-01 Keepalived: VIP = 192.168.1.12, VPORT = 443

Sep 12 14:13:11 example-01 Keepalived: VirtualHost = www.example.com

Sep 12 14:13:11 example-01 Keepalived: delay_loop = 10, lb_algo = rr

Sep 12 14:13:11 example-01 Keepalived: persistence timeout = 360

Sep 12 14:13:11 example-01 Keepalived: protocol = TCP

Sep 12 14:13:11 example-01 Keepalived: lb_kind = NAT

Sep 12 14:13:11 example-01 Keepalived: RIP = 10.20.40.10, RPORT = 443, WEIGHT = 1

Sep 12 14:13:11 example-01 Keepalived: RIP = 10.20.40.11, RPORT = 443, WEIGHT = 1

Sep 12 14:13:11 example-01 Keepalived: ------< Health checkers >------

Sep 12 14:13:11 example-01 Keepalived: 10.20.40.20:22

Sep 12 14:13:11 example-01 Keepalived: Keepalive method = TCP_CHECK

Sep 12 14:13:11 example-01 Keepalived: Connection timeout = 10

Sep 12 14:13:11 example-01 Keepalived: Connection port = 22

Sep 12 14:13:11 example-01 Keepalived: 10.20.40.21:22

Sep 12 14:13:11 example-01 Keepalived: Keepalive method = TCP_CHECK

Sep 12 14:13:11 example-01 Keepalived: Connection timeout = 10

Sep 12 14:13:11 example-01 Keepalived: Connection port = 22

Sep 12 14:13:11 example-01 Keepalived: 10.20.40.10:80

Sep 12 14:13:11 example-01 Keepalived: Keepalive method = HTTP_GET

Sep 12 14:13:11 example-01 Keepalived: Connection port = 80

Sep 12 14:13:11 example-01 Keepalived: Connection timeout = 10

Sep 12 14:13:11 example-01 Keepalived: Nb get retry = 3

Sep 12 14:13:11 example-01 Keepalived: Delay before retry = 10

Sep 12 14:13:11 example-01 Keepalived: Checked url = /test.html, digest = 422834d1d2b972eee9e5b875e4bd8c33

Sep 12 14:13:11 example-01 Keepalived: 10.20.40.11:80

Sep 12 14:13:11 example-01 Keepalived: Keepalive method = HTTP_GET

Sep 12 14:13:11 example-01 Keepalived: Connection port = 80

Sep 12 14:13:11 example-01 Keepalived: Connection timeout = 10

Sep 12 14:13:11 example-01 Keepalived: Nb get retry = 3

Sep 12 14:13:11 example-01 Keepalived: Delay before retry = 10

Sep 12 14:13:11 example-01 Keepalived: Checked url = /test.html, digest = 422834d1d2b972eee9e5b875e4bd8c33

Sep 12 14:13:11 example-01 Keepalived: 10.20.40.10:443

Sep 12 14:13:11 example-01 Keepalived: Keepalive method = SSL_GET

Sep 12 14:13:11 example-01 Keepalived: Connection port = 443

Sep 12 14:13:11 example-01 Keepalived: Connection timeout = 10

Sep 12 14:13:11 example-01 Keepalived: Nb get retry = 3

Sep 12 14:13:11 example-01 Keepalived: Delay before retry = 10

Sep 12 14:13:11 example-01 Keepalived: Checked url = /test.html, digest = 422834d1d2b972eee9e5b875e4bd8c33

Sep 12 14:13:11 example-01 Keepalived: 10.20.40.11:443

Sep 12 14:13:11 example-01 Keepalived: Keepalive method = SSL_GET

Sep 12 14:13:11 example-01 Keepalived: Connection port = 443

Sep 12 14:13:11 example-01 Keepalived: Connection timeout = 10

Sep 12 14:13:11 example-01 Keepalived: Nb get retry = 3

Sep 12 14:13:11 example-01 Keepalived: Delay before retry = 10

Sep 12 14:13:11 example-01 Keepalived: Checked url = /test.html, digest = 422834d1d2b972eee9e5b875e4bd8c33

And ipvasdm:

example-01:~# ipvsadm

IP Virtual Server version 1.0.4 (size=65536)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.1.12:ssh rr

-> 10.20.40.20:ssh Masq 1 0 0

-> 10.20.40.21:ssh Masq 1 0 1

TCP 192.168.1.11:http rr

-> 10.20.40.10:http Masq 1 0 0

-> 10.20.40.11:http Masq 1 0 0

TCP 192.168.1.11:http rr persistent 360

-> 10.20.40.10:https Masq 1 0 0

-> 10.20.40.11:https Masq 1 1 0

example-01:~ # ip addr list

1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 brd 127.255.255.255 scope host lo

2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100

link/ether 00:e0:81:21:bb:1c brd ff:ff:ff:ff:ff:ff

inet 192.168.1.9/24 brd 192.168.1.254 scope global eth0

inet 192.168.1.11/32 scope global eth0

inet 192.168.1.12/32 scope global eth0

3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100

link/ether 00:e0:81:21:bb:1d brd ff:ff:ff:ff:ff:ff

inet 10.20.40.2/24 brd 10.20.40.255 scope global eth1

inet 10.20.40.1/32 scope global eth1

Remeber, if we wanted to put this in a failover config, we could just add another box with the same config file (modified with a different lvs_id, state and priority) and start up keepalived on the backup box.

9. Example startup script for SuSE 8.0

#! /bin/sh

# Copyright (c) 1995-2002 SuSE Linux AG, Nuernberg, Germany.

# All rights reserved.

#

# Author of template: Kurt Garloff <feedback@suse.de>

# Modified for keepalived by Adam Fletcher <adamf+keepalived@csh.rit.edu>

#

# /etc/init.d/keepalived

#

#

# LSB compliant service control script; see http://www.linuxbase.org/spec/

#

# System startup script for some example service or daemon keepalived (template)

#

### BEGIN INIT INFO

# Provides: keepalived

# Required-Start: $remote_fs $syslog

# Required-Stop: $remote_fs $syslog

# Default-Start: 3 5

# Default-Stop: 0 1 2 6

# Description: Start keepalived to allow XY and provide YZ

# continued on second line by '#<TAB>'

### END INIT INFO

#

# Note on Required-Start: It does specify the init script ordering,

# not real dependencies. Depencies have to be handled by admin

# resp. the configuration tools (s)he uses.

# Source SuSE config (if still necessary, most info has been moved)

test -r /etc/rc.config && . /etc/rc.config

# Check for missing binaries (stale symlinks should not happen)

KEEPALIVED_BIN=/usr/local/sbin/keepalived

test -x $KEEPALIVED_BIN || exit 5

# Shell functions sourced from /etc/rc.status:

# rc_check check and set local and overall rc status

# rc_status check and set local and overall rc status

# rc_status -v ditto but be verbose in local rc status

# rc_status -v -r ditto and clear the local rc status

# rc_failed set local and overall rc status to failed

# rc_failed <num> set local and overall rc status to <num><num>

# rc_reset clear local rc status (overall remains)

# rc_exit exit appropriate to overall rc status

# rc_active checks whether a service is activated by symlinks

. /etc/rc.status

# First reset status of this service

rc_reset

# Return values acc. to LSB for all commands but status:

# 0 - success

# 1 - generic or unspecified error

# 2 - invalid or excess argument(s)

# 3 - unimplemented feature (e.g. "reload")

# 4 - insufficient privilege

# 5 - program is not installed

# 6 - program is not configured

# 7 - program is not running

#

# Note that starting an already running service, stopping

# or restarting a not-running service as well as the restart

# with force-reload (in case signalling is not supported) are

# considered a success.

case "$1" in

start)

echo -n "Starting keepalived"

## Start daemon with startproc(8). If this fails

## the echo return value is set appropriate.

# NOTE: startproc returns 0, even if service is

# already running to match LSB spec.

startproc $KEEPALIVED_BIN -d

# Remember status and be verbose

rc_status -v

;;

stop)

echo -n "Shutting down keepalived"

## Stop daemon with killproc(8) and if this fails

## set echo the echo return value.

killproc -TERM $KEEPALIVED_BIN

# masquerade the rest of the 10.20.40.0/24 network

# through the external IP

# Remember status and be verbose

rc_status -v

;;

try-restart)

## Stop the service and if this succeeds (i.e. the

## service was running before), start it again.

## Note: try-restart is not (yet) part of LSB (as of 0.7.5)

$0 status >/dev/null && $0 restart

# Remember status and be quiet

rc_status

;;

restart)

## Stop the service and regardless of whether it was

## running or not, start it again.

$0 stop

$0 start

# Remember status and be quiet

rc_status

;;

force-reload)

## Signal the daemon to reload its config. Most daemons

## do this on signal 1 (SIGHUP).

## If it does not support it, restart.

echo -n "Reload service keepalived"

## if it supports it:

killproc -HUP $KEEPALIVED_BIN

touch /var/run/keepalived.pid

rc_status -v

## Otherwise:

#$0 stop && $0 start

#rc_status

;;

reload)

## Like force-reload, but if daemon does not support

## signalling, do nothing (!)

# If it supports signalling:

echo -n "Reload service keepalived"

killproc -HUP $KEEPALIVED_BIN

touch /var/run/keepalived.pid

rc_status -v

## Otherwise if it does not support reload:

#rc_failed 3

#rc_status -v

;;

status)

echo -n "Checking for service keepalived: "

## Check status with checkproc(8), if process is running

## checkproc will return with exit status 0.

# Return value is slightly different for the status command:

# 0 - service running

# 1 - service dead, but /var/run/ pid file exists

# 2 - service dead, but /var/lock/ lock file exists

# 3 - service not running

# NOTE: checkproc returns LSB compliant status values.

checkproc $KEEPALIVED_BIN

rc_status -v

;;

probe)

## Optional: Probe for the necessity of a reload,

## print out the argument which is required for a reload.

test /etc/keepalived/keepalived.conf -nt /var/run/keepalived.pid && echo reload

;;

*)

echo "Usage: $0 {start|stop|status|try-restart|restart|force-reload|reload|probe}"

exit 1

;;

esac

rc_exit

10. Troubleshooting

Run keepalived with the -d option and watch your /var/log/messages
Look at the output of ipvsadm
Look at the output of ip addr list

Important caveats:

Turn on IP forwarding (echo "1" > /proc/sys/net/ipv4/ip_forward)
Get the latest LVS - don't go to a mirror of linuxvirtualserver.org unless you are sure it has the latest version (i got burned by this for a day!)
The realserver must use the load balancer as it's default gateway.
The realserver must not be on both the internal and external network, they must be on separate segements/LANs/VLANs.
You *must* have a vrrp_instance section defined for keepalived to setup the VIPs.
In VRRP, a lower number for the priority means that VRRP instance has less say - in other words, a lower priority is a lower number.
VRRP instances can not have the same router_id - keepalived won't give any errors, but you won't see that VRRP instance's IP addresses.
While keepalived won't complain if you leave out some options in the config file, it's not recommended. Just because you put a TCP_CHECK on a realserver that listens on port 80, don't leave out the connect_port 80 directive!

These may seem obvious, but they are all things that happened to me during my setup of keepalived.

If you have further questions, and have read this and all other keepalive documentation, please subscribe to the the keepalived-devel mailing list, available athttp://www.keepalived.org/mailinglist.html

11. Credits

Keepalived is written and maintained by Alexandre Cassen. It is primarily through his effort that it exists at all, and I thank him for that efffort.

This HOWTO is written by Adam Fletcher, adamf+keepalived@csh.rit.edu.

This document is Copyright 2002 by Adam Fletcher Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published
by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is available athttp://www.gnu.org/copyleft/fdl.html