| LVS / IPVS / Keepalived Load Balancing HOWTO TTK Ciar, ttk (at) archive.org HOWTO document, 2004-12-02 This document very briefly discusses installing and configuring Keepalived and Linux Virtual Server (LVS) on Linux for the purpose of providing load balancing of a service across multiple hosts. Table of Contents 1. Introduction 2. Installation 3. Configuring Using Keepalive 4. Configuring Using Just LVS Alone 5. Making It Go 6. Monitoring 7. Caveats 8. Things To Do 1. INTRODUCTION Linux Virtual Server is a function of the Linux kernel which allows for the expedient forwarding of connections to multiple servers on a network. Keepalived is a userspace daemon which wraps the LVS interface (ipvsadm) and provides for rapid and transparent failover of multiple virtual hosts (hosts forwarding connections to real hosts) and multiple real hosts (hosts accepting connections and providing services). The purpose of using LVS is to divide processing load across multiple servers for best performance. Keepalived makes this robust by co-ordinating the removal of servers from the LVS pool as they go down, and adding them back into the pool when they come back up. Return To Table of Contents 2. INSTALLATION This section is intentionally light, since the necessary packages come with adequate installation documentation. The meat is in the configuration section, following this one. IPVS is an optional feature of the Linux kernel. Your kernel must be compiled with the IPVS options enabled. Joerg has compiled correct kernels for each of our archivectures. They are located in ops03:/0/nnssi/kernel/2.6.9/, and are bzImage-via, bzImage-athlon, and bzImage-p4. Put the appropriate parameters into admin:/0/cluster/cluster.txt for each virtual servers. Changing kernels is not necessary for real servers. (Note: Real servers must be able to restrict their ARP replies to addresses not bound to physical interfaces, which requires a kernel patch, but all kernels currently used at The Archive already have this patch, so this does not require additional configuration work.) Additionally, ipvsadm must be installed. On debian systems this is as simple as running 'apt-get install ipvsadm'. Otherwise the v1.1.7 package must be obtained from www.linuxvirtualserver.org and installed. Keepalived can be obtained from http://www.keepalived.org/. Follow the instructions in its internal documentation to install. It is strongly recommended that the correct kernel be installed on the virtual servers, and then ipvsadm installed, and then keepalived installed. Changing versions of any package after any other package has been installed may give rise to interoperational problems. Return To Table of Contents 3. CONFIGURATION WITH KEEPALIVED Since configuration of load balancing is in some ways simpler with keepalived, it will be described first. The following section will describe a standalone LVS configuration. Before configuration begins, you will need the following resources: * Two or more virtual servers, which will forward connections to real servers (for instance: lvs01, lvs02). Choose one virtual server to be the "master". The rest will be "backup" servers. * Two or more real servers, which will provide the services (for instance: www1, www4, www6). (In the example below, the IP addresses for www1, www4, and www6 are 207.241.228.105, 207.241.228.168, and 207.241.229.86 respectively.) * One or more "floating" IP addresses which have not been assigned to any host. (In the example below, 207.241.224.241) Configuration of virtual servers: For each virtual server, the following lines must be entered into its /specials/rc.final file (located at admin:/0/cluster/Specials/): sysctl -w net.ipv4.ip_forward=1 /usr/sbin/keepalived -f /specials/keepalived.conf Appropriate keepalived.conf files must then be made for each virtual server. They should all be identical, except that the master server should specify master status. Brief descriptions of some parameters are described. For more complete descriptions, please view the Keepalived Users Guide at http://www.keepalived.org/pdf/UserGuide.pdf).
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 207.241.230.143 # <-- Replace this with real mail-dev.archive.org IP address
smtp_connect_timeout 30
lvs_id LVS_FOO # <-- Replace this with a unique name (like LVS_WB or LVS_CGI)
}
# The vrrp_sync_group defines the group of hosts amongst which the floating IP address
# should be shared. This one is very simple, but VRRP can be used to define very
# complicated failover organizations, involving multiple services, with different
# virtual host groups for each service.
vrrp_sync_group VGX { # <-- Replace VGX with a unique name
group {
VI_Z # <-- replace VI_Z with a unique name
}
}
# The vrrp_instance describes the VRRP parameters for this virtual server.
vrrp_instance VI_Z {
state MASTER # <-- make this BACKUP for backup virtual servers
interface eth0
virtual_router_id 42 # <-- replace 42 with a unique number
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1414 # <-- replace 1414 with a unique number
}
virtual_ipaddress {
207.241.224.241/21 # vip -- "floating" ip of LVS virtual host pool
}
}
# Here we describe the real servers being multiplexed off a given floating ip.
virtual_server 207.241.224.241 80 { # 80 = http service, for www load balancing
delay_loop 10 # number of seconds between real server connectvity checks
lb_algo rr # use round-robin connection allocation algorithm
lb_kind DR # use direct routing; can also be NAT, but don't do that.
persistence_timeout 9600 # Joerg says to use three hour timeouts, and I trust him.
protocol TCP # allocate connections to TCP port 80, as opposed to UDP port 80.
real_server 207.241.228.105 80 { # www1
weight 1 # higher number == give this machine more connections.
TCP_CHECK { # use simple TCP connectivity test for detecting up/down status.
connect_timeout 3 # server is down if connection takes > 3 seconds
connect_port 80 # test connection to TCP port 80
}
}
real_server 207.241.228.168 80 { # www4
weight 1
TCP_CHECK {
connect_timeout 3
connect_port 80
}
}
real_server 207.241.229.86 80 { # www6
weight 1
TCP_CHECK {
connect_timeout 3
connect_port 80
}
}
}
Configuration of real servers: For each real server, the following lines must be entered into its /specials/rc.final file (located at admin:/0/cluster/Specials/): ip route add 224.0.0.0/4 dev eth0 sysctl -w net.ipv4.conf.lo.hidden=1 sysctl -w net.ipv4.conf.all.hidden=1 ip addr add 207.241.224.241/32 dev lo Return To Table of Contents 4. CONFIGURATION OF STANDALONE LVS If you do not care about failover, or keepalived is giving you grief, LVS can be used by itself. Before configuration begins, you will need the following resources: * One virtual IP address (vip) which has not been allocated to any machine. (In the example below, 207.241.224.240). * One virtual server, which will forward connections to real servers. * Two or more real servers, which will provide the services (for instance: www1, www4, www6). (In the example below, the IP addresses for www1, www4, and www6 are 207.241.228.105, 207.241.228.168, and 207.241.229.86 respectively.) Make sure an LVS-enabled kernel and the ipvsadm utility has been installed on the virtual server (see INSTALLATION section). The following lines must be entered into the virtual server's /specials/rc.final file (located at admin:/0/cluster/Specials/): sysctl -w net.ipv4.ip_forward=1 ip addr add 207.241.224.240/21 dev eth0 ipvsadm -D -t 207.241.224.240:80 Futhermore, one line must be entered into the virtual server's /specials/rc.final file for every real server: ipvsadm -a -t 207.241.224.240:80 -r 207.241.228.105 -g ipvsadm -a -t 207.241.224.240:80 -r 207.241.228.168 -g ipvsadm -a -t 207.241.224.240:80 -r 207.241.229.86 -g Configuration of real servers: For each real server, the following lines must be entered into its /specials/rc.final file (located at admin:/0/cluster/Specials/): sysctl -w net.ipv4.conf.lo.hidden=1 sysctl -w net.ipv4.conf.all.hidden=1 ip addr add 207.241.224.240/32 dev lo Return To Table of Contents 5. MAKING IT GO To make it all go, either reboot the real and virtual servers (in that order), or type the commands into the shell, which you added to each machine's rc.final. There is also a script called lvstool, probably located a homeserver:/home/bill/bin/lvstool, which makes remote standalone LVS initialization easy. It should be run as root from admin. Maybe someday there will be an equivalent tool for using keepalived. Bug John Berry about allocating some software engineer's time to it. Return To Table of Contents 6. MONITORING One line should be added to cluster.txt for every VIP (floating IP), which will cause Nagios to monitor the VIP like ordinary servers: www 00:00:00:00:00:00 LB WWW-VIP www 207.241.224.241 -This will cause Nagios to perform PING and SSH checks, which will be done on whichever Virtual Server is currently holding the VIP. In this case (www), an apache check will also be performed on port 80, which will go through the VIP normally. The easiest way to monitor the state of the loadbalancer is with ipvsadm. Run it on a virtual server with the '-l' option to see the current state of LVS. List current stats: lvs01:~# ipvsadm -l IP Virtual Server version 1.2.0 (size=16384) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP iahost-241-224-241.archive.o rr persistent 9600 -> www6.archive.org:www Route 1 38 1668 -> www4.archive.org:www Route 1 33 1400 -> www1.archive.org:www Route 1 37 1874Other handy variants: lvs01:~# ipvsadm -l -n lvs01:~# ipvsadm -l -n --rate lvs01:~# ipvsadm -l -n --stats lvs01:~# ipvsadm -l -cParticularly recommended: lvs01:~# watch ipvsadm -l -n --rateSee "ipvsadm --help" for full options. If you are using keepalived, run "watch ipvsadm -l -n --rate" on each virtual server so you can see the connection forwarding "jump over" to a backup server when the master goes down. Return To Table of Contents 7. CAVEATS When Direct Routing (DR) mode is used, LVS only works correctly when the virtual servers and real servers are all on the same local network. It is possible to do more complex configurations using TUN and NAT, but this is more complex and outside the scope of this HOWTO. See the Keepalived Users Guide for more information. Whenever the IP of a real server changes, this change must be reflected in each keepalived.conf file, and then keepalived restarted. This can be done by using "ps aux | grep keepalived" to obtain the pid of keepalived (it will be the first process of the three that show up), "kill <pid>", and then "keepalived -f /specials/keepalived.conf". However, simply rebooting the server on any configuration change might be preferable. Users of keepalived have reported problems with the kill + re-run process, and there should be no interruption of services when a single server is rebooted anyway. The same goes for when the IP of your mail server changes. Return To Table of Contents 8. THINGS TO DO * Enhance lvstool with monitoring function. * Write analog to lvstool for keepalived. * Update this HOWTO with directord information. * Discuss issues with flapping servers. * Make up a Nagios check for monitoring. Return To Table of Contents |