lvs+keepalived实现高可用群集配置详解

wskickahn

　　lvs是一个开源的软件，由毕业于国防科技大学的章文嵩博士于1998年5月创立(中国人的项目)，可以实现LINUX平台下的简单负载均衡。LVS是Linux Virtual Server的缩写，意思是Linux虚拟服务器。本文将介绍lvs结合keepalived实现一个高科用的Linux群集系统.
　　lvs有三种工作模式NAT(地址转换),IP Tunneling(IP隧道)、Direct Routing(直接路由)。 工作效率最低的是NAT模式,但NAT模式可以用于各种系统,各种环境的负载均衡,只需要一个公网ip即可实现 IP Tunneling模式调度器将连接分发到不同的后端real server,然后由real server处理请求直接相应给用户,大大提高了调度器的调度效率,后端real server没有物理位置和逻辑关系的限制,后端real server可以在Lan/Wlan,但是后端real server必须支持IP隧道协议. DR(Direct Routing)是效率最高的,与IP Tunneling类似,都是处理一般连接,将请求给后端real server,然后由real server处理请求直接相应给用户,Direct Routing与IP Tunneling相比，没有IP封装的开销，但由于采用物理层,所以DR模式的调度器和后端real server必须在一个物理网段里,中间不能过路由器(也就是一个交换机相连).
　　lvs支持8种不同的调度算法轮叫(rr)、加权轮叫(wrr)、最小连接(lc)、加权最小连接(wlc)、基于局部性最小连接(lblc)、带复制的基于局部性最少链接(lblcr)、目标地址散列(dh)和源地址散列(sh).
　　下面就介绍如何来安装和配置lvs+keepalived
　　本文使用环境: 操作系统:CentOS 5.5 32bit
　　主调度器:192.168.3.101/24
　　备调度器:192.168.3.102/24
　　后端real server: 192.168.3.3/24 |192.168.3.102/24(我们这里使用备用lvs作为一个测试
　　vip(virtual ip):192.168.3.100/24
　　lvs在2.6的内核中是默认支持的,所以我们就不需要在来安装,但是我们需要安装用户配置工具ipvsadm
yum-yinstallipvsadm#分别在主从lvs上执行安装ipvsadm　　我们查看lvs是否支持:
lsmod|grepip_vs#ip_vs780811modprobe-l|grepip_vs/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs.ko/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_dh.ko/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_ftp.ko/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_lblc.ko/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_lblcr.ko/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_lc.ko/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_nq.ko/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_rr.ko/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_sed.ko/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_sh.ko/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_wlc.ko/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_wrr.ko　　本文介绍lvs的DR模式,首先部署keepalived.本博前面已经介绍如何来安装keepalived.这里就不在只简单的贴一下步骤:
　　在主备服务器上部署keepalived(因为前面已经rpm包安装了ipvsadm,所以就不需要重复安装):
vi/etc/sysctl.confnet.ipv4.ip_forward=1#此参数改为1sysctl-p#使修改生效　　安装依赖:
yum  -y install openssl-devel　　
# 下载并安装keepalived
　　
wget http://www.keepalived.org/software/keepalived-1.1.19.tar.gz
　　
tar -zxvf keepalived-1.1.19.tar.gz
　　
cd keepalived-1.1.19
　　
./configure --prefix=/ \ # 安装在默认位置(配置文件,二进制文件,启动脚本放到默认位置)
　　
--mandir=/usr/local/share/man/ \
　　
--with-kernel-dir=/usr/src/kernels/2.6.18-194.el5-i686/    # 需要内核的头文件
　　
make && make install
　　

　　在主备lvs上安装keepalived完毕后我们先来配置主lvs上的keepalived: 编辑配置文件/etc/keepalived/keepalived.conf:
!ConfigurationFileforkeepalivedglobal_defs{notification_email{coldnight@linuxzen.com#发生故障时发送的邮箱}notification_email_fromlinuxzen@linuxzen.com#使用哪个邮箱发送smtp_serverlinuxzen.com#发件服务器smtp_connect_timeout30router_idLVS_DEVEL}vrrp_instanceVI_1{stateMASTER#标示为主lvsinterfaceeth0#HA检测端口virtual_router_id51#主备的virtual_router_id必须相同priority100#优先级,备lvs要比主lvs稍小advert_int1#VRRPMulticast广播周期秒数authentication{#定义认证auth_typePASS#认证方式为口令认证auth_pass1111#定义口令}virtual_ipaddress{#定义vip192.168.3.100#多个vip可换行添加}}virtual_server192.168.3.10080{delay_loop6#每隔6秒查看realserver状态lb_algowlc#调度算法为加权最小连接数lb_kindDR#lvs工作模式为DR(直接路由)模式nat_mask255.255.255.0persistence_timeout50#同一IP的连接50秒内被分配到同一台realserver(测试时建议改为0)protocolTCP#用TCP监测realserver的状态real_server192.168.3.380{#定义realserverweight3#定义权重TCP_CHECK{#注意TCP_CHECK和{之间的空格,如果没有的话只会添加第一个realserverconnect_timeout3#三秒无响应超时nb_get_retry3delay_before_retry3connect_port80}}real_server192.168.3.10280{weight3TCP_CHECK{connect_timeout3nb_get_retry3delay_before_retry3connect_port80}}}　　配置备用lvs的keepalived,只需要将state MASTER 改为state BACKUP,降低priority 100 的值:
!ConfigurationFileforkeepalivedglobal_defs{notification_email{coldnight@linuxzen.com#发生故障时发送的邮箱}notification_email_fromlinuxzen@linuxzen.com#使用哪个邮箱发送smtp_serverlinuxzen.com#发件服务器smtp_connect_timeout30router_idLVS_DEVEL}vrrp_instanceVI_1{stateBACKUP#标示为备lvsinterfaceeth0#HA检测端口virtual_router_id51#主备的virtual_router_id必须相同priority99#优先级,备lvs要比主lvs稍小advert_int1#VRRPMulticast广播周期秒数authentication{#定义认证auth_typePASS#认证方式为口令认证auth_pass1111#定义口令}virtual_ipaddress{#定义vip192.168.3.100#多个vip可换行添加}}virtual_server192.168.3.10080{delay_loop6#每隔6秒查看realserver状态lb_algowlc#调度算法为加权最小连接数lb_kindDR#lvs工作模式为DR(直接路由)模式nat_mask255.255.255.0persistence_timeout50#同一IP的连接50秒内被分配到同一台realserverprotocolTCP#用TCP监测realserver的状态real_server192.168.3.380{#定义realserverweight3#定义权重TCP_CHECK{#注意TCP_CHECK和{之间的空格,如果没有的话只会添加第一个realserverconnect_timeout3#三秒无响应超时nb_get_retry3delay_before_retry3connect_port80}}real_server192.168.3.10280{weight3TCP_CHECK{connect_timeout3nb_get_retry3delay_before_retry3connect_port80}}}　　由于使用keepalived就不需要使用脚本来配置lvs调度器,但是这里我们还是会给出一个脚本内容,但我们不会用到这个脚本:lvs已经内置于内核,配置命令是ipvsadm,所以lvs的一些操作是通过ipvsadm来控制.下面我们就编写脚本来实现lvs的DR模式:
　　编写脚本lvsdr:
　　我们把lvsvi /etc/init.d/lvsdr添加如下内容
#!/bin/sh# 定义虚拟ipVIP=192.168.3.100　　
# 定义realserver,并已逗号分开RIPS=192.168.3.3,192.168.3.102 #,192.168.3.5,192.168.3.6# 定义提供服务的端口SERVICE=80
　　

　　
# 调用init.d脚本的标准库
　　
. /etc/rc.d/init.d/functions
　　
case$1 in
　　start)echo"Start LVS of DR Mode"# lvs dr模式不需要路由转发,但是keepalived需要#echo "0" > /proc/sys/net/ipv4/ip_forward# 开启icmp包重定向echo"1" > /proc/sys/net/ipv4/conf/all/send_redirects
　　echo"1" > /proc/sys/net/ipv4/conf/default/send_redirects
　　echo"1" > /proc/sys/net/ipv4/conf/eth0/send_redirects
　　# 绑定虚拟ip
　　ifconfig eth0:0 $VIP broadcast $VIP netmask 255.255.255.255 up
　　route add -host $VIP dev eth0:0
　　# 清除lvs规则
　　ipvsadm -C
　　# 添加一条虚拟服务器记录# -p指定一定的时间内将相同的客户端分配到同一台后端服务器# 用于解决session的问题,测试时或有别的解决方案时建议去掉
　　ipvsadm -A -t $VIP:$SERVICE -s wlc -p
　　

　　# 添加真实服务器记录for RIP in `echo$RIPS |sed  -e 's/,/\n/g'`doipvsadm -a -t $VIP:$SERVICE -r $RIP:$SERVICE -g -w 1
　　done# 设置tcp tcpfin  udp的超时连接值
　　ipvsadm --set 30 120 300
　　ipvsadm
　　;;
　　

　　stop)echo"Stop LVS DR"
　　ifconfig eth0:0 down
　　ipvsadm -C
　　;;
　　*)echo"Usage:$0 {start | stop}"exit 1
　　
esac
　　编辑完毕保存退出,然后给这个脚本执行权限:
chmod+x/etc/init.d/lvsdr　　然后就可以通过service命令来启动lvs dr模式
servicelvsdrstart　　将这个脚本分别放到主备lvs的/etc/init.d/下,赋予执行权限. 我们真正需要的是realserver的脚本,下面我们来编写realserver脚本,同样放在/etc/init.d/下,编辑rs脚本:
vi /etc/init.d/lvsrs　　
#!/bin/shVIP=192.168.3.100
　　
. /etc/rc.d/init.d/functions
　　
case$1 in
　　start)echo"lo:0 port starting"# 为了相应lvs调度器转发过来的包,需在本地lo接口上绑定vip
　　ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
　　# 限制arp请求echo"1" > /proc/sys/net/ipv4/conf/lo/arp_ignore
　　echo"2" > /proc/sys/net/ipv4/conf/lo/arp_announce
　　echo"1" > /proc/sys/net/ipv4/conf/all/arp_ignore
　　echo"2" > /proc/sys/net/ipv4/conf/all/arp_announce
　　;;
　　stop)echo"lo:0 port closing"
　　ifconfig lo:0 down
　　echo"0" > /proc/sys/net/ipv4/conf/lo/arp_ignore
　　echo"0" > /proc/sys/net/ipv4/conf/lo/arp_announce
　　echo"0" > /proc/sys/net/ipv4/conf/all/arp_ignore
　　echo"0" > /proc/sys/net/ipv4/conf/all/arp_announce
　　;;
　　*)echo"Usage: $0 {start | stop}"exit 1
　　
esac
　　给脚本赋予执行权限
chmod+x/etc/init.d/lvsrs　　并将这个脚本放到所有的realserver的/etc/init.d/下.下面开始测试:
　　先来确认下我们做的变动:主从lvs分别安装keepalived,并且在/etc/init.d/下添加了lvsdr脚本(不使用).
　　后端realserver分别在/etc/init.d/下添加了lvsrs脚本.我们先测试keepalived:
　　首先在主调度器上启动keepalived:
servicekeepalivedstart　　查看日志文件:
tail-50/var/log/messageMar2122:29:10masterkernel:deviceeth0leftpromiscuousmodeMar2122:29:10masterkernel:type=1700audit(1332340150.598:12):dev=eth0prom=0old_prom=256auid=4294967295ses=4294967295Apr1613:31:32masterKeepalived:StartingKeepalivedv1.1.19(04/16,2012)Apr1613:31:32masterKeepalived_healthcheckers:NetlinkreflectorreportsIP192.168.3.101addedApr1613:31:32masterKeepalived_healthcheckers:RegisteringKernelnetlinkreflectorApr1613:31:32masterKeepalived_healthcheckers:RegisteringKernelnetlinkcommandchannelApr1613:31:32masterKeepalived_healthcheckers:Openingfile'/etc/keepalived/keepalived.conf'.Apr1613:31:32masterKeepalived_healthcheckers:Configurationisusing:8897BytesApr1613:31:32masterKeepalived_healthcheckers:UsingLinkWatchkernelnetlinkreflector...Apr1613:31:32masterKeepalived:StartingHealthcheckchildprocess,pid=5369Apr1613:31:32masterKeepalived:StartingVRRPchildprocess,pid=5370Apr1613:31:32masterKeepalived_vrrp:NetlinkreflectorreportsIP192.168.3.101addedApr1613:31:32masterKeepalived_vrrp:RegisteringKernelnetlinkreflectorApr1613:31:32masterKeepalived_vrrp:RegisteringKernelnetlinkcommandchannelApr1613:31:32masterKeepalived_vrrp:RegisteringgratutiousARPsharedchannelApr1613:31:32masterKeepalived_vrrp:Openingfile'/etc/keepalived/keepalived.conf'.Apr1613:31:32masterKeepalived_vrrp:Configurationisusing:36512BytesApr1613:31:32masterKeepalived_vrrp:UsingLinkWatchkernelnetlinkreflector...Apr1613:31:32masterKeepalived_vrrp:VRRPsockpool:[ifindex(2),proto(112),fd(10,11)]Apr1613:31:33masterKeepalived_vrrp:VRRP_Instance(VI_1)TransitiontoMASTERSTATEApr1613:31:34masterKeepalived_vrrp:VRRP_Instance(VI_1)EnteringMASTERSTATEApr1613:31:34masterKeepalived_vrrp:VRRP_Instance(VI_1)settingprotocolVIPs.Apr1613:31:34masterKeepalived_healthcheckers:NetlinkreflectorreportsIP192.168.3.100addedApr1613:31:34masterKeepalived_vrrp:VRRP_Instance(VI_1)SendinggratuitousARPsoneth0for192.168.3.100Apr1613:31:34masterKeepalived_vrrp:NetlinkreflectorreportsIP192.168.3.100addedApr1613:31:39masterKeepalived_vrrp:VRRP_Instance(VI_1)SendinggratuitousARPsoneth0for192.168.3.100　　然后在备用调度器上启动keepalived然后查看日志:
Apr1613:33:35slaveKeepalived_vrrp:VRRP_Instance(VI_1)EnteringBACKUPSTATEApr1613:33:35slaveKeepalived_vrrp:VRRPsockpool:[ifindex(2),proto(112),fd(11,12)]Apr1613:33:35slaveKeepalived_healthcheckers:NetlinkreflectorreportsIP192.168.3.102addedApr1613:33:35slaveKeepalived_healthcheckers:RegisteringKernelnetlinkreflectorApr1613:33:35slaveKeepalived_healthcheckers:RegisteringKernelnetlinkcommandchannelApr1613:33:35slaveKeepalived_healthcheckers:Openingfile'/etc/keepalived/keepalived.conf'.Apr1613:33:35slaveKeepalived_healthcheckers:Configurationisusing:8895BytesApr1613:33:35slavekernel:IPVS:[wlc]schedulerregistered.Apr1613:33:35slaveKeepalived_healthcheckers:UsingLinkWatchkernelnetlinkreflector...　　在主调度器上执行
servicekeepalivedstop　　查看备用调度器日志:
tail-20/var/log/messageApr1613:39:44slaveKeepalived_vrrp:VRRP_Instance(VI_1)TransitiontoMASTERSTATEApr1613:39:45slaveKeepalived_vrrp:VRRP_Instance(VI_1)EnteringMASTERSTATEApr1613:39:45slaveKeepalived_vrrp:VRRP_Instance(VI_1)settingprotocolVIPs.Apr1613:39:45slaveKeepalived_vrrp:VRRP_Instance(VI_1)SendinggratuitousARPsoneth0for192.168.3.100Apr1613:39:45slaveKeepalived_vrrp:NetlinkreflectorreportsIP192.168.3.100addedApr1613:39:45slaveKeepalived_healthcheckers:NetlinkreflectorreportsIP192.168.3.100added　　我们看到keepalived已经成功切换.
　　然后我们使用ipvsadm命令查看(在此之前要确认后端realserver已经启动了web服务):
ipvsadmIPVirtualServerversion1.2.1(size=4096)ProtLocalAddress:PortSchedulerFlags->RemoteAddress:PortForwardWeightActiveConnInActConnTCP192.168.3.100:httpwlc->192.168.3.3:httpRoute300->192.168.3.102:httpRoute300　　然后分别启动后端realserver的lvsrs服务:
servielvsrsstart　　然后浏览器访问192.168.3.100,如果keepalived的persistence_timeout参数值为0,而且两个后端realserver是不同的内容,刷新就可以看到两个不同的页面交替.