Docker实现 LVS + Keepalived + bind 二级负载均衡主从模式集群

1. 概述

本文利用docker快速搭建一个由17个小节点构成的二层负载均衡集群。第一层负载均衡DNS采用Bind
实现,第二层负载均衡由LVS实现,同时主从切换用Keepalived来实现。

本文主要是验证二层负载均衡+主从切换的方案,由于虚拟机占用资源较大,故采用docker来验证。

关于LVS,Keepalived和bind的相关原理和各自的搭建步骤,可以参考其他的文档。


2. Docker环境安装配置

2.1 宿主机环境安装

yum install -y ipvsadm
ipvsadm               

2.2 Docker环境安装配置

2.2.1 安装docker
yum install -y docker
2.1.2 配置Docker镜像的源

打开 /etc/docker

vim /etc/docker/daemon.json

将daemon.json改成如下

{
"registry-mirrors":["https://almtd3fa.mirror.aliyuncs.com"]
}
2.2.3 启动docker
systemctl start docker.service
systemctl enable docker.service # 设置开机启动
2.2.4 登录docker

注册了docker账号后可以直接登录

docker login

3. 搭建集群

3.1 集群节点信息

本集群总共由17个节点构成,具体的地址映射关系如下所示

节点的名称DIPVIP
DNS-Server172.18.0.2172.18.0.100
------------
DS-1-Master172.18.1.2172.18.1.100
DS-1-Slave172.18.1.3172.18.1.100
RS-1-Master172.18.1.4172.18.1.101
RS-1-Slave172.18.1.5172.18.1.101
RS-2-Master172.18.1.6172.18.1.102
RS-2-Slave172.18.1.7172.18.1.102
RS-3-Master172.18.1.8172.18.1.103
RS-3-Slave172.18.1.9172.18.1.103
------------
DS-2-Master172.18.2.2172.18.2.100
DS-2-Slave172.18.2.3172.18.2.100
RS-4-Master172.18.2.4172.18.2.101
RS-4-Slave172.18.2.5172.18.2.101
RS-5-Master172.18.2.6172.18.2.102
RS-5-Slave172.18.2.7172.18.2.102
RS-6-Master172.18.2.8172.18.2.103
RS-6-Slave172.18.2.9172.18.2.103

DS-1和DS-2分别作为两个小集群的网关,每个网关掌握着3个RS节点。

外面的客户端通过DNS-Server获取DS-1和DS-2的地址,并由DS-1和DS-2进行消息的路由和转发。

除了DNS-Server节点外,所有的节点均拥有镜像节点,作为从节点Slave随时顶替主节点master。

整体架构图如下所示
在这里插入图片描述

3.2 制作单个节点的镜像

docker创建自定义网络,这是由于docker设置固定的ip需要用户自己设置subnet,在此例中
172.18.0.0/16是集群的子网。lvs-keepalived-bind-Cluster是集群所用子网的名称。

docker network create --subnet=172.18.0.0/16 lvs-keepalived-bind-Cluster

节点都采用centos7作为基础的镜像,在此基础上不断加入需要的组件。

# 启动DNS-Server
docker run -itd \
  --name DNS-Server \
  --privileged=true \
  --network lvs-keepalived-bind-Cluster \
  --ip 172.18.0.2 \
  --hostname DNS-Server \
  centos:centos7 /usr/sbin/init

进入DNS-Server

docker exec -it DNS-Server /bin/bash

在DNS-Server上进行环境安装

# 必需组件
yum install -y net-tools    # 安装ifconfig
yum install -y keepalived   # 安装keepalived
yum install -y httpd        # 安装apache服务
systemctl restart httpd     # 启动apache服务
systemctl enable httpd      # 开机启动apache服务
echo "I am DNS-Server -VIP: 172.18.0.2 -DIP: 172.18.0.100 " > /var/www/html/index.html

# 个性化设置
sed -i 's/PS1=.*$/PS1="\[\\\[\\e\[32;1m\\\]\\u\\\[\\e\[37;0m\\\]@\\h \\\[\\e\[36;1m\\\]\\w\\\[\\e\[0m\\\]\]\\\\$ \"/g' /etc/bashrc

提交本地docker镜像

# -a :提交的镜像作者;
# -c :使用Dockerfile指令来创建镜像;
# -m :提交时的说明文字;
# -p :在commit时,将容器暂停。
docker commit -a "Kotori" -m "cluster node for lvs + keepalived + bind project" DNS-Server xzsww123/cluster-node:v1.0

提交远端docker镜像

# docker push 注册用户名/镜像名
docker push xzsww123/cluster-node

3.3 启动其他节点

基于DNS-SERVER的镜像,建立并启动其他节点的容器

3.3.1 节点DS-1-Master的设置
############# 节点DS-1-Master的设置 #############
docker run -itd --name DS-1-Master --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.1.2 --hostname DS-1-Master xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it DS-1-Master /bin/bash

### 设置LVS-DS的相关
echo "I am DS-1-Master -VIP: 172.18.1.100 -DIP: 172.18.1.2" > /var/www/html/index.html
systemctl restart httpd
yum -y install ipvsadm
# 创建虚拟网卡和路由规则的开机脚本
echo "
ifconfig eth0:0 172.18.1.100 broadcast 172.18.1.100 netmask 255.255.255.255 up
route add -host 172.18.1.100 dev eth0
ipvsadm -C
ipvsadm -A -t 172.18.1.100:80 -s rr
ipvsadm -a -t 172.18.1.100:80 -r 172.18.1.101 -g
ipvsadm -a -t 172.18.1.100:80 -r 172.18.1.102 -g
ipvsadm -a -t 172.18.1.100:80 -r 172.18.1.103 -g
ipvsadm-save
" > /root/lvs-script-ds.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-ds.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-100
}

vrrp_instance VI_1 {  
    state MASTER      
    interface eth0
    mcast_src_ip 172.18.1.2
    virtual_router_id 100
    priority 200     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.1.100
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit
############# end of DS-1-Master #############
3.3.2 节点DS-1-Slave的设置
############# 节点DS-1-Slave的设置 #############
docker run -itd --name DS-1-Slave --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.1.3 --hostname DS-1-Slave xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it DS-1-Slave /bin/bash

### 设置LVS-DS的相关
echo "I am DS-1-Slave -VIP: 172.18.1.100 -DIP: 172.18.1.3" > /var/www/html/index.html
systemctl restart httpd
yum -y install ipvsadm
# 创建虚拟网卡和路由规则的开机脚本
echo "
ifconfig eth0:0 172.18.1.100 broadcast 172.18.1.100 netmask 255.255.255.255 up
route add -host 172.18.1.100 dev eth0:0
ipvsadm -C
ipvsadm -A -t 172.18.1.100:80 -s rr
ipvsadm -a -t 172.18.1.100:80 -r 172.18.1.101 -g
ipvsadm -a -t 172.18.1.100:80 -r 172.18.1.102 -g
ipvsadm -a -t 172.18.1.100:80 -r 172.18.1.103 -g
ipvsadm-save
" > /root/lvs-script-ds.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-ds.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-100
}

vrrp_instance VI_1 {  
    state BACKUP
    interface eth0    
    mcast_src_ip 172.18.1.3
    virtual_router_id 100
    priority 100     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.1.100
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit
############# end of DS-1-Slave #############
3.3.3 节点RS-1-Master的设置
############# 节点RS-1-Master的设置 #############
docker run -itd --name RS-1-Master --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.1.4 --hostname RS-1-Master xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it RS-1-Master /bin/bash

### 设置LVS-RS的相关
echo "I am RS-1-Master -VIP: 172.18.1.101 -DIP: 172.18.1.4" > /var/www/html/index.html
systemctl restart httpd
# 开机时创建虚拟网卡
echo "
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 172.18.1.100 broadcast 172.18.1.100 netmask 255.255.255.255 up
" > /root/lvs-script-rs.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-rs.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-101
}

vrrp_instance VI_1 {  
    state MASTER
    interface eth0    
    mcast_src_ip 172.18.1.4
    virtual_router_id 101
    priority 200     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.1.101
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit
############# end of RS-1-Master #############
3.3.4 节点RS-1-Slave的设置
############# 节点RS-1-Slave的设置 #############
docker run -itd --name RS-1-Slave --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.1.5 --hostname RS-1-Slave xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it RS-1-Slave /bin/bash

### 设置LVS-RS的相关
echo "I am RS-1-Slave -VIP: 172.18.1.101 -DIP: 172.18.1.5" > /var/www/html/index.html
systemctl restart httpd
# 开机时创建虚拟网卡
echo "
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 172.18.1.100 broadcast 172.18.1.100 netmask 255.255.255.255 up
" > /root/lvs-script-rs.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-rs.sh" > /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-101
}

vrrp_instance VI_1 {  
    state BACKUP
    interface eth0    
    mcast_src_ip 172.18.1.5
    virtual_router_id 101
    priority 100     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.1.101
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit
############# end of RS-1-Slave #############
3.3.5 节点RS-2-Master的设置
############# 节点RS-2-Master的设置 #############
docker run -itd --name RS-2-Master --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.1.6 --hostname RS-2-Master xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it RS-2-Master /bin/bash
### 设置LVS-RS的相关
echo "I am RS-2-Master -VIP: 172.18.1.102 -DIP: 172.18.1.6" > /var/www/html/index.html
systemctl restart httpd
# 开机时创建虚拟网卡
echo "
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 172.18.1.100 broadcast 172.18.1.100 netmask 255.255.255.255 up
" > /root/lvs-script-rs.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-rs.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-102
}

vrrp_instance VI_1 {  
    state MASTER
    interface eth0    
    mcast_src_ip 172.18.1.6
    virtual_router_id 102
    priority 200     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.1.102
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit    
############# end of RS-2-Master #############
3.3.6 节点RS-2-Slave的设置
############# 节点RS-2-Slave的设置 #############
docker run -itd --name RS-2-Slave --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.1.7 --hostname RS-2-Slave xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it RS-2-Slave /bin/bash
### 设置LVS-RS的相关
echo "I am RS-2-Slave -VIP: 172.18.1.102 -DIP: 172.18.1.7" > /var/www/html/index.html
systemctl restart httpd
# 开机时创建虚拟网卡
echo "
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 172.18.1.100 broadcast 172.18.1.100 netmask 255.255.255.255 up
" > /root/lvs-script-rs.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-rs.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-102
}

vrrp_instance VI_1 {  
    state BACKUP
    interface eth0    
    mcast_src_ip 172.18.1.7
    virtual_router_id 102
    priority 100     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.1.102
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit    
############# end of RS-2-Slave #############
3.3.7 节点RS-3-Master的设置
############# 节点RS-3-Master的设置 #############
docker run -itd --name RS-3-Master --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.1.8 --hostname RS-3-Master xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it RS-3-Master /bin/bash
### 设置LVS-RS的相关
echo "I am RS-3-Master -VIP: 172.18.1.103 -DIP: 172.18.1.8" > /var/www/html/index.html
systemctl restart httpd
# 开机时创建虚拟网卡
echo "
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 172.18.1.100 broadcast 172.18.1.100 netmask 255.255.255.255 up
" > /root/lvs-script-rs.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-rs.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-103
}

vrrp_instance VI_1 {  
    state MASTER
    interface eth0    
    mcast_src_ip 172.18.1.8
    virtual_router_id 103
    priority 200     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.1.103
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit
############# end of RS-3-Master #############
3.3.8 节点RS-3-Slave的设置
############# 节点RS-3-Slave的设置 #############
docker run -itd --name RS-3-Slave --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.1.9 --hostname RS-3-Slave xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it RS-3-Slave /bin/bash
### 设置LVS-RS的相关
echo "I am RS-3-Slave -VIP: 172.18.1.103 -DIP: 172.18.1.9" > /var/www/html/index.html
systemctl restart httpd
# 开机时创建虚拟网卡
echo "
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 172.18.1.100 broadcast 172.18.1.100 netmask 255.255.255.255 up
" > /root/lvs-script-rs.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-rs.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-103
}

vrrp_instance VI_1 {  
    state BACKUP
    interface eth0    
    mcast_src_ip 172.18.1.9
    virtual_router_id 103
    priority 100     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.1.103
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit
############# end of RS-3-Slave #############
3.3.9 节点DS-2-Master的设置
############# 节点DS-2-Master的设置 #############
docker run -itd --name DS-2-Master --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.2.2 --hostname DS-2-Master xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it DS-2-Master /bin/bash

### 设置LVS-DS的相关
echo "I am DS-2-Master -VIP: 172.18.2.100 -DIP: 172.18.2.2" > /var/www/html/index.html
systemctl restart httpd
yum -y install ipvsadm
# 创建虚拟网卡和路由规则的开机脚本
echo "
ifconfig eth0:0 172.18.2.100 broadcast 172.18.2.100 netmask 255.255.255.255 up
route add -host 172.18.2.100 dev eth0
ipvsadm -C
ipvsadm -A -t 172.18.2.100:80 -s rr
ipvsadm -a -t 172.18.2.100:80 -r 172.18.2.101 -g
ipvsadm -a -t 172.18.2.100:80 -r 172.18.2.102 -g
ipvsadm -a -t 172.18.2.100:80 -r 172.18.2.103 -g
ipvsadm-save
" > /root/lvs-script-ds.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-ds.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-100
}

vrrp_instance VI_1 {  
    state MASTER      
    interface eth0
    mcast_src_ip 172.18.2.2
    virtual_router_id 200
    priority 200     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.2.100
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit
############# end of DS-2-Master #############
3.3.10 节点DS-2-Slave的设置
############# 节点DS-2-Slave的设置 #############
docker run -itd --name DS-2-Slave --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.2.3 --hostname DS-2-Slave xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it DS-2-Slave /bin/bash

### 设置LVS-DS的相关
echo "I am DS-2-Slave -VIP: 172.18.2.100 -DIP: 172.18.2.3" > /var/www/html/index.html
systemctl restart httpd
yum -y install ipvsadm
# 创建虚拟网卡和路由规则的开机脚本
echo "
ifconfig eth0:0 172.18.2.100 broadcast 172.18.2.100 netmask 255.255.255.255 up
route add -host 172.18.2.100 dev eth0:0
ipvsadm -C
ipvsadm -A -t 172.18.2.100:80 -s rr
ipvsadm -a -t 172.18.2.100:80 -r 172.18.2.101 -g
ipvsadm -a -t 172.18.2.100:80 -r 172.18.2.102 -g
ipvsadm -a -t 172.18.2.100:80 -r 172.18.2.103 -g
ipvsadm-save
" > /root/lvs-script-ds.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-ds.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-100
}

vrrp_instance VI_1 {  
    state BACKUP
    interface eth0    
    mcast_src_ip 172.18.2.3
    virtual_router_id 200
    priority 100     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.2.100
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit
############# end of DS-2-Slave #############
3.3.11 节点RS-4-Master的设置
############# 节点RS-4-Master的设置 #############
docker run -itd --name RS-4-Master --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.2.4 --hostname RS-4-Master xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it RS-4-Master /bin/bash

### 设置LVS-RS的相关
echo "I am RS-4-Master -VIP: 172.18.2.101 -DIP: 172.18.2.4" > /var/www/html/index.html
systemctl restart httpd
# 开机时创建虚拟网卡
echo "
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 172.18.2.100 broadcast 172.18.2.100 netmask 255.255.255.255 up
" > /root/lvs-script-rs.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-rs.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-101
}

vrrp_instance VI_1 {  
    state MASTER
    interface eth0    
    mcast_src_ip 172.18.2.4
    virtual_router_id 201
    priority 200     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.2.101
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit
############# end of RS-4-Master #############
3.3.12 节点RS-4-Slave的设置
############# 节点RS-1-Slave的设置 #############
docker run -itd --name RS-4-Slave --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.2.5 --hostname RS-4-Slave xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it RS-4-Slave /bin/bash

### 设置LVS-RS的相关
echo "I am RS-4-Slave -VIP: 172.18.2.101 -DIP: 172.18.2.5" > /var/www/html/index.html
systemctl restart httpd
# 开机时创建虚拟网卡
echo "
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 172.18.2.100 broadcast 172.18.1.100 netmask 255.255.255.255 up
" > /root/lvs-script-rs.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-rs.sh" > /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-101
}

vrrp_instance VI_1 {  
    state BACKUP
    interface eth0    
    mcast_src_ip 172.18.2.5
    virtual_router_id 201
    priority 100     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.2.101
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit
############# end of RS-5-Slave #############
3.3.13 节点RS-5-Master的设置
############# 节点RS-2-Master的设置 #############
docker run -itd --name RS-5-Master --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.2.6 --hostname RS-5-Master xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it RS-5-Master /bin/bash
### 设置LVS-RS的相关
echo "I am RS-5-Master -VIP: 172.18.2.102 -DIP: 172.18.2.6" > /var/www/html/index.html
systemctl restart httpd
# 开机时创建虚拟网卡
echo "
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 172.18.2.100 broadcast 172.18.2.100 netmask 255.255.255.255 up
" > /root/lvs-script-rs.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-rs.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-102
}

vrrp_instance VI_1 {  
    state MASTER
    interface eth0    
    mcast_src_ip 172.18.2.6
    virtual_router_id 202
    priority 200     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.2.102
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit    
############# end of RS-5-Master #############
3.3.14 节点RS-5-Slave的设置
############# 节点RS-2-Slave的设置 #############
docker run -itd --name RS-5-Slave --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.2.7 --hostname RS-5-Slave xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it RS-5-Slave /bin/bash
### 设置LVS-RS的相关
echo "I am RS-5-Slave -VIP: 172.18.2.102 -DIP: 172.18.2.7" > /var/www/html/index.html
systemctl restart httpd
# 开机时创建虚拟网卡
echo "
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 172.18.2.100 broadcast 172.18.2.100 netmask 255.255.255.255 up
" > /root/lvs-script-rs.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-rs.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-102
}

vrrp_instance VI_1 {  
    state BACKUP
    interface eth0    
    mcast_src_ip 172.18.2.7
    virtual_router_id 202
    priority 100     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.2.102
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit    
############# end of RS-5-Slave #############
3.3.15 节点RS-6-Master的设置
############# 节点RS-6-Master的设置 #############
docker run -itd --name RS-6-Master --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.2.8 --hostname RS-6-Master xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it RS-6-Master /bin/bash
### 设置LVS-RS的相关
echo "I am RS-6-Master -VIP: 172.18.2.103 -DIP: 172.18.2.8" > /var/www/html/index.html
systemctl restart httpd
# 开机时创建虚拟网卡
echo "
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 172.18.2.100 broadcast 172.18.2.100 netmask 255.255.255.255 up
" > /root/lvs-script-rs.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-rs.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-103
}

vrrp_instance VI_1 {  
    state MASTER
    interface eth0    
    mcast_src_ip 172.18.2.8
    virtual_router_id 203
    priority 200     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.2.103
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit
############# end of RS-6-Master #############
3.3.16 节点RS-6-Slave的设置
############# 节点RS-6-Slave的设置 #############
docker run -itd --name RS-6-Slave --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.2.9 --hostname RS-6-Slave xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it RS-6-Slave /bin/bash
### 设置LVS-RS的相关
echo "I am RS-6-Slave -VIP: 172.18.2.103 -DIP: 172.18.2.9" > /var/www/html/index.html
systemctl restart httpd
# 开机时创建虚拟网卡
echo "
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
ifconfig lo:0 172.18.2.100 broadcast 172.18.2.100 netmask 255.255.255.255 up
" > /root/lvs-script-rs.sh
# 将开机脚本加入到开机启动
echo "/bin/sh /root/lvs-script-rs.sh" >> /etc/rc.d/rc.local
# 改变开机脚本权限
chmod +x /etc/rc.d/rc.local
source /etc/rc.d/rc.local

### 设置keepalived的相关
yum install -y keepalived
echo "
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost      
   smtp_server 127.0.0.1    
   smtp_connect_timeout 30  
   router_id vip-103
}

vrrp_instance VI_1 {  
    state BACKUP
    interface eth0    
    mcast_src_ip 172.18.2.9
    virtual_router_id 203
    priority 100     
    advert_int 1     
    authentication {
        auth_type PASS
        auth_pass MrUse
    }
    virtual_ipaddress {
        172.18.2.103
    }
}
" > /etc/keepalived/keepalived.conf
systemctl daemon-reload             #重新加载配置
systemctl start keepalived.service  #启动keepalived服务
systemctl enable keepalived.service #开机启动keepalived服务
systemctl status keepalived.service #查看当前状态
exit
############# end of RS-6-Slave #############

4. 创建DNS服务器容器

直接用已经生成的镜像创建DNS服务器的容器,如果已存在这个容器就先删除

docker run -itd --name DNS-Server --privileged=true --network lvs-keepalived-bind-Cluster --ip 172.18.0.2 --hostname DNS-Server xzsww123/cluster-node:v1.0 /usr/sbin/init
docker exec -it DNS-Server /bin/bash

# 安装和启动bind
yum install -y bind bind-utils bind-devel bind-libs bind-chroot
systemctl start named
systemctl enable named
systemctl status named

# 修改域主服务器设置 /etc/named.conf
vi /etc/named.conf

# 改成如下
options {
	listen-on port 53 { 172.18.0.2; };
	listen-on-v6 port 53 { ::1; };
  rrset-order { order random; };
	directory "/var/named";
	dump-file "/var/named/data/cache_dump.db";
	statistics-file "/var/named/data/named_stats.txt";
	memstatistics-file "/var/named/data/named_mem_stats.txt";
	recursing-file  "/var/named/data/named.recursing";
	secroots-file   "/var/named/data/named.secroots";
	allow-query { any; };                 
	recursion yes;  
	dnssec-enable yes;
	dnssec-validation yes;
	bindkeys-file "/etc/named.root.key"; #大大
	managed-keys-directory "/var/named/dynamic";
	pid-file "/run/named/named.pid";
	session-keyfile "/run/named/session.key";
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};

zone "." IN {
	type hint;
	file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";


# 修改区域配置文件设置 /etc/named.rfc1912.zones
vi /etc/named.rfc1912.zones

# 添加如下代码
zone "Odyssey.com" IN {
type master;           
file "Odyssey.com.zone";
};

# 创建自己的zone文件
vi  /var/named/Odyssey.com.zone
# 改成如下
$TTL 600
@   IN  SOA Odyssey.com.   root.qq.com(
                            1811201955   
                            1H           
                            1M          
                            1W           
                            10M)         

@   IN  NS  ns1
ns1 IN A 172.18.0.2

@   IN  A   172.18.1.100
    IN  A   172.18.2.100

www IN  A   172.18.1.100
www IN  A   172.18.2.100

*   IN  A   255.255.255.255

# 重新加载服务
rndc reload

5. 整体方案测试

以上集群搭建完毕,接下来进行简单的测试

先在宿主机上安装nscd,用来刷新DNS缓存

yum install -y nscd

先在宿主机上改变dns (最好断掉外网,因为外网上也有Odyssey.com这个域名)

echo "
DNS1=172.18.0.2
" >> /etc/sysconfig/network-scripts/ifcfg-ens33

service network restart

然后执行

host -t a www.Odyssey.com 172.18.0.2

得到如下结果

Using domain server:
Name: 172.18.0.2
Address: 172.18.0.2#53
Aliases:

www.Odyssey.com has address 172.18.1.100
www.Odyssey.com has address 172.18.2.100

然后直接访问www.Odyssey.com

curl www.Odyssey.com

可以看到访问请求轮流在RS间循环,但是DNS轮询机制会受到多方面的影响,如:A记录的TTL时间长
短的影响;别的 DNS 服务器 Cache 的影响;windows 客户端也有一个DNS Cache。这些都会影响
DNS轮询的效果。因此 DNS 的轮询机制并不能做为一个 load balancing的解决方案,只能作为一
个 load distribution 方案。因此执行上面的指令未必会切换DS服务器,这是根据每台服务器上的
DNS过期时间等都有关系。


附录

docker stop $(docker ps -a | grep xzsww123 | awk '{print $1 }')   #停止所有已经退出的容器
docker rm $(docker ps -a | grep xzsww123 | awk '{print $1 }')     #删除所有已经退出的容器

参考资料

[1] LVS负载均衡之DR模式部署

[2] keepalived 配置文件解析

[3] Linux运维实战之DNS(bind)服务器的安装与配置

[4] linux实现DNS轮询实现负载平衡

Logo

权威|前沿|技术|干货|国内首个API全生命周期开发者社区

更多推荐