企业级Kubernetes高可用集群实战：从零构建多Master架构

1. 高可用集群架构设计核心思路

在生产环境中，单Master节点的Kubernetes集群存在明显的单点故障风险。当Master节点宕机时，整个集群的控制平面将完全瘫痪。通过部署多Master节点配合负载均衡组件，可以实现控制平面的高可用性，确保API Server、Controller Manager和Scheduler等关键组件持续提供服务。

典型的高可用架构包含以下核心组件：

• 多Master节点：至少3个节点组成etcd集群，采用Raft协议保证数据一致性
• 负载均衡层：Keepalived + HAProxy组合实现VIP漂移和流量分发
• 网络插件：Calico/Flannel等CNI实现Pod间通信
• 证书管理：kubeadm自动生成的CA证书或自定义证书

# 典型生产环境节点规划示例
+-------------------+       +-------------------+
|  Load Balancer 1  |-------|  Load Balancer 2  |
| (Keepalived+HAProxy)      | (Keepalived+HAProxy) |
+-------------------+       +-------------------+
          |                          |
          +------------+-------------+
                       |
          +------------+-------------+
          |            |             |
+-------------------+ +-------------------+ +-------------------+
|    Master Node 1   | |    Master Node 2   | |    Master Node 3   |
| (etcd,API Server)  | | (etcd,API Server)  | | (etcd,API Server)  |
+-------------------+ +-------------------+ +-------------------+
          |                   |                   |
          +---------------------------------------+
                       |
          +---------------------------------------+
          |                   |                   |
+-------------------+ +-------------------+ +-------------------+
|    Worker Node 1   | |    Worker Node 2   | |    Worker Node 3   |
+-------------------+ +-------------------+ +-------------------+

2. 基础环境准备与系统优化

在开始部署前，所有节点需要完成统一的系统配置。以下操作需要在所有Master和Worker节点上执行：

内核参数调优：

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
kernel.panic = 10
kernel.panic_on_oops = 1
EOF
sysctl --system

禁用Swap并优化cgroup驱动：

swapoff -a
sed -i '/swap/s/^\(.*\)$/#\1/g' /etc/fstab

# 配置docker使用systemd cgroup
mkdir -p /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "registry-mirrors": ["https://registry.cn-hangzhou.aliyuncs.com"]
}
EOF

时间同步与主机名解析：

# 配置NTP时间同步
timedatectl set-timezone Asia/Shanghai
yum install -y chrony
systemctl enable --now chronyd

# 设置主机名解析（所有节点）
cat >> /etc/hosts <<EOF
192.168.1.10 k8s-master1
192.168.1.11 k8s-master2
192.168.1.12 k8s-master3
192.168.1.100 k8s-vip
EOF

3. 负载均衡层部署与配置

Keepalived和HAProxy的组合为Kubernetes API Server提供了高可用的访问入口。建议至少部署两个负载均衡节点以避免单点故障。

Keepalived配置（主节点示例）：

yum install -y keepalived

cat > /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}

vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 50
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 42f8d4e7
    }
    virtual_ipaddress {
        192.168.1.100/24
    }
    track_script {
        chk_haproxy
    }
}
EOF

systemctl enable --now keepalived

HAProxy配置（所有LB节点）：

yum install -y haproxy

cat > /etc/haproxy/haproxy.cfg <<'EOF'
global
    log /dev/log local0
    maxconn 20000
    stats socket /var/lib/haproxy/stats

defaults
    mode tcp
    timeout connect 5s
    timeout client 50s
    timeout server 50s

frontend k8s-api
    bind *:6443
    default_backend k8s-masters

backend k8s-masters
    balance roundrobin
    option tcp-check
    server k8s-master1 192.168.1.10:6443 check inter 3s fall 3 rise 2
    server k8s-master2 192.168.1.11:6443 check inter 3s fall 3 rise 2
    server k8s-master3 192.168.1.12:6443 check inter 3s fall 3 rise 2

listen stats
    bind *:8404
    mode http
    stats enable
    stats uri /admin?stats
    stats auth admin:SecurePass123
EOF

systemctl enable --now haproxy

关键提示：生产环境中建议将HAProxy的监控端口(8404)通过防火墙限制访问，仅允许管理网络访问

4. Kubernetes控制平面初始化

首个Master节点初始化：

kubeadm init \
  --control-plane-endpoint "k8s-vip:6443" \
  --upload-certs \
  --pod-network-cidr=10.244.0.0/16 \
  --service-cidr=10.96.0.0/12 \
  --image-repository registry.aliyuncs.com/google_containers

初始化成功后，按照提示配置kubectl：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

添加其他控制平面节点： 使用初始化时输出的join命令，例如：

kubeadm join k8s-vip:6443 \
  --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:xxxxxxxx \
  --control-plane \
  --certificate-key yyyyyyyy

网络插件安装（Calico示例）：

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

验证集群状态：

kubectl get nodes -o wide
kubectl get pods -n kube-system

5. 生产环境关键配置与优化

证书自动续期配置：

# 修改kube-controller-manager配置
kubectl edit cm kube-controller-manager -n kube-system
# 添加以下参数：
--experimental-cluster-signing-duration=87600h0m0s
--feature-gates=RotateKubeletServerCertificate=true

API Server高可用参数调优：

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
  extraArgs:
    audit-log-maxbackup: "10"
    audit-log-maxsize: "100"
    event-ttl: "720h"
    profiling: "false"
  extraVolumes:
  - name: audit-log
    hostPath: /var/log/kubernetes/audit
    mountPath: /var/log/kubernetes/audit
    readOnly: false
    pathType: DirectoryOrCreate

节点维护与隔离操作：

# 安全排空节点
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# 标记节点不可调度
kubectl cordon <node-name>

# 恢复节点
kubectl uncordon <node-name>

6. 常见故障排查指南

证书问题排查：

# 检查证书有效期
openssl x509 -noout -dates -in /etc/kubernetes/pki/apiserver.crt

# 手动更新证书
kubeadm certs renew all

网络连通性测试：

# 创建测试Pod
kubectl run net-test --image=nicolaka/netshoot -- sleep 3600

# 进入Pod测试
kubectl exec -it net-test -- bash
ping k8s-vip
curl -k https://k8s-vip:6443
nslookup kubernetes.default.svc.cluster.local

组件日志查看：

# 查看kubelet日志
journalctl -u kubelet -f

# 查看容器运行时日志
journalctl -u containerd -f

# 查看特定Pod日志
kubectl logs -n kube-system <pod-name> --tail=100

7. 集群扩展与升级策略

添加Worker节点：

# 使用初始化时生成的token
kubeadm token list

# 若无有效token则创建新token
kubeadm token create --print-join-command

# Worker节点加入命令示例
kubeadm join k8s-vip:6443 \
  --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:xxxxxxxx

版本升级方案：

# 1. 升级kubeadm
yum install -y kubeadm-1.28.0 --disableexcludes=kubernetes

# 2. 升级控制平面
kubeadm upgrade plan
kubeadm upgrade apply v1.28.0

# 3. 升级kubelet和kubectl
yum install -y kubelet-1.28.0 kubectl-1.28.0 --disableexcludes=kubernetes
systemctl restart kubelet

集群备份与恢复：

# 备份etcd
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /tmp/etcd-snapshot.db

# 恢复etcd
ETCDCTL_API=3 etcdctl snapshot restore /tmp/etcd-snapshot.db \
  --data-dir /var/lib/etcd-backup