Kubernetes集群的高可用性设计与实践：从理论到落地

🔥 硬核开场

各位技术大佬们，今天咱们来聊聊Kubernetes集群的高可用性设计。别跟我说你的K8s集群就一个master节点，那都不叫生产环境！在生产环境中，高可用是底线，是生命线。今天susu就带你们从理论到实践，一步步构建高可用的Kubernetes集群，从多master部署到etcd集群配置，从网络规划到故障转移，全给你整明白！

📋 核心内容

1. Kubernetes高可用的核心概念

• 什么是高可用：系统在面对各种故障时仍能保持正常运行的能力
• Kubernetes的高可用目标：99.95%的可用性，每年 downtime 不超过4小时
• 高可用的关键组件：多master节点、etcd集群、负载均衡、网络冗余

2. 高可用集群架构设计

2.1 多Master架构

+---------------------+
|    Load Balancer    |
+---------------------+
          |
+---------------------+
|  Master Node 1      |
| kube-apiserver      |
| kube-controller-manager |
| kube-scheduler      |
+---------------------+
          |
+---------------------+
|  Master Node 2      |
| kube-apiserver      |
| kube-controller-manager |
| kube-scheduler      |
+---------------------+
          |
+---------------------+
|  Master Node 3      |
| kube-apiserver      |
| kube-controller-manager |
| kube-scheduler      |
+---------------------+
          |
+---------------------+
|   etcd Cluster      |
| (3/5 nodes)         |
+---------------------+
          |
+---------------------+
|  Worker Nodes       |
| (multiple)          |
+---------------------+

2.2 etcd集群配置

etcd是Kubernetes的核心存储，必须高可用。

# 安装etcd
ETCD_VER=v3.5.4
DOWNLOAD_URL=https://github.com/etcd-io/etcd/releases/download
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o etcd.tar.gz
tar xzvf etcd.tar.gz
cd etcd-${ETCD_VER}-linux-amd64
cp etcd etcdctl /usr/local/bin/

# 配置etcd集群
# 在node1上执行
etcd --name etcd1 --initial-advertise-peer-urls http://192.168.1.101:2380 \
  --listen-peer-urls http://192.168.1.101:2380 \
  --listen-client-urls http://192.168.1.101:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://192.168.1.101:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster etcd1=http://192.168.1.101:2380,etcd2=http://192.168.1.102:2380,etcd3=http://192.168.1.103:2380 \
  --initial-cluster-state new

# 在node2上执行
etcd --name etcd2 --initial-advertise-peer-urls http://192.168.1.102:2380 \
  --listen-peer-urls http://192.168.1.102:2380 \
  --listen-client-urls http://192.168.1.102:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://192.168.1.102:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster etcd1=http://192.168.1.101:2380,etcd2=http://192.168.1.102:2380,etcd3=http://192.168.1.103:2380 \
  --initial-cluster-state new

# 在node3上执行
etcd --name etcd3 --initial-advertise-peer-urls http://192.168.1.103:2380 \
  --listen-peer-urls http://192.168.1.103:2380 \
  --listen-client-urls http://192.168.1.103:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://192.168.1.103:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster etcd1=http://192.168.1.101:2380,etcd2=http://192.168.1.102:2380,etcd3=http://192.168.1.103:2380 \
  --initial-cluster-state new

# 验证etcd集群状态
etcdctl endpoint health --endpoints=http://192.168.1.101:2379,http://192.168.1.102:2379,http://192.168.1.103:2379

3. 部署多Master Kubernetes集群

3.1 安装kubeadm

# 安装依赖
apt-get update && apt-get install -y apt-transport-https curl

# 添加Kubernetes GPG key
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

# 添加Kubernetes apt源
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

# 安装kubeadm, kubelet, kubectl
apt-get update && apt-get install -y kubelet kubeadm kubectl

# 锁定版本
apt-mark hold kubelet kubeadm kubectl

3.2 初始化第一个Master节点

# 初始化Master节点
kubeadm init --control-plane-endpoint "192.168.1.200:6443" --upload-certs \
  --pod-network-cidr=10.244.0.0/16 \
  --service-cidr=10.96.0.0/12 \
  --etcd-servers=https://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.103:2379 \
  --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt \
  --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key

# 配置kubectl
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

# 安装网络插件
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

3.3 添加其他Master节点

# 获取加入命令
kubeadm token create --print-join-command

# 在其他Master节点上执行（包含--control-plane参数）
kubeadm join 192.168.1.200:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash> --control-plane --certificate-key <certificate-key>

3.4 配置负载均衡器

# 安装HAProxy
apt-get install -y haproxy

# 配置HAProxy
cat <<EOF > /etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode http
    option httplog
    option dontlognull
    timeout connect 5000
    timeout client 50000
    timeout server 50000

frontend kubernetes
    bind 192.168.1.200:6443
    mode tcp
    option tcplog
    default_backend kubernetes-master-nodes

backend kubernetes-master-nodes
    mode tcp
    option tcplog
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server master1 192.168.1.101:6443 check
    server master2 192.168.1.102:6443 check
    server master3 192.168.1.103:6443 check
EOF

# 重启HAProxy
systemctl restart haproxy
systemctl enable haproxy

4. 高可用网络配置

4.1 配置Calico网络插件

# 安装Calico
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

# 验证网络状态
kubectl get pods -n kube-system
kubectl get nodes

4.2 配置网络策略

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

5. 高可用存储配置

5.1 配置StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: high-availability
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  iopsPerGB: "10"
  encrypted: "true"
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

5.2 配置PersistentVolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: high-availability-pvc
spec:
  storageClassName: high-availability
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

6. 高可用应用部署

6.1 部署高可用应用示例

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-high-availability
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - nginx
            topologyKey: "kubernetes.io/hostname"
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
  - port: 80
    targetPort: 80
  type: LoadBalancer

7. 监控与告警

7.1 部署Prometheus和Grafana

# 使用Helm安装Prometheus和Grafana
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace

# 查看监控组件
kubectl get pods -n monitoring

7.2 配置高可用告警规则

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-high-availability-alerts
  namespace: monitoring
spec:
  groups:
  - name: kubernetes-master
    rules:
    - alert: MasterDown
      expr: kube_node_status_condition{condition="Ready",node=~"master.*"} == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Master node down"
        description: "Master node {{ $labels.node }} is down for more than 5 minutes"

    - alert: EtcdDown
      expr: etcd_server_has_leader{job="etcd"} == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Etcd cluster down"
        description: "Etcd cluster has no leader for more than 5 minutes"

8. 故障转移与恢复

8.1 模拟Master节点故障

# 模拟Master节点故障
ssh master1 "sudo systemctl stop kube-apiserver kube-controller-manager kube-scheduler"

# 查看集群状态
kubectl get nodes
kubectl get pods -n kube-system

# 恢复Master节点
ssh master1 "sudo systemctl start kube-apiserver kube-controller-manager kube-scheduler"

# 验证集群状态
kubectl get nodes
kubectl get pods -n kube-system

8.2 模拟etcd节点故障

# 模拟etcd节点故障
ssh etcd1 "sudo systemctl stop etcd"

# 查看etcd集群状态
etcdctl endpoint health --endpoints=http://192.168.1.101:2379,http://192.168.1.102:2379,http://192.168.1.103:2379

# 恢复etcd节点
ssh etcd1 "sudo systemctl start etcd"

# 验证etcd集群状态
etcdctl endpoint health --endpoints=http://192.168.1.101:2379,http://192.168.1.102:2379,http://192.168.1.103:2379

🛠️ 最佳实践

1. 集群规划：

   • Master节点数量应为奇数（3或5个），确保etcd集群的高可用
   • 每个Master节点配置至少2CPU/4GB内存
   • Worker节点根据实际工作负载配置

2. 网络配置：

   • 使用Calico或Cilium等成熟的网络插件
   • 配置网络策略，限制Pod间通信
   • 确保网络带宽足够，特别是etcd节点间的通信

3. 存储配置：

   • 使用高可用的存储解决方案
   • 为关键应用配置PersistentVolumeClaim
   • 定期备份etcd数据

4. 监控与告警：

   • 部署Prometheus和Grafana监控集群状态
   • 配置关键指标的告警
   • 建立监控Dashboard，实时查看集群健康状态

5. 安全配置：

   • 启用RBAC，限制用户权限
   • 配置Pod安全策略
   • 定期更新Kubernetes版本和组件

6. 灾备与恢复：

   • 定期备份etcd数据
   • 制定详细的故障转移和恢复流程
   • 定期进行故障演练，确保团队熟悉恢复流程

📊 总结

Kubernetes集群的高可用性设计是生产环境的必备条件。通过本文的实践，你应该已经掌握了：

• 多Master架构的设计与部署
• etcd集群的配置与管理
• 负载均衡器的配置
• 网络和存储的高可用配置
• 应用的高可用部署
• 监控与告警的设置
• 故障转移与恢复的流程

记住，高可用性不是一蹴而就的，需要持续的维护和优化。在实际生产环境中，要根据业务需求和资源情况，选择合适的高可用方案。

---

susu碎碎念：

• Master节点数量不要太多，3个就足够了，多了会增加复杂度
• etcd集群一定要使用奇数节点，确保选举机制正常工作
• 负载均衡器本身也要高可用，避免成为单点故障
• 定期备份etcd数据，这是恢复集群的最后希望
• 监控告警要设置合理，避免告警风暴

觉得有用？点个赞再走！咱们下期见～ 🔥