在 vSphere 部署高可用的 KubeSphere

对于生产环境，我们需要考虑集群的高可用性。如果关键组件（例如 kube-apiserver，kube-scheduler 和 kube-controller-manager）都在同一主节点上运行，则一旦主节点出现故障，Kubernetes 和 KubeSphere 将不可用。因此，我们需要通过为负载均衡器配置多个主节点来设置高可用性集群。您可以使用任何云负载平衡器或任何硬件负载平衡器（例如F5）。另外，Keepalived 和HAproxy 或 Nginx 也是创建高可用性集群的替代方法。

本教程为您提供了一个示例，说明如何使用 keepalived + haproxy 对 kube-apiserver 进行负载均衡，实现高可用 kubernetes 集群。

1. 前提条件

• 请遵循该指南，确保您已经知道如何将 KubeSphere 与多节点集群一起安装。有关用于安装的 config yaml 文件的详细信息，请参阅多节点安装。本教程重点介绍如何配置负载均衡器。
• 您需要一个 VMware vSphere 帐户来创建VM资源。
• 考虑到数据的持久性，对于生产环境，我们建议您准备持久性存储并预先创建 StorageClass 。为了进行开发和测试，您可以使用集成的 OpenEBS 直接将 LocalPV设置为存储服务。

2. 部署架构

3. 创建主机

本示例创建 9 台 CentOS Linux release 7.6.1810（Core） 的虚拟机，每台配置为 8 Core 16 GB 100 G。

主机 IP	主机名称	角色
10.10.71.214	master1	master1, etcd
10.10.71.73	master2	master2, etcd
10.10.71.62	master3	master3, etcd
10.10.71.75	node1	node
10.10.71.76	node2	node
10.10.71.79	node3	node
10.10.71.67	vip	vip
10.10.71.77	lb-0	lb（keepalived + haproxy）
10.10.71.66	lb-1	lb（keepalived + haproxy）

选择可创建的资源池，点击右键-新建虚拟机（创建虚拟机入口请好几个，自己选择）

选择创建类型，创建新虚拟机。

填写虚拟机名称和存放文件夹。

选择计算资源。

选择存储。

选择兼容性，这里是 ESXi 7.0 及更高版本。

选择客户机操作系统，Linux CentOS 7 （64 位）。

自定义硬件，这里操作系统是挂载的 ISO 文件（打开电源时连接），网络是 VLAN71（勾选）。

清单，确认无误后，点击确定。

4. 部署 keepalived+haproxy

1. yum 安装

#在主机为lb-0和lb-1中部署keepalived+haproxy
#即IP为10.10.71.77与10.10.71.66的服务器上安装部署haproxy、keepalived、psmisc
yum install keepalived haproxy psmisc -y

2. 配置 haproxy

在IP为 10.10.71.77 与 10.10.71.66 的服务器 ，配置 haproxy (两台 lb 机器配置一致即可，注意后端服务地址)。

#Haproxy 配置 /etc/haproxy/haproxy.cfg
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    log                     global
    option                  httplog
    option                  dontlognull
    timeout connect         5000
    timeout client          5000
    timeout server          5000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  kube-apiserver
    bind *:6443
    mode tcp
    option tcplog
    default_backend kube-apiserver
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kube-apiserver
    mode tcp
    option tcplog
    balance     roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server kube-apiserver-1 10.10.71.214:6443 check
    server kube-apiserver-2 10.10.71.73:6443 check
    server kube-apiserver-3 10.10.71.62:6443 check

#启动之前检查语法是否有问题
haproxy -f /etc/haproxy/haproxy.cfg -c
#启动 Haproxy，并设置开机自启动
systemctl restart haproxy && systemctl enable haproxy
#停止 Haproxy
systemctl stop haproxy

3. 配置 keepalived

# 主 haproxy 77 lb-0-10.10.71.77
#/etc/keepalived/keepalived.conf
global_defs {
  notification_email {
  }
  smtp_connect_timeout 30    #连接超时时间
  router_id LVS_DEVEL01 ##相当于给这个服务器起个昵称
  vrrp_skip_check_adv_addr
  vrrp_garp_interval 0
  vrrp_gna_interval 0
}
vrrp_script chk_haproxy {
  script "killall -0 haproxy"
  interval 2
  weight 2
}
vrrp_instance haproxy-vip {
  state MASTER  #主服务器 是MASTER
  priority 100  #主服务器优先级要比备服务器高
  interface ens192                        #实例绑定的网卡
  virtual_router_id 60 #定义一个热备组，可以认为这是60号热备组
  advert_int 1 #1秒互相通告一次，检查对方死了没。
  authentication {
    auth_type PASS #认证类型
    auth_pass 1111 #认证密码  这些相当于暗号
  }
  unicast_src_ip 10.10.71.77      #当前机器地址
  unicast_peer {
    10.10.71.66                       #peer中其它机器地址
  }
  virtual_ipaddress {
    #vip地址
    10.10.71.67/24 
  }
  track_script {
    chk_haproxy
  }
}

#备 haproxy 66 lb-1-10.10.71.66
#/etc/keepalived/keepalived.conf
global_defs {
  notification_email {
  }
  router_id LVS_DEVEL02 ##相当于给这个服务器起个昵称
  vrrp_skip_check_adv_addr
  vrrp_garp_interval 0
  vrrp_gna_interval 0
}
vrrp_script chk_haproxy {
  script "killall -0 haproxy"
  interval 2
  weight 2
}
vrrp_instance haproxy-vip {
  state BACKUP #备份服务器 是 backup
  priority 90 #优先级要低（把备份的90修改为100）
  interface ens192                        #实例绑定的网卡
  virtual_router_id 60
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  unicast_src_ip 10.10.71.66      #当前机器地址
  unicast_peer {
    10.10.71.77                         #peer 中其它机器地址
  }
  virtual_ipaddress {
    #加/24
    10.10.71.67/24 
  }
  track_script {
    chk_haproxy
  }
}

#启动 keepalived，设置开机自启动
systemctl restart keepalived && systemctl enable keepalived
systemctl stop keepaliv
systemctl start keepalived    #开启 keepalived服务

4. 验证可用性

• 使用 ip a s 查看各 lb 节点 vip 绑定情况
• 暂停vip所在节点 haproxy：systemctl stop haproxy
• 再次使用 ip a s 查看各 lb 节点 vip 绑定情况，查看 vip 是否发生漂移
• 或者使用 systemctl status -l keepalived 命令查看

5. 获取安装程序可执行文件

#下载 installer 至一台目标机器
curl -O -k https://kubernetes.pek3b.qingstor.com/tools/kubekey/kk
chmod +x kk

6. 创建多节点群集

您可以使用高级安装来控制自定义参数或创建多节点群集。具体来说，通过指定配置文件来创建集群。

1. kubekey 部署 k8s 集群

# 创建配置文件(一个示例配置文件)|包含 kubesphere 的配置文件
./kk create config --with-kubesphere v3.0.0 -f ~/config-sample.yaml
#如果重复安装，镜像就不用再下载了，可以 skip
./kk create cluster -f ~/config-sample.yaml --debug --skip-pull-images
#删除 cluster
./kk delete cluster -f ~/config-sample.yaml --debug --skip-pull-images
#小提示，如果安装过程中碰到 `Failed to add worker to cluster: Failed to exec command...`
kubeadm reset

2. 多集群配置

#vi ~/config-sample.yaml
apiVersion: kubekey.kubesphere.io/v1alpha1
kind: Cluster
metadata:
  name: config-sample
spec:
  hosts:
  - {name: master1, address: 10.10.71.214, internalAddress: 10.10.71.214, password: P@ssw0rd!}
  - {name: master2, address: 10.10.71.73, internalAddress: 10.10.71.73, password: P@ssw0rd!}
  - {name: master3, address: 10.10.71.62, internalAddress: 10.10.71.62, password: P@ssw0rd!}
  - {name: node1, address: 10.10.71.75, internalAddress: 10.10.71.75, password: P@ssw0rd!}
  - {name: node2, address: 10.10.71.76, internalAddress: 10.10.71.76, password: P@ssw0rd!}
  - {name: node3, address: 10.10.71.79, internalAddress: 10.10.71.79, password: P@ssw0rd!}
  roleGroups:
    etcd:
    - master1
    - master2
    - master3
    master: 
    - master1
    - master2
    - master3
    worker:
    - node1
    - node2
    - node3
  controlPlaneEndpoint:
    domain: lb.kubesphere.local
    # vip
    address: "10.10.71.67"                    
    port: "6443"
  kubernetes:
    version: v1.17.9
    imageRepo: kubesphere
    clusterName: cluster.local
    masqueradeAll: false  # masqueradeAll tells kube-proxy to SNAT everything if using the pure iptables proxy mode. [Default: false]
    maxPods: 110  # maxPods is the number of pods that can run on this Kubelet. [Default: 110]
    nodeCidrMaskSize: 24  # internal network node size allocation. This is the size allocated to each node on your network. [Default: 24]
    proxyMode: ipvs  # mode specifies which proxy mode to use. [Default: ipvs]
  network:
    plugin: calico
    calico:
      ipipMode: Always  # IPIP Mode to use for the IPv4 POOL created at start up. If set to a value other than Never, vxlanMode should be set to "Never". [Always | CrossSubnet | Never] [Default: Always]
      vxlanMode: Never  # VXLAN Mode to use for the IPv4 POOL created at start up. If set to a value other than Never, ipipMode should be set to "Never". [Always | CrossSubnet | Never] [Default: Never]
      vethMTU: 1440  # The maximum transmission unit (MTU) setting determines the largest packet size that can be transmitted through your network. [Default: 1440]
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
  registry:
    registryMirrors: []
    insecureRegistries: []
    privateRegistry: ""
  storage:
    defaultStorageClass: localVolume
    localVolume:
      storageClassName: local  

---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
  name: ks-installer
  namespace: kubesphere-system
  labels:
    version: v3.0.0
spec:
  local_registry: ""
  persistence:
    storageClass: ""
  authentication:
    jwtSecret: ""
  etcd:
    monitoring: true        # Whether to install etcd monitoring dashboard
    endpointIps: 192.168.0.7,192.168.0.8,192.168.0.9  # etcd cluster endpointIps
    port: 2379              # etcd port
    tlsEnable: true
  common:
    mysqlVolumeSize: 20Gi # MySQL PVC size
    minioVolumeSize: 20Gi # Minio PVC size
    etcdVolumeSize: 20Gi  # etcd PVC size
    openldapVolumeSize: 2Gi   # openldap PVC size
    redisVolumSize: 2Gi # Redis PVC size
    es:  # Storage backend for logging, tracing, events and auditing.
      elasticsearchMasterReplicas: 1   # total number of master nodes, it's not allowed to use even number
      elasticsearchDataReplicas: 1     # total number of data nodes
      elasticsearchMasterVolumeSize: 4Gi   # Volume size of Elasticsearch master nodes
      elasticsearchDataVolumeSize: 20Gi    # Volume size of Elasticsearch data nodes
      logMaxAge: 7                     # Log retention time in built-in Elasticsearch, it is 7 days by default.
      elkPrefix: logstash              # The string making up index names. The index name will be formatted as ks-<elk_prefix>-log
      # externalElasticsearchUrl:
      # externalElasticsearchPort:
  console:
    enableMultiLogin: false  # enable/disable multiple sing on, it allows an account can be used by different users at the same time.
    port: 30880
  alerting:                # Whether to install KubeSphere alerting system. It enables Users to customize alerting policies to send messages to receivers in time with different time intervals and alerting levels to choose from.
    enabled: false
  auditing:                # Whether to install KubeSphere audit log system. It provides a security-relevant chronological set of records，recording the sequence of activities happened in platform, initiated by different tenants.
    enabled: false         
  devops:                  # Whether to install KubeSphere DevOps System. It provides out-of-box CI/CD system based on Jenkins, and automated workflow tools including Source-to-Image & Binary-to-Image
    enabled: false
    jenkinsMemoryLim: 2Gi      # Jenkins memory limit
    jenkinsMemoryReq: 1500Mi   # Jenkins memory request
    jenkinsVolumeSize: 8Gi     # Jenkins volume size
    jenkinsJavaOpts_Xms: 512m  # The following three fields are JVM parameters
    jenkinsJavaOpts_Xmx: 512m
    jenkinsJavaOpts_MaxRAM: 2g
  events:                  # Whether to install KubeSphere events system. It provides a graphical web console for Kubernetes Events exporting, filtering and alerting in multi-tenant Kubernetes clusters.
    enabled: false
  logging:                 # Whether to install KubeSphere logging system. Flexible logging functions are provided for log query, collection and management in a unified console. Additional log collectors can be added, such as Elasticsearch, Kafka and Fluentd.
    enabled: false
    logsidecarReplicas: 2
  metrics_server:                    # Whether to install metrics-server. IT enables HPA (Horizontal Pod Autoscaler).
    enabled: true
  monitoring:                        #
    prometheusReplicas: 1            # Prometheus replicas are responsible for monitoring different segments of data source and provide high availability as well.
    prometheusMemoryRequest: 400Mi   # Prometheus request memory
    prometheusVolumeSize: 20Gi       # Prometheus PVC size
    alertmanagerReplicas: 1          # AlertManager Replicas
  multicluster:
    clusterRole: none  # host | member | none  # You can install a solo cluster, or specify it as the role of host or member cluster
  networkpolicy:       # Network policies allow network isolation within the same cluster, which means firewalls can be set up between certain instances (Pods).
    enabled: false     
  notification:        # It supports notification management in multi-tenant Kubernetes clusters. It allows you to set AlertManager as its sender, and receivers include Email, Wechat Work, and Slack.
    enabled: false
  openpitrix:          # Whether to install KubeSphere Application Store. It provides an application store for Helm-based applications, and offer application lifecycle management
    enabled: false
  servicemesh:         # Whether to install KubeSphere Service Mesh (Istio-based). It provides fine-grained traffic management, observability and tracing, and offer visualization for traffic topology
    enabled: false

3. 输出结果

**************************************************
#####################################################
###              Welcome to KubeSphere!           ###
#####################################################
Console: http://10.10.71.214:30880
Account: admin
Password: P@88w0rd
NOTES：
  1. After logging into the console, please check the
     monitoring status of service components in
     the "Cluster Management". If any service is not
     ready, please wait patiently until all components 
     are ready.
  2. Please modify the default password after login.
#####################################################
https://kubesphere.io             2020-08-15 23:32:12
#####################################################