openEuler 系统 Kubernetes + Harbor 小规模生产环境详细部署指南（5节点）

一、环境规划

1.1 节点架构（5节点高可用）

节点角色	主机名	IP地址	配置要求	部署组件
Master1	k8s-master1	192.168.100.177	4 vCPU / 8GB / 100GB	K8s控制平面+etcd
Master2	k8s-master2	192.168.100.178	4 vCPU / 8GB / 100GB	K8s控制平面+etcd
Master3	k8s-master3	192.168.100.179	4 vCPU / 8GB / 100GB	K8s控制平面+etcd
Worker1	k8s-worker1	192.168.100.180	4 vCPU / 8GB / 100GB	K8s工作节点
Worker2	k8s-worker2	192.168.100.181	4 vCPU / 8GB / 100GB	K8s工作节点+Harbor+Kuboard
VIP		192.168.100.200	-	负载均衡虚拟IP

1.2 软件版本推荐（生产环境）

组件	版本	说明
openEuler	22.03 LTS SP4 / 24.03 LTS	推荐LTS版本，生产稳定
Kubernetes	1.29.x / 1.30.x	长期支持版本
Harbor	2.14.x	最新稳定版
Kuboard	v3.x	Kubernetes管理面板
容器运行时	containerd 1.6.x	openEuler内置支持
Docker	26.x	Harbor依赖
HAProxy	2.8.x	负载均衡
Keepalived	2.2.x	高可用VIP

1.3 网络规划

网络类型	CIDR	说明
Pod网络	10.244.0.0/16	Calico网络插件
Service网络	10.96.0.0/12	K8s服务网络
集群API	192.168.100.200:6443	VIP访问入口
Harbor服务	192.168.100.181:443 私有仓库域名：harbor-Warehouses	HTTPS访问
Kuboard服务	192.168.100.200:30080	NodePort访问
docker	172.98.0.1/172.99.0.1	docker IP网段

---

二、系统初始化（所有5个节点执行）

2.1 配置主机名和hosts

# ==================== Master1 ====================
hostnamectl set-hostname k8s-master1

# ==================== Master2 ====================
hostnamectl set-hostname k8s-master2

# ==================== Master3 ====================
hostnamectl set-hostname k8s-master3

# ==================== Worker1 ====================
hostnamectl set-hostname k8s-worker1

# ==================== Worker2 ====================
hostnamectl set-hostname k8s-worker2

# ==================== 所有节点配置hosts ====================
cat >> /etc/hosts << EOF
192.168.100.177 k8s-master1
192.168.100.178 k8s-master2
192.168.100.179 k8s-master3
192.168.100.180 k8s-worker1
192.168.100.181 k8s-worker2 harbor-Warehouses
192.168.100.200 k8s-vip
EOF

2.2 关闭防火墙和SELinux

# 所有节点执行
# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

# 关闭SELinux（生产环境建议配置策略而非完全关闭）
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

# 关闭swap（K8s要求）
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

# 验证
free -h  # 确认swap为0
getenforce  # 确认Disabled

2.3 配置内核参数（生产优化）

# 所有节点执行
cat > /etc/sysctl.d/k8s.conf << EOF
# 网络桥接
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1

# 内存管理
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
vm.max_map_count=262144

# 网络连接优化
net.ipv4.tcp_keepalive_time=600
net.ipv4.tcp_keepalive_intvl=30
net.ipv4.tcp_keepalive_probes=10
net.netfilter.nf_conntrack_max=1048576

# 文件描述符
fs.file-max=2097152
fs.inotify.max_user_watches=524288
fs.inotify.max_user_instances=512

# 端口范围
net.ipv4.ip_local_port_range=1024 65535
EOF

# 加载内核模块
cat > /etc/modules-load.d/k8s.conf << EOF
br_netfilter
overlay
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF

# 应用配置
modprobe br_netfilter
modprobe overlay
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack
sysctl --system

# 验证
sysctl net.bridge.bridge-nf-call-iptables
sysctl net.ipv4.ip_forward

2.4 配置时间同步（生产关键）

# 所有节点执行
dnf install -y chrony

# 配置NTP服务器（使用国内源）
cat > /etc/chrony.conf << EOF
pool ntp.aliyun.com iburst
pool ntp.tencent.com iburst
pool cn.pool.ntp.org iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony
EOF

# 启动服务
systemctl enable --now chronyd

# 验证时间同步
chronyc sources -v
chronyc tracking

# 配置时区
timedatectl set-timezone Asia/Shanghai
timedatectl status

2.5 配置SSH免密登录（便于管理）

# 在Master1节点执行
[root@k8s-master1 ~]# ssh-keygen -t rsa -b 4096 -N ""
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): # 回车
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:Oej475yPGaQzHrfAz3j/rOQNO0o/II6uDnGxg++4T24 root@k8s-master1
The key's randomart image is:
+---[RSA 4096]----+
|                 |
|                 |
|  .              |
| . o   . .       |
|o +   . S        |
| + . +.o..       |
|. o .oO.+.o      |
| *E .o.%.Xo=     |
|o=*o. ++%+B=+    |
+----[SHA256]-----+
[root@k8s-master1 ~]#

# 分发公钥到所有节点
[root@k8s-master1 ~]# for node in k8s-master1 k8s-master2 k8s-master3 k8s-worker1 k8s-worker2; do
    ssh-copy-id root@$node
done
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'k8s-master1 (192.168.100.177)' can't be established.
ED25519 key fingerprint is SHA256:j8ghKLG9MQvUHdv/3HEYjWhDs/nRPd5pmhKMyAHNrA0.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes # 输入yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Authorized users only. All activities may be monitored and reported.
root@k8s-master1's password:  # 输入密码

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@k8s-master1'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'k8s-master2 (192.168.100.178)' can't be established.
ED25519 key fingerprint is SHA256:nmjNJnjEnWU6er+j0sJ0aqFi76ZH0Au7PUmFE0AJ1dU.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Authorized users only. All activities may be monitored and reported.
root@k8s-master2's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@k8s-master2'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'k8s-master3 (192.168.100.179)' can't be established.
ED25519 key fingerprint is SHA256:XlZAlGVlgAEdEw7XpDb+61MGzOGv0Xqy3ETbz4XzWwY.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Authorized users only. All activities may be monitored and reported.
root@k8s-master3's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@k8s-master3'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'k8s-worker1 (192.168.100.180)' can't be established.
ED25519 key fingerprint is SHA256:4wThGlt+ZVyhV+zkLjaKUrNOupeize9vKmNDZEYB/Zc.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Authorized users only. All activities may be monitored and reported.
root@k8s-worker1's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@k8s-worker1'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'k8s-worker2 (192.168.100.181)' can't be established.
ED25519 key fingerprint is SHA256:a5ug4Rcobs4xnTe0+LlRLZDpftA/QB7TLppw8t6ORlk.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Authorized users only. All activities may be monitored and reported.
root@k8s-worker2's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@k8s-worker2'"
and check to make sure that only the key(s) you wanted were added.

[root@k8s-master1 ~]# 


# 验证
[root@k8s-master1 ~]# ssh k8s-master2 "hostname"

Authorized users only. All activities may be monitored and reported.
k8s-master2
[root@k8s-master1 ~]# ssh k8s-worker2 "hostname"

Authorized users only. All activities may be monitored and reported.
k8s-worker2
[root@k8s-master1 ~]# 

---

三、负载均衡配置（Keepalived + HAProxy）

3.1 安装HAProxy和Keepalived（Master1和Master2）

# 在Master1和Master2、Master3节点执行
dnf install -y haproxy keepalived

# 启用服务
systemctl enable --now haproxy

3.2 配置HAProxy

# 在Master1和Master2、Master3节点执行
cat > /etc/haproxy/haproxy.cfg << EOF
global
    log         127.0.0.1 local2
    
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon 
       
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------  
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#--------------------------------------------------------------------- 
frontend kubernetes-apiserver
    mode                 tcp
    bind                 *:16443
    option               tcplog
    default_backend      kubernetes-apiserver    
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
    mode        tcp
    balance     roundrobin
    server k8s-master1 192.168.100.177:6443 check-ssl verify none inter 3s fall 3 rise 2
    server k8s-master2 192.168.100.178:6443 check-ssl verify none inter 3s fall 3 rise 2
    server k8s-master3 192.168.100.179:6443 check-ssl verify none inter 3s fall 3 rise 2
EOF

# 先检查配置语法
haproxy -c -f /etc/haproxy/haproxy.cfg

# 重启服务
systemctl restart haproxy
systemctl status haproxy

3.3 配置Keepalived

# ==================== Master1 (主节点) ====================
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
}
vrrp_script chk_haproxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    weight 2
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33          # 根据实际网卡名修改
    virtual_router_id 51
    priority 100  # 不通点: master，100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.100.200/24 dev ens33 label ens33:1
    }
    track_script {
        chk_haproxy
    }
}
EOF

# ==================== Master2 (备节点) ====================
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
}
vrrp_script chk_haproxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    weight 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33          # 根据实际网卡名修改
    virtual_router_id 51
    priority 90   # 不通点: BACKUP，90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.100.200/24 dev ens33 label ens33:1
    }
    track_script {
        chk_haproxy
    }
}
EOF

# ==================== Master3 (备节点) ====================
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
}
vrrp_script chk_haproxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    weight 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33          # 根据实际网卡名修改
    virtual_router_id 51
    priority 80   # 不通点: BACKUP，80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.100.200/24 dev ens33 label ens33:1
    }
    track_script {
        chk_haproxy
    }
}
EOF

# 重启服务
systemctl restart keepalived
systemctl status keepalived

# 验证VIP(在master1)
ip addr show ens33 | grep 192.168.100.200

---

四、Kubernetes集群部署

4.1 安装容器运行时（containerd）

# 以下操作，所有5个节点执行
dnf install -y containerd cri-tools

# 生成默认配置
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

# 修改配置（systemd cgroup驱动）
grep 'SystemdCgroup' /etc/containerd/config.toml  # 先查看是否是false，再进行修改
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

[root@k8s-master1 ~]# grep 'sandbox_image' /etc/containerd/config.toml 
    sandbox_image = "registry.k8s.io/pause:3.6"
[root@k8s-master1 ~]# 
[root@k8s-master1 ~]# sed -i 's#sandbox_image = "registry.k8s.io/pause:3.6"#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"#' /etc/containerd/config.toml
[root@k8s-master1 ~]# 
[root@k8s-master1 ~]# grep 'sandbox_image' /etc/containerd/config.toml 
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"
[root@k8s-master1 ~]# 
# 替换这个配置，就是把「国内访问不了的国外基础镜像地址」，改成「国内阿里云的高速镜像地址」，解决 K8s 集群因为网络问题无法启动的致命问题。



# 配置镜像加速
[root@k8s-master1 ~]# vim /etc/containerd/config.toml 
...省略N
    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      #[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
      # 配置华为云加速器，添加以下内容。
      # 1.登录 华为云 SWR 控制台。
      # 2.左侧：镜像资源 → 镜像中心 → 镜像加速器
      # 3.复制你的专属地址（格式：https://<你的账号ID>.mirror.swr.myhuaweicloud.com）
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
        endpoint = ["https://448b823acc1c4d5b8cf5c81ea9bfce60.mirror.swr.myhuaweicloud.com"]
      # 下面2行是配置私有镜像仓库地址
      # [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor-Warehouses"]
      #  endpoint = ["https://harbor-Warehouses"]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

...省略N
[root@k8s-master1 ~]#

# 重启containerd
systemctl daemon-reload
systemctl enable --now containerd
systemctl restart containerd

# 让 crictl 直接指定连接 containerd，不再遍历废弃套接字
# 创建 crictl 配置文件，指定 containerd 套接字
cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

# 验证
crictl info
crictl version

4.2 安装Kubernetes组件

# 所有5个节点执行
# 添加Kubernetes仓库（阿里云镜像）
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.29/rpm
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF

# 安装指定版本
dnf install -y kubelet-1.29.0 kubeadm-1.29.0 kubectl-1.29.0 --disableexcludes=kubernetes

# 设置开机启动
systemctl enable --now kubelet

# 验证
kubeadm version
kubelet --version
kubectl version --client

4.3 初始化第一个Master节点

# 仅在Master1节点执行
# 创建kubeadm配置文件
cat > kubeadm-config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.29.0
controlPlaneEndpoint: "192.168.100.200:6443"   # VIP地址
imageRepository: registry.aliyuncs.com/google_containers
networking:
  podSubnet: "10.244.0.0/16"     # Calico默认网段
  serviceSubnet: "10.96.0.0/12"
apiServer:
  certSANs:
  - "192.168.100.200"
  - "k8s-vip"
  - "192.168.100.177"
  - "192.168.100.178"
  - "192.168.100.179"
  - "127.0.0.1"
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  criSocket: "unix:///var/run/containerd/containerd.sock"
  name: "k8s-master1"
  #kubeletExtraArgs: 存放kubelet 数据到指定位置data下
    #root-dir: /data/kubelet
EOF

# 创建目录
mkdir -p /data/kubelet

# 预拉取镜像
[root@k8s-master1 ~]# kubeadm config images pull --config kubeadm-config.yaml
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.29.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.29.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.29.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.29.0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.11.1
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.9
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.10-0
[root@k8s-master1 ~]# 

# 初始化集群
[root@k8s-master1 ~]# kubeadm init --config kubeadm-config.yaml --upload-certs
...省略N
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.100.200:16443 --token ssgfna.28wqzvg8gtqxhmat \
	--discovery-token-ca-cert-hash sha256:7512eb20b91cfa28bc0c58dda91cccc4b233bcfb0ed8825f8937399b47c256be \
	--control-plane --certificate-key 462c328aa602952af6595f1c4eb3de6844898f0ecd6da01fe96721d52d35d8d1

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.100.200:16443 --token ssgfna.28wqzvg8gtqxhmat \
	--discovery-token-ca-cert-hash sha256:7512eb20b91cfa28bc0c58dda91cccc4b233bcfb0ed8825f8937399b47c256be 
[root@k8s-master1 ~]#


# 配置kubectl
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

# 保存证书密钥（用于其他Master加入）
# 记录输出中的 --certificate-key 值

4.4 添加其他Master节点

# 在Master1获取join命令
kubeadm token create --print-join-command  # 生成工作节点加入集群的完整命令
kubeadm init phase upload-certs --upload-certs  # 上传控制平面证书到集群，为多主节点高可用集群设置提供证书共享机制。

# 在Master2和Master3节点执行（使用输出的命令）
[root@k8s-master2 ~]# kubeadm join 192.168.100.200:16443 --token ssgfna.28wqzvg8gtqxhmat \
    --discovery-token-ca-cert-hash sha256:7512eb20b91cfa28bc0c58dda91cccc4b233bcfb0ed8825f8937399b47c256be \
    --control-plane --certificate-key 462c328aa602952af6595f1c4eb3de6844898f0ecd6da01fe96721d52d35d8d1
...省略N

To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

[root@k8s-master2 ~]# mkdir -p $HOME/.kube
        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
        sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@k8s-master2 ~]# 


# 配置kubectl（Master2和Master3）
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

4.5 部署网络插件（Calico）

# 在Master1节点执行
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

# 验证Pod状态
kubectl get pods -n kube-system -o wide
kubectl get nodes

4.6 添加Worker节点

# 在Master1获取worker join命令
kubeadm token create --print-join-command

# 在Worker1和Worker2节点执行
[root@k8s-worker1 ~]# kubeadm join 192.168.100.200:16443 --token ssgfna.28wqzvg8gtqxhmat \
    --discovery-token-ca-cert-hash sha256:7512eb20b91cfa28bc0c58dda91cccc4b233bcfb0ed8825f8937399b47c256be
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[root@k8s-worker1 ~]#

# 在Master1验证
kubectl get nodes

---

五、Harbor私有仓库部署（生产配置）

5.1 安装Docker和Docker Compose（Worker2节点）

# 在Worker2节点执行
cat > /etc/yum.repos.d/docker-ce.repo <<'EOF'
[docker]
name=Docker
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/8/x86_64/stable
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
EOF

# 安装Docker
dnf install -y docker-ce

# 配置Docker（支持Harbor）
cat > /etc/docker/daemon.json <<'EOF'
{
  "data-root": "/data/docker",
  "registry-mirrors": [ "https://448b823acc1c4d5b8cf5c81ea9bfce60.mirror.swr.myhuaweicloud.com" ],
  "bip": "172.98.0.1/16",
  "default-address-pools": [
    {
      "base": "172.99.0.0/16",
      "size": 24
    }
  ],
  "iptables": true,
  "ip-forward": true,
  "ip-masq": true
}
EOF

# 启动Docker
systemctl daemon-reload
systemctl enable --now docker
systemctl restart docker

# 安装Docker Compose
curl -L "https://github.com/docker/compose/releases/download/v2.26.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

# 验证
docker --version
docker-compose --version

5.2 生成自签名证书（生产环境建议使用正式证书）

# 在Worker2节点执行
mkdir -p /opt/harbor/certs
cd /opt/harbor/certs

# 生成CA证书
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -sha256 -days 3650 \
  -key ca.key \
  -out ca.crt \
  -subj "/C=CN/ST=Beijing/L=Beijing/O=Harbor/OU=Registry/CN=Harbor CA"

# 生成服务器证书
openssl genrsa -out harbor.key 4096
openssl req -new -sha256 \
  -key harbor.key \
  -out harbor.csr \
  -subj "/C=CN/ST=Beijing/L=Beijing/O=Harbor/OU=Registry/CN=192.168.100.181"

# 使用CA签名
openssl x509 -req -sha256 \
  -days 3650 \
  -in harbor.csr \
  -CA ca.crt \
  -CAkey ca.key \
  -CAcreateserial \
  -out harbor.crt \
  -extfile <(echo "subjectAltName=DNS:192.168.100.181,DNS:harbor-Warehouses,DNS:k8s-worker2,DNS:harbor.local,IP:192.168.100.181")

# 分发证书到所有K8s节点
for node in k8s-master1 k8s-master2 k8s-master3 k8s-worker1 k8s-worker2; do
    scp ca.crt root@$node:/etc/pki/ca-trust/source/anchors/
    ssh root@$node "update-ca-trust force-enable && update-ca-trust extract"
done

# 配置Docker信任证书
mkdir -p /etc/docker/certs.d/192.168.100.181
cp ca.crt /etc/docker/certs.d/192.168.100.181/ca.crt
systemctl restart docker

5.3 下载和配置Harbor

# 在Worker2节点执行
cd /opt
wget https://github.com/goharbor/harbor/releases/download/v2.14.0/harbor-online-installer-v2.14.0.tgz
tar -xvf harbor-online-installer-v2.14.0.tgz
cd harbor

# 复制配置文件
cp harbor.yml.tmpl harbor.yml

# 编辑配置文件
cat > harbor.yml << EOF
hostname: harbor-Warehouses

http:
  port: 80

https:
  port: 443
  certificate: /opt/harbor/certs/harbor.crt
  private_key: /opt/harbor/certs/harbor.key

harbor_admin_password: Harbor12345

database:
  password: root123
  max_idle_conns: 50
  max_open_conns: 1000

data_volume: /data/harbor

trivy:
  ignore_unfixed: false
  skip_update: false
  offline_scan: false
  skip_java_db_download: false

jobservice:
  max_job_workers: 10

notification:
  webhook_job_max_retry: 10

log:
  level: info
  local:
    rotate_count: 50
    rotate_size: 200M
    location: /var/log/harbor

_proxy_cache_url_suffix: ""
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - trivy
EOF

5.4 安装Harbor

# 在Worker2节点执行
# 创建数据目录
mkdir -p /data/harbor

# 执行安装
./install.sh

# 验证服务
docker-compose ps
docker-compose logs -f

# 访问测试
curl -k https://192.168.100.181

5.5 配置K8s使用Harbor

# 在Master1节点执行
# 创建命名空间
kubectl create namespace harbor-system

# 创建镜像拉取密钥
kubectl create secret docker-registry harbor-secret \
  --docker-server=https://192.168.100.181 \
  --docker-username=admin \
  --docker-password=Harbor12345 \
  --docker-email=admin@example.com \
  -n harbor-system

# 创建默认拉取密钥（所有命名空间）
kubectl create secret docker-registry harbor-secret \
  --docker-server=https://192.168.100.181 \
  --docker-username=admin \
  --docker-password=Harbor12345 \
  --docker-email=admin@example.com \
  -n default

# 配置默认ServiceAccount
kubectl patch serviceaccount default \
  -p '{"imagePullSecrets": [{"name": "harbor-secret"}]}' \
  -n default

---

六、Kuboard 管理面板部署

6.1 部署架构说明

• 运行节点：在 k8s-master1

• 访问方式：通过 VIP（192.168.100.200）

• 资源需求：10GB 磁盘

• 高可用策略：Kuboard 本身是无状态的，通过 Kubernetes 多副本保证高可用（如果部署多副本）

6.2 部署 Kuboard（使用 Kubernetes 方式）

第一步：创建本地存储目录

# 在 master1 上执行
[root@k8s-master1 ~]# mkdir -p /opt/kuboard-data
[root@k8s-master1 ~]# chmod 777 /opt/kuboard-data

第二步：创建 yaml 文件

• yaml 文件：Kuboard.yaml

# ==============================================
#  Kuboard 持久化 + 可在 Master 运行
#  数据保留在本地 k8s-master1 节点上/opt/kuboard-data，防止 pod 删除后数据丢失问题
# ==============================================

---
# 1. 命名空间
apiVersion: v1
kind: Namespace
metadata:
  name: kuboard

---
# 2. 本地持久化卷（数据永久保存）
apiVersion: v1
kind: PersistentVolume
metadata:
  name: kuboard-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: "local-storage"
  local:
    path: /opt/kuboard-data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-master1

---
# 3. 持久化声明
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: kuboard-pvc
  namespace: kuboard
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: "local-storage"

---
# 4. 权限账号
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kuboard-admin
  namespace: kuboard

---
# 5. 集群管理员权限
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kuboard-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kuboard-admin
  namespace: kuboard

---
# 6. Kuboard 应用部署
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kuboard
  namespace: kuboard
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kuboard
  template:
    metadata:
      labels:
        app: kuboard
    spec:
      nodeSelector:
        kubernetes.io/hostname: k8s-master1
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      serviceAccountName: kuboard-admin
      securityContext:
        runAsUser: 0
        fsGroup: 0
      containers:
      - name: kuboard
        image: swr.cn-east-2.myhuaweicloud.com/kuboard/kuboard:v3.5.2.3
        ports:
        - containerPort: 80
        env:
        - name: KUBOARD_DISABLE_AUDIT
          value: "true"
        volumeMounts:
        - name: data
          mountPath: /data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: kuboard-pvc

---
# 7. 对外访问端口
apiVersion: v1
kind: Service
metadata:
  name: kuboard
  namespace: kuboard
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30080
  selector:
    app: kuboard

第三步：应用 Kuboard.yaml

[root@k8s-master1 ~]# kubectl apply  -f kuboard.yaml 
namespace/kuboard created
persistentvolume/kuboard-pv created
persistentvolumeclaim/kuboard-pvc created
serviceaccount/kuboard-admin created
clusterrolebinding.rbac.authorization.k8s.io/kuboard-admin unchanged
deployment.apps/kuboard created
service/kuboard created
[root@k8s-master1 ~]# 

第四步：查看状态

[root@k8s-master1 ~]# kubectl get pods -n kuboard -w
NAME                       READY   STATUS    RESTARTS   AGE
kuboard-55cbf4d4f5-wd4hv   1/1     Running   0          16s

6.3 访问 Kuboard 并登录

• 打开浏览器访问：http://192.168.100.200:30080
• 用户：admin
• 密码：Kuboard123

---

添加 K8s 集群

• 点击 添加集群

• 点击KubConfig
  • 在k8s-master1 上执行 cat /etc/kubernetes/admin.conf 输出的内容复制出来，将内容复制到KubConfig框。
  • 填写名称。
  • 描述。
  • 最后点击确定。

---

• 点击左边的 使用ServiceAccount kuboard-admin 后点击集群概要。

• 查看当前节点

---

6.4 部署 kuboard （docker 方式）

• 可选

docker run -d \
  --restart=unless-stopped \
  --name=kuboard \
  -p 30080:80/tcp \
  -p 10081:10081/tcp \
  -e KUBOARD_ENDPOINT="http://在已安装docker的主机IP:30080" \
  -e KUBOARD_AGENT_SERVER_TCP_PORT="10081" \
  -v /data/kuboard-data:/data \
  eipwork/kuboard:v3

逐参数解释

1. docker run

Docker 基础命令，作用：创建并启动一个容器。

2. -d

全称 --detach，后台运行容器

• 容器不会占用当前终端窗口，启动后直接返回容器 ID
• 退出终端容器依然正常运行

3. --restart=unless-stopped

容器重启策略

• 规则：除非手动停止容器，否则 Docker 会自动重启容器
• 场景：服务器重启、容器异常崩溃时，自动恢复 Kuboard 服务

4. --name=kuboard

给容器指定名称

• 方便后续管理（停止、删除、查看日志）
• 不指定的话，Docker 会自动生成随机名称

5. -p 30080:80/tcp

端口映射（核心）

格式：主机端口:容器端口/协议

• 主机端口：30080（你访问 Kuboard 网页的端口）
• 容器端口：80（Kuboard 内部服务端口）
• 作用：访问 主机IP:30080 → 转发到容器内的 80 端口，打开 Kuboard 网页

6. -p 10081:10081/tcp

Agent 通信端口映射

• Kuboard 管理 Kubernetes 集群时，Agent 客户端通过这个端口和服务端通信
• 主机端口 = 容器端口 = 10081，直接映射即可

7. -e KUBOARD_ENDPOINT="http://主机IP:30080"

环境变量（核心配置）

• KUBOARD_ENDPOINT：Kuboard 对外访问地址
• 必须填写安装 Docker 的这台服务器的真实 IP
• 作用：Kuboard 页面、Agent 客户端依赖这个地址通信

8. -e KUBOARD_AGENT_SERVER_TCP_PORT="10081"

Agent 通信端口环境变量

• 告诉 Kuboard：Agent 客户端连接的端口是 10081
• 必须和上面的端口映射 10081 保持一致

9. -v /data/kuboard-data:/data

数据卷挂载（持久化存储）

格式：主机目录:容器内目录

• 主机目录：/data/kuboard-data（服务器本地文件夹）
• 容器目录：/data（Kuboard 存储数据的目录）
• 作用：容器删除 / 重建，配置、用户数据不会丢失（持久化）

10. eipwork/kuboard:v3

Docker 镜像名称 + 版本

• eipwork/kuboard：官方镜像名
• v3：指定使用 v3 版本

七、备份和恢复

7.1 Harbor备份脚本

# 创建备份脚本
cat > /opt/harbor/backup.sh << EOF
#!/bin/bash
BACKUP_DIR="/data/harbor/backup"
DATE=\$(date +%Y%m%d_%H%M%S)
mkdir -p \$BACKUP_DIR/\$DATE

# 备份数据库
docker exec -t harbor-db pg_dump -U postgres -d registry > \$BACKUP_DIR/\$DATE/registry.sql

# 备份配置文件
cp /opt/harbor/harbor.yml \$BACKUP_DIR/\$DATE/

# 备份证书
cp -r /opt/harbor/certs \$BACKUP_DIR/\$DATE/

# 压缩备份
tar -czf \$BACKUP_DIR/harbor_backup_\$DATE.tar.gz -C \$BACKUP_DIR \$DATE

# 清理旧备份（保留7天）
find \$BACKUP_DIR -name "harbor_backup_*.tar.gz" -mtime +7 -delete

echo "Backup completed: harbor_backup_\$DATE.tar.gz"
EOF
chmod +x /opt/harbor/backup.sh

# 配置定时任务
crontab -e
# 每天凌晨2点备份
0 2 * * * /opt/harbor/backup.sh >> /var/log/harbor_backup.log 2>&1

7.2 etcd备份

# 在Master1节点执行
ETCDCTL_API=3 etcdctl snapshot save /opt/etcd/backup/etcd-snapshot-$(date +%Y%m%d_%H%M%S).db \
--endpoints=https://192.168.100.177:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key

# 配置定时备份
crontab -e
# 每天凌晨3点备份etcd
0 3 * * * ETCDCTL_API=3 etcdctl snapshot save /opt/etcd/backup/etcd-snapshot-$(date +\%Y\%m\%d_\%H\%M\%S).db --endpoints=https://192.168.100.177:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key

---

八、验证与测试

8.1 集群状态检查

# 检查节点状态
kubectl get nodes -o wide

# 检查系统组件
kubectl get pods -n kube-system -o wide

# 检查etcd集群
ETCDCTL_API=3 etcdctl member list \
  --endpoints=https://192.168.100.177:2379,https://192.168.100.178:2379,https://192.168.100.179:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# 检查HAProxy状态
systemctl status haproxy

# 检查Keepalived状态
systemctl status keepalived
ip addr show ens33

8.2 Harbor功能测试

# 登录Harbor
docker login 192.168.100.181 -u admin -p Harbor12345

# 推送测试镜像
docker pull nginx:latest
docker tag nginx:latest 192.168.100.181/library/nginx:latest
docker push 192.168.100.181/library/nginx:latest

# 在K8s中部署
cat > test-harbor.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-harbor
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: test-harbor
  template:
    metadata:
      labels:
        app: test-harbor
    spec:
      containers:
      - name: nginx
        image: 192.168.100.181/library/nginx:latest
        ports:
        - containerPort: 80
      imagePullSecrets:
      - name: harbor-secret
EOF

kubectl apply -f test-harbor.yaml
kubectl get pods

8.3 Kuboard功能测试

# 获取Kuboard登录Token
kubectl -n kuboard get secret kuboard-user -o go-template='{{.data.token}}' | base64 -d

# 访问Kuboard UI
# 浏览器打开 http://192.168.100.177:30080
# 使用Token登录，验证能否查看集群资源

8.4 高可用测试

# 测试Master故障转移
# 1. 停止Master1的kubelet
systemctl stop kubelet  # 在Master1执行

# 2. 验证集群仍可用
kubectl get nodes  # 在任意Master执行

# 3. 测试VIP漂移
# 停止Master1的keepalived
systemctl stop keepalived  # 在Master1执行

# 4. 验证VIP已漂移到Master2
ip addr show ens33 | grep 192.168.100.177  # 在Master2执行

# 5. 恢复服务
systemctl start keepalived  # 在Master1执行
systemctl start kubelet  # 在Master1执行

---

九、运维手册

9.1 常用命令速查

# K8s集群管理
kubectl get nodes                    # 查看节点
kubectl get pods -A                  # 查看所有Pod
kubectl describe node <node-name>    # 节点详情
kubectl top nodes                    # 节点资源使用
kubectl top pods -A                  # Pod资源使用

# 证书管理
kubeadm certs check-expiration       # 检查证书过期时间
kubeadm certs renew all              # 更新所有证书

# Harbor管理
docker-compose ps                    # 查看Harbor服务状态
docker-compose logs -f               # 查看Harbor日志
docker-compose restart               # 重启Harbor

# Kuboard管理
kubectl get pods -n kuboard          # 查看Kuboard状态
kubectl logs -f -n kuboard deployment/kuboard  # 查看Kuboard日志


# 负载均衡管理
systemctl status haproxy             # HAProxy状态
systemctl status keepalived          # Keepalived状态
ip addr show ens33                    # 查看VIP

9.2 故障排查流程

问题	排查步骤	解决方案
Pod无法启动	1. kubectl describe pod 2. kubectl logs 3. 检查镜像拉取	检查Harbor连接、imagePullSecrets
节点NotReady	1. systemctl status kubelet 2. journalctl -u kubelet 3. 检查网络插件	重启kubelet、检查CNI
API无法访问	1. 检查VIP 2. 检查HAProxy 3. 检查Master节点	恢复Keepalived、重启HAProxy
Harbor无法访问	1. docker-compose ps 2. 检查证书 3. 检查端口	重启Harbor、更新证书
Kuboard无法访问	1. kubectl get pods -n kuboard 2. kubectl logs -n kuboard deployment/kuboard 3. 检查Service NodePort	重启Pod、检查网络策略
etcd故障	1. etcdctl member list 2. 检查etcd日志 3. 从备份恢复	恢复etcd快照

9.3 定期维护任务

任务	频率	说明
证书检查	每月	检查K8s和Harbor证书过期时间
备份验证	每周	验证备份文件可恢复
安全更新	每月	更新系统和安全补丁
日志清理	每周	清理旧日志文件
镜像清理	每月	清理Harbor旧镜像
资源审计	每月	审计集群资源使用情况

---

十、性能优化建议

10.1 内核参数优化

# 生产环境建议添加
cat >> /etc/sysctl.d/k8s.conf << EOF
# 网络连接优化
net.core.somaxconn=65535
net.core.netdev_max_backlog=65535
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_tw_reuse=1

# 文件描述符
fs.nr_open=1048576
EOF
sysctl --system