kubeasz部署k8s集群保姆级教程
·
docker以及harbor安装包
通过网盘分享的文件：docker
链接: https://pan.baidu.com/s/1ReA9jUj_K83sWBKQjWg5Uw?pwd=6666 提取码: 6666
–来自百度网盘超级会员v4的分享
一：k8s集群环境搭建
k8s集群环境主要是kubernetes管理端服务(kube-apiserver、kube-controller-manager、kube-scheduler)的高可用实现，以及node节点上的(kubelet、kube-proxy)客户端服务的部署。
Kubernetes设计架构：
https://www.kubernetes.org.cn/kubernetes%E8%AE%BE%E8%AE%A1%E6%9E%B6%E6%9E%84
1.1：k8s高可用集群环境规划信息
按照实际环境需求，进行规划与部署相应的单master或者多master的高可用k8s运行环境。
1.1.1：单master环境
见 kubeadm 安装k8s
1.1.2：多master环境（架构图）
1.1.3：服务器统计：
| 类型 | 服务器IP地址 | 备注 |
|---|---|---|
| Ansible(2台) | 192.168.60.125/126 | K8S集群部署服务器，可以和其他服务器混用 |
| K8S Master(3台) | 192.168.60.125/126/127 | K8s控制端，通过一个VIP做主备高可用 |
| Harbor(2台) | 192.168.60.131/132 | 高可用镜像服务器 |
| Etcd(最少3台) | 192.168.60.119/120/121 | 保存k8s集群数据的服务器 |
| Hproxy(2台) | 192.168.60.129/130 | 高可用etcd代理服务器 |
| Node节点(2-N台) | 192.168.60.122/123/124/xxx | 真正运行容器的服务器，高可用环境至少两台 |
1.2：服务器准备：
服务器可以是私有云的虚拟机或物理机，也可以是公有云环境的虚拟机环境，如果是公司托管的IDC环境，可以直接将harbor和node节点部署在物理机环境，master节点、etcd、负载均衡等可以是虚拟机。
服务器信息统计
| 类型 | 服务器IP | 主机名 | VIP |
|---|---|---|---|
| K8S Master1 | 192.168.60.125 | k8s-master1 | 192.168.60.188 |
| K8S Master2 | 192.168.60.126 | k8s-master2 | 192.168.60.188 |
| K8S Master3 | 192.168.60.127 | k8s-master3 | 192.168.60.188 |
| Harbor1 | 192.168.60.131 | k8s-harbor1 | |
| Harbor2 | 192.168.60.132 | k8s-harbor2 | |
| etcd节点1 | 192.168.60.119 | k8s-etcd1 | |
| etcd节点2 | 192.168.60.120 | k8s-etcd2 | |
| etcd节点3 | 192.168.60.121 | k8s-etcd3 | |
| Haproxy1 | 192.168.60.129 | k8s-ha1 | |
| Haproxy2 | 192.168.60.130 | k8s-ha2 | |
| Node节点1 | 192.168.60.122 | k8s-node1 | |
| Node节点2 | 192.168.60.123 | k8s-node2 | |
| Node节点3 | 192.168.60.124 | k8s-node3 |
1.3：k8s集群软件清单：
见当前目录下Excel文件 kubernetes软件清单
API端口：
端口: 192.168.60.188:6443 #需要配置在负载均衡上实现反向代理
操作系统: centos7
k8s版本: v1.23.1
calico: v3.19.3
1.4：基础环境准备：
系统主机名配置、IP配置、系统参数优化，以及依赖的负载均衡和Harbor部署
1.4.1：系统配置：
主机名、iptables、防火墙、内核参数与资源限制等系统配置 略
所有服务器时间同步：
步骤 1：安装时间同步工具（所有节点执行）
CentOS/RHEL 系统推荐用 chrony（比 ntp 更轻量、适配性更好）：
# 安装 chrony
yum -y install chrony
# 停止并禁用旧的 ntp 服务（如果有）
systemctl stop ntpd && systemctl disable ntpd
步骤 2：配置 chrony 同步源（所有节点执行）
编辑 /etc/chrony.conf，优先使用国内公共 NTP 服务器，或内网 NTP 服务器：
# 备份原配置
cp /etc/chrony.conf /etc/chrony.conf.bak
# 覆盖配置（用国内阿里云/腾讯云 NTP 源）
cat > /etc/chrony.conf <<EOF
# 国内公共 NTP 服务器
server ntp.aliyun.com iburst
server time1.cloud.tencent.com iburst
server ntp1.ntsc.ac.cn iburst
# 允许本地网络同步（仅 master/etcd 节点需要，node 节点可注释）
allow 192.168.60.0/24
# 同步硬件时钟
rtcsync
# 本地时间漂移补偿
makestep 1.0 3
# 日志文件
logdir /var/log/chrony
EOF
步骤 3：启动并验证时间同步（所有节点执行）
# 启动 chrony 并设置开机自启
systemctl start chronyd && systemctl enable chronyd
# 强制同步时间（立即生效）
chronyc -a makestep
# 验证同步状态（看到 "^*" 开头的服务器即为同步成功）
chronyc sources -v
# 再次检查时间（所有节点时间应完全一致）
date
1.4.2：高可用负载均衡：
k8s高可用反向代理
参见博客 http://blogs.studylinux.net/?p=4579
1.4.2.1：keepalived：
两台都需要改配置文件，这里只改一台演示
[root@k8s-haproxy1 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.60.188/24 dev ens33 label ens33:0
192.168.60.189/24 dev ens33 label ens33:1
192.168.60.190/24 dev ens33 label ens33:2
}
}
1.4.2.2：haproxy：
配置haproxy：vim /etc/haproxy/haproxy.cfg
listen k8s-api-6443
bind 192.168.60.188:6443
mode tcp
server master1 192.168.60.125:6443 check inter 3s fall 3 rise 1
server master2 192.168.60.126:6443 check inter 3s fall 3 rise 1
server master3 192.168.60.127:6443 check inter 3s fall 3 rise 1
此时会发现两台haproxy只能启动一台，另一台启动会失败，这是正常现象因为正常情况下keepalived的vip只会绑定在一台haproxy上面，另一台没有vip的haproxy就会启动失败，此时我们需要修改系统内核运行参数：
net.ipv4.ip_nonlocal_bind = 1
重启haproxy：
1.4.3：安装docker：
各master及node节点安装docker。
root@k8s-node3:/usr/local/src# pwd
/usr/local/src
root@k8s-node3:/usr/local/src# tar xvf docker-19.03.15-binary-install.tar.gz
root@k8s-node3:/usr/local/src# bash docker-install.sh
root@k8s-node3:/usr/local/src# reboot
1.4.4：Harbor之https：
内部镜像将统一保存在内部Harbor服务器，不再通过互联网在线下载。
root@k8s-harbor1:/usr/local/src# pwd
/usr/local/src
root@k8s-harbor1:/usr/local/src# tar xf docker-19.03.15-binary-install.tar.gz
root@k8s-harbor1:/usr/local/src# bash docker-install.sh
root@k8s-harbor1:/usr/local/src# mkdir /app
root@k8s-harbor1:/usr/local/src# cd /apps/
root@k8s-harbor1:/apps# tar xvf harbor-offline-installer-v2.2.1.tgz
root@k8s-harbor1:/apps# cd harbor/
root@k8s-harbor1:/apps/harbor# mkdir certs
#生成ca
[root@k8s-harbor1 certs]# openssl genrsa -out ./harbor-ca.key
Generating RSA private key, 2048 bit long modulus
...................................................................+++
....................+++
e is 65537 (0x10001)
[root@k8s-harbor1 certs]# openssl req -x509 -new -nodes -key ./harbor-ca.key -subj "/CN=harbor.k8s.local" -days 7120 -out ./harbor-ca.crt
[root@k8s-harbor1 certs]# ll
总用量 8
-rw-r--r-- 1 root root 1115 3月 6 10:21 harbor-ca.crt
-rw-r--r-- 1 root root 1675 3月 6 10:21 harbor-ca.key
root@k8s-harbor1:/apps/harbor/certs# cd /apps/harbor/
root@k8s-harbor1:/apps/harbor# cp harbor.yml.tmpl harbor.yml
root@k8s-harbor1:/apps/harbor# vim harbor.yml
root@k8s-harbor1:/apps/harbor# grep -v "#" harbor.yml | grep -v "^#"
hostname: harbor.k8s.local
http:
port: 80
https:
port: 443
certificate: /apps/harbor/certs/harbor-ca.crt
private_key: /apps/harbor/certs/harbor-ca.key
harbor_admin_password: 1
root@harbor1:/apps/harbor# ./install.sh --help
root@harbor1:/apps/harbor# ./install.sh --with-trivy
1.4.4.1: 部署节点同步harbor crt证书:
master1:~# mkdir /etc/docker/certs.d/harbor.k8s.local -p
root@harbor1:/apps/harbor/certs# pwd
/apps/harbor/certs
root@harbor1:/apps/harbor/certs# scp /apps/harbor/certs/harbor-ca.crt \
192.168.60.125:/etc/docker/certs.d/harbor.k8s.local
master1:~# vim /etc/hosts #添加host文件解析
192.168.60.131 harbor.k8s.local
master1:~# systemctl restart docker #重启docker
1.4.4.2: node登录harbor:
root@k8s-master1:~# docker login harbor.k8s.local
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
Harbor 管理界面截图：
地址：harbor.k8s.local/harbor/projects
左侧菜单：项目、日志、系统管理（用户管理、机器人账户、仓库管理等）
默认项目：library
1.4.4.3: 测试push镜像到harbor:
root@k8s-master1:~# docker pull alpine
[root@k8s-master1 harbor.k8s.local]# docker tag alpine:latest harbor.k8s.local/library/alpine:latest
[root@k8s-master1 harbor.k8s.local]# docker push harbor.k8s.local/library/alpine:latest
The push refers to repository [harbor.k8s.local/library/alpine]
989e799e6349: Pushed
latest: digest: sha256:8637808eae0e7c2a2875ab58a1c7f74999823afdb67b4570e3bf778eeccc119b size: 527
Harbor 界面中可看到该镜像信息：
1.5：手动二进制部署：
见 k8s 1.11 ubuntu1804部署文档
1.6：ansible部署：
1.6.1：基础环境准备：
各节点安装python 2.7
~# apt-get install python2.7 -y
~# ln -s /usr/bin/python2.7 /usr/bin/python
部署节点安装ansible
root@k8s-master1:/usr/local/src# apt install -y ansible
root@k8s-master1:/usr/local/src# ansible --version
ansible [core 2.11.1]
config file = None
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
ansible collection location =
/root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.6.9 (default, Jan 26 2021, 15:33:00) [GCC 8.4.0]
jinja version = 3.0.1
libyaml = True
# ssh-keygen #生成密钥对
# apt-get install sshpass #安装sshpass命令用于同步公钥到各k8s服务器
分发公钥脚本：
#!/bin/bash
# 批量推送SSH公钥到目标节点
# 配置项（建议根据实际情况修改）
USER="root" # SSH登录用户名
PASSWORD="123456" # 登录密码（生产环境建议改为从环境变量传入）
# 目标主机列表（每行一个IP，空行会自动跳过）
IP_LIST="
192.168.60.125
192.168.60.126
192.168.60.127
192.168.60.131
192.168.60.132
192.168.60.119
192.168.60.120
192.168.60.121
192.168.60.129
192.168.60.130
192.168.60.122
192.168.60.123
192.168.60.124
"
# 检查sshpass是否安装
if ! command -v sshpass &> /dev/null; then
echo "错误：未安装sshpass，请先执行 yum install -y sshpass 或 apt install -y sshpass"
exit 1
fi
# 循环推送公钥
for node in ${IP_LIST}; do
# 跳过空行
if [ -z "${node}" ]; then
continue
fi
echo "==== 开始处理节点 ${node} ===="
# 核心命令：修正参数顺序，指定用户名，关闭主机密钥检查
sshpass -p "${PASSWORD}" ssh-copy-id -o StrictHostKeyChecking=no ${USER}@${node}
# 判断执行结果
if [ $? -eq 0 ]; then
echo -e "${node} 公钥推送成功 ✅\n"
else
echo -e "${node} 公钥推送失败 ❌\n"
fi
done
echo "==== 所有节点处理完成 ===="
同步docker证书脚本：
#!/bin/bash
set -euo pipefail # 开启严格模式：遇到错误立即退出、未定义变量报错、管道失败则整体失败
# 配置项（建议放到脚本顶部，方便修改）
SSH_USER="root" # 指定SSH连接用户，避免权限问题
SSH_PASS="1" # 建议生产环境通过环境变量传入，而非硬编码
HARBOR_DOMAIN="harbor.k8s.local"
HARBOR_IP="192.168.60.131"
CERT_PATH="/apps/harbor/certs/harbor-ca.crt"
Target_PATH="/etc/docker/certs.d/harbor.k8s.local/"
DOCKER_AUTH_DIR="/root/.docker"
# 目标主机列表（数组形式，更规范）
declare -a NODE_IPS=(
"192.168.60.125"
"192.168.60.126"
"192.168.60.127"
"192.168.60.122"
"192.168.60.123"
"192.168.60.124"
)
# 定义错误处理函数
error_exit() {
echo "[$(date +'%Y-%m-%d %H:%M:%S')] ERROR: $1" >&2
exit 1
}
# 遍历所有节点执行配置
for node in "${NODE_IPS[@]}"; do
echo -e "\n===== 开始配置节点: ${node} ====="
# 1. 推送SSH密钥（关闭严格主机密钥检查）
echo "步骤1: 推送SSH密钥到${node}"
sshpass -p "${SSH_PASS}" ssh-copy-id -i /root/.ssh/id_rsa.pub -o StrictHostKeyChecking=no "${SSH_USER}@${node}"
if [ $? -eq 0 ]; then
echo "✅ ${node} SSH密钥推送成功"
else
error_exit "${node} SSH密钥推送失败，终止配置"
fi
# 2. 创建Harbor证书目录
echo "步骤2: 创建Harbor证书目录"
ssh "${SSH_USER}@${node}" "mkdir -p $(dirname ${Target_PATH})"
if [ $? -eq 0 ]; then
echo "✅ ${node} Harbor证书目录创建成功"
else
error_exit "${node} Harbor证书目录创建失败"
fi
# 3. 拷贝Harbor证书
echo "步骤3: 拷贝Harbor证书到${node}"
scp "${CERT_PATH}" "${SSH_USER}@${node}:${Target_PATH}"
if [ $? -eq 0 ]; then
echo "✅ ${node} Harbor证书拷贝成功"
else
error_exit "${node} Harbor证书拷贝失败"
fi
# 4. 配置hosts映射（转义引号，避免冲突）
echo "步骤4: 配置${node}的hosts文件"
ssh "${SSH_USER}@${node}" "echo '${HARBOR_IP} ${HARBOR_DOMAIN}' >> /etc/hosts"
if [ $? -eq 0 ]; then
echo "✅ ${node} hosts配置成功"
else
error_exit "${node} hosts配置失败"
fi
echo -e "===== 节点${node}配置完成 =====\n"
done
echo "所有节点配置全部完成！"
执行脚本同步：
k8s-master1:~# bash scp.sh
root@s2:~# vim ~/.vimrc #取消vim 自动缩进功能
set paste
1.6.2：下载项目及组件：
root@k8s-master1:~# export release=3.2.0
root@k8s-master1:~# wget https://github.com/easzlab/kubeasz/releases/download/${release}/ezdown
root@k8s-master1:~# chmod a+x ezdown
root@k8s-master1:~# ./ezdown -D
[root@k8s-master1 harbor.k8s.local]# ll /etc/kubeasz/down/
总用量 1192036
-rw------- 1 root root 384354816 3月 6 11:03 calico_v3.19.3.tar
-rw------- 1 root root 46967296 3月 6 11:04 coredns_1.8.6.tar
-rw------- 1 root root 224458240 3月 6 11:05 dashboard_v2.4.0.tar
-rw-r--r-- 1 root root 62436240 7月 30 2025 docker-19.03.15.tgz
-rw------- 1 root root 70554112 3月 6 11:05 flannel_v0.15.1.tar
-rw------- 1 root root 106171392 3月 6 11:04 k8s-dns-node-cache_1.21.1.tar
-rw------- 1 root root 179055104 3月 6 11:08 kubeasz_3.2.0.tar
-rw------- 1 root root 34463744 3月 6 11:06 metrics-scraper_v1.0.7.tar
-rw------- 1 root root 65683968 3月 6 11:06 metrics-server_v0.5.2.tar
-rw------- 1 root root 45085184 3月 6 11:08 nfs-provisioner_v4.0.2.tar
-rw------- 1 root root 692736 3月 6 11:07 pause_3.6.tar
-rw------- 1 root root 692736 3月 6 11:07 pause.tar
1.6.3：生成hosts文件：
root@k8s-master1:~# cd /etc/kubeasz/
root@k8s-master1:/etc/kubeasz# pwd
/etc/kubeasz
root@k8s-master1:/etc/kubeasz# ./ezctl new k8s-cluster1
2021-06-10 11:22:54 DEBUG generate custom cluster files in
/etc/kubeasz/clusters/k8s-cluster1
2021-06-10 11:22:54 DEBUG set version of common plugins
2021-06-10 11:22:54 DEBUG cluster k8s-cluster1: files successfully created.
2021-06-10 11:22:54 INFO next steps 1: to config '/etc/kubeasz/clusters/k8s-cluster1/hosts'
2021-06-10 11:22:54 INFO next steps 2: to config '/etc/kubeasz/clusters/k8s-cluster1/config.yml'
root@k8s-master1:/etc/kubeasz# vim clusters/k8s-cluster1/hosts
root@k8s-master1:/etc/kubeasz# vim clusters/k8s-cluster1/config.yml
1.6.3.1：编辑ansible hosts文件：
指定etcd节点、master节点、node节点、VIP、运行时、网络组建类型、service IP与pod IP范围等配置信息。
root@k8s-master1:/etc/kubeasz/clusters/k8s-cluster1# pwd
/etc/kubeasz/clusters/k8s-cluster1
root@k8s-master1:/etc/kubeasz/clusters/k8s-cluster1# grep -v ^# hosts
[etcd]
192.168.60.119
192.168.60.120
192.168.60.121
[kube_master]
192.168.60.125
192.168.60.126
192.168.60.127
[kube_node]
192.168.60.122
192.168.60.123
192.168.60.124
[harbor]
[ex_lb]
192.168.60.129 LB_ROLE=backup EX_APISERVER_VIP=192.168.60.188 EX_APISERVER_PORT=6443
192.168.60.130 LB_ROLE=master EX_APISERVER_VIP=192.168.60.188 EX_APISERVER_PORT=6443
[chrony]
[all:vars]
SECURE_PORT="6443"
CONTAINER_RUNTIME="docker"
CLUSTER_NETWORK="calico"
PROXY_MODE="ipvs"
SERVICE_CIDR="10.100.0.0/16"
CLUSTER_CIDR="10.200.0.0/16"
NODE_PORT_RANGE="30000-65000"
CLUSTER_DNS_DOMAIN="cluster.local"
bin_dir="/usr/local/bin"
base_dir="/etc/kubeasz"
cluster_dir="{{ base_dir }}/clusters/k8s-cluster1"
ca_dir="/etc/kubernetes/ssl"
[root@k8s-master1 k8s-cluster1]# grep -v ^# config.yml
INSTALL_SOURCE: "online"
OS_HARDEN: false
ntp_servers:
- "ntp1.aliyun.com"
- "time1.cloud.tencent.com"
- "0.cn.pool.ntp.org"
local_network: "0.0.0.0/0"
CA_EXPIRY: "876000h"
CERT_EXPIRY: "876000h"
CLUSTER_NAME: "cluster1"
CONTEXT_NAME: "context-{{ CLUSTER_NAME }}"
K8S_VER: "1.23.1"
ETCD_DATA_DIR: "/var/lib/etcd"
ETCD_WAL_DIR: ""
ENABLE_MIRROR_REGISTRY: true
SANDBOX_IMAGE: "easzlab/pause:3.6"
CONTAINERD_STORAGE_DIR: "/var/lib/containerd"
DOCKER_STORAGE_DIR: "/var/lib/docker"
ENABLE_REMOTE_API: false
INSECURE_REG: '["127.0.0.1/8","192.168.60.131","192.168.60.132"]'
MASTER_CERT_HOSTS:
- "10.1.1.1"
- "k8s.test.io"
#- "www.test.com"
NODE_CIDR_LEN: 24
KUBELET_ROOT_DIR: "/var/lib/kubelet"
MAX_PODS: 500
KUBE_RESERVED_ENABLED: "no"
SYS_RESERVED_ENABLED: "no"
BALANCE_ALG: "roundrobin"
FLANNEL_BACKEND: "vxlan"
DIRECT_ROUTING: false
flannelVer: "v0.15.1"
flanneld_image: "easzlab/flannel:{{ flannelVer }}"
flannel_offline: "flannel_{{ flannelVer }}.tar"
CALICO_IPV4POOL_IPIP: "Always"
IP_AUTODETECTION_METHOD: "can-reach={{ groups['kube_master'][0] }}"
CALICO_NETWORKING_BACKEND: "brid"
calico_ver: "v3.19.3"
calico_ver_main: "{{ calico_ver.split('.')[0] }}.{{ calico_ver.split('.')[1] }}"
calico_offline: "calico_{{ calico_ver }}.tar"
ETCD_CLUSTER_SIZE: 1
cilium_ver: "v1.4.1"
cilium_offline: "cilium_{{ cilium_ver }}.tar"
OVN_DB_NODE: "{{ groups['kube_master'][0] }}"
kube_ovn_ver: "v1.5.3"
kube_ovn_offline: "kube_ovn_{{ kube_ovn_ver }}.tar"
OVERLAY_TYPE: "full"
FIREWALL_ENABLE: "true"
kube_router_ver: "v0.3.1"
busybox_ver: "1.28.4"
kuberouter_offline: "kube-router_{{ kube_router_ver }}.tar"
busybox_offline: "busybox_{{ busybox_ver }}.tar"
dns_install: "no"
corednsVer: "1.8.6"
ENABLE_LOCAL_DNS_CACHE: true
dnsNodeCacheVer: "1.21.1"
LOCAL_DNS_CACHE: "169.254.20.10"
metricsserver_install: "no"
metricsVer: "v0.5.2"
dashboard_install: "no"
dashboardVer: "v2.4.0"
dashboardMetricsScraperVer: "v1.0.7"
ingress_install: "no"
ingress_backend: "traefik"
traefik_chart_ver: "10.3.0"
prom_install: "no"
prom_namespace: "monitor"
prom_chart_ver: "12.10.6"
nfs_provisioner_install: "no"
nfs_provisioner_namespace: "kube-system"
nfs_provisioner_ver: "v4.0.2"
nfs_storage_class: "managed-nfs-storage"
nfs_server: "192.168.1.10"
nfs_path: "/data/nfs"
HARBOR_VER: "v2.1.3"
HARBOR_DOMAIN: "harbor.k8s.local"
HARBOR_TLS_PORT: 8443
HARBOR_SELF_SIGNED_CERT: true
HARBOR_WITH_NOTARY: false
HARBOR_WITH_TRIVY: false
HARBOR_WITH_CLAIR: false
HARBOR_WITH_CHARTMUSEUM: true
1.6.4：部署k8s集群：
通过ansible脚本初始化环境及部署k8s高可用集群
1.6.4.1：环境初始化
root@k8s-master1:/etc/kubeasz# ./ezctl help setup
Usage: ezctl setup <cluster> <step>
available steps:
01 prepare to prepare CA/certs & kubeconfig & other system settings
02 etcd to setup the etcd cluster
03 container-runtime to setup the container runtime(docker or containerd)
04 kube-master to setup the master nodes
05 kube-node to setup the worker nodes
06 network to setup the network plugin
07 cluster-addon to setup other useful plugins
90 all to run 01~07 all at once
10 ex-lb to install external loadbalance for accessing k8s from outside
11 harbor to install a new harbor server or to integrate with an existed one
examples: ./ezctl setup test-k8s 01 (or ./ezctl setup test-k8s prepare)
./ezctl setup test-k8s 02 (or ./ezctl setup test-k8s etcd)
./ezctl setup test-k8s all
./ezctl setup test-k8s 04 -t restart_master
root@k8s-master1:/etc/kubeasz# ./ezctl setup k8s-cluster1 01 #准备CA和基础系统设置
1.6.4.2：部署etcd集群：
可更改启动脚本路径及版本等自定义配置
root@k8s-master1:/etc/kubeasz# ./ezctl help setup
root@k8s-master1:/etc/kubeasz# ./ezctl setup k8s-cluster1 02 #部署etcd集群
各etcd服务器验证etcd服务：
# 1. 创建脚本
cat > check_etcd_health.sh <<EOF
#!/bin/bash
# 极简版 etcd 集群健康度检测
NODE_IPS="192.168.60.119 192.168.60.120 192.168.60.121"
for ip in \${NODE_IPS}; do
etcdctl endpoint health --endpoints=https://\${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem
done
EOF
# 2. 添加执行权限
chmod +x check_etcd_health.sh
# 3. 执行
./check_etcd_health.sh
[root@k8s-etcd2 ~]# ./check_etcd_health.sh
https://192.168.60.119:2379 is healthy: successfully committed proposal: took = 15.657273ms
https://192.168.60.120:2379 is healthy: successfully committed proposal: took = 11.800273ms
https://192.168.60.121:2379 is healthy: successfully committed proposal: took = 13.692926ms
注：以上返回信息表示etcd集群运行正常，否则异常！
1.6.4.3：部署docker:
master与node节点必须安装docker，docker可以自行yum或者安装包安装也可以使用二进制安装，因此此步骤为可选步骤！
基础容器镜像配置
# 查找基础镜像配置
[root@k8s-master1 kubeasz]# grep SANDBOX_IMAGE ./clusters/* -R
./clusters/k8s-cluster1/config.yml:SANDBOX_IMAGE: "easzlab/pause:3.6"
[root@k8s-master1 kubeasz]# docker pull easzlab/pause:3.6
3.6: Pulling from easzlab/pause
Digest: sha256:74bf6fc6be13c4ec53a86a5acf9fdbc6787b176db0693659ad6ac89f115e182c
Status: Image is up to date for easzlab/pause:3.6
docker.io/easzlab/pause:3.6
[root@k8s-master1 kubeasz]# docker tag easzlab/pause:3.6 harbor.k8s.local/baseimages/easzlab/pause:3.6
[root@k8s-master1 kubeasz]# docker push harbor.k8s.local/baseimages/easzlab/pause:3.6
The push refers to repository [harbor.k8s.local/baseimages/easzlab/pause]
1021ef88c797: Pushed
3.6: digest: sha256:74bf6fc6be13c4ec53a86a5acf9fdbc6787b176db0693659ad6ac89f115e182c size: 526
# 编辑配置文件修改基础镜像地址
root@k8s-master1:/etc/kubeasz# vim ./clusters/k8s-cluster1/config.yml
48 # [containerd]基础容器镜像
49 SANDBOX_IMAGE: "harbor.k8s.local/baseimages/easzlab/pause:3.6"
# 执行容器运行时部署
root@k8s-master1:/etc/kubeasz# ./ezctl setup k8s-cluster1 03
node节点验证docker
root@k8s-node2:~# docker version
Client: Docker Engine - Community
Version: 19.03.15
API version: 1.40
Go version: go1.13.15
Git commit: 99e3ed8
Built: Sat Jan 30 03:11:43 2021
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.15
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 99e3ed8
Built: Sat Jan 30 03:18:13 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.3.9
GitCommit: ea765aba0d05254012b0b9e595e995c09186427f
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
1.6.4.4：部署master：
可选更改启动脚本参数及路径等自定义功能
root@k8s-master1:/etc/kubeasz# ./ezctl help setup
root@k8s-master1:/etc/kubeasz# ./ezctl setup k8s-cluster1 04
验证服务器：
[root@k8s-master1 kubeasz]# kubectl get no
NAME STATUS ROLES AGE VERSION
192.168.60.125 Ready,SchedulingDisabled master 19m v1.23.1
192.168.60.126 Ready,SchedulingDisabled master 19m v1.23.1
192.168.60.127 Ready,SchedulingDisabled master 19m v1.23.1
1.6.4.5：部署node：
node节点必须安装docker
root@k8s-master1:/etc/kubeasz# ./ezctl setup k8s-cluster1 05
验证服务器：
[root@k8s-master1 kubeasz]# kubectl get no
NAME STATUS ROLES AGE VERSION
192.168.60.122 Ready node 63s v1.23.1
192.168.60.123 Ready node 65s v1.23.1
192.168.60.124 Ready node 63s v1.23.1
192.168.60.125 Ready,SchedulingDisabled master 22m v1.23.1
192.168.60.126 Ready,SchedulingDisabled master 22m v1.23.1
192.168.60.127 Ready,SchedulingDisabled master 22m v1.23.1
1.6.4.5：部署网络服务calico
可选更改calico服务启动脚本路径，csr证书信息
root@k8s-master1:/etc/kubeasz# vim ./clusters/k8s-cluster1/config.yml
# ------------------------------------------- calico
# [calico]设置 CALICO_IPV4POOL_IPIP=“off”,可以提高网络性能，条件限制详见 docs/setup/calico.md
CALICO_IPV4POOL_IPIP: "Always"
# [calico]设置 calico-node使用的host IP，bgp邻居通过该地址建立，可手工指定也可以自动发现
IP_AUTODETECTION_METHOD: "can-reach={{ groups['kube_master'][0] }}"
# [calico]设置calico 网络 backend: brid, vxlan, none
CALICO_NETWORKING_BACKEND: "brid"
# [calico]更新支持calico 版本: [v3.3.x] [v3.4.x] [v3.8.x] [v3.15.x]
calico_ver: "v3.19.3"
# [calico]calico 主版本
calico_ver_main: "{{ calico_ver.split('.')[0] }}.{{ calico_ver.split('.')[1] }}"
# [calico]离线镜像tar包
calico_offline: "calico_{{ calico_ver }}.tar"
[root@k8s-master1 kubeasz]# grep image roles/calico/templates/calico-v3.19.yaml.j2
image: docker.io/calico/cni:{{ calico_ver }}
image: docker.io/calico/pod2daemon-flexvol:{{ calico_ver }}
image: docker.io/calico/node:{{ calico_ver }}
image: docker.io/calico/kube-controllers:{{ calico_ver }}
root@k8s-master1:/etc/kubeasz# docker pull docker.io/calico/cni:v3.19.3
root@k8s-master1:/etc/kubeasz# docker tag calico/cni:v3.19.3 harbor.k8s.local/baseimages/calico/cni:v3.19.3
root@k8s-master1:/etc/kubeasz# docker push harbor.k8s.local/baseimages/calico/cni:v3.19.3
root@k8s-master1:/etc/kubeasz# docker pull calico/pod2daemon-flexvol:v3.19.3
root@k8s-master1:/etc/kubeasz# docker tag docker.io/calico/pod2daemon-flexvol:v3.19.3 harbor.k8s.local/baseimages/calico-pod2daemon-flexvol:v3.19.3
root@k8s-master1:/etc/kubeasz# docker push harbor.k8s.local/baseimages/calico-pod2daemon-flexvol:v3.19.3
root@k8s-master1:/etc/kubeasz# docker pull calico/node:v3.19.3
root@k8s-master1:/etc/kubeasz# docker tag calico/node:v3.19.3 harbor.k8s.local/baseimages/calico-node:v3.19.3
root@k8s-master1:/etc/kubeasz# docker push harbor.k8s.local/baseimages/calico-node:v3.19.3
root@k8s-master1:/etc/kubeasz# docker pull calico/kube-controllers:v3.19.3
root@k8s-master1:/etc/kubeasz# docker tag calico/kube-controllers:v3.19.3 harbor.k8s.local/baseimages/calico-kube-controllers:v3.19.3
root@k8s-master1:/etc/kubeasz# docker push harbor.k8s.local/baseimages/calico-kube-controllers:v3.19.3
修改镜像地址：
root@k8s-master1:/etc/kubeasz# vim roles/calico/templates/calico-v3.15.yaml.j2
root@k8s-master1:/etc/kubeasz# grep images roles/calico/templates/calico-v3.15.yaml.j2
image: harbor.k8s.local/baseimages/calico-cni:v3.19.3
image: harbor.k8s.local/baseimages/calico-pod2daemon-flexvol:v3.19.3
image: harbor.k8s.local/baseimages/calico-node:v3.19.3
image: harbor.k8s.local/baseimages/calico-kube-controllers:v3.19.3
root@k8s-master1:/etc/kubeasz# ./ezctl help setup
root@k8s-master1:/etc/kubeasz# ./ezctl setup k8s-cluster1 06
验证calico:
[root@k8s-master1 kubeasz]# calicoctl node status
Calico process is running.
IPv4 BGP status
+----------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+----------------+-------------------+-------+----------+-------------+
| 192.168.60.126 | node-to-node mesh | up | 09:56:52 | Established |
| 192.168.60.127 | node-to-node mesh | up | 09:56:53 | Established |
| 192.168.60.122 | node-to-node mesh | up | 09:56:54 | Established |
| 192.168.60.123 | node-to-node mesh | up | 09:57:10 | Established |
| 192.168.60.124 | node-to-node mesh | up | 09:56:54 | Established |
+----------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
1.6.6：验证网络：
# kubectl run net-test1 --image=alpine sleep 360000 #创建pod测试夸主机网络通信是否正常
# kubectl run net-test2 --image=alpine sleep 360000
[root@k8s-master1 kubeasz]# kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
net-test1 1/1 Running 0 71s 10.200.107.193 192.168.60.124 <none> <none>
net-test2 1/1 Running 0 66s 10.200.169.129 192.168.60.123 <none> <none>
[root@k8s-master1 kubeasz]# kubectl exec -it net-test1 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # ping 10.200.169.129
PING 10.200.169.129 (10.200.169.129): 56 data bytes
64 bytes from 10.200.169.129: seq=0 ttl=62 time=1.282 ms
64 bytes from 10.200.169.129: seq=1 ttl=62 time=0.494 ms
^C
--- 10.200.169.129 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.494/0.888/1.282 ms
/ #
1.6.5：集群管理：
集群管理主要是添加master、添加node、删除master与删除node等节点管理及监控
当前集群状态：
[root@k8s-master1 kubeasz]# kubectl get no
NAME STATUS ROLES AGE VERSION
192.168.60.122 Ready node 44m v1.23.1
192.168.60.123 Ready node 44m v1.23.1
192.168.60.124 Ready node 44m v1.23.1
192.168.60.125 Ready,SchedulingDisabled master 65m v1.23.1
192.168.60.126 Ready,SchedulingDisabled master 65m v1.23.1
192.168.60.127 Ready,SchedulingDisabled master 65m v1.23.1
1.6.5.1: 添加node节点:
root@k8s-master1:/etc/kubeasz# ./ezctl help
root@k8s-master1:/etc/kubeasz# ./ezctl add-node k8s-cluster1 192.168.60.128
TASK [calico : 轮询等待calico-node 运行，视下载镜像速度而定] *******************
changed: [192.168.60.128]
TASK [推送cluster-addon的离线镜像包] ****************************************
changed: [192.168.60.128] => (item=/etc/kubeasz/download/coredns_1.8.0.tar)
changed: [192.168.60.128] => (item=/etc/kubeasz/download/dashboard_v2.2.0.tar)
changed: [192.168.60.128] => (item=/etc/kubeasz/download/metrics-scraper_v1.0.6.tar)
changed: [192.168.60.128] => (item=/etc/kubeasz/download/metrics-server_v0.3.6.tar)
TASK [导入离线镜像（若执行失败，可忽略）] ************************************
changed: [192.168.60.128]
PLAY RECAP *********************************************************************
192.168.60.128 : ok=98 changed=84 unreachable=0 failed=0 skipped=183 rescued=0 ignored=1
验证节点：
root@k8s-master1:/etc/kubeasz# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.60.122 Ready node 44m v1.23.1
192.168.60.123 Ready node 44m v1.23.1
192.168.60.124 Ready node 44m v1.23.1
192.168.60.128 Ready node 44m v1.23.1
192.168.60.125 Ready,SchedulingDisabled master 65m v1.23.1
192.168.60.126 Ready,SchedulingDisabled master 65m v1.23.1
192.168.60.127 Ready,SchedulingDisabled master 65m v1.23.1
1.6.5.2: 添加master节点:
root@k8s-master1:/etc/kubeasz# ./ezctl help
root@k8s-master1:/etc/kubeasz# ./ezctl add-master k8s-cluster1 192.168.60.133
1.6.5.3: 验证当前节点:
root@k8s-master1:/etc/kubeasz# kubectl get node
root@k8s-master1:/etc/kubeasz# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.60.122 Ready node 44m v1.23.1
192.168.60.123 Ready node 44m v1.23.1
192.168.60.124 Ready node 44m v1.23.1
192.168.60.128 Ready node 44m v1.23.1
192.168.60.125 Ready,SchedulingDisabled master 65m v1.23.1
192.168.60.126 Ready,SchedulingDisabled master 65m v1.23.1
192.168.60.127 Ready,SchedulingDisabled master 65m v1.23.1
192.168.60.133 Ready,SchedulingDisabled master 65m v1.23.1
1.6.5.4: 验证calico状态:
（后续可执行 calicoctl node status 验证 BGP 对等连接是否正常）
# 示例验证命令
calicoctl node status
# 预期所有节点 BGP 状态为 up，Established
[root@k8s-master1 kubeasz]# calicoctl node status
Calico process is running.
IPv4 BGP status
+----------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+----------------+-------------------+-------+----------+-------------+
| 192.168.60.126 | node-to-node mesh | up | 09:56:52 | Established |
| 192.168.60.127 | node-to-node mesh | up | 09:56:53 | Established |
| 192.168.60.122 | node-to-node mesh | up | 09:56:54 | Established |
| 192.168.60.123 | node-to-node mesh | up | 09:57:10 | Established |
| 192.168.60.124 | node-to-node mesh | up | 09:56:54 | Established |
| 192.168.60.128 | node-to-node mesh | up | 09:57:54 | Established |
| 192.168.60.133 | node-to-node mesh | up | 09:57:34 | Established |
+----------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
1.6.5.5: 验证node节点路由:
[root@k8s-node1 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.60.2 0.0.0.0 UG 100 0 0 ens33
10.200.36.64 0.0.0.0 255.255.255.192 U 0 0 0 *
10.200.107.192 192.168.60.124 255.255.255.192 UG 0 0 0 tunl0
10.200.135.192 192.168.60.127 255.255.255.192 UG 0 0 0 tunl0
10.200.159.128 192.168.60.125 255.255.255.192 UG 0 0 0 tunl0
10.200.169.128 192.168.60.123 255.255.255.192 UG 0 0 0 tunl0
10.200.224.0 192.168.60.126 255.255.255.192 UG 0 0 0 tunl0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
192.168.60.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
1.7: DNS服务:
目前常用的dns组件有kube-dns和coredns两个，即到目前k8s版本1.17.X都可以使用，kube-dns和coredns用于解析k8s集群中service name所对应得到IP地址。
镜像仓库地址：https://console.cloud.google.com/gcr/images/google-containers/GLOBAL
1.7.1：部署coredns
将kube-dns更换为coredns：
https://github.com/coredns/coredns
coredns 1.2/1.3/1.4/1.5版本
# docker tag gcr.io/google-containers/coredns:1.2.6 harbor.magedu.net/baseimages/coredns:1.2.6
# docker push harbor.magedu.net/baseimages/coredns:1.2.6
1.6及以上新版本
https://github.com/coredns/deployment/tree/master/kubernetes#16部署方式
# unzip deployment-master.zip
# cd deployment-master/kubernetes/
# 生成yaml文件的两种方式，2选1
# cp coredns.yaml.sed coredns-1.8.3.yaml #1.手动编辑yaml文件
# ./deploy.sh > coredns.yaml #2.自动生成yaml文件
[root@k8s-master1 ~]# cat coredns.yaml
# __MACHINE_GENERATED_WARNING__
apiVersion: v1
kind: ServiceAccount
metadata:
name: coredns
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: Reconcile
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: EnsureExists
name: system:coredns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:coredns
subjects:
- kind: ServiceAccount
name: coredns
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: EnsureExists
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . 223.5.5.5 {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "CoreDNS"
spec:
# replicas: not specified here:
# 1. In order to make Addon Manager do not reconcile this replicas parameter.
# 2. Default is 1.
# 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
labels:
k8s-app: kube-dns
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
priorityClassName: system-cluster-critical
serviceAccountName: coredns
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: k8s-app
operator: In
values: ["kube-dns"]
topologyKey: kubernetes.io/hostname
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
nodeSelector:
kubernetes.io/os: linux
containers:
- name: coredns
image: coredns/coredns:1.8.7
imagePullPolicy: IfNotPresent
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 70Mi
args: [ "-conf", "/etc/coredns/Corefile" ]
volumeMounts:
- name: config-volume
mountPath: /etc/coredns
readOnly: true
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
livenessProbe:
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /ready
port: 8181
scheme: HTTP
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- all
readOnlyRootFilesystem: true
dnsPolicy: Default
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile
---
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
annotations:
prometheus.io/port: "9153"
prometheus.io/scrape: "true"
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "CoreDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.100.0.2
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
- name: metrics
port: 9153
protocol: TCP
创建资源并查看：
[root@k8s-master1 ~]# kubectl apply -f coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
[root@k8s-master1 ~]# kubectl get po -n kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-7db6b45f67-lgtc6 1/1 Running 0 16h
1.7.2：域名解析测试：
[root@k8s-master1 ~]# kubectl exec -it net-test1 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # cat /etc/resolv.conf
nameserver 10.100.0.2
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
/ # ping www.baidu.com
PING www.baidu.com (110.242.69.21): 56 data bytes
64 bytes from 110.242.69.21: seq=0 ttl=127 time=23.703 ms
64 bytes from 110.242.69.21: seq=1 ttl=127 time=22.947 ms
64 bytes from 110.242.69.21: seq=2 ttl=127 time=22.479 ms
^C
--- www.baidu.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 22.479/23.043/23.703 ms
/ #
1.7.3：踩坑点：
CoreDNS 解析异常问题
一、问题现象
-
Pod 内
/etc/resolv.conf中nameserver指向169.254.20.10（node-local-dns 监听地址），正常应该指向我们在coredns.yaml里设置的：10.100.0.2 。但该地址无法解析，执行nslookup kube-dns.kube-system.svc.cluster.local 169.254.20.10提示连接超时；
-
Pod 间无法通过名称解析（如
ping net-test2提示bad address 'net-test2'），核心 DNS 服务 kube-dns 的 ClusterIP（10.100.0.2）虽正常，但 Pod 未指向该地址；
-
排查发现集群部署的 node-local-dns 组件未正常运行，导致
169.254.20.10地址无服务响应。
二、根因分析
- kubelet 配置文件中
clusterDNS字段默认配置为169.254.20.10（node-local-dns 地址），但该组件部署异常/未运行，无法提供 DNS 解析服务； - 集群核心 DNS 服务 kube-dns（ClusterIP：
10.100.0.2）实际正常，但 Pod 因 kubelet 配置未指向该地址，导致解析链路中断； - 仅修改部分节点的 kubelet 配置时，未修改节点上的 Pod 仍使用失效的
169.254.20.10，解析问题未彻底解决。
三、解决措施
-
全节点修改 kubelet 配置：在所有 master/worker 节点编辑
/var/lib/kubelet/config.yaml，将clusterDNS字段从169.254.20.10改为 kube-dns 真实 ClusterIP10.100.0.2；
-
重启 kubelet 生效：所有节点执行
systemctl restart kubelet，使新的 DNS 配置生效； -
重建存量 Pod：旧 Pod 不会自动更新
/etc/resolv.conf，需删除重建（如kubectl delete pod net-test1 net-test2），新 Pod 会加载正确的nameserver: 10.100.0.2； -
验证解析：重建后的 Pod 内可正常解析集群内服务（如
nslookup kube-dns.kube-system.svc.cluster.local），Pod 间可通过 Service 名称正常访问。
四、关键结论
- CoreDNS 解析异常的核心并非 kube-dns 服务本身故障，而是 kubelet 配置的 DNS 地址指向了失效的 node-local-dns 组件；
- kubelet 的
clusterDNS配置为节点级独立配置，需在所有节点统一修改才能确保全集群 Pod 解析正常；
. Pod 内/etc/resolv.conf中nameserver指向169.254.20.10（node-local-dns 监听地址），正常应该指向我们在coredns.yaml里设置的：10.100.0.2 。但该地址无法解析，执行nslookup kube-dns.kube-system.svc.cluster.local 169.254.20.10提示连接超时；
[外链图片转存中…(img-pLA99L5K-1773113298250)] - Pod 间无法通过名称解析（如
ping net-test2提示bad address 'net-test2'），核心 DNS 服务 kube-dns 的 ClusterIP（10.100.0.2）虽正常，但 Pod 未指向该地址；
[外链图片转存中…(img-9mgoao0n-1773113298251)] - 排查发现集群部署的 node-local-dns 组件未正常运行，导致
169.254.20.10地址无服务响应。
更多推荐
9
0- 0
已为社区贡献3条内容
所有评论(0)