K8s-部署 etcd 集群.04
内容转载自:https://github.com/opsnull/follow-me-install-kubernetes-cluster/blob/master/04.%E9%83%A8%E7%BD%B2etcd%E9%9B%86%E7%BE%A4.md上一篇:K8s-部署 kubectl 命令行工具.0304.部署 etcd 集群etcd 是基于 Raft 的分布式 key-val...
·
04.部署 etcd 集群
etcd 是基于 Raft 的分布式 key-value 存储系统,由 CoreOS 开发,常用于服务发现、共享配置以及并发控制(如 leader 选举、分布式锁等)。kubernetes 使用 etcd 存储所有运行数据。
本文档介绍部署一个三节点高可用 etcd 集群的步骤:
- 下载和分发 etcd 二进制文件;
- 创建 etcd 集群各节点的 x509 证书,用于加密客户端(如 etcdctl) 与 etcd 集群、etcd 集群之间的数据流;
- 创建 etcd 的 systemd unit 文件,配置服务参数;
- 检查集群工作状态;
etcd 集群各节点的名称和 IP 如下:
- k8s-etcd-0001:172.27.129.104
- k8s-etcd-0002:172.27.129.105
- k8s-etcd-0003:172.27.129.106
下载和分发 etcd 二进制文件
到 https://github.com/coreos/etcd/releases 页面下载最新版本的发布包:
wget https://github.com/coreos/etcd/releases/download/v3.3.7/etcd-v3.3.7-linux-amd64.tar.gz tar -xvf etcd-v3.3.7-linux-amd64.tar.gz
分发二进制文件到集群etcd节点:
source /opt/k8s/bin/environment.sh for etcd_ip in ${ETCD_IP[@]} do echo ">>> ${netcd_ip}" scp etcd-v3.3.7-linux-amd64/etcd* k8s@${etcd_ip}:/opt/k8s/bin ssh k8s@${etcd_ip} "chmod +x /opt/k8s/bin/*" done
创建 etcd 证书和私钥
cat > etcd-csr.json <<EOF { "CN": "etcd", "hosts": [ "127.0.0.1", "172.27.129.104", "172.27.129.105", "172.27.129.106" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "4Paradigm" } ] } EOF
- hosts 字段指定授权使用该证书的 etcd 节点 IP 或域名列表,这里将 etcd 集群的三个节点 IP 都列在其中;
生成证书和私钥:
cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \ -ca-key=/etc/kubernetes/cert/ca-key.pem \ -config=/etc/kubernetes/cert/ca-config.json \ -profile=kubernetes etcd-csr.json | cfssljson -bare etcd ls etcd*
分发生成的证书和私钥到各 etcd 节点:
source /opt/k8s/bin/environment.sh for etcd_ip in ${ETCD_IP[@]} do echo ">>> ${etcd_ip}" ssh root@${etcd_ip} "mkdir -p /etc/etcd/cert && chown -R k8s /etc/etcd/cert" scp etcd*.pem k8s@${etcd_ip}:/etc/etcd/cert/ done
创建 etcd 的 systemd unit 模板文件
source /opt/k8s/bin/environment.sh cat > etcd.service.template <<EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos [Service] User=k8s Type=notify WorkingDirectory=/var/lib/etcd/ ExecStart=/opt/k8s/bin/etcd \\ --data-dir=/var/lib/etcd \\ --name=##ETCD_NAME## \\ --cert-file=/etc/etcd/cert/etcd.pem \\ --key-file=/etc/etcd/cert/etcd-key.pem \\ --trusted-ca-file=/etc/kubernetes/cert/ca.pem \\ --peer-cert-file=/etc/etcd/cert/etcd.pem \\ --peer-key-file=/etc/etcd/cert/etcd-key.pem \\ --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\ --peer-client-cert-auth \\ --client-cert-auth \\ --listen-peer-urls=https://##ETCD_IP##:2380 \\ --initial-advertise-peer-urls=https://##ETCD_IP##:2380 \\ --listen-client-urls=https://##ETCD_IP##:2379,http://127.0.0.1:2379 \\ --advertise-client-urls=https://##ETCD_IP##:2379 \\ --initial-cluster-token=etcd-cluster-0 \\ --initial-cluster=${ETCD_NODES} \\ --initial-cluster-state=new Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
User
:指定以 k8s 账户运行;WorkingDirectory
、--data-dir
:指定工作目录和数据目录为/var/lib/etcd
,需在启动服务前创建这个目录;--name
:指定节点名称,当--initial-cluster-state
值为new
时,--name
的参数值必须位于--initial-cluster
列表中;--cert-file
、--key-file
:etcd server 与 client 通信时使用的证书和私钥;--trusted-ca-file
:签名 client 证书的 CA 证书,用于验证 client 证书;--peer-cert-file
、--peer-key-file
:etcd 与 peer 通信使用的证书和私钥;--peer-trusted-ca-file
:签名 peer 证书的 CA 证书,用于验证 peer 证书;
为各节点创建和分发 etcd systemd unit 文件
替换模板文件中的变量,为各节点创建 systemd unit 文件:
source /opt/k8s/bin/environment.sh for (( i=0; i < 3; i++ )) do sed -e "s/##ETCD_NAME##/${ETCD_NAMES[i]}/" -e "s/##ETCD_IP##/${ETCD_IP[i]}/" etcd.service.template > etcd-${ETCD_IP[i]}.service done ls *.service
- ETCD_NAMES 和 ETCD_IP 为相同长度的 bash 数组,分别为节点名称和对应的 IP;
分发生成的 systemd unit 文件:
source /opt/k8s/bin/environment.sh for etcd_ip in ${ETCD_IP[@]} do echo ">>> ${etcd_ip}" ssh root@${etcd_ip} "mkdir -p /var/lib/etcd && chown -R k8s /var/lib/etcd" scp etcd-${etcd_ip}.service root@${etcd_ip}:/etc/systemd/system/etcd.service done
- 必须先创建 etcd 数据目录和工作目录;
- 文件重命名为 etcd.service;
启动 etcd 服务
source /opt/k8s/bin/environment.sh for etcd_ip in ${ETCD_IP[@]} do echo ">>> ${etcd_ip}" ssh root@${etcd_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd" done
- etcd 进程首次启动时会等待其它节点的 etcd 加入集群,命令
systemctl restart etcd
会卡住一段时间,为正常现象。 - (如卡住,可Ctrl+c中断命令重新执行一次循环)
检查启动结果
source /opt/k8s/bin/environment.sh for etcd_ip in ${ETCD_IP[@]} do echo ">>> ${etcd_ip}" ssh k8s@${etcd_ip} "systemctl status etcd|grep Active" done
确保状态为 active (running)
,否则查看日志,确认原因:
$ journalctl -u etcd
验证服务状态
注意:部署完 etcd 集群后,在任一 etcd(k8s-etcd-0001到k8s-etcd-0003) 节点上执行如下命令:
source /opt/k8s/bin/environment.sh for etcd_ip in ${ETCD_IP[@]} do echo ">>> ${etcd_ip}" ETCDCTL_API=3 /opt/k8s/bin/etcdctl \ --endpoints=https://${etcd_ip}:2379 \ --cacert=/etc/kubernetes/cert/ca.pem \ --cert=/etc/etcd/cert/etcd.pem \ --key=/etc/etcd/cert/etcd-key.pem endpoint health done
预期输出:
https://172.27.129.104:2379 is healthy: successfully committed proposal: took = 2.192932ms https://172.27.129.105:2379 is healthy: successfully committed proposal: took = 3.546896ms https://172.27.129.106:2379 is healthy: successfully committed proposal: took = 3.013667ms
输出均为 healthy
时表示集群服务正常。
更多推荐
已为社区贡献27条内容
所有评论(0)