【运维】K8S集群部署系列之ETCD集群搭建（四）

ETCD集群扩容和缩容本文将介绍生产环境下如何对ETCD集群进行扩容和缩容。新节点环境准备（node3）下载安装包并初始化环境mkdir /home/k8scd /home/k8swget https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gztar -zx...

Let's Golang

1369人浏览 · 2019-08-13 17:12:42

Let's Golang · 2019-08-13 17:12:42 发布

ETCD集群扩容和缩容

本文将介绍生产环境下如何对ETCD集群进行扩容和缩容。

新节点环境准备（node3）

下载安装包并初始化环境

mkdir /home/k8s
cd /home/k8s
wget https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz
tar -zxvf etcd-v3.3.13-linux-amd64.tar.gz
mv etcd-v3.3.13-linux-amd64 etcd
chmod -R +x etcd/
cp -f ./{etcd,etcdctl} /usr/bin/
cp -f ./{etcd,etcdctl} /usr/local/bin/
mkdir -p /opt/etcd/{etc,data,pki}

网络准备

cat >>/etc/hosts<< EOF
192.168.159.3 master
192.168.159.4 node1
192.168.159.5 node2
192.168.159.6 node3
EOF

# 防火墙设置，开放2379和2380端口,如果启动防火墙但未放开端口则集群状态为“degraded”，
# 开启防火墙的节点状态为“are all unreachable”
# 2379端口提供给客户端访问集群，客户端如：etcdctl
# 2380端口提供给集群节点间通信
systemctl start firewalld
firewall-cmd --zone=public --add-port=2379/tcp --permanent
firewall-cmd --zone=public --add-port=2380/tcp --permanent
firewall-cmd --reload
firewall-cmd --list-all

生成`node3`对等证书

由于已有集群开启了服务端内部的TLS安全认证，因此需要在nodes准备好证书，
但旧的证书中hosts列表并不包含node3节点的主机IP，因此需要重新生成node3节点的peer证书。

cat > /home/k8s/cfssl/ssl/etcd4-peer-csr.json << EOF 
{
    "CN": "etcd4-peer",
    "hosts": [
        "192.168.159.6"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "ChengDu",
            "O": "JSQ",
            "OU": "k8s",
            "ST": "SiChuan"
        }
    ]
}
EOF

cfssl gencert --ca=ca.pem --ca-key=ca-key.pem --config=ca-config.json --profile=peer etcd4-peer-csr.json  | cfssljson -bare etcd4-peer
scp etcd4-* root@192.168.159.6:/opt/etcd/pki/

注意：此处hosts列表中只包含了node3节点的主机IP，实际上为了更加方便的进行动态的扩容和缩容操作，对等证书和服务器证书的hosts列表最好只包含本机的IP地址。

向集群中添加普通节点`node3`

添加新节点

命令调用方式：etcdctl member add <memberName> <peerURLS>；
注意此处添加的节点名和链接应该与新增节点的配置一致；
在已有集群的任意节点（最好为集群主节点，即isLeader=true的节点）执行如下命令。

etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member add etcd-4 http://192.168.159.6:2380

查看集群状态

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member list
46899d42c87d524e: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=true
6bdd9302771bc9c5: name=etcd-3 peerURLs=https://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
a3ec213779ea2c81: name=etcd-1 peerURLs=https://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=false
e1b7f9d6e4ff0f36[unstarted]: peerURLs=https://192.168.159.6:2380

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-health
member 46899d42c87d524e is healthy: got healthy result from https://192.168.159.4:2379
member 6bdd9302771bc9c5 is healthy: got healthy result from https://192.168.159.5:2379
member a3ec213779ea2c81 is healthy: got healthy result from https://192.168.159.3:2379
member e1b7f9d6e4ff0f36 is unreachable: no available published client urls
cluster is healthy

添加`node3`节点配置

此处ETCD_INITIAL_CLUSTER_STATE必须为existing，表示向已有集群新增节点；
由于已有集群已开启TLS安全验证，因此必须配置相关证书，如果是普通集群则无须配置；
ETCD_PEER_CLIENT_CERT_AUTH=false表示集群内部访问该节点服务端无须进行TLS验证。

mkdir -p /opt/etcd/{data,etc}
cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-4"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.6:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.159.6:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.6:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.159.6:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380,etcd-4=http://192.168.159.6:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="existing"

#[Security]
ETCD_PEER_CERT_FILE="/opt/etcd/pki/etcd4-peer.pem"    
ETCD_PEER_KEY_FILE="/opt/etcd/pki/etcd4-peer-key.pem"   
ETCD_PEER_CLIENT_CERT_AUTH="false"
ETCD_PEER_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem" 
EOF

添加`node3`节点服务文件

cat >  /usr/lib/systemd/system/etcd.service << EOF    
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--peer-cert-file=${ETCD_PEER_CERT_FILE} \
--peer-key-file=${ETCD_PEER_KEY_FILE} \
--peer-client-cert-auth=${ETCD_PEER_CLIENT_CERT_AUTH} \
--peer-trusted-ca-file=${ETCD_PEER_TRUSTED_CA_FILE}
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

启动`node3`节点

systemctl daemon-reload && systemctl start etcd

再次查看集群状态

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member list
46899d42c87d524e: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=true
6bdd9302771bc9c5: name=etcd-3 peerURLs=https://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
a3ec213779ea2c81: name=etcd-1 peerURLs=https://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=false
e1b7f9d6e4ff0f36: name=etcd-4 peerURLs=http://192.168.159.6:2380 clientURLs=http://192.168.159.6:2379 isLeader=false

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-health
member 46899d42c87d524e is healthy: got healthy result from https://192.168.159.4:2379
member 6bdd9302771bc9c5 is healthy: got healthy result from https://192.168.159.5:2379
member a3ec213779ea2c81 is healthy: got healthy result from https://192.168.159.3:2379
member e1b7f9d6e4ff0f36 is healthy: got healthy result from http://192.168.159.6:2379
cluster is healthy

至此完成向集群添加一个普通节点的操作，下一步继续进行节点的移除操作。

将`node3`节点从集群中移除

集群节点查看

通过查看集群节点确定需要移除节点的ID。

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member list
46899d42c87d524e: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=true
6bdd9302771bc9c5: name=etcd-3 peerURLs=https://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
a3ec213779ea2c81: name=etcd-1 peerURLs=https://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=false
e1b7f9d6e4ff0f36: name=etcd-4 peerURLs=http://192.168.159.6:2380 clientURLs=http://192.168.159.6:2379 isLeader=false

移除节点

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member remove e1b7f9d6e4ff0f36
Removed member e1b7f9d6e4ff0f36 from cluster

查看集群状态

可以看到节点etcd-4已经从集群移除。

[root@master pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem member list
46899d42c87d524e: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=true
6bdd9302771bc9c5: name=etcd-3 peerURLs=https://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
a3ec213779ea2c81: name=etcd-1 peerURLs=https://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=false

[root@master pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem cluster-health
member 46899d42c87d524e is healthy: got healthy result from https://192.168.159.4:2379
member 6bdd9302771bc9c5 is healthy: got healthy result from https://192.168.159.5:2379
member a3ec213779ea2c81 is healthy: got healthy result from https://192.168.159.3:2379
cluster is healthy

查看`node3`节点的服务状态

可以看到服务已被节点a3ec213779ea2c81通知停止通信。

[root@localhost ~]# systemctl status etcd
● etcd.service
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; bad; vendor preset: disabled)
   Active: inactive (dead)

8月 13 16:36:22 localhost.localdomain etcd[1344]: failed to dial a3ec213779ea2c81 on stream MsgApp v2 (context canceled)
8月 13 16:36:22 localhost.localdomain etcd[1344]: peer a3ec213779ea2c81 became inactive (message send to peer failed)
8月 13 16:36:22 localhost.localdomain etcd[1344]: stopped streaming with peer a3ec213779ea2c81 (stream MsgApp v2 reader)
8月 13 16:36:22 localhost.localdomain etcd[1344]: stopped streaming with peer a3ec213779ea2c81 (stream Message reader)
8月 13 16:36:22 localhost.localdomain etcd[1344]: stopped peer a3ec213779ea2c81
8月 13 16:39:07 localhost.localdomain systemd[1]: [/usr/lib/systemd/system/etcd.service:1] Assignment outside of section. Ignoring.
8月 13 16:39:07 localhost.localdomain systemd[1]: [/usr/lib/systemd/system/etcd.service:2] Assignment outside of section. Ignoring.
8月 13 16:39:07 localhost.localdomain systemd[1]: [/usr/lib/systemd/system/etcd.service:3] Assignment outside of section. Ignoring.
8月 13 16:39:07 localhost.localdomain systemd[1]: [/usr/lib/systemd/system/etcd.service:4] Assignment outside of section. Ignoring.
8月 13 16:39:07 localhost.localdomain systemd[1]: [/usr/lib/systemd/system/etcd.service:5] Assignment outside of section. Ignoring.

向集群中添加安全节点`node3`

生成node3节点的服务端证书

由于集群已开启服务端的TLS认证，因此node3节点需要有相应的服务器证书；
但旧的服务端证书中hosts列表并不包含node3节点的主机IP，因此需要重新生成node3节点的server证书

cat > etcd4-csr.json << EOF 
{
    "CN": "etcd4",
    "hosts": [
        "192.168.159.6"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "ChengDu",
            "O": "JSQ",
            "OU": "k8s",
            "ST": "SiChuan"
        }
    ]
}
EOF

cfssl gencert --ca=ca.pem --ca-key=ca-key.pem --config=ca-config.json --profile=server etcd4-csr.json  | cfssljson -bare etcd4
scp etcd4* root@192.168.159.6:/opt/etcd/pki/

[root@localhost pki]# ls etcd4*
etcd4.csr  etcd4-csr.json  etcd4-key.pem  etcd4-peer.csr  etcd4-peer-csr.json  etcd4-peer-key.pem  etcd4-peer.pem  etcd4.pem

集群的客户端证书由于没有hosts主机列表，因此可以通用。

从普通节点升级为`TLS`认证的安全节点

参照【运维】K8S集群部署系列之ETCD集群搭建（三）

直接将`node3`添加为`TLS`认证的安全节点

以`https`方式添加`node3`节点

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member add etcd-4 https://192.168.159.6:2380
Added member named etcd-4 with ID 1e7da56305348d0d to cluster

ETCD_NAME="etcd-4"
ETCD_INITIAL_CLUSTER="etcd-4=https://192.168.159.6:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380,etcd-1=https://192.168.159.3:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

查看集群状态

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member list
1e7da56305348d0d[unstarted]: peerURLs=https://192.168.159.6:2380
46899d42c87d524e: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=true
6bdd9302771bc9c5: name=etcd-3 peerURLs=https://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
a3ec213779ea2c81: name=etcd-1 peerURLs=https://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=false

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-health
member 1e7da56305348d0d is unreachable: no available published client urls
member 46899d42c87d524e is healthy: got healthy result from https://192.168.159.4:2379
member 6bdd9302771bc9c5 is healthy: got healthy result from https://192.168.159.5:2379
member a3ec213779ea2c81 is healthy: got healthy result from https://192.168.159.3:2379
cluster is degraded

修改`node3`节点配置

此处修改字段ETCD_LISTEN_PEER_URLS、ETCD_LISTEN_CLIENT_URLS、ETCD_INITIAL_ADVERTISE_PEER_URLS、ETCD_ADVERTISE_CLIENT_URLS、ETCD_INITIAL_CLUSTER、ETCD_PEER_CLIENT_CERT_AUTH；
新增字段ETCD_CERT_FILE、ETCD_KEY_FILE、ETCD_CLIENT_CERT_AUTH、ETCD_TRUSTED_CA_FILE；
注意字段ETCD_INITIAL_CLUSTER_STATE的值必须为existing。

cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-4"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.159.6:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.6:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.159.6:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.6:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380,etcd-4=https://192.168.159.6:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="existing"

#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd4.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd4-key.pem"

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"

ETCD_PEER_CERT_FILE="/opt/etcd/pki/etcd4-peer.pem"    
ETCD_PEER_KEY_FILE="/opt/etcd/pki/etcd4-peer-key.pem"   
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem" 
EOF

安全启动`node3`的服务文件

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--cert-file=${ETCD_CERT_FILE} \
--key-file=${ETCD_KEY_FILE} \
--client-cert-auth=${ETCD_CLIENT_CERT_AUTH} \
--trusted-ca-file=${ETCD_TRUSTED_CA_FILE} \
--peer-cert-file=${ETCD_PEER_CERT_FILE} \
--peer-key-file=${ETCD_PEER_KEY_FILE} \
--peer-client-cert-auth=${ETCD_PEER_CLIENT_CERT_AUTH} \
--peer-trusted-ca-file=${ETCD_PEER_TRUSTED_CA_FILE}
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

删除`node3`节点的旧数据

这一步很重要，否则node3节点无法正确启动。

rm -rf /opt/etcd/data/*

启动`node3`节点

systemctl daemon-reload && systemctl start etcd

再次查看集群状态

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member list
1e7da56305348d0d: name=etcd-4 peerURLs=https://192.168.159.6:2380 clientURLs=https://192.168.159.6:2379 isLeader=false
46899d42c87d524e: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=true
6bdd9302771bc9c5: name=etcd-3 peerURLs=https://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
a3ec213779ea2c81: name=etcd-1 peerURLs=https://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=false

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-health
member 1e7da56305348d0d is healthy: got healthy result from https://192.168.159.6:2379
member 46899d42c87d524e is healthy: got healthy result from https://192.168.159.4:2379
member 6bdd9302771bc9c5 is healthy: got healthy result from https://192.168.159.5:2379
member a3ec213779ea2c81 is healthy: got healthy result from https://192.168.159.3:2379
cluster is healthy

至此ETCD集群搭建及其动态扩容和缩容介绍完毕，下一篇我们将对etcdctl的基本操作进行简单介绍。

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub

K8S/Kubernetes

k8s 火了！

2020，上云之年，产品云端化成为一种趋势。在一线城市，很多公司都已经构建了自己的私有云环境，比如阿里云、网易云、华为云等。而Kubernetes 作为基于容器编排领域的王者，具备扩展...

K8S/Kubernetes

所有评论(0)

查看更多评论

Let's Golang

@weixin_39974140

已为社区贡献7条内容

【运维】K8S集群部署系列之ETCD集群搭建（四）

Let's Golang

ETCD集群扩容和缩容

文章目录

新节点环境准备（node3）

下载安装包并初始化环境

网络准备

生成node3对等证书

向集群中添加普通节点node3

添加新节点

查看集群状态

添加node3节点配置

添加node3节点服务文件

启动node3节点

再次查看集群状态

将node3节点从集群中移除

集群节点查看

移除节点

查看集群状态

查看node3节点的服务状态

向集群中添加安全节点node3

生成node3节点的服务端证书

从普通节点升级为TLS认证的安全节点

直接将node3添加为TLS认证的安全节点

以https方式添加node3节点

查看集群状态

修改node3节点配置

安全启动node3的服务文件

删除node3节点的旧数据

启动node3节点

再次查看集群状态

所有评论(0)

Let's Golang

生成`node3`对等证书

向集群中添加普通节点`node3`

添加`node3`节点配置

添加`node3`节点服务文件

启动`node3`节点

将`node3`节点从集群中移除

查看`node3`节点的服务状态

向集群中添加安全节点`node3`

从普通节点升级为`TLS`认证的安全节点

直接将`node3`添加为`TLS`认证的安全节点

以`https`方式添加`node3`节点

修改`node3`节点配置

安全启动`node3`的服务文件

删除`node3`节点的旧数据

启动`node3`节点