【运维】K8S集群部署系列之ETCD集群搭建（三）

ETCD集群安全升级【运维】K8S集群部署系列之ETCD集群搭建（一）中已部署了由master、node1、node2三个节点组成的普通集群。创建TLS加密方式的ETCD安全集群可以采取删除旧集群重建和逐步升级的方式，相对于删除后重建的方式，逐步升级为安全集群可避免旧数据丢失，本文将采取逐步升级的方式。证书准备在【运维】K8S集群部署系列之ETCD集群搭建（二）中我们已经制作了本文...

Let's Golang

1245人浏览 · 2019-08-13 15:17:48

Let's Golang · 2019-08-13 15:17:48 发布

ETCD集群安全升级

文章目录

ETCD集群安全升级

【运维】K8S集群部署系列之ETCD集群搭建（一）中已部署了由master、node1、node2三个节点组成的普通集群。创建TLS加密方式的ETCD安全集群可以采取删除旧集群重建和逐步升级的方式，相对于删除后重建的方式，逐步升级为安全集群可避免旧数据丢失，本文将采取逐步升级的方式。

证书准备

在【运维】K8S集群部署系列之ETCD集群搭建（二）中我们已经制作了本文需要的相关证书。

证书拷贝

# 三个节点执行
mkdir -p /opt/etcd/pki
# 证书服务器执行
scp -P 22 *.pem root@192.168.159.3:/opt/etcd/pki/
scp -P 22 *.csr root@192.168.159.3:/opt/etcd/pki/
scp -P 22 *.pem root@192.168.159.4:/opt/etcd/pki/
scp -P 22 *.csr root@192.168.159.4:/opt/etcd/pki/
scp -P 22 *.pem root@192.168.159.5:/opt/etcd/pki/
scp -P 22 *.csr root@192.168.159.5:/opt/etcd/pki/

[root@master ~]# ls /opt/etcd/pki/
ca.csr  ca-key.pem  ca.pem  etcd.csr  etcdctl.csr  etcdctl-key.pem  etcdctl.pem  etcd-key.pem  etcd.pem  peer.csr  peer-key.pem  peer.pem

三步走升级为TLS安全集群

Step1 开启集群外部`TLS`安全认证

外部访问即客户端访问服务端，etcdctl和k8s的apiserver服务都属于外部客户端，该小节将用到服务器证书etcd.pem及其秘钥文件etcd-key.pem。

修改master节点配置

本节更新字段ETCD_LISTEN_CLIENT_URLS和ETCD_ADVERTISE_CLIENT_URLS中的链接为https方式
新增字段ETCD_CERT_FILE和ETCD_KEY_FILE。

cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.3:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.3:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.3:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.3:2379"
ETCD_INITIAL_CLUSTER="etcd-1=http://192.168.159.3:2380,etcd-2=http://192.168.159.4:2380,etcd-3=http://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"
EOF

字段解释

ETCD_CERT_FILE，服务器证书，可以使用对等证书；
ETCD_KEY_FILE，服务器证书私钥，可以使用对等证书私钥。

修改master节点服务文件

此处新增参数--cert-file和--key-file

cat >  vim /usr/lib/systemd/system/etcd.service << EOF    
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--cert-file=${ETCD_CERT_FILE} \
--key-file=${ETCD_KEY_FILE}
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

重启master节点并验证

重启master节点
systemctl daemon-reload && systemctl restart etcd

集群状态查看

# 此时没加CA根证书，提示master节点不可达
etcdctl cluster-health
# 加上CA根证书，集群验证通过,master节点链接变为https模式
etcdctl -ca-file /opt/etcd/pki/ca.pem cluster-health

member 8ada33a16cb8b5f9 is healthy: got healthy result from http://192.168.159.4:2379
member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379
member e689a191b9fab04f is healthy: got healthy result from http://192.168.159.5:2379
cluster is healthy

修改node1节点配置

cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-2"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.4:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.4:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.4:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.4:2379"
ETCD_INITIAL_CLUSTER="etcd-1=http://192.168.159.3:2380,etcd-2=http://192.168.159.4:2380,etcd-3=http://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"
EOF

修改node1节点服务文件

同master设置

重启node1节点并验证

重启master节点
systemctl daemon-reload && systemctl restart etcd

集群状态查看

etcdctl cluster-health # 此时没加CA根证书，提示master节点不可达
etcdctl -ca-file /opt/etcd/pki/ca.pem cluster-health # 加上CA根证书，集群验证通过,节点链接变为https模式

member 8ada33a16cb8b5f9 is healthy: got healthy result from https://192.168.159.4:2379
member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379
member e689a191b9fab04f is healthy: got healthy result from http://192.168.159.5:2379
cluster is healthy

修改node2节点配置

cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-3"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.5:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.5:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.5:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.5:2379"
ETCD_INITIAL_CLUSTER="etcd-1=http://192.168.159.3:2380,etcd-2=http://192.168.159.4:2380,etcd-3=http://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"
EOF

修改node2节点服务文件

同master设置

重启node2节点并验证

重启master节点
systemctl daemon-reload && systemctl restart etcd
集群状态查看

    etcdctl cluster-health # 此时没加CA根证书，提示master节点不可达
    etcdctl -ca-file /opt/etcd/pki/ca.pem cluster-health # 加上CA根证书，集群验证通过,节点链接变为https模式

    member 8ada33a16cb8b5f9 is healthy: got healthy result from https://192.168.159.4:2379
    member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379
    member e689a191b9fab04f is healthy: got healthy result from https://192.168.159.5:2379
    cluster is healthy

Step2 开启客户端验证

即服务端对etcdctl等客户端的验证。

修改master节点配置

本节新增字段ETCD_CLIENT_CERT_AUTH和ETCD_TRUSTED_CA_FILE

cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.3:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.3:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.3:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.3:2379"
ETCD_INITIAL_CLUSTER="etcd-1=http://192.168.159.3:2380,etcd-2=http://192.168.159.4:2380,etcd-3=http://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"
EOF

字段解释

ETCD_CLIENT_CERT_AUTH，开启客户端证书验证
ETCD_TRUSTED_CA_FILE，用于验证客户端验证的CA根证书

修改master节点服务文件

本节新增参数--client-cert-auth和--trusted-ca-file

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--cert-file=${ETCD_CERT_FILE} \
--key-file=${ETCD_KEY_FILE} \
--client-cert-auth=${ETCD_CLIENT_CERT_AUTH} \
--trusted-ca-file=${ETCD_TRUSTED_CA_FILE}
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

重启master节点并验证

重启master节点
systemctl daemon-reload && systemctl restart etcd

集群状态查看

# 此时没加CA根证书，提示master节点不可达
etcdctl cluster-health
# 加上CA根证书，master不可访问，因为没有配置客户端证书
etcdctl -ca-file /opt/etcd/pki/ca.pem cluster-health
# 加上CA根证书，集群正常,此处也可以使用对等证书及其私钥进行验证，只要有客户端验证功能即可
etcdctl --ca-file=/opt/etcd/pki/ca.pem --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-health

member 8ada33a16cb8b5f9 is healthy: got healthy result from https://192.168.159.4:2379
member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379
member e689a191b9fab04f is healthy: got healthy result from https://192.168.159.5:2379
cluster is healthy

修改node1节点配置

cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-2"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.4:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.4:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.4:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.4:2379"
ETCD_INITIAL_CLUSTER="etcd-1=http://192.168.159.3:2380,etcd-2=http://192.168.159.4:2380,etcd-3=http://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"
EOF

修改node1节点服务文件

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--cert-file=${ETCD_CERT_FILE} \
--key-file=${ETCD_KEY_FILE} \
--client-cert-auth=${ETCD_CLIENT_CERT_AUTH} \
--trusted-ca-file=${ETCD_TRUSTED_CA_FILE}
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

重启node1节点并验证

重启master节点
systemctl daemon-reload && systemctl restart etcd

集群状态查看

# 此时没加CA根证书，提示master节点不可达
etcdctl cluster-health
# 加上CA根证书，master不可访问，因为没有配置客户端证书
etcdctl -ca-file /opt/etcd/pki/ca.pem cluster-health
# 加上CA根证书，集群正常,也可以使用对等证书及其私钥进行验证
etcdctl --ca-file=/opt/etcd/pki/ca.pem --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-health

member 8ada33a16cb8b5f9 is healthy: got healthy result from https://192.168.159.4:2379
member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379
member e689a191b9fab04f is healthy: got healthy result from https://192.168.159.5:2379
cluster is healthy

修改node2节点配置

cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-3"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.5:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.5:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.5:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.5:2379"
ETCD_INITIAL_CLUSTER="etcd-1=http://192.168.159.3:2380,etcd-2=http://192.168.159.4:2380,etcd-3=http://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"
EOF

修改node2节点服务文件

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--cert-file=${ETCD_CERT_FILE} \
--key-file=${ETCD_KEY_FILE} \
--client-cert-auth=${ETCD_CLIENT_CERT_AUTH} \ # 开启客户端验证
--trusted-ca-file=${ETCD_TRUSTED_CA_FILE}     # 生成客户端证书的CA证书
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

重启node2节点并验证

重启master节点
systemctl daemon-reload && systemctl restart etcd

集群状态查看

# 此时没加CA根证书，提示master节点不可达
etcdctl cluster-health 
# 加上CA根证书，master不可访问，因为没有配置客户端证书
etcdctl --ca-file /opt/etcd/pki/ca.pem cluster-health 
# 加上CA根证书，集群正常,也可以使用对等证书及其私钥进行验证
etcdctl --ca-file=/opt/etcd/pki/ca.pem --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-health

member 8ada33a16cb8b5f9 is healthy: got healthy result from https://192.168.159.4:2379
member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379
member e689a191b9fab04f is healthy: got healthy result from https://192.168.159.5:2379
cluster is healthy

Step3 开启集群内部`TLS`安全认证

开启集群节点服务器间的内部通信TLS安全认证。

查看集群节点标识

etcdctl -ca-file=/opt/etcd/pki/ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem member list

可以看到此时peerURLs任然是http的方式。

8ada33a16cb8b5f9: name=etcd-2 peerURLs=http://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=false
df5c33b8666738a6: name=etcd-1 peerURLs=http://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=true
e689a191b9fab04f: name=etcd-3 peerURLs=http://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false

更新节点`peerURLs`链接为`https`方式

为了避免日志中出现无关报错，先将peerURLs链接更新为https是必要的；
命令调用方式：etcdctl member update <memberID> <peerURLs>。

etcdctl -ca-file=/opt/etcd/pki/ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem member update df5c33b8666738a6 https://192.168.159.3:2380
etcdctl -ca-file=/opt/etcd/pki/ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem member update 8ada33a16cb8b5f9 https://192.168.159.4:2380
etcdctl -ca-file=/opt/etcd/pki/ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem member update e689a191b9fab04f https://192.168.159.5:2380

注意：
1、由于已开启集群的服务端和客户端验证，因此需要加上相关证书进行调用；
2、节点的memberID需要与上一步查看的必须一致；
3、节点的peerURLs的端口号和IP地址与上一步查看的必须一致。

再次查看集群状态

此时peerURLs链接全部为https方式。

[root@master pki]# etcdctl -ca-file=/opt/etcd/pki/ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem member list
8ada33a16cb8b5f9: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=false
df5c33b8666738a6: name=etcd-1 peerURLs=http://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=true
e689a191b9fab04f: name=etcd-3 peerURLs=http://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false

[root@master pki]# etcdctl -ca-file=/opt/etcd/pki/ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem cluster-health
member 8ada33a16cb8b5f9 is healthy: got healthy result from https://192.168.159.4:2379
member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379
member e689a191b9fab04f is healthy: got healthy result from https://192.168.159.5:2379
cluster is healthy

节点开启内部`TLS`验证

通过上述操作集群内部https通信并没有真正建立，因为PEER_URLS的侦听地址和相关证书还没有配置；
如果单个节点的PEER_URLS开启https，则其余节点都需要配置相应证书和修改集群客户端侦听地址ETCD_INITIAL_CLUSTER，
集群内部才能正确通信。

查看当前的`ETCD`服务状态

可以看到日志中有TLS相关的错误信息。

[root@master pki]# systemctl status etcd -l
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since 四 2019-08-08 14:11:42 CST; 8min ago
 Main PID: 2720 (etcd)
   CGroup: /system.slice/etcd.service
           └─2720 /home/k8s/etcd/etcd --name=etcd-1 --data-dir=/opt/etcd/data --listen-peer-urls=https://192.168.159.3:2380 --listen-client-urls=https://192.168.159.3:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.159.3:2379 --initial-advertise-peer-urls=https://192.168.159.3:2380 --initial-cluster=etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/opt/etcd/pki/etcd.pem --key-file=/opt/etcd/pki/etcd-key.pem --client-cert-auth=true --trusted-ca-file=/opt/etcd/pki/ca.pem

8月 08 14:11:42 master etcd[2720]: ready to serve client requests
8月 08 14:11:42 master etcd[2720]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
8月 08 14:11:42 master systemd[1]: Started Etcd Server.
8月 08 14:11:42 master etcd[2720]: rejected connection from "192.168.159.3:46646" (error "tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage", ServerName "")
8月 08 14:11:42 master etcd[2720]: WARNING: 2019/08/08 14:11:42 Failed to dial 192.168.159.3:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
8月 08 14:11:42 master etcd[2720]: peer e689a191b9fab04f became active
8月 08 14:11:42 master etcd[2720]: established a TCP streaming connection with peer e689a191b9fab04f (stream MsgApp v2 writer)
8月 08 14:11:42 master etcd[2720]: established a TCP streaming connection with peer e689a191b9fab04f (stream MsgApp v2 reader)
8月 08 14:11:42 master etcd[2720]: established a TCP streaming connection with peer e689a191b9fab04f (stream Message reader)
8月 08 14:11:42 master etcd[2720]: established a TCP streaming connection with peer e689a191b9fab04f (stream Message writer)

修改master节点配置

本节更新字段ETCD_LISTEN_PEER_URLS、ETCD_INITIAL_ADVERTISE_PEER_URLS和ETCD_INITIAL_CLUSTER;
新增字段ETCD_PEER_CERT_FILE、ETCD_PEER_KEY_FILE、ETCD_PEER_CLIENT_CERT_AUTH和ETCD_PEER_TRUSTED_CA_FILE

cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.159.3:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.3:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.159.3:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.3:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"

ETCD_PEER_CERT_FILE="/opt/etcd/pki/peer.pem"    
ETCD_PEER_KEY_FILE="/opt/etcd/pki/peer-key.pem"   
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem" 
EOF

字段解释

ETCD_PEER_CERT_FILE， 服务端内部通信对等证书
ETCD_PEER_KEY_FILE， 服务端内部通信对等证书私钥
ETCD_PEER_CLIENT_CERT_AUTH，开启内部通信TLS验证
ETCD_PEER_TRUSTED_CA_FILE， 用于验证对等证书的CA根证书

修改node1节点配置

cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-2"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.159.4:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.4:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.159.4:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.4:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"

ETCD_PEER_CERT_FILE="/opt/etcd/pki/peer.pem"    
ETCD_PEER_KEY_FILE="/opt/etcd/pki/peer-key.pem"   
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem" 
EOF

修改node2节点配置

cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.159.5:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.5:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.159.5:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.5:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"

ETCD_PEER_CERT_FILE="/opt/etcd/pki/peer.pem"    
ETCD_PEER_KEY_FILE="/opt/etcd/pki/peer-key.pem"   
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem" 
EOF

修改服务文件

本节在三个节点同时做以下操作。
新增参数peer-cert-file、--peer-key-file、--peer-client-cert-auth和--peer-trusted-ca-file

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--cert-file=${ETCD_CERT_FILE} \
--key-file=${ETCD_KEY_FILE} \
--client-cert-auth=${ETCD_CLIENT_CERT_AUTH} \
--trusted-ca-file=${ETCD_TRUSTED_CA_FILE} \
--peer-cert-file=${ETCD_PEER_CERT_FILE} \
--peer-key-file=${ETCD_PEER_KEY_FILE} \
--peer-client-cert-auth=${ETCD_PEER_CLIENT_CERT_AUTH} \
--peer-trusted-ca-file=${ETCD_PEER_TRUSTED_CA_FILE}
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

服务重启

systemctl daemon-reload && systemctl restart etcd

再次查看的`ETCD`服务状态

日志中没有再出现错误信息。

[root@master pki]# systemctl status etcd.service -l
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since 四 2019-08-08 14:28:19 CST; 20s ago
 Main PID: 2812 (etcd)
   CGroup: /system.slice/etcd.service
           └─2812 /home/k8s/etcd/etcd --name=etcd-1 --data-dir=/opt/etcd/data --listen-peer-urls=https://192.168.159.3:2380 --listen-client-urls=https://192.168.159.3:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.159.3:2379 --initial-advertise-peer-urls=https://192.168.159.3:2380 --initial-cluster=etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/opt/etcd/pki/etcd.pem --key-file=/opt/etcd/pki/etcd-key.pem --client-cert-auth=true --trusted-ca-file=/opt/etcd/pki/ca.pem --peer-cert-file=/opt/etcd/pki/peer.pem --peer-key-file=/opt/etcd/pki/peer-key.pem --peer-client-cert-auth=true --peer-trusted-ca-file=/opt/etcd/pki/ca.pem

8月 08 14:28:21 master etcd[2812]: df5c33b8666738a6 is starting a new election at term 167
8月 08 14:28:21 master etcd[2812]: df5c33b8666738a6 became candidate at term 168
8月 08 14:28:21 master etcd[2812]: df5c33b8666738a6 received MsgVoteResp from df5c33b8666738a6 at term 168
8月 08 14:28:21 master etcd[2812]: df5c33b8666738a6 [logterm: 167, index: 104] sent MsgVote request to e689a191b9fab04f at term 168
8月 08 14:28:21 master etcd[2812]: df5c33b8666738a6 [logterm: 167, index: 104] sent MsgVote request to 8ada33a16cb8b5f9 at term 168
8月 08 14:28:21 master etcd[2812]: raft.node: df5c33b8666738a6 lost leader 8ada33a16cb8b5f9 at term 168
8月 08 14:28:22 master etcd[2812]: df5c33b8666738a6 [term: 168] received a MsgVote message with higher term from 8ada33a16cb8b5f9 [term: 170]
8月 08 14:28:22 master etcd[2812]: df5c33b8666738a6 became follower at term 170
8月 08 14:28:22 master etcd[2812]: df5c33b8666738a6 [logterm: 167, index: 104, vote: 0] cast MsgVote for 8ada33a16cb8b5f9 [logterm: 167, index: 104] at term 170
8月 08 14:28:22 master etcd[2812]: raft.node: df5c33b8666738a6 elected leader 8ada33a16cb8b5f9 at term 170

至此集群已经完全升级为TLS安全通信方式，在生产环境中，该方式有效实现平滑升级，避免集群旧数据的丢失。
下一篇将继续介绍集群的扩容和节点删除操作。

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub

K8S/Kubernetes

k8s 火了！

2020，上云之年，产品云端化成为一种趋势。在一线城市，很多公司都已经构建了自己的私有云环境，比如阿里云、网易云、华为云等。而Kubernetes 作为基于容器编排领域的王者，具备扩展...

K8S/Kubernetes

所有评论(0)

查看更多评论

Let's Golang

@weixin_39974140

已为社区贡献7条内容

【运维】K8S集群部署系列之ETCD集群搭建（三）

Let's Golang

ETCD集群安全升级

文章目录

证书准备

证书拷贝

三步走升级为TLS安全集群

Step1 开启集群外部TLS安全认证

修改master节点配置

修改master节点服务文件

重启master节点并验证

修改node1节点配置

修改node1节点服务文件

重启node1节点并验证

修改node2节点配置

修改node2节点服务文件

重启node2节点并验证

Step2 开启客户端验证

修改master节点配置

修改master节点服务文件

重启master节点并验证

修改node1节点配置

修改node1节点服务文件

重启node1节点并验证

修改node2节点配置

修改node2节点服务文件

重启node2节点并验证

Step3 开启集群内部TLS安全认证

查看集群节点标识

更新节点peerURLs链接为https方式

再次查看集群状态

节点开启内部TLS验证

查看当前的ETCD服务状态

修改master节点配置

修改node1节点配置

修改node2节点配置

修改服务文件

服务重启

再次查看的ETCD服务状态

所有评论(0)

Let's Golang

Step1 开启集群外部`TLS`安全认证

Step3 开启集群内部`TLS`安全认证

更新节点`peerURLs`链接为`https`方式

节点开启内部`TLS`验证

查看当前的`ETCD`服务状态

再次查看的`ETCD`服务状态