1、etcd集群的搭建
环境准备及机器规划：（13台机器）

192.168.43.121 k8s-etcd1
192.168.43.122 k8s-etcd2
192.168.43.123 k8s-etcd3
192.168.43.124 k8s-master1
192.168.43.125 k8s-master2
192.168.43.126 k8s-master3
192.168.43.127 k8s-harbor
192.168.43.131 k8s-node1
192.168.43.132 k8s-node2
192.168.43.133 k8s-node3
192.168.43.134 k8s-node4
192.168.43.135 k8s-node5
192.168.43.136 k8s-node6

系统版本：

[root@k8s-etcd1 ~]# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)

1、操作系统初始化：(每台机器都要做)

关闭防火墙

[root@k8s-etcd1 ~]# systemctl stop firewalld
[root@k8s-etcd1 ~]# systemctl disable firewalld

关闭selinux

[root@k8s-etcd1 ~]# sed -i ‘s/enforcing/disabled/’ /etc/selinux/config # 永久
[root@k8s-etcd1 ~]# setenforce 0 # 临时

关闭swap

[root@k8s-etcd1 ~]# swapoff -a # 临时

[root@k8s-etcd1 ~]# sed -ri ‘s/.swap./#&/’ /etc/fstab # 永久

根据规划设置主机名

[root@k8s-etcd1 ~]# hostnamectl set-hostname k8s-etcd1

在master添加hosts

cat > /etc/hosts << EOF
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.43.121 k8s-etcd1
192.168.43.122 k8s-etcd2
192.168.43.123 k8s-etcd3
192.168.43.124 k8s-master1
192.168.43.125 k8s-master2
192.168.43.126 k8s-master3
192.168.43.127 k8s-harbor
192.168.43.131 k8s-node1
192.168.43.132 k8s-node2
192.168.43.133 k8s-node3
192.168.43.134 k8s-node4
192.168.43.135 k8s-node5
192.168.43.136 k8s-node6
EOF

将桥接的IPv4流量传递到iptables的链

[root@k8s-etcd1 ~]# cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

[root@k8s-etcd1 ~]# sysctl --system ===使之生效

• Applying /usr/lib/sysctl.d/00-system.conf …
• Applying /usr/lib/sysctl.d/10-default-yama-scope.conf …
  kernel.yama.ptrace_scope = 0
• Applying /usr/lib/sysctl.d/50-default.conf …
  kernel.sysrq = 16
  kernel.core_uses_pid = 1
  net.ipv4.conf.default.rp_filter = 1
  net.ipv4.conf.all.rp_filter = 1
  net.ipv4.conf.default.accept_source_route = 0
  net.ipv4.conf.all.accept_source_route = 0
  net.ipv4.conf.default.promote_secondaries = 1
  net.ipv4.conf.all.promote_secondaries = 1
  fs.protected_hardlinks = 1
  fs.protected_symlinks = 1
• Applying /etc/sysctl.d/99-sysctl.conf …
• Applying /etc/sysctl.d/k8s.conf …
• Applying /etc/sysctl.conf …

各服务器之间时间同步

[root@k8s-etcd1 ~]# yum install ntpdate -y
[root@k8s-etcd1 ~]# ntpdate time.windows.com

####安装ETCD集群

软件版本：etcd-v3.4.14

上传软件并解压

[root@k8s-etcd1 ~]# cd /usr/local/src/

[root@k8s-etcd1 src]# ls
etcd-v3.4.14-linux-amd64.tar.gz

2.1 准备cfssl证书生成工具

cfssl是一个开源的证书管理工具，使用json文件生成证书，相比openssl更方便使用。

找任意一台服务器操作，这里用Master节点。

[root@k8s-etcd1 src]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
–2021-02-23 20:26:17-- https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
Resolving pkg.cfssl.org (pkg.cfssl.org)… 104.18.22.229, 104.18.23.229, 2606:4700::6812:17e5, …
Connecting to pkg.cfssl.org (pkg.cfssl.org)|104.18.22.229|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 10376657 (9.9M) [application/octet-stream]
Saving to: ‘cfssl_linux-amd64’

100%[======================================>] 10,376,657 2.90MB/s in 3.4s

2021-02-23 20:26:24 (2.90 MB/s) - ‘cfssl_linux-amd64’ saved [10376657/10376657]

You have new mail in /var/spool/mail/root
[root@k8s-etcd1 src]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
–2021-02-23 20:26:25-- https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
Resolving pkg.cfssl.org (pkg.cfssl.org)… 104.18.23.229, 104.18.22.229, 2606:4700::6812:17e5, …
Connecting to pkg.cfssl.org (pkg.cfssl.org)|104.18.23.229|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 2277873 (2.2M) [application/octet-stream]
Saving to: ‘cfssljson_linux-amd64’

100%[======================================>] 2,277,873 177KB/s in 12s

2021-02-23 20:26:38 (189 KB/s) - ‘cfssljson_linux-amd64’ saved [2277873/2277873]

[root@k8s-etcd1 src]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
–2021-02-23 20:26:41-- https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
Resolving pkg.cfssl.org (pkg.cfssl.org)… 104.18.23.229, 104.18.22.229, 2606:4700::6812:17e5, …
Connecting to pkg.cfssl.org (pkg.cfssl.org)|104.18.23.229|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 6595195 (6.3M) [application/octet-stream]
Saving to: ‘cfssl-certinfo_linux-amd64’

100%[======================================>] 6,595,195 1.13MB/s in 6.7s

2021-02-23 20:26:50 (958 KB/s) - ‘cfssl-certinfo_linux-amd64’ saved [6595195/6595195]

[root@k8s-etcd1 src]# ls
cfssl-certinfo_linux-amd64 cfssl_linux-amd64
cfssljson_linux-amd64 etcd-v3.4.14-linux-amd64.tar.gz
[root@k8s-etcd1 src]# chmod +x cfssl*
[root@k8s-etcd1 src]# ls
cfssl-certinfo_linux-amd64 cfssl_linux-amd64
cfssljson_linux-amd64 etcd-v3.4.14-linux-amd64.tar.gz
[root@k8s-etcd1 src]# mv cfssl_linux-amd64 /usr/local/bin/cfssl
[root@k8s-etcd1 src]# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
[root@k8s-etcd1 src]# mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
[root@k8s-etcd1 src]# ls
etcd-v3.4.14-linux-amd64.tar.gz

2.2 生成Etcd证书

1. 自签证书颁发机构（CA）

创建工作目录：

mkdir -p ~/TLS/{etcd,k8s}

cd TLS/etcd

自签CA：

[root@k8s-etcd1 etcd]# cat > ca-config.json << EOF
{
“signing”: {
“default”: {
“expiry”: “87600h”
},
“profiles”: {
“www”: {
“expiry”: “87600h”,
“usages”: [
“signing”,
“key encipherment”,
“server auth”,
“client auth”
]
}
}
}
}
EOF

[root@k8s-etcd1 etcd]# cat > ca-csr.json << EOF
{
“CN”: “etcd CA”,
“key”: {
“algo”: “rsa”,
“size”: 2048
},
“names”: [
{
“C”: “CN”,
“L”: “Beijing”,
“ST”: “Beijing”
}
]
}
EOF

[root@k8s-etcd1 etcd]# ls
ca-config.json ca-csr.json

生成证书：

[root@k8s-etcd1 etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2021/02/23 20:30:50 [INFO] generating a new CA key and certificate from CSR
2021/02/23 20:30:50 [INFO] generate received request
2021/02/23 20:30:50 [INFO] received CSR
2021/02/23 20:30:50 [INFO] generating key: rsa-2048
2021/02/23 20:30:50 [INFO] encoded CSR
2021/02/23 20:30:50 [INFO] signed certificate with serial number 162279841843832490601547299208397940470138892893
You have new mail in /var/spool/mail/root
[root@k8s-etcd1 etcd]# ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
[root@k8s-etcd1 etcd]# ll *.pem
-rw------- 1 root root 1679 Feb 23 20:30 ca-key.pem
-rw-r–r-- 1 root root 1265 Feb 23 20:30 ca.pem

2. 使用自签CA签发Etcd HTTPS证书

创建证书申请文件：

[root@k8s-etcd1 etcd]# cat > server-csr.json << EOF
{
“CN”: “etcd”,
“hosts”: [
“192.168.43.121”,
“192.168.43.122”,
“192.168.43.123”
],
“key”: {
“algo”: “rsa”,
“size”: 2048
},
“names”: [
{
“C”: “CN”,
“L”: “BeiJing”,
“ST”: “BeiJing”
}
]
}
EOF

注：上述文件hosts字段中IP为所有etcd节点的集群内部通信IP，一个都不能少！为了方便后期扩容可以多写几个预留的IP。

生成证书：

[root@k8s-etcd1 etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2021/02/23 20:35:48 [INFO] generate received request
2021/02/23 20:35:48 [INFO] received CSR
2021/02/23 20:35:48 [INFO] generating key: rsa-2048
2021/02/23 20:35:49 [INFO] encoded CSR
2021/02/23 20:35:49 [INFO] signed certificate with serial number 136553168563275032268347225984581968746877896122
2021/02/23 20:35:49 [WARNING] This certificate lacks a “hosts” field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 (“Information Requirements”).
You have new mail in /var/spool/mail/root

[root@k8s-etcd1 etcd]# ll server*.pem
-rw------- 1 root root 1675 Feb 23 20:35 server-key.pem
-rw-r–r-- 1 root root 1338 Feb 23 20:35 server.pem

2.4 部署Etcd集群

以下在节点1上操作，为简化操作，待会将节点1生成的所有文件拷贝到节点2和节点3.

1. 创建工作目录并解压二进制包

[root@k8s-etcd1 etcd]# mkdir /opt/etcd/{bin,cfg,ssl} -p

[root@k8s-etcd1 src]# ls
etcd-v3.4.14-linux-amd64.tar.gz
[root@k8s-etcd1 src]# tar xf etcd-v3.4.14-linux-amd64.tar.gz
[root@k8s-etcd1 src]# ls
etcd-v3.4.14-linux-amd64 etcd-v3.4.14-linux-amd64.tar.gz

###将etcd和etcdctl放到创建的/opt/etcd/bin下面去

[root@k8s-etcd1 src]# cd etcd-v3.4.14-linux-amd64
[root@k8s-etcd1 etcd-v3.4.14-linux-amd64]# ls
Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md
[root@k8s-etcd1 etcd-v3.4.14-linux-amd64]# \cp -r etcd etcdctl /opt/etcd/bin/

2. 创建etcd配置文件

[root@k8s-etcd1 cfg]# cat /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME=“etcd-1”
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS=“https://192.168.43.121:2380”
ETCD_LISTEN_CLIENT_URLS=“https://192.168.43.121:2379”

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://192.168.43.121:2380”
ETCD_ADVERTISE_CLIENT_URLS=“https://192.168.43.121:2379”
ETCD_INITIAL_CLUSTER=“etcd-1=https://192.168.43.121:2380,etcd-2=https://192.168.43.122:2380,etcd-3=https://192.168.43.123:2380”
ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
ETCD_INITIAL_CLUSTER_STATE=“new”

参数解释：

ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群
启动服务配置文件：

[root@k8s-etcd1 system]# cat etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

4. 拷贝刚才生成的证书

把刚才生成的证书拷贝到配置文件中的路径：

[root@k8s-etcd1 etcd]# ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem server.csr server-csr.json server-key.pem server.pem
[root@k8s-etcd1 etcd]# cp /root/TLS/etcd/*.pem /opt/etcd/ssl

5. 将/opt/etcd目录拷贝到另外两台机器上面去

[root@k8s-etcd1 opt]# scp -r etcd k8s-etcd2:/opt/

[root@k8s-etcd1 opt]# scp -r etcd k8s-etcd3:/opt/

#k8s-etcd2的配置文件

[root@k8s-etcd2 ~]# cat /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME=“etcd-2”
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS=“https://192.168.43.122:2380”
ETCD_LISTEN_CLIENT_URLS=“https://192.168.43.122:2379”

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://192.168.43.122:2380”
ETCD_ADVERTISE_CLIENT_URLS=“https://192.168.43.122:2379”
ETCD_INITIAL_CLUSTER=“etcd-1=https://192.168.43.121:2380,etcd-2=https://192.168.43.122:2380,etcd-3=https://192.168.43.123:2380”
ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
ETCD_INITIAL_CLUSTER_STATE=“new”

#k8s-etcd3的配置文件

[root@k8s-etcd3 ~]# cat /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME=“etcd-3”
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS=“https://192.168.43.123:2380”
ETCD_LISTEN_CLIENT_URLS=“https://192.168.43.123:2379”

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://192.168.43.123:2380”
ETCD_ADVERTISE_CLIENT_URLS=“https://192.168.43.123:2379”
ETCD_INITIAL_CLUSTER=“etcd-1=https://192.168.43.121:2380,etcd-2=https://192.168.43.122:2380,etcd-3=https://192.168.43.123:2380”
ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
ETCD_INITIAL_CLUSTER_STATE=“new”

[root@k8s-etcd1 opt]# scp -r /usr/lib/systemd/system/etcd.service root@192.168.43.122:/usr/lib/systemd/system/
root@192.168.43.122’s password:
etcd.service 100% 522 357.1KB/s 00:00
[root@k8s-etcd1 opt]# scp -r /usr/lib/systemd/system/etcd.service root@192.168.43.123:/usr/lib/systemd/system/
root@192.168.43.123’s password:
etcd.service 100% 522 409.7KB/s 00:00

##启动程序并加入开机自启动

[root@k8s-etcd1 etcd]# systemctl daemon-reload && systemctl start etcd && systemctl status etcd
● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-02-23 21:04:10 CST; 13s ago
Main PID: 9458 (etcd)
CGroup: /system.slice/etcd.service
└─9458 /opt/etcd/bin/etcd --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server…

Feb 23 21:04:10 k8s-etcd1 etcd[9458]: raft2021/02/23 21:04:10 INFO: a8d953f42a387c42 [logterm: 1, index: 3, vote: 0] cast MsgVote for ebf750e0b… term 499
Feb 23 21:04:10 k8s-etcd1 etcd[9458]: raft2021/02/23 21:04:10 INFO: raft.node: a8d953f42a387c42 elected leader ebf750e0b069a7fe at term 499
Feb 23 21:04:10 k8s-etcd1 etcd[9458]: set the initial cluster version to 3.0
Feb 23 21:04:10 k8s-etcd1 etcd[9458]: enabled capabilities for version 3.0
Feb 23 21:04:10 k8s-etcd1 etcd[9458]: published {Name:etcd-1 ClientURLs:[https://192.168.43.121:2379]} to cluster 1ce34f03b11d2468
Feb 23 21:04:10 k8s-etcd1 etcd[9458]: ready to serve client requests
Feb 23 21:04:10 k8s-etcd1 systemd[1]: Started Etcd Server.
Feb 23 21:04:10 k8s-etcd1 etcd[9458]: serving client requests on 192.168.43.121:2379
Feb 23 21:04:10 k8s-etcd1 etcd[9458]: updated the cluster version from 3.0 to 3.4
Feb 23 21:04:10 k8s-etcd1 etcd[9458]: enabled capabilities for version 3.4
Hint: Some lines were ellipsized, use -l to show in full.

####测试集群是否健康

[root@k8s-etcd3 etcd]# ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints=“https://192.168.43.121:2379,https://192.168.43.122:2379,https://192.168.43.123:2379” endpoint health
https://192.168.43.123:2379 is healthy: successfully committed proposal: took = 16.678196ms
https://192.168.43.122:2379 is healthy: successfully committed proposal: took = 21.300035ms
https://192.168.43.121:2379 is healthy: successfully committed proposal: took = 22.545504ms
You have new mail in /var/spool/mail/root

###如出现上诉显示就代表etcd集群搭建成功了