部署环境介绍

单Master架构图

服务器环境

• 操作系统：centos7.4 双核4G内存
  

实验前准备

1. 关闭Selinux，以及防火墙(Master与Node操作相同)

sed -i '/^SELINUX/s#enforcing#disabled#' /etc/selinux/config
systemctl stop firewalld

2. 修改主机名

#Master：
hostnamectl set-hostname master
su
#Node1：
hostnamectl set-hostname node1
su
#Node2：
hostnamectl set-hostname node2
su

Etcd群集搭建

Tips：一般来说生产环境会用5台etcd，本次实验因为资源有限只使用3台etcd

生成自签证书

1. 编写下载生成自签证书的cfssl命令

vim cfssl.sh

curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

2. 生成证书

mkdir k8s
cd k8s/
mkdir k8s-cert
mkdir etcd-cert
cd etcd-cert/
#(如果时间有问题需要ntpdate time.windows.com)

#生成CA证书
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json <<EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

3. 生成etcd域名证书

#生成etcd域名证书（IP）（一定要包含每个节点的IP）
cat > server-csr.json <<EOF
{
    "CN": "etcd",
    "hosts": [
    "192.168.18.10",
    "192.168.18.20",
    "192.168.18.30"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "NanJing",
            "ST": "NanJing"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

4. 查看部署好生成的证书
   

Etcd群集部署

1. 下载二进制包
   二进制包下载地址：https://github.com/coreos/etcd/releases/tag/v3.3.10

2. 创建etcd的功能目录

mkdir software
mv etcd-v3.3.10-linux-amd64.tar.gz ~/software/
cd software/ 
tar -zxvf etcd-v3.3.10-linux-amd64.tar.gz
cd etcd-v3.3.10-linux-amd64/
mkdir -p /opt/etcd/{cfg,bin,ssl}
mv etcd etcdctl /opt/etcd/bin/

3. 脚本生成Etcd配置文件及systemctl启动

• 执行etcd.sh脚本（发现服务无法正常启动，执行下面步骤）

  chmod +x etcd.sh
  ./etcd.sh etcd01 192.168.18.10 etcd02=https://192.168.18.20:2380,etcd03=https://192.168.18.30:2380

• etcd.sh脚本
  #!/bin/bash

# example: ./etcd.sh etcd01 192.168.1.10 etcd02=https://192.168.1.11:2380,etcd03=https://192.168.1.12:2380

ETCD_NAME=$1
ETCD_IP=$2
ETCD_CLUSTER=$3

WORK_DIR=/opt/etcd

cat <<EOF >$WORK_DIR/cfg/etcd
#[Member]
ETCD_NAME="${ETCD_NAME}"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

cat <<EOF >/usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=${WORK_DIR}/cfg/etcd
ExecStart=${WORK_DIR}/bin/etcd \
--name=\${ETCD_NAME} \
--data-dir=\${ETCD_DATA_DIR} \
--listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=\${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=${WORK_DIR}/ssl/server.pem \
--key-file=${WORK_DIR}/ssl/server-key.pem \
--peer-cert-file=${WORK_DIR}/ssl/server.pem \
--peer-key-file=${WORK_DIR}/ssl/server-key.pem \
--trusted-ca-file=${WORK_DIR}/ssl/ca.pem \
--peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

4. 拷贝证书

cp ~/k8s/etcd-cert/{ca,server-key,server}.pem /opt/etcd/ssl/

5. 将证书拷贝到两个node节点

scp -r /opt/etcd/ root@192.168.18.20:/opt
scp -r /opt/etcd/ root@192.168.18.30:/opt

6. 将systemctl启动文件拷贝到两个node节点

scp /usr/lib/systemd/system/etcd.service root@192.168.18.20:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/etcd.service root@192.168.18.30:/usr/lib/systemd/system/

7. 修改两个node节点的配置文件

vim /opt/etcd/cfg/etcd

• node1：
  
• node2
  

8. 启动所有节点的etcd服务

systemctl daemon-reload
systemctl start etcd
systemctl enable etcd

• 我们可以通过查看日志来etcd群集是否部署完成

tail -f /var/log/message

• 没有连接上的示意图
  

• 连接上的示意图
  

9. 部署完成之后查看健康状态

 /opt/etcd/bin/etcdctl \
--ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem \
--endpoints="https://192.168.18.10:2379,https://192.168.18.20:2379,https://192.168.18.30:2379" \
cluster-health
 

• 群集健康如图所示：
  
• 如果群集不健康，我们可以通过查看日志或journalctl -u etcd查看

Etcd配置文件的参数介绍

• ETCD_NAME 节点名称
• ETCD_DATA_DIR 数据目录
• ETCD_LISTEN_PEER_URLS 集群通信监听地址
• ETCD_LISTEN_CLIENT_URLS 客户端访问监听地址
• ETCD_INITIAL_ADVERTISE_PEER_URLS 集群通告地址
• ETCD_ADVERTISE_CLIENT_URLS 客户端通告地址
• ETCD_INITIAL_CLUSTER 集群节点地址
• ETCD_INITIAL_CLUSTER_TOKEN 集群Token
• ETCD_INITIAL_CLUSTER_STATE 加入集群的当前状态，new是新集群，existing表示加入