K8S/Kubernetes

网上文档也多，安装的时候，还是踩了几个坑。

现在作一个安装记录吧。

1，先作自签名的证书ca-csr.json(为了和k8s共用根证书，可能将信息调为k8s)。

{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "ca": {
    "expiry": "438000h"
    },
    "names": [
    {
      "C": "CN",
      "ST": "ShangHai",
      "L": "ShangHai",
      "O": "kubernetes",
      "OU": "system"
    }
  ]
}

2,生成根证书和key。

cfssl gencert -initca  ca-csr.json | cfssljson -bare ca

这一步，会生成ca.pem和ca-key.pem文件。

3，有了根证书之后，建一个ca-config.json文件，用它来作为签发其它证书的配置文件。

{
    "signing": {
        "default": {
            "expiry": "438000h"
        },
        "profiles": {
            "server": {
                "expiry": "438000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "438000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "438000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

4，生成一个etcd_server_demo.json。

{
    "CN": "etcdServer",
    "hosts": [
        "127.0.0.1",
        "localhost",
	"etcd_ip_address"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "O": "etcd",
            "OU": "etcd Security",
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai"
        }
    ]
}

这个json文件，我们暂时不生成证书，因为不同的etcd节点，需要证书认证的IP不一样，我们可以通过脚本把IP传进去，即时生成证书。

5，生成一个etcd_peer.json文件

{
    "CN": "etcdPeer",
    "hosts": [
	"127.0.0.1",
	"localhost",
　　　　　"peer*.ip",
	"server.etcd.cluster.local",
	"peer1.etcd.cluster.local",
	"peer2.etcd.cluster.local",
	"peer3.etcd.cluster.local",
	"peer4.etcd.cluster.local",
	"peer5.etcd.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
	    "O": "etcd",
	    "OU": "etcd Security",
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai"
        }
    ]
}

6，使用上面的json文件，生成peer之间证书(未启用peer认证，因为是内部网，只启用了apiserver到2379的tls认证)。

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=etcd_ca-config.json -profile=peer etcd-peer.json | cfssljson -bare peer

 

7，生成一个etcd_client.json文件

{
    "CN": "etcdClient",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
	    "O": "etcd",
	    "OU": "etcd Security",
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai"
        }
    ]
}

 

8，使用上面的json文件，生成一个客户端认证的证书(这个证书，改了名字之后，可用于k8s的healthcheck，也可用于apiserver-etcd-client)。

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=etcd_ca-config.json -profile=client etcd-client.json | cfssljson -bare client

9,生成了这些证书之后，将所有证书和cfssl放到pik/目录下，使用下面这个etcd.sh脚本，进行etcd的集群安装。

#! /usr/bin/env bash

set -e
set -u
set -x


THIS_HOST=$1

REGISTRY=harbor/k8s.gcr.io/etcd-amd64
ETCD_VERSION=3.2.18
TOKEN=k8s-etcd-token
CLUSTER_STATE=new
HOST_1=1.2.3.4
HOST_2=5.6.7.8
HOST_3=9.0.1.2
CLUSTER=${HOST_1}=http://${HOST_1}:2380,${HOST_2}=http://${HOST_2}:2380,${HOST_3}=http://${HOST_3}:2380
DATA_DIR=/etcd/etcd-data
PKI_SRC_DIR=etcd/pki
PKI_DST_DIR=/pki
LOCAL_IP=$(hostname -I)
# 以上变量,三个节点相同

#判断输入的IP是否在集群内，是否为本机IP
if [[ ${THIS_HOST} != ${HOST_1} && ${THIS_HOST} != ${HOST_2} && ${THIS_HOST} != ${HOST_3} ]]; then
  echo "ip not in the etcd cluster host."
  exit 110
fi

if ! [[ ${LOCAL_IP} =~ ${THIS_HOST} ]]; then
  echo "ip not in the local ip."
  exit 110
fi


# 替换IP生成证书
/bin/cp -rf pki/etcd_server_demo.json pki/etcd_server.json
sed -i "s/etcd_ip_address/${THIS_HOST}/g" pki/etcd_server.json
pki/cfssl gencert -ca=pki/ca.pem -ca-key=pki/ca-key.pem -config=pki/ca-config.json -profile=server pki/etcd_server.json|pki/cfssljson -bare pki/server


# 运行docker
docker run \
   -p 2379:2379 \
   -p 2380:2380 \
   --volume=${DATA_DIR}:/etcd-data \
   --volume=${PKI_SRC_DIR}:${PKI_DST_DIR} \
   --name etcd ${REGISTRY}:${ETCD_VERSION} \
   /usr/local/bin/etcd \
   --data-dir=/etcd-data --name ${THIS_HOST} \
   --initial-advertise-peer-urls http://${THIS_HOST}:2380 \
   --listen-peer-urls http://0.0.0.0:2380 \
   --advertise-client-urls https://${THIS_HOST}:2379 \
   --listen-client-urls https://0.0.0.0:2379 \
   --initial-cluster ${CLUSTER} \
   --initial-cluster-state ${CLUSTER_STATE} \
   --initial-cluster-token ${TOKEN} \
   --cert-file=${PKI_DST_DIR}/server.pem \
   --key-file=${PKI_DST_DIR}/server-key.pem \
   --trusted-ca-file=${PKI_DST_DIR}/ca.pem

 

10，安装集群。将etcd.sh里的HOST_1~3更改为计划集群的IP地址。使用，在每台机器上，使用sh etcd.sh [本地IP]，即可安装。

11，常用的测试集群和测试集群的命令，还带tls的。

./etcdctl --ca-file=pki/ca.pem --cert-file=pki/client.pem --key-file=pki/client-key.pem --endpoints=https://1.2.3.4:2379 cluster-health

./etcdctl --ca-file=pki/ca.pem --cert-file=pki/client.pem --key-file=pki/client-key.pem --endpoints=https://1.2.3.4:2379 member list

./etcdctl --ca-file=pki/ca.pem --cert-file=pki/client.pem --key-file=pki/client-key.pem --endpoints=https://1.2.3.4:2379 set /message hello

./etcdctl --ca-file=pki/ca.pem --cert-file=pki/client.pem --key-file=pki/client-key.pem --endpoints=https://1.2.3.4:2379 get /message

curl -k --cert pki/client.pem  --key pki/client-key.pem https://1.2.3.4:2379/v2/keys/message