centos7 使用 kubeadm 安装 kubernetes

时间: 20191118
系统: CentOS Linux release 7.6.1810
vm: k8s-master: 192.168.116.4 k8s-node01:192.168.116.5

使用最新版的 kubeadm

[root@k8s-master ~]# kubeadm config images list
W1118 08:44:56.950660   14419 version.go:101] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1118 08:44:56.950738   14419 version.go:102] falling back to the local client version: v1.16.3
k8s.gcr.io/kube-apiserver:v1.16.3
k8s.gcr.io/kube-controller-manager:v1.16.3
k8s.gcr.io/kube-scheduler:v1.16.3
k8s.gcr.io/kube-proxy:v1.16.3
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2

国内无法访问 k8s.gcr.io
docker hub 中没有 mirrorgooglecontainers/kube-apiserver:v1.16.3
所以选择其它版本的 kubeadm
作者选择 kubeadm-1.14.0，试过其它的一些版本，有些依赖有问题

安装步骤；
k8s-master 执行第 1-8 步
k8s-node01 执行第 1,2,3,4,9 步
1 安装 docker
参考文章 https://blog.csdn.net/Man_In_The_Night/article/details/85791469

配置 docker 镜像加速器
使用阿里云 docker image 加速器
修改 /etc/docker/daemon.json 文件并添加上 registry-mirrors 键值,没有就新建/etc/docker/daemon.json 文件，添加如下内容：

[root@k8s-master ~]# cat /etc/docker/daemon.json
{
  "registry-mirrors": ["https://yourcode.mirror.aliyuncs.com"],
  "bip": "192.167.1.1/24"
}
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl restart docker

https://yourcode.mirror.aliyuncs.com 为你的阿里云加速地址，可以去阿里云申请，获取方法：
登录阿里云 https://cr.console.aliyun.com/, 容器镜像服务—镜像中心—镜像加速，查看加速器地址
bip 用来设置 docker 的 ip 地址范围

2 在 yum 仓库中添加 kubernetes

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装 kubeadm

[root@k8s-master ~]# yum install -y kubelet-1.14.0 kubeadm-1.14.0 kubectl-1.14.0 --disableexcludes=kubernetes
[root@k8s-master ~]# systemctl enable --now kubelet

3 国内镜像设置

[root@k8s-master ~]# cat alipull 
#!/bin/bash
images=(
    kube-apiserver:v1.14.0
    kube-controller-manager:v1.14.0
    kube-scheduler:v1.14.0
    kube-proxy:v1.14.0
    pause:3.1
    etcd:3.3.10
)

for imageName in ${images[@]} ; do
    docker pull mirrorgooglecontainers/$imageName
    docker tag mirrorgooglecontainers/$imageName k8s.gcr.io/$imageName
done
[root@k8s-master ~]# sh alipull

[root@k8s-master ~]# docker pull coredns/coredns:1.3.1
[root@k8s-master ~]# docker tag coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1

4 关闭防火墙，selinux

[root@k8s-master ~]# systemctl stop firewalld
[root@k8s-master ~]# systemctl disable firewalld
[root@k8s-master ~]# setenforce 0

在 sysctl config 文件中 设置 net.bridge.bridge-nf-call-iptables=1

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

5 kubeadm 初始化

[root@k8s-master ~]#  kubeadm init --kubernetes-version=v1.14.0 --apiserver-advertise-address=192.168.116.4 --pod-network-cidr=192.169.0.0/16
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.116.4:6443 --token 86hsm1.nm4nwdfdzlk2hxey \
    --discovery-token-ca-cert-hash sha256:7bcaa1b8d34c28eb2bacb88d94f5d5f53e8f73ec183962e28ce995e6c11417cb  

–kubernetes-version 为 kubernetes 版本
–apiserver-advertise-address 一般设置为 master ip
–pod-network-cidr pod 的 ip 地址段，根据所选择的 CNI 不同而不同，calico 默认是 192.168.0.0/16，如果修改安装 CNI 的时候也需要相应修改。作者安装 calico，主机网段和 192.168.0.0/16 冲突，安装前修改配置为 192.169.0.0/16
保存 kubeadm init 输出最后一段 “kubeadm join …" ，后面添加节点需要用到

6 测试：

[root@k8s-master ~]# curl https://127.0.0.1:6443 -k
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}

7 基础调试
以 root 用户运行 kubernetes，添加环境变量 KUBECONFIG=/etc/kubernetes/admin.conf

[root@k8s-master ~]# vim /etc/profile
...
export KUBECONFIG=/etc/kubernetes/admin.conf
...
[root@k8s-master ~]# source /etc/profile

默认情况下由于安全原因你的 cluster 不会调度 pods 在你的 master 上。如果你想让你的 master 也参与调度,去掉 master 节点的污点 taint

[root@k8s-master ~]# kubectl taint nodes --all node-role.kubernetes.io/master-
node/k8s-master untainted

或者

kubectl taint nodes k8s-master node-role.kubernetes.io/master-

查看 node，状态为 NotReady

[root@k8s-master ~]# kb get node
NAME         STATUS     ROLES    AGE     VERSION
k8s-master   NotReady   master   6m33s   v1.14.0

kubectl describe 查看节点 NotReady 原因

[root@k8s-master ~]# kubectl describe node k8s-master
Name:               k8s-master
Roles:              master
...
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
...
  Ready            False   Mon, 18 Nov 2019 20:39:51 +0800   Mon, 18 Nov 2019 20:31:43 +0800   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
  ...

因为没有安装 cni 插件

8 安装 calico
由于 kubeadm init 修改了 --pod-network-cidr=192.169.0.0，所以需要相应修改 calico 的 yaml 文件

[root@k8s-master ~]# wget https://docs.projectcalico.org/v3.8/manifests/calico.yaml
[root@k8s-master ~]# cat calico.yam
...
            - name: CALICO_IPV4POOL_CIDR
              value: "192.169.0.0/16"
              ...
[root@k8s-master ~]# kubectl apply -f calico.yaml

至此，单节点已经可以正常工作

9 添加多节点（k8s-node01 执行）

[root@k8s-node01 ~]# kubeadm join 192.168.116.4:6443 --token 86hsm1.nm4nwdfdzlk2hxey \
>     --discovery-token-ca-cert-hash sha256:7bcaa1b8d34c28eb2bacb88d94f5d5f53e8f73ec183962e28ce995e6c11417cb
[preflight] Running pre-flight checks
   ....
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[root@k8s-master Chapter05]# kb get nodes
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   13h   v1.14.0
k8s-node01   Ready    <none>   40m   v1.14.0

10 恢复 init 前的状态（k8s-master 执行）

kubeadm reset
kubeadm reset remove-etcd-member

11 非 master 节点使用 kubectl 查看集群信息
copy k8s-master 节点的 /etc/kubernetes/admin.conf 到 k8s-node01 的 /root/.kube/config

[root@k8s-node01 ~]# kubectl get po
NAME           READY   STATUS    RESTARTS   AGE
kubia-manual   1/1     Running   0          74m

/root/.kube/ 是默认位置，config 为默认文件，如果不使用默认文件，可以使用 --kubecofnig 指定特定路径

[root@k8s-node01 ~]# kubectl --kubeconfig ./admin.conf get po
NAME           READY   STATUS    RESTARTS   AGE
kubia-manual   1/1     Running   0          80m

参考文章：
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
https://juejin.im/post/5b8a4536e51d4538c545645c