k8s官网关于kubeadm升级k8s集群文档
    k8s的更新非常快，几乎每几周就会发布一个小版本。因此使用集群一年之后，可能和最新稳定版本的k8s相差几个大版本，如果想要使用k8s新版本的一些新功能，是必须要升级k8s集群的。如果追求稳定就不必要升级了。
    使用kubeadm升级集群非常简单、安全，下面就演示案例使用kubeadm将集群从1.15.1升级到1.16.1。另外由于kubeadm不支持跨大版本的升级，只能1个、1个小版本的升级。事实上也没必要跨多个小版本升级，因为这对于集群的稳定性和安全性也是巨大的挑战。
    k8s集群升级是主要是对k8s的几个核心组件(例如api-server、etcd、controller-manager、coredns、scheduler)进行升级。需要注意为了保证集群的高可用，集群最好是一主一备。

升级步骤：

1. 升级master节点的kubeadm和kubectl。
2. 在master节点使用kubeadm升级集群。
3. 升级master节点的kubelet。
4. 验证节点的状态。
5. 升级其他节点kubelet。
6. 升级其他节点的kubeadm、kubectl。

1. 更新集群

1.1 升级kubeadm和kubectl的版本

[root@k8s-master01 ~]# yum install -y kubeadm-1.16.1 kubectl-1.16.1

注意这里不能下载kubelet，因为kubelet是最后升级版本的。同时这里不能下载最新版本的kubeadm和kubectl
如果下载了错误版本的kubeadm、kubectl、kubelet可以使用yum downgrade XXX-指定的版本。
这里需要升级到哪个版本就下载哪个版本的kubeadm和kubectl

1.2 查看、验证kubeadm的升级计划

先查看一下当前的k8s集群、kubeadm、kubectl、kubelet的版本

[root@k8s-master01 ~]# kubectl get node -o wide
NAME           STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
k8s-master01   Ready    master   21m   v1.15.1   192.168.1.180   <none>        CentOS Linux 7 (Core)   4.4.236-1.el7.elrepo.x86_64   docker://19.3.12
k8s-node01     Ready    <none>   20m   v1.15.1   192.168.1.181   <none>        CentOS Linux 7 (Core)   4.4.236-1.el7.elrepo.x86_64   docker://19.3.12
k8s-node02     Ready    <none>   20m   v1.15.1   192.168.1.182   <none>        CentOS Linux 7 (Core)   4.4.236-1.el7.elrepo.x86_64   docker://19.3.12
[root@k8s-master01 ~]# kubelet --version
Kubernetes v1.15.1
[root@k8s-master01 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:18:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:09:21Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
[root@k8s-master01 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:15:32Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

可以看到kubeadm、kubectl、kubelet都是1.15.1版本。

查看一下升级到1.16.1执行计划

[root@k8s-master01 ~]# kubeadm upgrade plan 1.16.1
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.15.1
[upgrade/versions] kubeadm version: v1.15.1

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
Kubelet     3 x v1.15.1   1.16.1

Upgrade to the latest version in the v1.15 series:

COMPONENT            CURRENT   AVAILABLE
API Server           v1.15.1   1.16.1
Controller Manager   v1.15.1   1.16.1
Scheduler            v1.15.1   1.16.1
Kube Proxy           v1.15.1   1.16.1
CoreDNS              1.3.1     1.3.1
Etcd                 3.3.10    3.3.10

You can now apply the upgrade by executing the following command:

	kubeadm upgrade apply 1.16.1

Note: Before you can perform this upgrade, you have to update kubeadm to 1.16.1.

_____________________________________________________________________

如果出现Kubernetes master stuck on NotReady after failed to upgrade 可以查看NotReady after failed to upgrade

这里是可以升级到1.16.1版本的。

先将配置文件保存下载。这里可也以使用初始化集群的配置文件，修改k8s的版本即可。

[root@k8s-master01 ~]# kubeadm config view > kubeadm-config.yaml

修改镜像仓库的地址和k8s的版本

阿里云镜像仓库的地址：registry.cn-hangzhou.aliyuncs.com/google_containers

使用--dry-run尝试运行

kubeadm upgrade apply 1.16.1 --dry-run
···
[dryrun] Finished dryrunning successfully!

最后输出Finished dryrunning successfully!

1.3 升级集群

正式执行命令升级集群

[root@k8s-master01 ~]# kubeadm upgrade apply v1.16.1 --config kubeadm-config.yaml

这里kubeadm会先去拉取所需要的的镜像，可以提前先将需要的镜像下载好。
等待大概2分钟后，升级成功。

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.16.1". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

1.4 验证升级集群的效果

查看一下kubectl的版本

[root@k8s-master01 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T16:51:36Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

可以看到已经升级成功了。

查看Pod的信息

[root@k8s-master01 ~]# kubectl get pod --all-namespaces 
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   coredns-67c766df46-m4tkb               1/1     Running   0          4m2s
kube-system   coredns-67c766df46-zs47g               1/1     Running   0          4m2s
kube-system   etcd-k8s-master01                      1/1     Running   0          4m39s
kube-system   kube-apiserver-k8s-master01            1/1     Running   0          4m38s
kube-system   kube-controller-manager-k8s-master01   1/1     Running   0          4m34s
kube-system   kube-flannel-ds-7pvj8                  1/1     Running   1          58m
kube-system   kube-flannel-ds-k94tq                  1/1     Running   1          58m
kube-system   kube-flannel-ds-vjgt6                  1/1     Running   1          58m
kube-system   kube-proxy-98pkg                       1/1     Running   0          3m42s
kube-system   kube-proxy-fb5xt                       1/1     Running   0          3m53s
kube-system   kube-proxy-rt66c                       1/1     Running   0          3m47s
kube-system   kube-scheduler-k8s-master01            1/1     Running   0          4m31s

核心的Pod都是正常运行的

2. 更新kubelet

2.1 更新主节点的kubelet

查看一下node的信息

[root@k8s-master01 ~]# kubectl get node
NAME           STATUS   ROLES    AGE   VERSION
k8s-master01   Ready    master   61m   v1.15.1
k8s-node01     Ready    <none>   59m   v1.15.1
k8s-node02     Ready    <none>   59m   v1.15.1

可以看到node的VERSION还是1.15.1，这是因为node没有升级kubelet的版本。
同时在kubeadm升级k8s集群日志输出的最后也提示升级kubelet

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

接下来根据提示升级kubelet

#升级kubelet
[root@k8s-master01 ~]# yum install -y kubelet-1.16.1
#重启kubelet
[root@k8s-master01 ~]# systemctl daemon-reload && systemctl restart kubelet
#查看node信息
[root@k8s-master01 ~]# kubectl get nodes 
NAME           STATUS   ROLES    AGE   VERSION
k8s-master01   Ready    master   64m   v1.16.1
k8s-node01     Ready    <none>   63m   v1.15.1
k8s-node02     Ready    <none>   63m   v1.15.1

注意这里千万不能下载最新版本的kubelet，需要下载指定版本的kubelet。
如果出现Kubernetes master stuck on NotReady after failed to upgrade 可以查看NotReady after failed to upgrade
使用yum downgrade kubelet-XXX命令将kubelet降级到指定的版本。

2.2 更新其他节点上的kubelet

需要注意的是，当更新节点上的kubelet时，可以设置节点为不可调度，将运行在节点上的Pod驱离，防止意外发生。
下载指定版本的kubelet，重启kubelet

yum install -y kubelet-1.16.1 && systemctl daemon-reload && systemctl restart kubelet

2.3 验证更新的结果

最后在master节点查看node信息

[root@k8s-master01 ~]# kubectl get nodes 
NAME           STATUS   ROLES    AGE   VERSION
k8s-master01   Ready    master   69m   v1.16.1
k8s-node01     Ready    <none>   68m   v1.16.1
k8s-node02     Ready    <none>   67m   v1.16.1

到此k8s集群的升级工作完成了。

3. 升级其他节点的kubeadm、kubectl

#在node01节点
[root@k8s-node01 ~]# yum install -y kubeadm-1.16.1 kubectl-1.16.1
#在node02节点
[root@k8s-node02 ~]# yum install -y kubeadm-1.16.1 kubectl-1.16.1