k8s滚动升级_kubernetes之滚动更新

kubernetes之滚动更新滚动更新滚动更新是一次只更新一小部分副本，成功后，在更新更多的副本，最终完成所有副本的更新，滚动更新的好处是零停机，整个过程始终有副本再运行，从而保证业务的连续性下面我们不熟三副本应用，初始镜像为httpd：2.2 然后将其更新到httpd：2.4httpd：2.2配置文件：[root@master music]# cathttpd.ymlapiVersion: ..

weixin_39763033

732人浏览 · 2021-01-27 03:39:07

weixin_39763033 · 2021-01-27 03:39:07 发布

kubernetes之滚动更新

滚动更新

滚动更新是一次只更新一小部分副本，成功后，在更新更多的副本，最终完成所有副本的更新，滚动更新的好处是零停机，整个过程始终有副本再运行，从而保证业务的连续性

下面我们不熟三副本应用，初始镜像为httpd：2.2 然后将其更新到httpd：2.4

httpd：2.2配置文件：

[root@master music]# cathttpd.yml

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

run: apache

spec:

replicas:3selector:

matchLabels:

run: apache

template:

metadata:

labels:

run: apache

spec:

containers:-name: httpd

image: httpd:2.4ports:- containerPort: 80

查看一下pod：

[root@master music]# kubectl get pod -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES

http-deploy-849cf97446-6k8jj 1/1 Running 0 2m28s 10.244.1.54 node1 http-deploy-849cf97446-l987p 1/1 Running 0 2m28s 10.244.1.55 node1 http-deploy-849cf97446-mtsqf 1/1 Running 0 2m28s 10.244.2.42 node2

在查看一下当前版本：

[root@master music]# kubectl get replicasets.apps -o wide

NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR

http-deploy-849cf97446 3 3 3 10m httpd httpd:2.2 pod-template-hash=849cf97446,run=apache

现在我们来滚动更新：把配置文件htppd.yml镜像httpd：2.2 更改为 httpd2.4，然后重新执行

现在我们再来看看

[root@master music]# kubectl get replicasets.apps -o wide

NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR

http-deploy-77c8788b9b 3 3 3 39s httpd httpd:2.4 pod-template-hash=77c8788b9b,run=apache

http-deploy-849cf97446 0 0 0 13m httpd httpd:2.2 pod-template-hash=849cf97446,run=apache

发现了变化镜像2.2变成了2.4，重新创建了pod 镜像为2.4

[root@master music]# kubectl describe deployment

Name: http-deploy

Namespace: default

CreationTimestamp: Mon,20 Jul 2020 20:08:32 +0800Labels: run=apache

Annotations: deployment.kubernetes.io/revision: 2kubectl.kubernetes.io/last-applied-configuration:

{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"l

Selector: run=apache

Replicas:3 desired | 3 updated | 3 total | 3 available | 0unavailable

StrategyType: RollingUpdate

MinReadySeconds:0RollingUpdateStrategy:25% max unavailable, 25%max surge

Pod Template:

Labels: run=apache

Containers:

httpd:

Image: httpd:2.4Port:80/TCP

Host Port:0/TCP

Environment:Mounts:Volumes:Conditions:

Type Status Reason---- ------ ------Available True MinimumReplicasAvailable

Progressing True NewReplicaSetAvailable

OldReplicaSets:NewReplicaSet: http-deploy-77c8788b9b (3/3replicas created)

Events:

Type Reason Age From Message---- ------ ---- ---- -------Normal ScalingReplicaSet 17m deployment-controller Scaled up replica set http-deploy-849cf974

Normal ScalingReplicaSet 5m9s deployment-controller Scaled up replica set http-deploy-77c8788b

Normal ScalingReplicaSet 4m52s deployment-controller Scaled down replica set http-deploy-849cf9

Normal ScalingReplicaSet 4m52s deployment-controller Scaled up replica set http-deploy-77c8788b

Normal ScalingReplicaSet 4m35s deployment-controller Scaled down replica set http-deploy-849cf9

Normal ScalingReplicaSet 4m35s deployment-controller Scaled up replica set http-deploy-77c8788b

Normal ScalingReplicaSet 4m34s deployment-controller Scaled down replica set http-deploy-849cf9

每次只更新替换一个pod，每次更换的pod数量是可以定制的，kubernetes提供了两个参数maxSurge和 maxUnavailable，来精细更换pod数量

回滚

kubectl apply 每次更新应用时 kubernetes都会记录下当然的配置，，保存为一个 revision(版次)，这样就可以回滚到某个指定的revision

就是在执行的时候后面跟上一个参数， --record

下面我们来创建三个配置文件，三个文件版本不一样就可以我们用httpd：2.37，httpd：2.38，httpd：2.39

[root@master music]# cathttpd.yml

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

run: apache

spec:

replicas:3selector:

matchLabels:

run: apache

template:

metadata:

labels:

run: apache

spec:

containers:-name: httpd

image: httpd:2.4.37 ##其余两个在这里就不写在这里了，把镜像版本改了就可以了ports:- containerPort: 80

执行：

[root@master music]# kubectl apply -f httpd.yml --record

deployment.apps/http-deploy created

[root@master music]# kubectl apply-f httpd1.yml --record

deployment.apps/http-deploy configured

[root@master music]# kubectl apply-f httpd2.yml --record

deployment.apps/http-deploy configured

通过查看可以看到每一次的更新。

[root@master music]# kubectl get deployments.apps -o wide

NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR

http-deploy 3/3 3 3 5m14s httpd httpd:2.4.39 run=apache

这是由2.4.37更新到2.4.39

--record的作用是将当前的命令记录到revision记录中，这样我们就可以知道每个revision对应的是那个配置文件了，通过

kubectl rollout history deployment 查看revision历史记录

[root@master music]# kubectl rollout history deployment

deployment.apps/http-deploy

REVISION CHANGE-CAUSE1 kubectl apply --filename=httpd.yml --record=true

2 kubectl apply --filename=httpd1.yml --record=true

3 kubectl apply --filename=httpd2.yml --record=true

如果想要回到某个版本，比如说最初的2.4.37.可以执行命令

[root@master music]# kubectl rollout history deployment ##先查看一下历史版本

deployment.apps/http-deploy

REVISION CHANGE-CAUSE1 kubectl apply --filename=httpd.yml --record=true

2 kubectl apply --filename=httpd1.yml --record=true

3 kubectl apply --filename=httpd2.yml --record=true[root@master music]# kubectl get deployments.apps-o wide

NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR

http-deploy 3/3 3 3 21m httpd httpd:2.4.39 run=apache

[root@master music]# kubectl rollout undo deployment--to-revision=1deployment.apps/http-deploy rolled back

[root@master music]# kubectl get deployments.apps-o wide

NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR

http-deploy 3/3 3 3 22m httpd httpd:2.4.37 run=apache

可以看到我们回到了我们制定的最开始的版本，此时。版本历史也会发生相应变化

[root@master music]# kubectl rollout history deployment

deployment.apps/http-deploy

REVISION CHANGE-CAUSE2 kubectl apply --filename=httpd1.yml --record=true

3 kubectl apply --filename=httpd2.yml --record=true

4 kubectl apply --filename=httpd.yml --record=true

之前的1变成了4

Health Check

强大的自愈能力是k8s这类容器编排引擎的一个重要特性，自愈的默认实现方式是自动重启发生故障的容器，除此之外，用户还可以利用liveness和readiness探测机制设置更精细的健康检查，进而实现如下需求

1:0停机部署

2：避免部署无效的镜像

3：更加安全的滚动升级

默认的健康检查

下面我们来模拟一个容器发生故障的场景，pod配置如下

[root@master health]# cathealth.yml

apiVersion: v1

kind: Pod

metadata:

labels:

test: healthcheck

spec:

restartPolicy: OnFailure

containers:-name: healthcheck

image: busybox

args:- /bin/bash- -c- sleep 10;exit 1

pod的restartpolicy 设置为onfailure，默认为always

sleep10；exit1 模拟容器启动10秒后发生故障

执行创建pod 命名为healthcheck

[root@master health]# kubectl get pods

NAME READY STATUS RESTARTS AGE

healthcheck0/1 CrashLoopBackOff 6 7m37s

可见容器已经启动了6次

liveness探测

liveness探测让用户可以自定义判断容器是否健康的条件，如果探测失败，k8s就会重启容器

案例

[root@master health]# catliveness.yml

apiVersion: v1

kind: Pod

metadata:

labels:

test: liveness

spec:

restartPolicy: OnFailure

containers:-name: liveness

image: busybox

args:- /bin/sh

- -c- touch /tmp/healthy;sleep 30;rm -rf /tmp/healthy;sleep 600livenessProbe:

exec:

command:- cat

- /tmp/healthy

initialDelaySeconds:10periodSeconds:5

执行以后进程是首先创建文件/tmp/healthy,30秒以后删除，如果文件存在则健康，否则就会认为是故障

可以通过查看日志

kubectl describe pod liveness

[root@master health]# kubectl describe pod liveness

Name: liveness

Namespace: default

Priority:0Node: node2/192.168.172.136Start Time: Mon,20 Jul 2020 22:01:31 +0800Labels: test=liveness

Annotations: kubectl.kubernetes.io/last-applied-configuration:

{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"test":"liveness"},"name":"liveness","namespace":"default"},"spec":...

Status: Running

IP:10.244.2.50IPs:

IP:10.244.2.50Containers:

liveness:

Container ID: docker://5a535ca4965f649b90161b72521c4bc75c52097f7a6f0f816dee991a0000156e

Image: busybox

Image ID: docker-pullable://busybox@sha256:9ddee63a712cea977267342e8750ecbc60d3aab25f04ceacfa795e6fce341793

Port: Host Port:Args:/bin/sh

-ctouch /tmp/healthy;sleep 30;rm -rf /tmp/healthy;sleep 600State: Waiting

Reason: CrashLoopBackOff

Last State: Terminated

Reason: Error

Exit Code:137Started: Mon,20 Jul 2020 22:10:13 +0800Finished: Mon,20 Jul 2020 22:11:27 +0800Ready: False

Restart Count:6Liveness: exec [cat /tmp/healthy] delay=10s timeout=1s period=5s #success=1 #failure=3Environment:Mounts:/var/run/secrets/kubernetes.io/serviceaccount from default-token-ptz8b (ro)

Conditions:

Type Status

Initialized True

Ready False

ContainersReady False

PodScheduled True

Volumes:

default-token-ptz8b:

Type: Secret (a volume populated by a Secret)

SecretName: default-token-ptz8b

Optional:falseQoS Class: BestEffort

Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for300s

node.kubernetes.io/unreachable:NoExecute for300s

Events:

Type Reason Age From Message---- ------ ---- ---- -------Normal Scheduled 12m default-scheduler Successfully assigned default/liveness to node2

Normal Pulled 9m43s (x3 over 12m) kubelet, node2 Successfully pulled image"busybox"Normal Created 9m43s (x3 over 12m) kubelet, node2 Created container liveness

Normal Started 9m43s (x3 over 12m) kubelet, node2 Started container liveness

Normal Killing 8m58s (x3 over 11m) kubelet, node2 Container liveness failed liveness probe, will be restarted

Normal Pulling 8m28s (x4 over 12m) kubelet, node2 Pulling image"busybox"Warning Unhealthy 7m48s (x10 over 12m) kubelet, node2 Liveness probe failed:cat: can't open'/tmp/healthy': No such file or directory

Warning BackOff 2m50s (x4 over 3m3s) kubelet, node2 Back-off restarting failed container

[root@master health]# kubectl get pods

NAME READY STATUS RESTARTS AGE

liveness1/1 Running 0 27s

Readiness探测

liveness探测可以告诉kubernetes什么时候通过重启容器实现自愈，readiness探测告诉kubernetes什么时候可以将容器加入到service负载均衡池中，对外提供服务

[root@master health]# cat readiness.yml

apiVersion: v1

kind: Pod

metadata:

labels:

test: readiness

spec:

restartPolicy: OnFailure

containers:-name: readiness

image: busybox

args:- /bin/sh- -c- touch /tmp/healthy;sleep 30;rm -rf /tmp/healthy;sleep 600readinessProbe:

exec:

command:-cat- /tmp/healthy

initialDelaySeconds:10periodSeconds:5

查看：

[root@master health]# kubectl getpods

NAME READY STATUS RESTARTS AGE

liveness1/1 Running 615h

readiness0/1 Running 0 20s

[root@master health]# kubectl describe pod readiness

Name: readiness

Namespace:defaultPriority:0Node: node2/192.168.172.136Start Time: Tue,21 Jul 2020 14:12:41 +0800Labels: test=readiness

Annotations: kubectl.kubernetes.io/last-applied-configuration:

{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"test":"readiness"},"name":"readiness","namespace":"default"},"spec...

Status: Running

IP:10.244.2.52IPs:

IP:10.244.2.52Containers:

readiness:

Container ID: docker://22465a3dd79db1bead14ec352348c544c76a5d1b7808882b328563531f32317b

Image: busybox

Image ID: docker-pullable://busybox@sha256:9ddee63a712cea977267342e8750ecbc60d3aab25f04ceacfa795e6fce341793

Port: Host Port:Args:/bin/sh-c

touch/tmp/healthy;sleep 30;rm -rf /tmp/healthy;sleep 600State: Running

Started: Tue,21 Jul 2020 14:12:58 +0800Ready: False

Restart Count:0Readiness: exec [cat/tmp/healthy] delay=10s timeout=1s period=5s #success=1 #failure=3Environment:Mounts:/var/run/secrets/kubernetes.io/serviceaccount from default-token-ptz8b (ro)

Conditions:

Type Status

Initialized True

Ready False

ContainersReady False

PodScheduled True

Volumes:default-token-ptz8b:

Type: Secret (a volume populated by a Secret)

SecretName:default-token-ptz8b

Optional:falseQoS Class: BestEffort

Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for300s

node.kubernetes.io/unreachable:NoExecute for300s

Events:

Type Reason Age From Message---- ------ ---- ---- -------Normal Scheduled 87sdefault-scheduler Successfully assigned default/readiness to node2

Normal Pulling 86s kubelet, node2 Pulling image"busybox"Normal Pulled 70s kubelet, node2 Successfully pulled image"busybox"Normal Created 70s kubelet, node2 Created container readiness

Normal Started 70s kubelet, node2 Started container readiness

Warning Unhealthy 0s (x8 over 35s) kubelet, node2 Readiness probe failed: cat: can't open'/tmp/healthy': No such file or directory

下面对Liveness探测和Readiness探测做个比较:

Liveness探测和Readiness探测是两种Health Check机制,如果不特意配置, Kubernetes将对两种探测采取相同的默认行为,即通过判断容器启动进程的返回值是否为零来判断探测是否成功。

两种探测的配置方法完全一样,支持的配置參数也一样。不同之处在于探测失败后的行为: Liveness探测是重启容器;Readiness探测则是将容器设置为不可用,不接收Service转发的请求。

Liveness探测和Readiness探测是独立执行的,二者之间没有依赖,所以可以单独使用,也可以同时使用。用Liveness探测判断容器是否需要重启以实现自愈;用Readiness探测判断容器是否已经准备好对外提供服务。

在 Scale up 中使用 health check

对于多副本应用,当执行Scale Up操作时,新副本会作为backend被添加到Service的负责均衡中,与已有副本一起处理客户的请求。考虑到应用启动通常都需要一个准备阶段,比如加载缓存数据,连接数据库等,从容器启动到正真能够提供服务是需要一段时间的。我们可以通过Readiness探测判断容器是否就绪,避免将请求发送到还没有ready的backend.

案例：

[root@master music]# cat server1.yml

apiVersion: v1

kind: Service

metadata:

spec:

selector:

run: httpd

ports:-protocol: TCP

port:8080targetPort:80

[root@master music]# cat web-svc.yml

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: web

spec:

replicas:3selector:

matchLabels:

app: web

template:

metadata:

labels:

app: web

spec:

containers:-image: httpd

ports:- containerPort: 8080readinessProbe:

httpGet:

scheme: HTTP

path:/healthy

port:8080initialDelaySeconds:10periodSeconds:5

上面配置的作用是:

容器启动10秒之后开始探测。如果http://container-ip]:8080/healthy返回代码不是200-400,表示容器没有就绪,不接收Service web-svc的请求。

每隔5秒再探测一次。

直到返回代码为200-400,表明容器已经就绪,然后将其加入到web-svc的负载均衡中,开始处理客户请求。探测会继续以5秒的间隔执行,如果连续发生3次失败,容器又会从负载均衡中移除,直到下次探测成功重新加入。

在滚动更新中使用health check

在滚动更新的时候正常副本需要10秒的时间准备工作，在此之间无法响应业务需求，如果因为人为配置错误，导致副本无法启动，那么是一个很严重的后果，在滚动更新中使用健康检查就是为了防止发生错误，新的副本只有通过readiness探测才会被添加到service如果没有通过将不会全部更换副本，业务还是可以正常运行

案例：

[root@master music]# cat web-svc.yml

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: app

spec:

replicas:10selector:

matchLabels:

run: app

template:

metadata:

labels:

run: app

spec:

containers:-image: busybox

args:- /bin/sh- -c- sleep 10;touch /tmp/healthy;sleep 30000readinessProbe:

exec:

command:-cat- /tmp/healthy

initialDelaySeconds:10periodSeconds:5

[root@master music]# kubectl getdeployments.app

NAME READY UP-TO-DATE AVAILABLE AGE

app10/10 10 10 100s

[root@master music]# kubectl getpods

NAME READY STATUS RESTARTS AGE

app-79fc8699cd-2c6tt 1/1 Running 03m7s

app-79fc8699cd-5svr7 1/1 Running 03m7s

app-79fc8699cd-97hml 1/1 Running 03m7s

app-79fc8699cd-bnbzh 1/1 Running 03m7s

app-79fc8699cd-czghg 1/1 Running 03m7s

app-79fc8699cd-j9dxt 1/1 Running 03m7s

app-79fc8699cd-ltqwx 1/1 Running 03m7s

app-79fc8699cd-prv8f 1/1 Running 03m7s

app-79fc8699cd-wh4nv 1/1 Running 03m7s

app-79fc8699cd-znpws 1/1 Running 0 3m7s

通过readiness检测后每10秒会启动一个副本

接下来滚动更新应用

[root@master music]# cat caoyi.yml

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: app

spec:

strategy:

rollingUpdate:

maxSurge:35%maxUnavailable:35%replicas:10selector:

matchLabels:

run: app

template:

metadata:

labels:

run: app

spec:

containers:-image: busybox

args:- /bin/sh- -c- sleep 30000readinessProbe:

exec:

command:-cat- /tmp/healthy

initialDelaySeconds:10periodSeconds:5

查看

[root@master music]# kubectl getdeployments.apps

NAME READY UP-TO-DATE AVAILABLE AGE

app7/10 7 7 4m15s

新副本中不存在/tmp/healthy，所以是无法通过检测的

[root@master music]# kubectl describe deployment app

Name: app

Namespace:defaultCreationTimestamp: Tue,21 Jul 2020 15:51:18 +0800Labels: app=app

Annotations: deployment.kubernetes.io/revision: 2kubectl.kubernetes.io/last-applied-configuration:

{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"kubernetes.io/change-cause":"kubectl apply --filename=caoyi.yml --...

kubernetes.io/change-cause: kubectl apply --filename=caoyi.yml --record=trueSelector: run=app

Replicas:10 desired | 7 updated | 14 total | 7 available | 7unavailable

StrategyType: RollingUpdate

MinReadySeconds:0RollingUpdateStrategy:35% max unavailable, 35%max surge

Pod Template:

Labels: run=app

Containers:

app:

Image: busybox

Port:Host Port:Args:/bin/sh-c

sleep30000Readiness: exec [cat/tmp/healthy] delay=10s timeout=1s period=5s #success=1 #failure=3Environment:Mounts:Volumes:Conditions:

Type Status Reason---- ------ ------Available True MinimumReplicasAvailable

Progressing True ReplicaSetUpdated

OldReplicaSets: app-79fc8699cd (7/7replicas created)

NewReplicaSet: app-77cdb45995 (7/7replicas created)

Events:

Type Reason Age From Message---- ------ ---- ---- -------Normal ScalingReplicaSet 6m37s deployment-controller Scaled up replica set app-79fc8699cd to 10Normal ScalingReplicaSet 4m30s deployment-controller Scaled up replica set app-77cdb45995 to 4Normal ScalingReplicaSet 4m30s deployment-controller Scaled down replica set app-79fc8699cd to 7Normal ScalingReplicaSet 4m30s deployment-controller Scaled up replica set app-77cdb45995 to 7

maxSurge

此参数控制滚动更新过程中副本总数的超过DESIRED的上限。maxSurge可以是具体的整数(比如3) ,也可以是百分百,向上取整。maxSurge默认值为25%在上面的例子中, DESIRED为10,那么副本总数的最大值为:roundUp(10 + 10 * 25%) = 13所以我们看到CURRENT就是13

maxUnavailable

此参数控制滚动更新过程中,不可用的副本相占DESIRED的最大比例。maxUnavailable可以是具体的整数(比如3) ,也可以是百分百,向下取整。maxUnavailable默认值为25%

回滚

先查看一下历史版本：

[root@master music]# kubectl rollout history deployment app

deployment.apps/app

REVISION CHANGE-CAUSE1 kubectl apply --filename=web-svc.yml --record=true

2 kubectl apply --filename=caoyi.yml --record=true

然后回滚：

[root@master music]# kubectl rollout undo deployment app --to-revision=1deployment.apps/app rolled back

查看：

[root@master music]# kubectl getpods

NAME READY STATUS RESTARTS AGE

app-79fc8699cd-27klw 1/1 Running 016m

app-79fc8699cd-d8ggf 1/1 Running 058s

app-79fc8699cd-gzl5p 1/1 Running 016m

app-79fc8699cd-mwt67 1/1 Running 016m

app-79fc8699cd-n65dj 1/1 Running 058s

app-79fc8699cd-rjsc9 1/1 Running 058s

app-79fc8699cd-v4lg8 1/1 Running 016m

app-79fc8699cd-wnlhj 1/1 Running 016m

app-79fc8699cd-wxvc2 1/1 Running 016m

app-79fc8699cd-zp5r4 1/1 Running 0 16m

可以看到已经回滚到第一个版本

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub