环境

基于记录 - k8s 入门搭建 (1.16.0, helloweb) 搭建的环境。三个节点:192.168.199.200 (master, k8s0)、201 (ks81)、202 (k8s2)。

StatefulSet 用来管理有状态应用(的一组pod)。本篇主要参考 https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set,介绍基础sts。

hostPath 存储

在k8s里,容器与容器、pod与pod、节点与节点之间常常需要共享存储(storage, volume, PersistentVolume)。aws,azure,gce提供商用存储。本篇练习使用hostPath 类型的存储,参考 https://kubernetes.io/docs/concepts/storage/volumes/#hostpath。

hostPath 存储将节点(主机) 的文件/目录映射到pod里,例如将主机的 “/myapp/html” 目录映射为pod里的 “/usr/share/nginx/html”。hostPath 最主要的问题是需要维护在不同主机上同一目录内容的异同。

在三个节点创建测试目录、及建立一套相同内容的存储目录:

mkdir -p /test/k8s/basic_stateful_set
cd /test/k8s/basic_stateful_set

mkdir -p mnt/data
echo 'dat-0' > mnt/data/index.html
mkdir -p mnt/data2
echo 'dat-1' > mnt/data2/index.html
mkdir -p mnt/data3
echo 'dat-2' > mnt/data3/index.html
mkdir -p mnt/data4
echo 'dat-3' > mnt/data4/index.html
mkdir -p mnt/data5
echo 'dat-4' > mnt/data5/index.html

以下在k8s0 操作。

创建pv1.yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv1
  labels:
    type: local
spec:
  # storageClassName: manual
  capacity:
    storage: 1Mi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/test/k8s/basic_stateful_set/mnt/data"

类似再创建pv2~pv5.yaml,文件内容里相应修改pv2->data2, …, pv5->data5。

创建5个持久存储(PersistentVolume):

[root@k8s0 basic_stateful_set]# for i in 1 2 3 4 5; do kc create -f pv$i.yaml; done
persistentvolume/pv1 created
persistentvolume/pv2 created
persistentvolume/pv3 created
persistentvolume/pv4 created
persistentvolume/pv5 created
[root@k8s0 basic_stateful_set]# kc get pv
NAME   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pv1    1Mi        RWO            Retain           Available                                   29s
pv2    1Mi        RWO            Retain           Available                                   28s
pv3    1Mi        RWO            Retain           Available                                   28s
pv4    1Mi        RWO            Retain           Available                                   27s
pv5    1Mi        RWO            Retain           Available                                   27s

RWO (ReadWriteOnce) 意思是存储仅可被单个节点读写。

创建StatefulSet

创建web.yaml:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Mi

一个sts 创建2个有状态pod (replicas=2)。pod 里image=nginx 容器有映射的存储目录"/usr/share/nginx/html"。通过"name=www" 对应到pvc (volumeClaim)。
pod并不是直接引用pv,而是pod -> pvc -> pv。pvc 申明要使用pv。创建pvc时kubernetes 检查如果有符合条件的pv 则自动绑定该pv。

[root@k8s0 basic_stateful_set]# #创建
[root@k8s0 basic_stateful_set]# kc apply -f web.yaml 
service/nginx created
statefulset.apps/web created
[root@k8s0 basic_stateful_set]# 
[root@k8s0 basic_stateful_set]# kc get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   15d
nginx        ClusterIP   None         <none>        80/TCP    92s
[root@k8s0 basic_stateful_set]# #sts "web" 的2个pod:web-0, web-1 - 序号自动递增
[root@k8s0 basic_stateful_set]# kc get sts
NAME   READY   AGE
web    2/2     96s
[root@k8s0 basic_stateful_set]# kc get pod
NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          118s
web-1   1/1     Running   0          113s
[root@k8s0 basic_stateful_set]# #pvc 的名称也是自动创建
[root@k8s0 basic_stateful_set]# kc get pvc
NAME        STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pv1      1Mi        RWO                           53s
www-web-1   Bound    pv5      1Mi        RWO                           49s
[root@k8s0 basic_stateful_set]# #pv1,pv5绑定了,剩下三个可用
[root@k8s0 basic_stateful_set]# kc get pv
NAME   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM               STORAGECLASS   REASON   AGE
pv1    1Mi        RWO            Retain           Bound       default/www-web-0                           2m20s
pv2    1Mi        RWO            Retain           Available                                               2m19s
pv3    1Mi        RWO            Retain           Available                                               2m19s
pv4    1Mi        RWO            Retain           Available                                               2m18s
pv5    1Mi        RWO            Retain           Bound       default/www-web-1                           2m18s

上面web-0,web-1 就组成一个StatefulSet。

sts里的pod

如上,sts 里pod 有固定、唯一的标志:web-0、web-1。

每个pod 也有固定的hostname:

[root@k8s0 basic_stateful_set]# for i in 0 1; do kc exec web-$i -- sh -c 'hostname'; done
web-0
web-1

web-0使用pv1 - “dat-0”,web-1使用pv5 - “dat-4”:

[root@k8s0 basic_stateful_set]# kc get pvc
NAME        STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pv1      1Mi        RWO                           5m28s
www-web-1   Bound    pv5      1Mi        RWO                           5m24s
[root@k8s0 basic_stateful_set]# kc get pod -o wide
NAME    READY   STATUS    RESTARTS   AGE     IP               NODE   NOMINATED NODE   READINESS GATES
web-0   1/1     Running   0          5m35s   10.100.109.103   k8s2   <none>           <none>
web-1   1/1     Running   0          5m31s   10.100.166.206   k8s1   <none>           <none>
[root@k8s0 basic_stateful_set]# curl 10.100.109.103
dat-0
[root@k8s0 basic_stateful_set]# curl 10.100.166.206
dat-4

执行“kc get pod -w -l app=nginx” 监视pod。
另开shell-2 执行“kc delete pod -l app=nginx”。
在原shell 可以看到2个pod 删除后,sts又自动重建了它们。

[root@k8s0 basic_stateful_set]# kc get pod -o wide
NAME    READY   STATUS    RESTARTS   AGE     IP               NODE   NOMINATED NODE   READINESS GATES
web-0   1/1     Running   1          12m     10.100.109.105   k8s2   <none>           <none>
web-1   1/1     Running   0          8m40s   10.100.166.255   k8s1   <none>           <none>
[root@k8s0 basic_stateful_set]# #hostname 保持着
[root@k8s0 basic_stateful_set]# for i in 0 1; do kc exec web-$i -- sh -c 'hostname'; done
web-0
web-1
[root@k8s0 basic_stateful_set]# curl 10.100.109.105
dat-0
[root@k8s0 basic_stateful_set]# curl 10.100.166.255
dat-4
[root@k8s0 basic_stateful_set]# kc get pvc
NAME        STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pv1      1Mi        RWO                           22m
www-web-1   Bound    pv5      1Mi        RWO                           22m

删除pod时pvc不会自动删除,下次再创建pod时sts会重用原来的pvc。就是在正常情况下pod即使重建 也仍使用原pvc -> 原pv。但是注意:

  • 例如k8s2停机,下次pod 只能重建在k8s1上。所以要保证每台机器有一套相同内容的存储
  • 如果删除了pvc,下次重建时可能绑定不同pv

伸缩sts

指通过kc scalekc patch 命令改变replicas数。

扩容

监视:kc get pods -w -l app=nginx
在shell-2:kc scale sts web --replicas=5
监视结果:

NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   1          31m
web-1   1/1     Running   0          27m

web-2   0/1     Pending   0          0s
web-2   0/1     Pending   0          0s
web-2   0/1     Pending   0          2s
web-2   0/1     ContainerCreating   0          2s
web-2   0/1     ContainerCreating   0          3s
web-2   1/1     Running             0          7s
web-3   0/1     Pending             0          0s
web-3   0/1     Pending             0          0s
web-3   0/1     Pending             0          2s
web-3   0/1     ContainerCreating   0          2s
web-3   0/1     ContainerCreating   0          3s
web-3   1/1     Running             0          2m29s
web-4   0/1     Pending             0          0s
web-4   0/1     Pending             0          0s
web-4   0/1     Pending             0          1s
web-4   0/1     ContainerCreating   0          1s
web-4   0/1     ContainerCreating   0          3s
web-4   1/1     Running             0          5s

sts会按顺序扩容,例如只有web-3 Running之后才会创建web-4。

缩容

同样:kc get pods -w -l app=nginx
在shell-2:kc patch sts web -p ‘{“spec”:{“replicas”:3}}’
监视结果:

NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   1          40m
web-1   1/1     Running   0          36m
web-2   1/1     Running   0          9m37s
web-3   1/1     Running   0          9m30s
web-4   1/1     Running   0          7m1s

web-4   1/1     Terminating   0          7m58s
web-4   0/1     Terminating   0          7m59s
web-4   0/1     Terminating   0          8m
web-4   0/1     Terminating   0          8m3s
web-4   0/1     Terminating   0          8m3s
web-3   1/1     Terminating   0          10m
web-3   0/1     Terminating   0          10m
web-3   0/1     Terminating   0          10m
web-3   0/1     Terminating   0          10m

web-4 Terminating 之后才会删除web-3。

[root@k8s0 basic_stateful_set]# kc get pod -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP               NODE   NOMINATED NODE   READINESS GATES
web-0   1/1     Running   1          44m   10.100.109.105   k8s2   <none>           <none>
web-1   1/1     Running   0          40m   10.100.166.255   k8s1   <none>           <none>
web-2   1/1     Running   0          13m   10.100.166.193   k8s1   <none>           <none>
[root@k8s0 basic_stateful_set]# kc get pvc -l app=nginx
NAME        STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pv1      1Mi        RWO                           53m
www-web-1   Bound    pv5      1Mi        RWO                           53m
www-web-2   Bound    pv2      1Mi        RWO                           13m
www-web-3   Bound    pv3      1Mi        RWO                           13m
www-web-4   Bound    pv4      1Mi        RWO                           11m

当sts的pod 被删除或缩容而删时,其pvc 仍旧会保留。

更新sts

Kubernetes 1.7之后,sts支持自动化更新。可以用来升级容器镜像、资源限制、label、annotation。spec.updateStrategy 可以指定两种更新策略:RollingUpdate (默认) 和OnDelete

RollingUpdate策略

RollingUpdate 策略会反方向(reverse ordinal order)更新sts的pod。

设置:

kc patch statefulset web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate"}}}'

监视:kc get po -l app=nginx -w
在shell-2:

kc patch sts web --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"YYY/helloweb"}]'

更新为入门搭建篇的helloweb 镜像,注意“YYY” 替换为docker hub 账户名。

监视结果:

NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   1          64m
web-1   1/1     Running   0          60m
web-2   1/1     Running   0          94s

web-2   1/1     Terminating   0          2m15s
web-2   0/1     Terminating   0          2m16s
web-2   0/1     Terminating   0          2m19s
web-2   0/1     Terminating   0          2m19s
web-2   0/1     Pending       0          0s
web-2   0/1     Pending       0          0s
web-2   0/1     ContainerCreating   0          0s
web-2   0/1     ContainerCreating   0          2s
web-2   1/1     Running             0          6s
web-1   1/1     Terminating         0          61m
web-1   0/1     Terminating         0          61m
web-1   0/1     Terminating         0          61m
web-1   0/1     Terminating         0          61m
web-1   0/1     Pending             0          1s
web-1   0/1     Pending             0          1s
web-1   0/1     ContainerCreating   0          1s
web-1   0/1     ContainerCreating   0          2s
web-1   1/1     Running             0          4s
web-0   1/1     Terminating         1          65m
web-0   0/1     Terminating         1          65m
web-0   0/1     Terminating         1          65m
web-0   0/1     Terminating         1          65m
web-0   0/1     Pending             0          0s
web-0   0/1     Pending             0          0s
web-0   0/1     ContainerCreating   0          0s
web-0   0/1     ContainerCreating   0          1s
web-0   1/1     Running             0          2s

可见是按2、1、0的反序逐一更新。

检查:

[root@k8s0 basic_stateful_set]# for p in 0 1 2; do kc get po web-$p --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'; echo; done
YYY/helloweb
YYY/helloweb
YYY/helloweb

分段更新 (Staging an Update)

设置partition = 3:

kc patch sts web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":3}}}}'

更新sts 镜像=nginx:

kc patch sts web --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"nginx"}]'

检查镜像仍是helloweb:

[root@k8s0 basic_stateful_set]# for p in 0 1 2; do kc get po web-$p --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'; echo; done
YYY/helloweb
YYY/helloweb
YYY/helloweb

监视:kc get po -l app=nginx -w
在shell-2:kc delete po web-2
监视结果:

NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          15m
web-1   1/1     Running   0          15m
web-2   1/1     Running   0          15m

web-2   1/1     Terminating   0          16m
web-2   0/1     Terminating   0          16m
web-2   0/1     Terminating   0          16m
web-2   0/1     Terminating   0          16m
web-2   0/1     Pending       0          0s
web-2   0/1     Pending       0          0s
web-2   0/1     ContainerCreating   0          0s
web-2   0/1     ContainerCreating   0          1s
web-2   1/1     Running             0          2s

web-2 重建后镜像仍是helloweb:

[root@k8s0 basic_stateful_set]# kc get po web-2 --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'
YYY/helloweb

因为web-0、1、2的ordinal 都小于partition 3,所以它们在更新时仍保留旧设置 (restore original container)。

金丝雀发布 (Rolling Out a Canary)

通过少量减少partition 来测试程序。

监视:kc get po -l app=nginx -w
在shell-2 减小partition = 2:

kc patch sts web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'

监视结果:

NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          32m
web-1   1/1     Running   0          32m
web-2   1/1     Running   0          16m

web-2   1/1     Terminating   0          16m
web-2   0/1     Terminating   0          16m
web-2   0/1     Terminating   0          16m
web-2   0/1     Terminating   0          16m
web-2   0/1     Pending       0          0s
web-2   0/1     Pending       0          0s
web-2   0/1     ContainerCreating   0          0s
web-2   0/1     ContainerCreating   0          1s
web-2   1/1     Running             0          3s

由于web-2的ordinal 2 现在>= partition 2,所以它自动更新了。
检查:

[root@k8s0 ~]# kc get po web-2 --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'
nginx

其镜像修改成nginx。

继续监视:kc get po -l app=nginx -w
在shell-2删除web-1:kc delete po web-1
监视结果:

NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          39m
web-1   1/1     Running   0          39m
web-2   1/1     Running   0          6m21s

web-1   1/1     Terminating   0          39m
web-1   0/1     Terminating   0          39m
web-1   0/1     Terminating   0          39m
web-1   0/1     Terminating   0          39m
web-1   0/1     Terminating   0          39m
web-1   0/1     Pending       0          0s
web-1   0/1     Pending       0          0s
web-1   0/1     ContainerCreating   0          1s
web-1   0/1     ContainerCreating   0          2s
web-1   1/1     Running             0          5s

web-1重建了。检查其镜像:

[root@k8s0 basic_stateful_set]# kc get po web-1 --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'
YYY/helloweb

web-1 镜像仍没修改,因为它的ordinal 1 < partition 2。

当设置了partition后,修改.spec.template 只会修改ordinal >= partition 的pod。Ordinal < partition 的pod 删除/重建时仍保留旧设置 (restore original configuration)。

分段发布 (Phased Roll Outs)

目前partition = 2。

监视:kc get po -l app=nginx -w
在shell-2 设置partition = 0:

kc patch sts web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":0}}}}'

监视结果:

NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          54m
web-1   1/1     Running   0          14m
web-2   1/1     Running   0          21m

web-1   1/1     Terminating   0          14m
web-1   0/1     Terminating   0          14m
web-1   0/1     Terminating   0          14m
web-1   0/1     Terminating   0          14m
web-1   0/1     Pending       0          0s
web-1   0/1     Pending       0          0s
web-1   0/1     ContainerCreating   0          0s
web-1   0/1     ContainerCreating   0          2s
web-1   1/1     Running             0          7m55s
web-0   1/1     Terminating         1          62m
web-0   0/1     Terminating         1          62m
web-0   0/1     Terminating         1          62m
web-0   0/1     Terminating         1          62m
web-0   0/1     Pending             0          0s
web-0   0/1     Pending             0          0s
web-0   0/1     ContainerCreating   0          0s
web-0   0/1     ContainerCreating   0          1s
web-0   1/1     Running             0          2s

看到按1、0的反序重建了pod。检查镜像:

[root@k8s0 basic_stateful_set]# for p in 0 1 2; do kc get po web-$p --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'; echo; done
nginx
nginx
nginx

看到所有pod都已更新。

OnDelete策略

Kubernetes 1.6及之前的行为。当更新.spec.template 时,不会自动更新pod。

删除sts

Non-Cascading Delete

非级联删除:删除sts时不删除pod。

web-0、1、2。
非级联删除:kc delete sts web --cascade=false
三个pod仍在。
删除web-0:kc delete pod web-0
只剩web-1、2。
重新执行:kc apply -f web.yaml
监视到先重建web-0、再删除web-2 (因为yaml里replicas=2)。

检查:

[root@k8s0 basic_stateful_set]# for i in 0 1; do kc exec -it web-$i -- cat /usr/share/nginx/html/index.html; done
dat-0
dat-4

web-0、1使用的pv没变(只要节点还是原来的),因为删除sts/pod 时,不会自动删除pv。

Cascading Delete

级联删除:删除sts时也删除pod。

web-0、1。
级联删除:kc delete statefulset web
2个pod 均删除了。

注意级联删除还留下nginx service没有删除。删除它:
kc delete service nginx

再次重建:

[root@k8s0 basic_stateful_set]# kc apply -f web.yaml
service/nginx created
statefulset.apps/web created

检查:

[root@k8s0 basic_stateful_set]# kc get pod -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP               NODE   NOMINATED NODE   READINESS GATES
web-0   1/1     Running   0          38s   10.100.166.214   k8s1   <none>           <none>
web-1   1/1     Running   0          36s   10.100.109.113   k8s2   <none>           <none>
[root@k8s0 basic_stateful_set]# for i in 0 1; do kc exec -it web-$i -- cat /usr/share/nginx/html/index.html; done
dat-0
dat-4
[root@k8s0 basic_stateful_set]# curl 10.100.166.214
dat-0
[root@k8s0 basic_stateful_set]# curl 10.100.109.113
dat-4
[root@k8s0 basic_stateful_set]# for i in 0 1; do kc exec -it web-$i -- sh -c 'hostname'; done
web-0
web-1

如前,因为pvc & pv 没有删除,所以重建后应用的行为通常不会改变。

终于,删除sts和svc:

[root@k8s0 basic_stateful_set]# kc delete svc nginx
service "nginx" deleted
[root@k8s0 basic_stateful_set]# kc delete sts web
statefulset.apps "web" deleted
[root@k8s0 basic_stateful_set]# kc get pvc
NAME        STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pv1      1Mi        RWO                           3h3m
www-web-1   Bound    pv5      1Mi        RWO                           3h3m
www-web-2   Bound    pv2      1Mi        RWO                           144m
www-web-3   Bound    pv3      1Mi        RWO                           143m
www-web-4   Bound    pv4      1Mi        RWO                           141m
[root@k8s0 basic_stateful_set]# kc get pv
NAME   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pv1    1Mi        RWO            Retain           Bound    default/www-web-0                           3h4m
pv2    1Mi        RWO            Retain           Bound    default/www-web-2                           3h4m
pv3    1Mi        RWO            Retain           Bound    default/www-web-3                           3h4m
pv4    1Mi        RWO            Retain           Bound    default/www-web-4                           3h4m
pv5    1Mi        RWO            Retain           Bound    default/www-web-1                           3h4m

(注:期间经过多次重试,所以5个pvc和pv均已绑定,刚才实际在用的只有2个)

Pod Management Policy

.spec.podManagementPolicy 属性:

  • “OrderedReady”:默认。上面所述严格按ordinal 顺序/反序来创建/删除 pod
  • “Parallel”:不严格遵照ordinal 顺序

Cleanup

删除本教程创建的pvc和hostPath pv:
kc delete pvc -l app=nginx
for i in 1 2 3 4 5; do kc delete pv pv$i; done

至于三个节点上实际的存储目录(/test/k8s/basic_stateful_set/mnt/data*) 请自行处理。

Logo

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐