2018年05月04日 17:16:53 迷途的攻城狮（798570156） 阅读数：2258

 版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/chenleiking/article/details/80197975

K8S的应用更新、回滚、伸缩

1、应用结构

以一个简单的deployment为例来进行说明：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 4
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: 192.168.101.88:5000/nginx:1.7.9
        ports:
        - containerPort: 80

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22

一个相对完整的应用可能还包括：volume、env、service、ingress等

root@master-1:~/kubernetes/example# kubectl apply -f nginx-deployment.yaml 
deployment.apps "nginx-deployment" created
root@master-1:~/kubernetes/example# kubectl get all
NAME                                    READY     STATUS    RESTARTS   AGE
pod/nginx-deployment-85b548b894-ct66d   1/1       Running   0          20s
pod/nginx-deployment-85b548b894-pfmvl   1/1       Running   0          20s
pod/nginx-deployment-85b548b894-v88qq   1/1       Running   0          20s
pod/nginx-deployment-85b548b894-zxml8   1/1       Running   0          20s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   7d

NAME                               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   4         4         4            4           20s

NAME                                          DESIRED   CURRENT   READY     AGE
replicaset.apps/nginx-deployment-85b548b894   4         4         4         20s

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17

忽略service/kubernetes～～

graph TB;
    deployment.apps/nginx-deployment --> replicaset.apps/nginx-deployment-85b548b894
    replicaset.apps/nginx-deployment-85b548b894 --> pod/nginx-deployment-85b548b894-ct66d
    replicaset.apps/nginx-deployment-85b548b894 --> pod/nginx-deployment-85b548b894-pfmvl
    replicaset.apps/nginx-deployment-85b548b894 --> pod/nginx-deployment-85b548b894-v88qq
    replicaset.apps/nginx-deployment-85b548b894 --> pod/nginx-deployment-85b548b894-zxml8

• 1
• 2
• 3
• 4
• 5
• 6

• 每个deployment会包含一个或者多个replicaset
• 每个replicaset会可以包含零个或者多个pod
• 每个replicaset对应deployment的一个revision
• 每次更新时，pod总是在一个replicaset中创建，然后在现有的replicaset中销毁

2、更新与回滚

只有spec.template.spec.containers.image或者spec.template.metadata.labels发生变化时才会出发更新rolout操作～～

• 更新

root@master-1:~/kubernetes/example# kubectl set image deployment nginx-deployment nginx=192.168.101.88:5000/nginx:1.9.1
deployment.apps "nginx-deployment" image updated

• 1
• 2

> gif图片太大，连接：https://pan.baidu.com/s/1cR7l4anB6FrQ27UNvJ1h0g

更新过程如图所示：

• 首先，因为没有匹配的replicaset，创建一个新的replicaset replicaset.apps/nginx-deployment-767fdf8df6
• 然后根据spec.strategy.rollingUpdate.maxUnavailable的值（默认25%）终止掉 replicaset.apps/nginx-deployment-85b548b894 中对应数量的pod
• 最后根据spec.strategy.rollingUpdate.maxSurge的值（默认25%）在 replicaset.apps/nginx-deployment-767fdf8df6 中启动对应数量多pod
• 最终，replicaset.apps/nginx-deployment-85b548b894 中的pod全部终止， replicaset.apps/nginx-deployment-767fdf8df6 中的pod达到预期值

• replicaset的名字组成：deploymentName + podTemplate的Hash值
• maxUnavailable：表示最大不可用pod数量，可以是整数或者百分比。这里的4个pod，25%就是一个，也就是说一次只能停止一个pod
• maxSurge：表示可超过预期值的pod数量，可以是整数或者百分比。预期值4，25%就是可以超出一个，也就是说同时可以存在5个可用的pod

graph TB;
    deployment.apps/nginx-deployment --> replicaset.apps/nginx-deployment-85b548b894
    deployment.apps/nginx-deployment --> replicaset.apps/nginx-deployment-767fdf8df6
    replicaset.apps/nginx-deployment-767fdf8df6 --> pod/nginx-deployment-767fdf8df6-5zj8c
    replicaset.apps/nginx-deployment-767fdf8df6 --> pod/nginx-deployment-767fdf8df6-7668l
    replicaset.apps/nginx-deployment-767fdf8df6 --> pod/nginx-deployment-767fdf8df6-s7rpq
    replicaset.apps/nginx-deployment-767fdf8df6 --> pod/nginx-deployment-767fdf8df6-vgm2m

• 1
• 2
• 3
• 4
• 5
• 6
• 7

• 回滚

### 查看更新记录 ###
root@master-1:~# kubectl rollout history deployment nginx-deployment
deployments "nginx-deployment"
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

• 1
• 2
• 3
• 4
• 5
• 6

默认的部署方式不会记录CHANGE-CAUSE，需要在部署的时候添加 --record 选项～～

spec.revisionHistoryLimit被用来设置保留的历史记录数量，默认为2，设置为0时将不能回滚～～

root@master-1:~/kubernetes/example# kubectl delete -f nginx-deployment.yaml 
deployment.apps "nginx-deployment" deleted
root@master-1:~/kubernetes/example# kubectl apply -f nginx-deployment.yaml --record
deployment.apps "nginx-deployment" created
root@master-1:~/kubernetes/example# kubectl set image deployment nginx-deployment nginx=192.168.101.88:5000/nginx:1.9.1
deployment.apps "nginx-deployment" image updated
root@master-1:~/kubernetes/example# kubectl rollout history deployment nginx-deployment
deployments "nginx-deployment"
REVISION  CHANGE-CAUSE
1         kubectl apply --filename=nginx-deployment.yaml --record=true
2         kubectl set image deployment nginx-deployment nginx=192.168.101.88:5000/nginx:1.9.1

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11

### 回滚到上一个版本 ###
root@master-1:~/kubernetes/example# kubectl rollout undo deployment nginx-deployment
deployment.apps "nginx-deployment" 
root@master-1:~/kubernetes/example# kubectl rollout history deployment nginx-deployment
deployments "nginx-deployment"
REVISION  CHANGE-CAUSE
2         kubectl set image deployment nginx-deployment nginx=192.168.101.88:5000/nginx:1.9.1
3         kubectl apply --filename=nginx-deployment.yaml --record=true

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8

回滚操作的执行流程与更新一样，都是按照spec.strategy.rollingUpdate.maxUnavailable和spec.strategy.rollingUpdate.maxSurge在当前replicaset中终止pod，在符合条件的replicaset中启动新的pod～～

### 回滚到指定版本 ###
root@master-1:~/kubernetes/example# kubectl rollout undo deployment nginx-deployment --to-revision=2
deployment.apps "nginx-deployment" 
root@master-1:~/kubernetes/example# kubectl rollout history deployment nginx-deployment
deployments "nginx-deployment"
REVISION  CHANGE-CAUSE
3         kubectl apply --filename=nginx-deployment.yaml --record=true
4         kubectl set image deployment nginx-deployment nginx=192.168.101.88:5000/nginx:1.9.1

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8

root@master-1:~/kubernetes/example# kubectl get all
NAME                                    READY     STATUS    RESTARTS   AGE
pod/nginx-deployment-767fdf8df6-56bmx   1/1       Running   0          42s
pod/nginx-deployment-767fdf8df6-bjbx4   1/1       Running   0          42s
pod/nginx-deployment-767fdf8df6-p6xll   1/1       Running   0          45s
pod/nginx-deployment-767fdf8df6-rf5tm   1/1       Running   0          44s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   7d

NAME                               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   4         4         4            4           6m

NAME                                          DESIRED   CURRENT   READY     AGE
replicaset.apps/nginx-deployment-767fdf8df6   4         4         4         6m
replicaset.apps/nginx-deployment-85b548b894   0         0         0         6m

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16

因为上面的更新与回滚一直在两个版本之间来回修改，deploymentName + podTemplate的Hash值并没有改变，所以没有产生新的replicaset，但是版本号回一直递增～～～

3、伸缩

3.1、手动伸缩

root@master-1:~/kubernetes/example# kubectl scale deployment nginx-deployment --replicas=10
deployment.extensions "nginx-deployment" scaled
root@master-1:~/kubernetes/example# kubectl get all
NAME                                    READY     STATUS    RESTARTS   AGE
pod/nginx-deployment-767fdf8df6-4zz4k   1/1       Running   0          11s
pod/nginx-deployment-767fdf8df6-56bmx   1/1       Running   0          5m
pod/nginx-deployment-767fdf8df6-9rqk6   1/1       Running   0          11s
pod/nginx-deployment-767fdf8df6-bjbx4   1/1       Running   0          5m
pod/nginx-deployment-767fdf8df6-ksrc2   1/1       Running   0          11s
pod/nginx-deployment-767fdf8df6-p6xll   1/1       Running   0          5m
pod/nginx-deployment-767fdf8df6-qcj9q   1/1       Running   0          11s
pod/nginx-deployment-767fdf8df6-rf5tm   1/1       Running   0          5m
pod/nginx-deployment-767fdf8df6-rljpn   1/1       Running   0          11s
pod/nginx-deployment-767fdf8df6-ts8sn   1/1       Running   0          11s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   7d

NAME                               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   10        10        10           10          11m

NAME                                          DESIRED   CURRENT   READY     AGE
replicaset.apps/nginx-deployment-767fdf8df6   10        10        10        11m
replicaset.apps/nginx-deployment-85b548b894   0         0         0         11m

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24

3.2、自动伸缩（Horizontal Pod Autoscaling）

自动伸缩需要部署Heapster和metrics-server～

root@master-1:~/kubernetes/example# cat nginx-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 4
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: 192.168.101.88:5000/nginx:1.7.9
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "25m"
          limits:
            memory: "128Mi"
            cpu: "50m"
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: nginx-deployment
  name: nginx-deployment-service
spec:
  ports:
    - port: 80
      targetPort: 80
      name: http
  selector:
    app: nginx

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-deployment-service-ingress
spec:
  rules:
    - host: nginx-deployment.chenlei.com
      http:
        paths:
          - backend:
              serviceName: nginx-deployment-service
              servicePort: 80

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35
• 36
• 37
• 38
• 39
• 40
• 41
• 42
• 43
• 44
• 45
• 46
• 47
• 48
• 49
• 50
• 51
• 52
• 53
• 54
• 55
• 56
• 57
• 58

• 使用HPA需要在部署文件中明确指定资源配置情况 resources ，最少需要配置 resources.requests.cpu
• 为了测试，向部署文件中增加了service和ingress

root@master-1:~/kubernetes/example# kubectl apply -f nginx-deployment.yaml 
deployment.apps "nginx-deployment" created
service "nginx-deployment-service" created
ingress.extensions "nginx-deployment-service-ingress" created

root@master-1:~/kubernetes/example# kubectl get all
NAME                                    READY     STATUS    RESTARTS   AGE
pod/nginx-deployment-5f79d4fc87-7k6cn   1/1       Running   0          9s
pod/nginx-deployment-5f79d4fc87-f5v27   1/1       Running   0          9s
pod/nginx-deployment-5f79d4fc87-ls6hz   1/1       Running   0          9s
pod/nginx-deployment-5f79d4fc87-rlwh4   1/1       Running   0          9s

NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/kubernetes                 ClusterIP   10.96.0.1       <none>        443/TCP   8d
service/nginx-deployment-service   ClusterIP   10.97.127.155   <none>        80/TCP    9s

NAME                               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   4         4         4            4           10s

NAME                                          DESIRED   CURRENT   READY     AGE
replicaset.apps/nginx-deployment-5f79d4fc87   4         4         4         9s

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21

root@master-1:~/kubernetes/example# kubectl autoscale deployment nginx-deployment --min=1 --max=10 --cpu-percent=20
deployment.apps "nginx-deployment" autoscaled

root@master-1:~/kubernetes/example# kubectl get all
NAME                                    READY     STATUS        RESTARTS   AGE
pod/nginx-deployment-5f79d4fc87-7k6cn   1/1       Running       0          2m
pod/nginx-deployment-5f79d4fc87-f5v27   0/1       Terminating   0          2m
pod/nginx-deployment-5f79d4fc87-ls6hz   0/1       Terminating   0          2m
pod/nginx-deployment-5f79d4fc87-rlwh4   0/1       Terminating   0          2m

NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/kubernetes                 ClusterIP   10.96.0.1       <none>        443/TCP   8d
service/nginx-deployment-service   ClusterIP   10.97.127.155   <none>        80/TCP    2m

NAME                               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   1         1         1            1           2m

NAME                                          DESIRED   CURRENT   READY     AGE
replicaset.apps/nginx-deployment-5f79d4fc87   1         1         1         2m

NAME                                                   REFERENCE                     TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/nginx-deployment   Deployment/nginx-deployment   0%/20%    1         10        4          34s

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22

HPA创建之后，会在30秒（根据controller-manager的参数配置）内根据CPU的负载情况自动调整POD数量。因为负载为零，所以自动终止了三个pod，仅保留一个～～～

kube-controller-manage的调度日志如下：

I0504 08:25:48.431218       1 horizontal.go:512] Successful rescale of nginx-deployment, old size: 4, new size: 1, reason: All metrics below target
I0504 08:25:48.431412       1 event.go:218] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"nginx-deployment", UID:"adb4b7a9-4f74-11e8-9807-0050568e3344", APIVersion:"autoscaling/v2beta1", ResourceVersion:"1314846", FieldPath:""}): type: 'Normal' reason: 'SuccessfulRescale' New size: 1; reason: All metrics below target
I0504 08:25:48.475085       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"default", Name:"nginx-deployment", UID:"79043755-4f74-11e8-9807-0050568e3344", APIVersion:"extensions", ResourceVersion:"1314901", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set nginx-deployment-5f79d4fc87 to 1
I0504 08:25:48.599598       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"nginx-deployment-5f79d4fc87", UID:"79075132-4f74-11e8-9807-0050568e3344", APIVersion:"extensions", ResourceVersion:"1314904", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: nginx-deployment-5f79d4fc87-ls6hz
I0504 08:25:48.599622       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"nginx-deployment-5f79d4fc87", UID:"79075132-4f74-11e8-9807-0050568e3344", APIVersion:"extensions", ResourceVersion:"1314904", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: nginx-deployment-5f79d4fc87-rlwh4
I0504 08:25:48.599634       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"nginx-deployment-5f79d4fc87", UID:"79075132-4f74-11e8-9807-0050568e3344", APIVersion:"extensions", ResourceVersion:"1314904", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: nginx-deployment-5f79d4fc87-f5v27
W0504 08:27:12.650959       1 reflector.go:341] k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:125: watch of <nil> ended with: very short watch: k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:125: Unexpected watch close - watch lasted less than a second and no items received

• 1
• 2
• 3
• 4
• 5
• 6
• 7

增加负载：

LiondeMacBook-Pro:~ lion$ for i in {1..10000}; do curl http://nginx-deployment.chenlei.com/ > /dev/null -s; done;

• 1

这里打开四个终端分别执行，然后观察pod伸缩情况

root@master-1:~/kubernetes/example# kubectl get all
NAME                                    READY     STATUS    RESTARTS   AGE
pod/nginx-deployment-5f79d4fc87-7k6cn   1/1       Running   0          12m
pod/nginx-deployment-5f79d4fc87-gvfpr   1/1       Running   0          4s
pod/nginx-deployment-5f79d4fc87-mzd58   1/1       Running   0          4s

NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/kubernetes                 ClusterIP   10.96.0.1       <none>        443/TCP   8d
service/nginx-deployment-service   ClusterIP   10.97.127.155   <none>        80/TCP    12m

NAME                               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   3         3         3            3           12m

NAME                                          DESIRED   CURRENT   READY     AGE
replicaset.apps/nginx-deployment-5f79d4fc87   3         3         3         12m

NAME                                                   REFERENCE                     TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/nginx-deployment   Deployment/nginx-deployment   44%/20%   1         10        1          11m

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18

当CPU压力增加时，kubernetes自动创建了两个新的pod来分摊压力，新增pod的数量取决于CPU压力打下

kube-controller-maanage的调度日志：

I0504 08:36:14.996758       1 horizontal.go:512] Successful rescale of nginx-deployment, old size: 1, new size: 3, reason: cpu resource utilization (percentage of request) above target
I0504 08:36:14.997180       1 event.go:218] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"nginx-deployment", UID:"adb4b7a9-4f74-11e8-9807-0050568e3344", APIVersion:"autoscaling/v2beta1", ResourceVersion:"1315533", FieldPath:""}): type: 'Normal' reason: 'SuccessfulRescale' New size: 3; reason: cpu resource utilization (percentage of request) above target
I0504 08:36:15.061094       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"default", Name:"nginx-deployment", UID:"79043755-4f74-11e8-9807-0050568e3344", APIVersion:"extensions", ResourceVersion:"1316131", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set nginx-deployment-5f79d4fc87 to 3
I0504 08:36:15.163244       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"nginx-deployment-5f79d4fc87", UID:"79075132-4f74-11e8-9807-0050568e3344", APIVersion:"extensions", ResourceVersion:"1316134", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: nginx-deployment-5f79d4fc87-mzd58
I0504 08:36:15.208366       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"nginx-deployment-5f79d4fc87", UID:"79075132-4f74-11e8-9807-0050568e3344", APIVersion:"extensions", ResourceVersion:"1316134", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: nginx-deployment-5f79d4fc87-gvfpr

• 1
• 2
• 3
• 4
• 5

压力较低之后又会消减pod数量

3.3、自定义指标

这一块内容留在下次补充～～～～

3.4、异常记录

• 问题一：

root@master-1:~/kubernetes/example# kubectl get hpa
NAME               REFERENCE                     TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
nginx-deployment   Deployment/nginx-deployment   <unknown>/10%   1         10        4          37s

root@master-1:~/kubernetes/example# kubectl get hpa
NAME               REFERENCE                     TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
nginx-deployment   Deployment/nginx-deployment   <unknown>/10%   1         10        4          37s

root@master-1:~/kubernetes/example# kubectl describe hpa nginx-deployment
Name:                                                  nginx-deployment
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Fri, 04 May 2018 04:46:16 -0400
Reference:                                             Deployment/nginx-deployment
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 10%
Min replicas:                                          1
Max replicas:                                          10
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Events:
  Type     Reason                        Age   From                       Message
  ----     ------                        ----  ----                       -------
  Warning  FailedGetResourceMetric       21s   horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedComputeMetricsReplicas  21s   horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29

kube-controller-manage日志：

I0504 08:47:16.381435       1 event.go:218] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"nginx-deployment", UID:"9b7a46ff-4f77-11e8-9807-0050568e3344", APIVersion:"autoscaling/v2beta1", ResourceVersion:"1317451", FieldPath:""}): type: 'Warning' reason: 'FailedGetResourceMetric' missing request for cpu on container nginx in pod default/nginx-deployment-85b548b894-2lzg8
E0504 08:47:16.391281       1 horizontal.go:189] failed to compute desired number of replicas based on listed metrics for Deployment/default/nginx-deployment: failed to get cpu utilization: missing request for cpu on container nginx in pod default/nginx-deployment-85b548b894-2lzg8
I0504 08:47:16.391308       1 event.go:218] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"nginx-deployment", UID:"9b7a46ff-4f77-11e8-9807-0050568e3344", APIVersion:"autoscaling/v2beta1", ResourceVersion:"1317451", FieldPath:""}): type: 'Warning' reason: 'FailedComputeMetricsReplicas' failed to get cpu utilization: missing request for cpu on container nginx in pod default/nginx-deployment-85b548b894-2lzg8

• 1
• 2
• 3

原因：

定义pod的时候，没有设置resources，HPA取不到CPU当前值

        resources:
          requests:
            memory: "64Mi"
            cpu: "25m"
          limits:
            memory: "128Mi"
            cpu: "50m"

• 1
• 2
• 3
• 4
• 5
• 6
• 7

• 问题二

root@master-1:~/kubernetes/example# kubectl autoscale deployment nginx-deployment --min=1 --max=10 --cpu-percent=10
deployment.apps "nginx-deployment" autoscaled
root@master-1:~/kubernetes/example# kubectl describe hpa nginx-deployment
Name:                                                  nginx-deployment
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Fri, 04 May 2018 04:53:20 -0400
Reference:                                             Deployment/nginx-deployment
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 10%
Min replicas:                                          1
Max replicas:                                          10
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
  Type     Reason                        Age   From                       Message
  ----     ------                        ----  ----                       -------
  Warning  FailedGetResourceMetric       27s   horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
  Warning  FailedComputeMetricsReplicas  27s   horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
root@master-1:~/kubernetes/example# 

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24

kube-controller-manage日志：

E0504 08:54:20.852828       1 horizontal.go:189] failed to compute desired number of replicas based on listed metrics for Deployment/default/nginx-deployment: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0504 08:54:20.852958       1 event.go:218] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"nginx-deployment", UID:"987a4389-4f78-11e8-9807-0050568e3344", APIVersion:"autoscaling/v2beta1", ResourceVersion:"1318418", FieldPath:""}): type: 'Warning' reason: 'FailedGetResourceMetric' unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

• 1
• 2

原因：

没有部署metrics-server服务，部署后任然无法解决的可参考GitHub上的这个问题。

4、参考资料

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/