k8s平台实现HPA过程中问题排查
k8s平台实现HPA过程中问题排查背景问题描述问题分析解决问题背景最近在着手学习HPA的过程,HPA需要metric-service进行采集数据。在实现过程中,遇到了一些问题,进行解决。问题描述查看当前HPA的情况。(base) root@gzk8s-master1:/home/cyye# kubectl get hpa -ANAMESPACENAMEREFERENCETARGETS
·
背景
最近在着手学习HPA的过程,HPA需要metric-service进行采集数据。在实现过程中,遇到了一些问题,进行解决。
问题描述
查看当前HPA的情况。
(base) root@gzk8s-master1:/home/cyye# kubectl get hpa -A
NAMESPACE NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
istio-system istio-ingressgateway Deployment/istio-ingressgateway 17%/80% 1 5 1 96d
istio-system istiod Deployment/istiod 1%/80% 1 5 1 96d
ztb-test test-hpa Deployment/test-hpa <unknown>/80% 1 5 2 18h
观察到TARGETS为< unknown >状态,但存在目标80%的阈值信息。初步可以判断是没有获取到当前的数值。
进一步获取详细信息。
事件描述为:
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 2m57s (x4340 over 18h) horizontal-pod-autoscaler missing request for cpu
问题分析
检查metric-service服务是否正常运行。
(base) root@gzk8s-master1:/home/cyye# kubectl get pods -A | grep metrics
kube-system metrics-server-56b49c5f5b-ntzjk 1/1 Running 93 133d
根据HPA的event里面日志信息,查阅相关资料,是由于HPA的target的Deployment管理的Pod没有设置CPU的request值。
添加的信息为:
resources:
requests:
cpu: 50m
memory: 50Mi
更新前后示意图如下所示:
解决问题
解决后的效果为:
(base) root@gzk8s-master1:/home/cyye# kubectl get hpa -A
NAMESPACE NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
istio-system istio-ingressgateway Deployment/istio-ingressgateway 16%/80% 1 5 1 96d
istio-system istiod Deployment/istiod 1%/80% 1 5 1 96d
ztb-test test-hpa Deployment/test-hpa 4%/80% 1 5 1 18h
后记
安装了metrics-server组件后,可以执行下述命令:
(base) root@gzk8s-master1:# kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
gzk8s-master1 13759m 57% 33135Mi 51%
(base) root@gzk8s-master1:# kubectl top pods
NAME CPU(cores) MEMORY(bytes)
details-v1-5f449bdbb9-rdl2j 5m 90Mi
productpage-v1-6f9df695b7-xbbsj 24m 135Mi
ratings-v1-857bb87c57-d7927 5m 81Mi
reviews-v1-68f9c47f69-vrm56 6m 190Mi
reviews-v2-5d56c488f5-lhh6r 13m 183Mi
reviews-v3-869ff44845-pwp4l 6m 168Mi
更多推荐
已为社区贡献3条内容
所有评论(0)