kubernetes+ prometheus自动伸缩的设计与实现(一)
这篇blog名字起的搞得和我写论文一样,xxxx的设计与实现。其实这个东西原理很简单,kubernetes的hpa使用的是heapster,heapster是k8s那帮家伙在搞的,所以k8s还是喜欢自己搞的东西,所以k8s的hpa默认使用的heapster,但在业内,还有一个比heapster更好的监控方案,那就是prometheus。如果按照写论文的方式,我这边应该分别介绍一下k8s和prome
这篇blog名字起的搞得和我写论文一样,xxxx的设计与实现。其实这个东西原理很简单,kubernetes的hpa使用的是heapster,heapster是k8s那帮家伙在搞的,所以k8s还是喜欢自己搞的东西,所以k8s的hpa默认使用的heapster,但在业内,还有一个比heapster更好的监控方案,那就是prometheus。如果按照写论文的方式,我这边应该分别介绍一下k8s和prometheus,但真的没有那个闲功夫,在此略过,我之前blog也做过它们的源码分析。
上面的图片展示了整个体系结构
下面就分析一下adapter的具体实现,这里需要结合上一篇blog关于api聚合的功能,这个adapter就是通过api聚合的方式注册到apiserver上面。
先看一个hpa的例子
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2alpha1
metadata:
name: wordpress
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: wordpress
minReplicas: 1
maxReplicas: 3
metrics:
- type: Pods
pods:
metricName: memory_usage_bytes
targetAverageValue: 100000
上面的代码添加一个内存使用量的hpa例子,通过targetAverageValue去设定阈值还有最大最小副本数,以及管理的deployment。在此额外说一下,hpa支持三种指标,分别是Object:描述k8s对象的某种指标,譬如ingress的hits-per-second。Pods:pod的平均指标,譬如transactions-processed-per-second,描述每个pod的事务每秒事务数,Resource是描述pod资源用量,譬如CPU或者内存。下面举一个object的例子,pod的http请求数。
- type: Object
object:
target:
kind: Service
name: sample-metrics-app
metricName: http_requests
targetValue: 100
先从hpa代码开始pkg/controller/podautoscaler/horizontal.go,里面的computeReplicasForMetrics方法,它是负责获取监控指标并且计算副本数的方法。针对不同资源调用不同的方法:
case autoscalingv2.ObjectMetricSourceType:
GetObjectMetricReplicas
...
case autoscalingv2.PodsMetricSourceType:
GetMetricReplicas
case autoscalingv2.ResourceMetricSourceType:
GetRawResourceReplicas
先看看GetObjectMetricReplicas这个方法
pkg/controller/podautoscaler/replica_calculator.go
func (c *ReplicaCalculator) GetObjectMetricReplicas(currentReplicas int32, targetUtilization int64, metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference) (replicaCount int32, utilization int64, timestamp time.Time, err error) {
utilization, timestamp, err = c.metricsClient.GetObjectMetric(metricName, namespace, objectRef)
if err != nil {
return 0, 0, time.Time{}, fmt.Errorf("unable to get metric %s: %v on %s %s/%s", metricName, objectRef.Kind, namespace, objectRef.Name, err)
}
usageRatio := float64(utilization) / float64(targetUtilization)
if math.Abs(1.0-usageRatio) <= c.tolerance {
// return the current replicas if the change would be too small
return currentReplicas, utilization, timestamp, nil
}
return int32(math.Ceil(usageRatio * float64(currentReplicas))), utilization, timestamp, nil
}
GetObjectMetric是一个接口,有两个方法,就是上面图所示的heapster和自定义custom接口。heapster这个就是调用heapster接口去获取性能指标,本blog着重介绍自定义性能指标,在启动controller-manager时候指定–horizontal-pod-autoscaler-use-rest-clients就可以使用自定义的性能指标了
pkg/controller/podautoscaler/metrics/rest_metrics_client.go
func (c *customMetricsClient) GetObjectMetric(metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference) (int64, time.Time, error) {
gvk := schema.FromAPIVersionAndKind(objectRef.APIVersion, objectRef.Kind)
var metricValue *customapi.MetricValue
var err error
if gvk.Kind == "Namespace" && gvk.Group == "" {
// handle namespace separately
// NB: we ignore namespace name here, since CrossVersionObjectReference isn't
// supposed to allow you to escape your namespace
metricValue, err = c.client.RootScopedMetrics().GetForObject(gvk.GroupKind(), namespace, metricName)
} else {
metricValue, err = c.client.NamespacedMetrics(namespace).GetForObject(gvk.GroupKind(), objectRef.Name, metricName)
}
if err != nil {
return 0, time.Time{}, fmt.Errorf("unable to fetch metrics from API: %v", err)
}
return metricValue.Value.MilliValue(), metricValue.Timestamp.Time, nil
}
上面的objectRef针对本blog只为
{Kind:Service,Name:wordpress,APIVersion:,},就是我们在yaml文件里面metrics里面定义。
上面通过vendor/k8s.io/metrics/pkg/client/custom_metrics/client.go
func (m *rootScopedMetrics) GetForObject(groupKind schema.GroupKind, name string, metricName string) (*v1alpha1.MetricValue, error) {
// handle namespace separately
if groupKind.Kind == "Namespace" && groupKind.Group == "" {
return m.getForNamespace(name, metricName)
}
resourceName, err := m.client.qualResourceForKind(groupKind)
if err != nil {
return nil, err
}
res := &v1alpha1.MetricValueList{}
err = m.client.client.Get().
Resource(resourceName).
Name(name).
SubResource(metricName).
Do().
Into(res)
if err != nil {
return nil, err
}
if len(res.Items) != 1 {
return nil, fmt.Errorf("the custom metrics API server returned %v results when we asked for exactly one", len(res.Items))
}
return &res.Items[0], nil
}
通过client发送https请求获取metrics。具体发送如下所示object:
https://localhost:6443/apis/custom-metrics.metrics.k8s.io/v1alpha1/namespaces/default/services/wordpress/requests-per-second
如果是pod则发送的请求是
https://localhost:6443/apis/custom-metrics.metrics.k8s.io/v1alpha1/namespaces/default/pods/%2A/memory_usage_bytes?labelSelector=app%3Dwordpress%2Ctier%3Dfrontend
至于group为啥是custom-metrics.metrics.k8s.io这个不是别的,是代码里面写死的,vendor/k8s.io/metrics/pkg/apis/custom_metrics/v1alpha1/register.go
// GroupName is the group name use in this package
const GroupName = "custom-metrics.metrics.k8s.io"
// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{Group: GroupName, Version: "v1alpha1"}
这样k8s的部分已经讲解完毕。下面就是adapter的部分了。
更多推荐
所有评论(0)