问题

怎样使用JVM (Micrometer)面板,监控Spring?这里不涉及Prometheus和Grafana,重点介绍与Micrometer与Springboot,k8s怎样集成。

pom.xml

引入依赖,如下:

<properties>
	<micrometer.version>1.12.5</micrometer.version>
	<micrometer-jvm-extras.version>0.2.2</micrometer-jvm-extras.version>
</properties>
...
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
  <version>${micrometer.version}</version>
</dependency>

<dependency>
    <groupId>io.github.mweirauch</groupId>
    <artifactId>micrometer-jvm-extras</artifactId>
    <version>${micrometer-jvm-extras.version}</version>
</dependency>

JVMConfig.java

启用micrometer-jvm-extras库监控内存指标:

import io.github.mweirauch.micrometer.jvm.extras.ProcessMemoryMetrics;
import io.micrometer.core.instrument.binder.MeterBinder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class JVMConfig {
    @Bean
    public MeterBinder processMemoryMetrics() {
        return new ProcessMemoryMetrics();
    }
}

application.yml

management:
  metrics:
    tags:
      application: ${spring.profiles.active}_${spring.application.name}
  endpoints:
    web:
      base-path: /actuator
      exposure:
        include: prometheus,health
server:
  tomcat:
    mbeanregistry:
      enabled: true

这里主要是三件事情:

  • 标记application名称;
  • 限制只能查询prometheus,health两个actuator查询;
  • 启用Tomcat指标。

prometheus.yml

配置prometheus抓取程序,如下:

global:
  scrape_interval: 30s
scrape_configs:
...
  # JVM (Micrometer)
  - job_name: 'kubernetes-service-endpoints'
    kubernetes_sd_configs:
    - role: endpoints
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
      action: replace
      target_label: __scheme__
      regex: (https?)
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
      action: replace
      target_label: __address__
      regex: (.+)(?::\d+);(\d+)
      replacement: $1:$2
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: application

这里主要就是prometheus抓取程序通过k8s自动发现endpoints,找到k8s中的Spring服务暴露的prometheus指标。需要注意__meta_kubernetes_service_annotation_开头的注解,需要与k8s的service中注解保持一致。
在实践中比较完整的prometheus抓取配置内容(上面之包含了对spring中prometheus指标抓取),参考如下:

global:
  scrape_interval: 30s
  external_labels:
    clusterArn: arn:aws:eks:us-east-1:xxx:cluster/uat
    cluster: uat
scrape_configs:
  # pod metrics
  - job_name: pod_exporter
    kubernetes_sd_configs:
      - role: pod
  # container metrics
  - job_name: cadvisor
    scheme: https
    authorization:
      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
    metric_relabel_configs:
      - source_labels: [instance]
        separator: ;
        regex: (.+)
        target_label: node
        replacement: $1
        action: replace
  # apiserver metrics
  - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    job_name: kubernetes-apiservers
    kubernetes_sd_configs:
    - role: endpoints
    relabel_configs:
    - action: keep
      regex: default;kubernetes;https
      source_labels:
      - __meta_kubernetes_namespace
      - __meta_kubernetes_service_name
      - __meta_kubernetes_endpoint_port_name
    scheme: https
  # kube proxy metrics
  - job_name: kube-proxy
    honor_labels: true
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - action: keep
      source_labels:
      - __meta_kubernetes_namespace
      - __meta_kubernetes_pod_name
      separator: '/'
      regex: 'kube-system/kube-proxy.+'
    - source_labels:
      - __address__
      action: replace
      target_label: __address__
      regex: (.+?)(\\:\\d+)?
      replacement: $1:10249
  # kube-state-metrics
  - job_name: kube-state-metrics
    honor_timestamps: true
    scrape_interval: 1m
    scrape_timeout: 1m
    metrics_path: /metrics
    scheme: http
    static_configs:
    - targets:
      - kube-state-metrics.kube-system.svc.cluster.local:8080
  # node-exporter
  - job_name: 'node-exporter'
    kubernetes_sd_configs:
    - role: node
    relabel_configs:
    - action: replace
      source_labels: [__address__]
      regex: '(.*):10250'
      replacement: '${1}:9100'
      target_label: __address__
  # JVM (Micrometer)
  - job_name: 'kubernetes-service-endpoints'
    kubernetes_sd_configs:
    - role: endpoints
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
      action: replace
      target_label: __scheme__
      regex: (https?)
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
      action: replace
      target_label: __address__
      regex: (.+)(?::\d+);(\d+)
      replacement: $1:$2
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: application

k8s.yaml

k8s的service部署配置,如下:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: {{ .Values.services.xxxxx.name }}
  name: {{ .Values.services.xxxxx.name }}
  namespace: {{ .Release.Namespace }}
  annotations:
    alb.ingress.kubernetes.io/healthcheck-path: {{ .Values.services.xxxxx.health.path }}
    alb.ingress.kubernetes.io/healthcheck-port: '{{ .Values.services.xxxxx.health.port }}'
    prometheus.io/path: {{ .Values.services.xxxxx.prometheus.path }}
    prometheus.io/port: '{{ .Values.services.xxxxx.prometheus.port }}'
    prometheus.io/scrape: "true"
spec:
  ports:
    - name: http
      port: {{ .Values.services.xxxxx.port }}
      targetPort: {{ .Values.services.xxxxx.port }}
  selector:
    app: {{ .Values.services.xxxxx.name }}
  type: ClusterIP

注意,这里的注解需要与prometheus抓取程序保持一致,如下图:
prometheus抓取成与k8s对应
这里还有一个就是k8s里面的注释不规则命名,如点,斜杆符号等,在prometheus抓取程序这边都被转化成下划线符号,反正prometheus抓取程序遇到不规则的k8s注释命名符号,都被转成下划线。
三个关键注释:

  • prometheus.io/path:设置springboot的暴露的prometheus指标路径,即spring actuator路径;
  • prometheus.io/port:设置springboot的暴露的prometheus指标端口,即spring actuator端口;
  • prometheus.io/scrape:是否启用prometheus抓取程序。

JVM (Micrometer)面板效果

grafana效果

总结

到这里就完成了对Spring项目添加prometheus指标过程,主要就是添加micrometer和micrometer-jvm-extras依赖,启用micrometer-jvm-extras的内存指标。放开spring actuator prometheus相关端点,启用tomcat指标,配置prometheus抓取程序,设置K8S Service的prometheus抓取程序注释配置。重新发布部署,在grafana查看效果。

参考:

Logo

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐