凌晨3点，手机疯狂报警。线上服务响应超时，用户投诉。登上服务器一看：内存用了95%，GC频率飙到每秒3次。如果2小时前就能发现内存缓慢上涨的趋势，这场事故完全可以避免。监控不是锦上添花，是生产环境的生命线。

---

一、监控体系全景图

┌─────────────────────────────────────────────────┐
│                监控体系架构                        │
├─────────────────────────────────────────────────┤
│                                                 │
│  数据采集层                                      │
│  ├── Node Exporter（节点指标）                    │
│  ├── kube-state-metrics（K8s资源状态）            │
│  ├── Actuator + Micrometer（应用指标）            │
│  └── cAdvisor（容器指标）                         │
│       │                                         │
│       ▼                                         │
│  数据存储层                                      │
│  └── Prometheus（时序数据库 + 拉取引擎）           │
│       │                                         │
│       ▼                                         │
│  展示层 + 告警层                                  │
│  ├── Grafana（可视化大屏）                        │
│  └── AlertManager（告警通知：钉钉/企微/邮件）      │
│                                                 │
└─────────────────────────────────────────────────┘

核心理念：Prometheus主动拉取（Pull）各组件的指标数据，而不是组件推送（Push）。这样即使监控系统和被监控对象解耦，扩展更方便。

---

二、一键部署Prometheus + Grafana

创建监控命名空间

# monitoring-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring

Prometheus配置

# prometheus-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s

    scrape_configs:
      # 采集K8s节点指标
      - job_name: 'node-exporter'
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_name]
          regex: 'node-exporter'
          action: keep

      # 采集K8s资源状态
      - job_name: 'kube-state-metrics'
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_name]
          regex: 'kube-state-metrics'
          action: keep

      # 采集容器指标（cAdvisor内置在kubelet中）
      - job_name: 'cadvisor'
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - source_labels: [__address__]
          regex: '(.*):10250'
          replacement: '${1}:4194'
          target_label: __address__

      # 采集Java应用指标
      - job_name: 'easy-platform'
        metrics_path: '/actuator/prometheus'
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - easy-platform
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_name]
          regex: 'easy-platform'
          action: keep

Prometheus部署

# prometheus-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
      - name: prometheus
        image: prom/prometheus:v2.48.0
        ports:
        - containerPort: 9090
        volumeMounts:
        - name: config
          mountPath: /etc/prometheus
        - name: data
          mountPath: /prometheus
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "2Gi"
            cpu: "1000m"
      volumes:
      - name: config
        configMap:
          name: prometheus-config
      - name: data
        emptyDir: {}    # 生产环境建议用PVC持久化
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  selector:
    app: prometheus
  ports:
  - port: 9090
    targetPort: 9090

Grafana部署

# grafana-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:10.2.0
        ports:
        - containerPort: 3000
        env:
        - name: GF_SECURITY_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana-secret
              key: admin-password
        resources:
          requests:
            memory: "256Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        volumeMounts:
        - name: grafana-data
          mountPath: /var/lib/grafana
      volumes:
      - name: grafana-data
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  selector:
    app: grafana
  ports:
  - port: 3000
    targetPort: 3000

Node Exporter部署（每个节点采集主机指标）

# node-exporter-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.7.0
        ports:
        - containerPort: 9100
        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --path.rootfs=/host/root
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
        - name: root
          mountPath: /host/root
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      - name: root
        hostPath:
          path: /

kube-state-metrics部署

# kube-state-metrics-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-state-metrics
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      containers:
      - name: kube-state-metrics
        image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.12.0
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  selector:
    app: kube-state-metrics
  ports:
  - port: 8080
    targetPort: 8080

一键部署

kubectl apply -f monitoring-namespace.yaml
kubectl create secret generic grafana-secret \
  --from-literal=admin-password=YourGrafanaPassword \
  -n monitoring
kubectl apply -f prometheus-config.yaml
kubectl apply -f prometheus-deployment.yaml
kubectl apply -f grafana-deployment.yaml
kubectl apply -f node-exporter-daemonset.yaml
kubectl apply -f kube-state-metrics-deployment.yaml

---

三、Spring Boot接入Prometheus

1. 添加依赖

<!-- pom.xml -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

2. 配置端点

# application.yaml
management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus,metrics
  metrics:
    export:
      prometheus:
        enabled: true
    tags:
      application: easy-platform    # 全局标签，区分不同应用

3. 自定义业务指标

@Component
public class OrderMetrics {

    private final Counter orderCounter;
    private final Timer orderTimer;
    private final Gauge activeConnections;

    public OrderMetrics(MeterRegistry registry) {
        // 订单创建计数器
        orderCounter = Counter.builder("order.created.total")
            .description("订单创建总数")
            .tag("type", "normal")
            .register(registry);

        // 订单处理耗时
        orderTimer = Timer.builder("order.process.time")
            .description("订单处理耗时")
            .register(registry);

        // 当前活跃连接数
        activeConnections = Gauge.builder("order.active.connections", this, 
            value -> getCurrentConnections())
            .description("当前活跃连接数")
            .register(registry);
    }

    public void recordOrder() {
        orderCounter.increment();
    }

    public Timer.Sample startTimer() {
        return Timer.start();
    }

    public void recordProcessTime(Timer.Sample sample) {
        sample.stop(orderTimer);
    }
}

4. 使用自定义指标

@Service
public class OrderService {

    private final OrderMetrics orderMetrics;

    public OrderService(OrderMetrics orderMetrics) {
        this.orderMetrics = orderMetrics;
    }

    public void createOrder(OrderRequest request) {
        Timer.Sample sample = orderMetrics.startTimer();
        try {
            // 业务逻辑...
            orderMetrics.recordOrder();
        } finally {
            orderMetrics.recordProcessTime(sample);
        }
    }
}

访问 http://localhost:8080/actuator/prometheus 可以看到指标数据：

# HELP order_created_total 订单创建总数
# TYPE order_created_total counter
order_created_total{application="easy-platform",type="normal"} 1523.0

# HELP order_process_time_seconds 订单处理耗时
# TYPE order_process_time_seconds summary
order_process_time_seconds_count{application="easy-platform"} 1523.0
order_process_time_seconds_sum{application="easy-platform"} 45.67

---

四、Grafana配置大屏

1. 添加数据源

访问Grafana → Configuration → Data Sources → Add Prometheus：

• URL: http://prometheus.monitoring.svc.cluster.local:9090
• 点击 Save & Test

2. 导入现成Dashboard

不用从零画，Grafana社区有大量现成模板：

Dashboard ID	用途	效果
6417	K8s节点监控	CPU/内存/磁盘/网络
15760	K8s Pod监控	容器资源使用排行
4701	JVM监控	堆内存/GC/线程
11955	Spring Boot监控	请求量/响应时间/错误率

导入方式：Dashboard → Import → 输入ID → Load

3. 关键监控面板配置

如果现成模板不满足，手动创建核心面板：

面板1：应用QPS（每秒请求数）

rate(http_server_requests_seconds_count{application="easy-platform"}[1m])

面板2：应用P99响应时间

histogram_quantile(0.99, 
  rate(http_server_requests_seconds_bucket{application="easy-platform"}[5m])
)

面板3：错误率

rate(http_server_requests_seconds_count{application="easy-platform",status=~"5.."}[1m])
  /
rate(http_server_requests_seconds_count{application="easy-platform"}[1m])

面板4：JVM堆内存使用

jvm_memory_used_bytes{application="easy-platform",area="heap"}
  /
jvm_memory_max_bytes{application="easy-platform",area="heap"}
  * 100

面板5：GC频率

rate(jvm_gc_pause_seconds_count{application="easy-platform"}[1m])

面板6：Pod重启次数

kube_pod_container_status_restarts_total{namespace="easy-platform"}

---

五、告警配置（AlertManager）

1. 告警规则

# alert-rules.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-alert-rules
  namespace: monitoring
data:
  alert_rules.yml: |
    groups:
    # 应用层告警
    - name: app-alerts
      rules:
      - alert: AppDown
        expr: up{job="easy-platform"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "应用不可用"
          description: "应用 {{ $labels.instance }} 已停止超过1分钟"

      - alert: HighErrorRate
        expr: |
          rate(http_server_requests_seconds_count{application="easy-platform",status=~"5.."}[1m])
            / rate(http_server_requests_seconds_count{application="easy-platform"}[1m]) > 0.05
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "错误率超过5%"
          description: "应用5xx错误率 {{ $value | humanizePercentage }}"

      - alert: HighResponseTime
        expr: |
          histogram_quantile(0.99, 
            rate(http_server_requests_seconds_bucket{application="easy-platform"}[5m])
          ) > 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P99响应时间超过3秒"
          description: "当前P99响应时间 {{ $value }}秒"

    # JVM告警
    - name: jvm-alerts
      rules:
      - alert: HeapMemoryHigh
        expr: |
          jvm_memory_used_bytes{application="easy-platform",area="heap"}
            / jvm_memory_max_bytes{application="easy-platform",area="heap"} > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "JVM堆内存使用超过85%"
          description: "当前堆内存使用率 {{ $value | humanizePercentage }}"

      - alert: GCFrequencyHigh
        expr: rate(jvm_gc_pause_seconds_count{application="easy-platform"}[1m]) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "GC频率过高"
          description: "每分钟GC {{ $value }} 次"

    # K8s层告警
    - name: k8s-alerts
      rules:
      - alert: PodCrashLooping
        expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Pod反复重启"
          description: "Pod {{ $labels.pod }} 在15分钟内重启 {{ $value }} 次"

      - alert: NodeMemoryHigh
        expr: |
          (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "节点内存使用超过90%"
          description: "节点 {{ $labels.instance }} 内存使用率 {{ $value | humanizePercentage }}"

      - alert: PVCAlmostFull
        expr: |
          kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "PVC存储使用超过85%"

2. AlertManager配置（企业微信通知）

# alertmanager-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: monitoring
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m

    route:
      group_by: ['alertname', 'severity']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      routes:
      - match:
          severity: critical
        receiver: 'wechat-critical'
        repeat_interval: 10m
      - match:
          severity: warning
        receiver: 'wechat-warning'

    receivers:
    - name: 'wechat-critical'
      webhook_configs:
      - url: 'http://alertmanager-webhook:8060/wechat'
        send_resolved: true

    - name: 'wechat-warning'
      webhook_configs:
      - url: 'http://alertmanager-webhook:8060/wechat'
        send_resolved: true

也可以配置钉钉通知，使用钉钉机器人Webhook更简单：

- name: 'dingtalk'
  webhook_configs:
  - url: 'https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN'
    send_resolved: true

3. 告警分级标准

级别	条件	通知方式	响应时间
🔴 P0 Critical	服务不可用、Pod反复重启	电话 + 钉钉/企微	5分钟内
🟠 P1 Warning	内存>85%、GC频繁、错误率>5%	钉钉/企微	30分钟内
🟡 P2 Info	PVC>70%、磁盘>80%	邮件	当天处理

---

六、生产环境监控大屏效果

最终监控大屏应包含以下6个区域：

┌──────────────────────────────────────────────────────┐
│                  🏢 应用总览                          │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌─────────┐  │
│  │ QPS: 520 │ │ P99: 86ms│ │ 错误率    │ │ 在线Pod  │  │
│  │ ↑ 12%    │ │ ↓ 5ms    │ │ 0.03%    │ │ 3/3     │  │
│  └──────────┘ └──────────┘ └──────────┘ └─────────┘  │
├──────────────────────────────────────────────────────┤
│  📈 请求量趋势(1h)          │  📈 响应时间趋势(1h)      │
│ ~~~~~~~~~~~~~~~~~~~~~     │  ~~~~~~~~~~~~~~~~~~~~~~  │
│  (折线图)                  │  (折线图P50/P90/P99)      │
├──────────────────────────────────────────────────────┤
│  ☕ JVM内存(实时)           │  🗑️ GC趋势(1h)            │
│  ┌──────────────────┐     │  ~~~~~~~~~~~~~~~~~~~~~~  │
│  │ Heap: 612M/1G    │     │  (Young GC / Full GC)    │
│  │ NonHeap: 187M    │     │                          │
│  └──────────────────┘     │                          │
├──────────────────────────────────────────────────────┤
│  🖥️ 节点资源              │  🔔 最近告警               │
│  Node1: CPU 45% Mem 62%  │  ⚠️ HeapMemoryHigh 10:30  │
│  Node2: CPU 38% Mem 71%  │  ✅ HighErrorRate 已恢复   │
│  Node3: CPU 52% Mem 58%  │                           │
└──────────────────────────────────────────────────────┘

---

七、常见问题排查手册

现象	查看指标	排查方向
接口变慢	http_server_requests_seconds P99	是所有接口还是某个接口？
内存持续上涨	jvm_memory_used_bytes	是否有内存泄漏？dump分析
GC频繁	jvm_gc_pause_seconds_count	堆是不是太小？大对象太多？
Pod重启	kube_pod_container_status_restarts_total	OOM? 健康检查失败?
CPU飙升	process_cpu_usage	死循环？正则回溯？
线程数暴涨	jvm_threads_live_threads	线程泄漏？连接池问题？

---

面试速答

面试官：说一下你们项目的监控体系是怎么搭建的？

答：我们用Prometheus + Grafana + AlertManager搭建了完整的监控体系。数据采集分三层：基础设施层用Node Exporter采集节点CPU/内存/磁盘，容器层用cAdvisor和kube-state-metrics采集Pod状态和资源使用，应用层用Spring Boot Actuator+Micrometer暴露JVM和业务指标，Prometheus每15秒拉取一次。Grafana上搭建了监控大屏，包含QPS、P99响应时间、错误率、JVM内存和GC等核心指标。告警方面，用AlertManager配置了三级告警：P0服务不可用直接电话通知，P1内存超85%或GC频繁通过钉钉通知，P2资源使用率偏高邮件通知。