云原生环境中的监控与可观测性最佳实践

🔥 核心概念

在云原生环境中，监控与可观测性是确保系统稳定运行的关键。它包括以下几个核心组件：

• Prometheus：时间序列数据库，用于存储和查询监控指标
• Grafana：可视化平台，用于展示监控数据
• Jaeger：分布式追踪系统，用于跟踪请求链路
• Loki：日志聚合系统，用于存储和查询日志

🚀 Prometheus部署与配置

1. 安装Prometheus

# 添加Helm仓库
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# 安装Prometheus
helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace

# 验证安装
kubectl get pods -n monitoring

2. 配置Prometheus

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 2
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      release: prometheus
  ruleSelector:
    matchLabels:
      release: prometheus
  resources:
    requests:
      memory: 4Gi
      cpu: 2
    limits:
      memory: 4Gi
      cpu: 2
  retention: 15d

3. 自定义监控指标

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics

📊 Grafana部署与配置

1. 安装Grafana

# 使用Helm安装Grafana
helm install grafana grafana/grafana --namespace monitoring

# 获取Grafana密码
kubectl get secret -n monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode

# 端口转发
kubectl port-forward -n monitoring svc/grafana 3000:3000

2. 配置Grafana数据源

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  prometheus.yaml: |
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.monitoring.svc.cluster.local
      access: proxy
      isDefault: true

3. 导入仪表盘

# 导入Kubernetes仪表盘
curl -X POST "http://localhost:3000/api/dashboards/import" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"dashboard": {...}, "overwrite": true}'

🕵️ Jaeger部署与配置

1. 安装Jaeger

# 使用Helm安装Jaeger
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm repo update
helm install jaeger jaegertracing/jaeger --namespace monitoring

# 端口转发
kubectl port-forward -n monitoring svc/jaeger-query 16686:16686

2. 配置Jaeger

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: monitoring
spec:
  strategy: allInOne
  allInOne:
    image: jaegertracing/all-in-one:1.46
    options:
      log-level: info
  storage:
    type: memory
  ingress:
    enabled: true
    hosts:
    - jaeger.example.com

3. 应用集成Jaeger

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: my-app:latest
        env:
        - name: JAEGER_SERVICE_NAME
          value: my-app
        - name: JAEGER_AGENT_HOST
          value: jaeger-agent.monitoring.svc.cluster.local
        - name: JAEGER_AGENT_PORT
          value: "6831"

📝 Loki部署与配置

1. 安装Loki

# 使用Helm安装Loki
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install loki grafana/loki-stack --namespace monitoring --set promtail.enabled=true

# 验证安装
kubectl get pods -n monitoring

2. 配置Loki

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: loki
  namespace: monitoring
spec:
  size: 1x.extra-small
  storage:
    schemas:
    - version: v12
      effectiveDate: 2023-01-01
  tenants:
    mode: openshift-logging

3. 配置Promtail

apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
  namespace: monitoring
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    clients:
    - url: http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push

    scrape_configs:
    - job_name: kubernetes-pods
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace

🔄 整合监控系统

1. 统一告警配置

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: app-alerts
  namespace: monitoring
spec:
  groups:
  - name: app
    rules:
    - alert: AppDown
      expr: up{job="my-app"} == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Application down"
        description: "Application {{ $labels.app }} is down for more than 5 minutes"

    - alert: HighCPUUsage
      expr: avg(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod) > 0.8
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High CPU usage"
        description: "Pod {{ $labels.pod }} has high CPU usage"

2. 集成Grafana仪表盘

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: kubernetes-dashboard
  namespace: monitoring
spec:
  json:
    "dashboard": {
      "id": null,
      "title": "Kubernetes Cluster",
      "panels": [
        {
          "title": "Cluster Health",
          "type": "grafana-worldmap-panel",
          "targets": [
            {
              "expr": "up{job=\"kubelet\"}"
            }
          ]
        },
        {
          "title": "CPU Usage",
          "type": "graph",
          "targets": [
            {
              "expr": "sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace)"
            }
          ]
        }
      ]
    }

📈 可观测性最佳实践

1. 标准化监控指标

// 定义监控指标
var (
    requestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "app_requests_total",
            Help: "Total number of requests",
        },
        []string{"method", "path", "status"},
    )
    requestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "app_request_duration_seconds",
            Help: "Request duration in seconds",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method", "path"},
    )
)

// 注册指标
func init() {
    prometheus.MustRegister(requestsTotal)
    prometheus.MustRegister(requestDuration)
}

// 使用指标
func handler(w http.ResponseWriter, r *http.Request) {
    start := time.Now()
    
    // 处理请求
    
    duration := time.Since(start).Seconds()
    requestsTotal.WithLabelValues(r.Method, r.URL.Path, strconv.Itoa(statusCode)).Inc()
    requestDuration.WithLabelValues(r.Method, r.URL.Path).Observe(duration)
}

2. 分布式追踪最佳实践

// 初始化Jaeger
func initJaeger() {
    cfg, err := jaeger.NewConfiguration(
        "my-app",
        jaeger.WithSampler(jaeger.NewConstSampler(true)),
        jaeger.WithReporter(jaeger.NewLoggingReporter()),
    )
    if err != nil {
        log.Fatal(err)
    }
    tracer, _, err := cfg.NewTracer()
    if err != nil {
        log.Fatal(err)
    }
    opentracing.SetGlobalTracer(tracer)
}

// 使用追踪
func handler(w http.ResponseWriter, r *http.Request) {
    span, ctx := opentracing.StartSpanFromContext(r.Context(), "handler")
    defer span.Finish()
    
    // 子操作
    childSpan, _ := opentracing.StartSpanFromContext(ctx, "database_query")
    // 执行数据库查询
    childSpan.Finish()
}

3. 日志最佳实践

// 初始化日志
func initLogger() {
    logrus.SetFormatter(&logrus.JSONFormatter{
        FieldMap: logrus.FieldMap{
            logrus.FieldKeyTime: "timestamp",
            logrus.FieldKeyLevel: "level",
            logrus.FieldKeyMsg: "message",
        },
    })
    logrus.SetOutput(os.Stdout)
    logrus.SetLevel(logrus.InfoLevel)
}

// 使用日志
func handler(w http.ResponseWriter, r *http.Request) {
    logrus.WithFields(logrus.Fields{
        "method": r.Method,
        "path": r.URL.Path,
        "ip": r.RemoteAddr,
    }).Info("Request received")
    
    // 处理请求
    
    logrus.WithFields(logrus.Fields{
        "method": r.Method,
        "path": r.URL.Path,
        "status": statusCode,
        "duration": duration,
    }).Info("Request processed")
}

🔧 监控系统维护

1. 数据保留策略

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  retention: 15d
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: standard
        resources:
          requests:
            storage: 100Gi

2. 监控系统扩容

# 扩容Prometheus
helm upgrade prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --set prometheus.resources.requests.memory=8Gi

# 扩容Grafana
helm upgrade grafana grafana/grafana --namespace monitoring --set resources.requests.memory=4Gi

3. 备份与恢复

# 备份Prometheus数据
kubectl exec -n monitoring prometheus-prometheus-community-kube-prometheus-prometheus-0 -- sh -c "tar -czf /tmp/prometheus-backup.tar.gz /prometheus"
kubectl cp monitoring/prometheus-prometheus-community-kube-prometheus-prometheus-0:/tmp/prometheus-backup.tar.gz prometheus-backup.tar.gz

# 恢复Prometheus数据
kubectl cp prometheus-backup.tar.gz monitoring/prometheus-prometheus-community-kube-prometheus-prometheus-0:/tmp/
kubectl exec -n monitoring prometheus-prometheus-community-kube-prometheus-prometheus-0 -- sh -c "tar -xzf /tmp/prometheus-backup.tar.gz -C /"

🚨 故障排查

1. 监控系统故障

# 检查Prometheus状态
kubectl get pods -n monitoring | grep prometheus

# 查看Prometheus日志
kubectl logs -n monitoring prometheus-prometheus-community-kube-prometheus-prometheus-0

# 检查Grafana状态
kubectl get pods -n monitoring | grep grafana

# 查看Grafana日志
kubectl logs -n monitoring grafana-0

2. 应用监控故障

# 检查应用Pod状态
kubectl get pods | grep my-app

# 查看应用日志
kubectl logs -l app=my-app

# 检查监控指标
kubectl port-forward -n monitoring svc/prometheus-server 9090:9090
# 访问 http://localhost:9090/graph

总结

云原生环境中的监控与可观测性是一个综合性的系统工程，需要从以下几个方面进行全面考虑：

1. Prometheus：收集和存储监控指标，提供强大的查询能力
2. Grafana：可视化监控数据，创建直观的仪表盘
3. Jaeger：跟踪分布式请求，分析系统性能瓶颈
4. Loki：聚合和查询日志，快速定位问题

通过这些工具的结合，可以构建一个完整的可观测性体系：

• 指标监控：了解系统的健康状态和性能指标
• 分布式追踪：跟踪请求在系统中的流转情况
• 日志管理：查看系统的详细运行日志
• 告警机制：及时发现和处理系统异常

在生产环境中，建议根据实际需求和规模，选择合适的监控方案，并定期进行维护和优化，以确保系统的稳定运行。

---

💡 小贴士：监控系统本身也需要被监控，建议为监控系统设置专门的监控和告警，以确保监控系统的可靠性。