K8s集群运行时异常趋势分析预警实操
K8s集群运行时异常趋势分析预警实操
|
项目 |
内容 |
|
适用版本 |
Kubernetes v1.28.15 + Prometheus 2.48 + Grafana 10.2 + Loki 2.9 + containerd 1.7.x |
|
核心目标 |
运行时异常实时感知、趋势预测、容量预警、自动化应急响应 |
|
编写日期 |
2026-08-18 |
|
文档版本 |
v1.0 |
1. 操作环境与前置准备
1.1 硬件要求
|
角色 |
CPU |
内存 |
硬盘 |
说明 |
|
Worker |
8核+ |
16GB+ |
200GB SSD |
业务节点 |
|
监控节点 |
8核+ |
16GB+ |
500GB SSD |
Prometheus+Grafana+Loki |
|
告警节点 |
4核+ |
8GB+ |
100GB SSD |
Alertmanager（可复用监控节点） |
监控节点建议独立部署，避免监控组件与业务争抢资源。大规模集群（50+节点）需独立部署。
1.2 软件版本矩阵
|
组件 |
版本 |
说明 |
|
Kubernetes |
v1.28.15 |
集群版本 |
|
containerd |
1.7.11+ |
容器运行时 |
|
Prometheus |
2.48.0+ |
指标存储与告警 |
|
Alertmanager |
0.26.0+ |
告警路由通知 |
|
Grafana |
10.2.0+ |
可视化看板 |
|
Loki |
2.9.0+ |
日志聚合分析 |
|
Promtail |
2.9.0+ |
日志采集器 |
|
node-exporter |
1.7.0+ |
节点指标采集 |
|
kube-state-metrics |
2.10.0+ |
K8s资源状态指标 |
|
kube-prometheus-stack |
最新 |
一键部署（可选） |
1.3 节点规划
|
主机名 |
IP地址 |
角色 |
部署组件 |
|
k8s-worker01 |
192.168.10.21 |
Worker |
业务Pod + node-exporter |
|
k8s-worker02 |
192.168.10.22 |
Worker |
业务Pod + node-exporter |
|
k8s-worker03 |
192.168.10.23 |
Worker |
业务Pod + node-exporter |
|
k8s-monitor |
192.168.10.40 |
监控 |
Prometheus/Grafana/Loki/Alertmanager |
1.4 存储规划
|
数据类型 |
保留周期 |
预估存储 |
存储路径 |
|
Loki日志 |
7天 |
200GB+ |
/data/loki |
|
Alertmanager |
永久 |
<1GB |
/data/alertmanager |
|
Grafana |
永久 |
<10GB |
/data/grafana |
2. 运行时异常监控体系架构
2.1 异常分类与监控维度
|
异常类别 |
监控对象 |
关键指标 |
预警级别 |
|
资源耗尽 |
CPU/内存/磁盘/网络 |
使用率、趋势斜率、预测值 |
P1/P2 |
|
Pod异常 |
容器状态 |
CrashLoopBackOff、OOMKilled、重启率 |
P1/P2 |
|
网络异常 |
连接/丢包/延迟 |
conntrack使用率、丢包率、TCP重传 |
P1/P2 |
|
存储异常 |
磁盘/IO |
磁盘使用率、IO等待、inode |
P1/P2 |
|
调度异常 |
调度器 |
调度失败数、Pending Pod数 |
P2 |
|
应用异常 |
业务容器 |
错误率、延迟、5xx比例 |
P1/P2 |
2.2 整体架构
┌──────────────────────────────────────────────────────────────────┐
│ K8s Cluster │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Worker1 │ │ Worker2 │ │ Worker3 │ │ Master │ │
│ │node-exp │ │node-exp │ │node-exp │ │kubelet │ │
│ │promtail │ │promtail │ │promtail │ │metrics │ │
│ │cAdvisor │ │cAdvisor │ │cAdvisor │ │ │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │ │
│ └──────────────┼──────────────┼──────────────┘ │
│ │ │ │
│ metrics │ logs │ │
│ ▼ ▼ │
│ ┌──────────────────────────┐ ┌──────────────────┐ │
│ │ Prometheus │ │ Loki │ │
│ │ (指标存储+告警规则) │ │ (日志聚合分析) │ │
│ └──────────┬───────────────┘ └────────┬─────────┘ │
│ │ alerts │ logs │
│ ▼ ▼ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Alertmanager │ │ Grafana │ │
│ │ (路由/通知/抑制) │◄──────►│ (可视化看板) │ │
│ └────────┬─────────┘ └──────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ 钉钉/企微/邮件 │ │
│ └──────────────────┘ │
└──────────────────────────────────────────────────────────────────┘
2.3 数据流转链路
采集层 存储层 分析层 通知层
───────── ───────── ───────── ─────────
node-exporter → Prometheus → 告警规则引擎 → Alertmanager → 钉钉
kubelet → Prometheus → 趋势预测模型 → Alertmanager → 企微
cAdvisor → Prometheus → 异常检测算法 → Grafana看板 → 邮件
Promtail → Loki → 日志关键字匹配 → 告警联动 → 短信
kube-state → Prometheus → 资源状态分析 → 容量预警
3. 指标监控采集部署
3.1 Prometheus部署
创建命名空间和配置：
kubectl create namespace monitoring
创建 prometheus-config.yaml：
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: monitoring
data:
prometheus.yml: |
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
cluster: k8s-prod
alerting:
alertmanagers:
- static_configs:
- targets:
- alertmanager.monitoring.svc.cluster.local:9093
rule_files:
- /etc/prometheus/rules/*.yml
scrape_configs:
# Prometheus自身
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
# node-exporter
- job_name: 'node-exporter'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_name]
regex: node-exporter
action: keep
# kubelet
- job_name: 'kubelet'
kubernetes_sd_configs:
- role: node
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
# cAdvisor (通过kubelet)
- job_name: 'cadvisor'
kubernetes_sd_configs:
- role: node
scheme: https
metrics_path: /metrics/cadvisor
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
# kube-state-metrics
- job_name: 'kube-state-metrics'
static_configs:
- targets: ['kube-state-metrics.monitoring.svc.cluster.local:8080']
# containerd (通过节点exporter或cRI)
- job_name: 'containerd'
static_configs:
- targets: ['192.168.10.21:9091','192.168.10.22:9091','192.168.10.23:9091']
部署Prometheus：
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
serviceAccountName: prometheus
containers:
- name: prometheus
image: harbor.local/monitoring/prometheus:v2.48.0
args:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention.time=15d
- --storage.tsdb.retention.size=100GB
- --web.enable-lifecycle
ports:
- containerPort: 9090
volumeMounts:
- name: config
mountPath: /etc/prometheus
- name: rules
mountPath: /etc/prometheus/rules
- name: storage
mountPath: /prometheus
resources:
requests:
cpu: "2"
memory: 4Gi
limits:
cpu: "4"
memory: 8Gi
volumes:
- name: config
configMap:
name: prometheus-config
- name: rules
configMap:
name: prometheus-rules
- name: storage
persistentVolumeClaim:
claimName: prometheus-pvc
---
apiVersion: v1
kind: Service
metadata:
name: prometheus
namespace: monitoring
spec:
selector:
app: prometheus
ports:
- port: 9090
targetPort: 9090
type: NodePort
3.2 node-exporter部署
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-exporter
namespace: monitoring
spec:
selector:
matchLabels:
app: node-exporter
template:
metadata:
labels:
app: node-exporter
spec:
hostNetwork: true
hostPID: true
containers:
- name: node-exporter
image: harbor.local/monitoring/node-exporter:v1.7.0
args:
- --path.procfs=/host/proc
- --path.sysfs=/host/sys
- --path.rootfs=/host/root
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+)($|/)
- --collector.systemd
- --collector.processes
- --collector.tcpstat
ports:
- containerPort: 9100
hostPort: 9100
volumeMounts:
- name: proc
mountPath: /host/proc
readOnly: true
- name: sys
mountPath: /host/sys
readOnly: true
- name: root
mountPath: /host/root
readOnly: true
volumes:
- name: proc
hostPath:
path: /proc
- name: sys
hostPath:
path: /sys
- name: root
hostPath:
path: /
---
apiVersion: v1
kind: Service
metadata:
name: node-exporter
namespace: monitoring
spec:
selector:
app: node-exporter
ports:
- port: 9100
targetPort: 9100
3.3 cAdvisor与kubelet指标
cAdvisor已集成在kubelet中，通过 /metrics/cadvisor 端点采集。关键指标：
|
指标名 |
说明 |
|
container_memory_working_set_bytes |
容器内存使用 |
|
container_network_receive_bytes_total |
容器网络接收 |
|
container_network_transmit_bytes_total |
容器网络发送 |
|
container_fs_usage_bytes |
容器磁盘使用 |
|
container_last_seen |
容器最后活跃时间 |
kubelet关键指标：
|
指标名 |
说明 |
|
kubelet_running_containers |
运行中容器数 |
|
kubelet_pod_start_duration_seconds |
Pod启动耗时 |
|
kubelet_cgroup_manager_duration_seconds |
cgroup操作耗时 |
|
kubelet_pleg_relist_duration_seconds |
PLEG耗时（过高表示运行时异常） |
3.4 containerd指标采集
containerd内置Prometheus指标支持，需开启：
# 修改containerd配置，开启metrics
# /etc/containerd/config.toml
[metrics]
address = "0.0.0.0:9091"
grpc_histogram = true
# 重启containerd
systemctl restart containerd
# 验证指标
curl http://localhost:9091/v1/metrics | head -20
containerd关键指标：
|
指标名 |
说明 |
|
containerd_grpc_request_duration_seconds |
gRPC请求耗时 |
|
containerd_snapshots_total |
快照数量 |
|
containerd_content_total |
内容存储数量 |
|
containerd_images_total |
镜像数量 |
3.5 kube-state-metrics部署
apiVersion: apps/v1
kind: Deployment
metadata:
name: kube-state-metrics
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: kube-state-metrics
template:
metadata:
labels:
app: kube-state-metrics
spec:
serviceAccountName: kube-state-metrics
containers:
- name: kube-state-metrics
image: harbor.local/monitoring/kube-state-metrics:v2.10.0
ports:
- containerPort: 8080
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
name: kube-state-metrics
namespace: monitoring
spec:
selector:
app: kube-state-metrics
ports:
- port: 8080
targetPort: 8080
4. 日志采集与异常分析
4.1 Loki部署
apiVersion: apps/v1
kind: Deployment
metadata:
name: loki
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: loki
template:
metadata:
labels:
app: loki
spec:
containers:
- name: loki
image: harbor.local/monitoring/loki:2.9.0
args:
- -config.file=/etc/loki/loki-config.yml
ports:
- containerPort: 3100
volumeMounts:
- name: config
mountPath: /etc/loki
- name: storage
mountPath: /data/loki
resources:
requests:
cpu: "1"
memory: 2Gi
limits:
cpu: "2"
memory: 4Gi
volumes:
- name: config
configMap:
name: loki-config
- name: storage
persistentVolumeClaim:
claimName: loki-pvc
---
apiVersion: v1
kind: ConfigMap
metadata:
name: loki-config
namespace: monitoring
data:
loki-config.yml: |
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /data/loki
storage:
filesystem:
chunks_directory: /data/loki/chunks
rules_directory: /data/loki/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
query_range:
results_cache:
cache:
enable_fifocache: true
fifocache:
max_size_items: 1024
validity: 24h
limits_config:
retention_period: 168h
max_query_length: 721h
max_streams_per_user: 0
reject_old_samples: true
reject_old_samples_max_age: 168h
schema_config:
configs:
- from: 2024-01-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
4.2 Promtail日志采集
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: promtail
namespace: monitoring
spec:
selector:
matchLabels:
app: promtail
template:
metadata:
labels:
app: promtail
spec:
serviceAccountName: promtail
containers:
- name: promtail
image: harbor.local/monitoring/promtail:2.9.0
args:
- -config.file=/etc/promtail/promtail-config.yml
volumeMounts:
- name: config
mountPath: /etc/promtail
- name: logs
mountPath: /var/log
readOnly: true
- name: containers
mountPath: /var/log/containers
readOnly: true
- name: pods
mountPath: /var/log/pods
readOnly: true
- name: journal
mountPath: /var/log/journal
readOnly: true
volumes:
- name: config
configMap:
name: promtail-config
- name: logs
hostPath:
path: /var/log
- name: containers
hostPath:
path: /var/log/containers
- name: pods
hostPath:
path: /var/log/pods
- name: journal
hostPath:
path: /var/log/journal
---
apiVersion: v1
kind: ConfigMap
metadata:
name: promtail-config
namespace: monitoring
data:
promtail-config.yml: |
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push
scrape_configs:
# 容器日志
- job_name: kubernetes-pods
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: ['__meta_kubernetes_pod_node_name']
target_label: 'node'
- source_labels: ['__meta_kubernetes_namespace']
target_label: 'namespace'
- source_labels: ['__meta_kubernetes_pod_name']
target_label: 'pod'
- source_labels: ['__meta_kubernetes_container_name']
target_label: 'container'
- replacement: '/var/log/pods/*$1/*.log'
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
# containerd系统日志
- job_name: containerd-system
static_configs:
- targets: [localhost]
labels:
job: containerd
__path__: /var/log/messages
pipeline_stages:
- match:
selector: '{job="containerd"} |= "containerd"'
stages:
- labels:
runtime: containerd
# kubelet日志
- job_name: kubelet-journal
journal:
json: false
max_age: 7d
path: /var/log/journal
labels:
job: kubelet
relabel_configs:
- source_labels: ['__journal__systemd_unit']
target_label: 'unit'
4.3 运行时日志关键字监控
在Loki中配置告警规则，监控运行时异常关键字：
# /etc/loki/rules/fake/runtime-alerts.yml
groups:
- name: runtime-errors
rules:
- alert: ContainerdCriError
expr: |
count_over_time({job="containerd"} |= "error" |= "cri" [5m]) > 5
for: 2m
labels:
severity: critical
annotations:
summary: "containerd CRI错误激增"
description: "节点{{ $labels.node }} containerd CRI错误5分钟内超过5次"
- alert: KubeletPlegHigh
expr: |
count_over_time({job="kubelet", unit="kubelet.service"} |= "PLEG" |= "unhealthy" [5m]) > 0
for: 1m
labels:
severity: critical
annotations:
summary: "kubelet PLEG不健康"
description: "节点{{ $labels.node }} kubelet PLEG异常，可能导致Pod无法调度"
- alert: OomKilledDetected
expr: |
count_over_time({namespace=~".+"} |= "OOMKilled" [10m]) > 0
for: 0m
labels:
severity: warning
annotations:
summary: "检测到OOMKilled"
description: "Pod {{ $labels.pod }} 因OOM被杀死"
- alert: CrashLoopBackOff
expr: |
count_over_time({namespace=~".+"} |= "CrashLoopBackOff" [5m]) > 0
for: 1m
labels:
severity: warning
annotations:
summary: "Pod CrashLoopBackOff"
description: "Pod {{ $labels.pod }} 进入CrashLoopBackOff状态"
4.4 日志异常模式匹配
常用异常日志模式：
|
异常模式 |
关键字 |
级别 |
|
CRI调用失败 |
failed to get sandbox / rpc error |
P1 |
|
kubelet节点异常 |
PLEG is not healthy / runtime operation timed out |
P0 |
|
镜像拉取失败 |
Failed to pull image / manifest unknown |
P2 |
|
OOM内存溢出 |
OOMKilled / out of memory |
P1 |
|
磁盘压力 |
disk pressure / image garbage collection |
P1 |
|
网络异常 |
network plugin cni / failed to setup network |
P1 |
|
调度失败 |
FailedScheduling / insufficient cpu/memory |
P2 |
Loki查询示例：
# 查询所有运行时错误
{job=~"containerd|kubelet"} |= "error" != "EOF"
# 按节点统计错误数
sum by (node) (count_over_time({job="containerd"} |= "error" [1h]))
# 趋势对比（同比上周）
sum(count_over_time({namespace="prod"} |= "500" [1h]))
/
sum(count_over_time({namespace="prod"} |= "500" [1h] offset 168h))
5. 运行时健康监控体系
5.1 containerd健康指标
# 查看containerd服务状态
systemctl is-active containerd
systemctl show containerd --property=NRestarts,ActiveEnterTimestamp
# 查看containerd指标
curl -s http://localhost:9091/v1/metrics | grep -E 'grpc_requests|grpc_request_duration'
# Prometheus查询
# containerd gRPC错误率
rate(containerd_grpc_requests_total{code=~"Internal|Unavailable|Unknown"}[5m])
# containerd gRPC延迟P99
histogram_quantile(0.99, rate(containerd_grpc_request_duration_seconds_bucket[5m]))
5.2 kubelet健康监控
# kubelet健康检查
curl -sk https://localhost:10250/healthz
# 应返回 ok
# kubelet就绪检查
curl -sk https://localhost:10250/readyz
# PLEG耗时（超过3s异常）
curl -sk https://localhost:10250/metrics | grep kubelet_pleg_relist_duration
# Prometheus告警表达式
# PLEG P99延迟 > 3s
histogram_quantile(0.99, rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) > 3
# kubelet重启
changes(kubelet_running_pods[5m]) < 0 and time() - kubelet_node_start_timestamp < 300
5.3 节点资源趋势监控
关键Prometheus查询：
# CPU使用率（按节点）
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# 内存使用率
100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))
# 磁盘使用率
100 - (100 * node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"})
# 磁盘IO等待
rate(node_pressure_io_waiting_seconds_total[5m])
# 网络接收速率
rate(node_network_receive_bytes_total{device!~"lo|cali.*|tunl0|vxlan"}[5m]) * 8
# 网络发送速率
rate(node_network_transmit_bytes_total{device!~"lo|cali.*|tunl0|vxlan"}[5m]) * 8
# TCP重传率
rate(node_netstat_Tcp_RetransSegs[5m]) / rate(node_netstat_Tcp_OutSegs[5m]) * 100
# conntrack使用率
node_nf_conntrack_entries / node_nf_conntrack_entries_limit * 100
5.4 Pod异常状态监控
# CrashLoopBackOff Pod数
sum(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}) by (namespace, pod)
# OOMKilled次数
increase(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[1h])
# Pending Pod数（调度失败）
sum(kube_pod_status_phase{phase="Pending"}) by (namespace)
# 容器重启率（5分钟内重启>2次）
increase(kube_pod_container_status_restarts_total[5m]) > 2
# Pod不可用
sum(kube_pod_status_phase{phase=~"Failed|Unknown"}) by (namespace, pod)
# 容器CPU throttling（被限流）
rate(container_cpu_cfs_throttled_seconds_total[5m]) > 0
5.5 网络异常监控
# conntrack使用率 > 80%
node_nf_conntrack_entries / node_nf_conntrack_entries_limit > 0.8
# TCP重传率 > 5%
rate(node_netstat_Tcp_RetransSegs[5m]) / rate(node_netstat_Tcp_OutSegs[5m]) > 0.05
# 网卡丢包率
rate(node_network_receive_drop_total[5m]) > 10
rate(node_network_transmit_drop_total[5m]) > 10
# 连接数异常增长
increase(node_netstat_Tcp_CurrEstab[10m]) > 1000
# Calico节点异常
calico Felix健康检查失败数
6. 异常趋势分析模型
6.1 静态阈值告警
最基础的告警方式，适用于明确的资源上限：
- alert: NodeCpuHigh
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
for: 10m
labels:
severity: warning
annotations:
summary: "节点CPU使用率超过85%"
- alert: NodeMemoryCritical
expr: 100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 95
for: 5m
labels:
severity: critical
6.2 动态基线告警
基于历史同期数据计算动态阈值，适用于业务有明显周期性的场景：
# 当前值 vs 上周同期基线（偏差>50%告警）
(
avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
-
avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m] offset 168h))
)
/
avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m] offset 168h))
> 0.5
6.3 同比环比趋势分析
# 环比（当前小时 vs 上一小时）
sum(rate(container_cpu_usage_seconds_total[1h]))
/
sum(rate(container_cpu_usage_seconds_total[1h] offset 1h))
# 同比（当前 vs 上周同时段）
sum(rate(container_network_receive_bytes_total[1h]))
/
sum(rate(container_network_receive_bytes_total[1h] offset 168h))
# 7日移动平均
avg_over_time((sum by(instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m])))[7d:])
6.4 异常突变检测
使用3-sigma原则检测异常突变：
# 当前值偏离7日均值超过3倍标准差
(
sum by(instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))
-
avg_over_time((sum by(instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m])))[7d:])
)
>
3 * stddev_over_time((sum by(instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m])))[7d:])
6.5 容量趋势预测
线性回归预测资源耗尽时间：
# 磁盘使用率趋势预测（预测24小时后使用率）
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 24*3600) < 0
# 内存耗尽预测（预测何时内存不足）
predict_linear(node_memory_MemAvailable_bytes[6h], 3600) < 0
# conntrack表满预测
predict_linear(node_nf_conntrack_entries[1h], 3600) > node_nf_conntrack_entries_limit * 0.95
# 按当前增长率预测磁盘满的时间（小时）
(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"})
/
(-1 * deriv(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[1h]))
/ 3600 < 24
7. 告警规则配置
7.1 运行时异常告警规则
创建 prometheus-rules.yaml：
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-rules
namespace: monitoring
data:
runtime-alerts.yml: |
groups:
- name: runtime-health
rules:
# containerd服务异常
- alert: ContainerdDown
expr: up{job="containerd"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "containerd服务停止"
description: "节点{{ $labels.instance }} containerd已停止超过1分钟"
# containerd gRPC错误率
- alert: ContainerdGrpcErrorHigh
expr: |
rate(containerd_grpc_requests_total{code=~"Internal|Unavailable|Unknown|DeadlineExceeded"}[5m])
/
rate(containerd_grpc_requests_total[5m]) > 0.05
for: 5m
labels:
severity: warning
annotations:
summary: "containerd gRPC错误率过高"
description: "节点{{ $labels.instance }} gRPC错误率超过5%"
# containerd gRPC延迟
- alert: ContainerdGrpcLatencyHigh
expr: histogram_quantile(0.99, rate(containerd_grpc_request_duration_seconds_bucket[5m])) > 2
for: 5m
labels:
severity: warning
annotations:
summary: "containerd gRPC延迟过高"
description: "节点{{ $labels.instance }} P99延迟超过2秒"
# kubelet异常
- alert: KubeletDown
expr: up{job="kubelet"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "kubelet服务停止"
# PLEG异常
- alert: KubeletPlegHighLatency
expr: histogram_quantile(0.99, rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) > 3
for: 5m
labels:
severity: critical
annotations:
summary: "kubelet PLEG延迟过高"
description: "节点{{ $labels.instance }} PLEG P99延迟超过3秒，Pod可能无法正常调度"
# kubelet重启
- alert: KubeletRestarted
expr: changes(kubelet_node_start_timestamp[15m]) > 0
for: 0m
labels:
severity: warning
annotations:
summary: "kubelet发生重启"
7.2 资源趋势告警规则
resource-alerts.yml: |
groups:
- name: resource-trend
rules:
# CPU高
- alert: NodeCpuUsageHigh
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
for: 10m
labels:
severity: warning
annotations:
summary: "节点CPU使用率超过85%"
description: "节点{{ $labels.instance }} CPU使用率{{ $value }}%"
# CPU持续高（趋势）
- alert: NodeCpuSustainedHigh
expr: avg_over_time((100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100))[30m:]) > 80
for: 30m
labels:
severity: critical
annotations:
summary: "节点CPU持续高负载"
# 内存高
- alert: NodeMemoryHigh
expr: 100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 90
for: 10m
labels:
severity: warning
# 磁盘即将满（预测）
- alert: DiskWillFillIn24h
expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay",mountpoint="/"}[6h], 24*3600) < 0
for: 1h
labels:
severity: critical
annotations:
summary: "磁盘预计24小时内写满"
description: "节点{{ $labels.instance }} 根分区预计24小时内耗尽"
# 磁盘使用率高
- alert: NodeDiskHigh
expr: 100 - (100 * node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) > 85
for: 10m
labels:
severity: warning
# 内存耗尽预测
- alert: MemoryWillExhaust
expr: predict_linear(node_memory_MemAvailable_bytes[6h], 3600) < 0
for: 15m
labels:
severity: critical
annotations:
summary: "内存预计1小时内耗尽"
7.3 网络异常告警规则
network-alerts.yml: |
groups:
- name: network-anomaly
rules:
# conntrack使用率
- alert: ConntrackHigh
expr: node_nf_conntrack_entries / node_nf_conntrack_entries_limit > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "conntrack使用率超过80%"
# conntrack即将满（预测）
- alert: ConntrackWillFull
expr: predict_linear(node_nf_conntrack_entries[1h], 1800) > node_nf_conntrack_entries_limit * 0.95
for: 10m
labels:
severity: critical
# TCP重传率
- alert: TcpRetransHigh
expr: rate(node_netstat_Tcp_RetransSegs[5m]) / rate(node_netstat_Tcp_OutSegs[5m]) > 0.05
for: 10m
labels:
severity: warning
annotations:
summary: "TCP重传率超过5%"
# 网卡丢包
- alert: NetworkPacketLoss
expr: rate(node_network_receive_drop_total[5m]) > 10 or rate(node_network_transmit_drop_total[5m]) > 10
for: 5m
labels:
severity: warning
# 网络带宽跑满
- alert: NetworkBandwidthFull
expr: (rate(node_network_receive_bytes_total{device!~"lo|cali.*|tunl0|vxlan"}[5m]) * 8) > (node_network_speed_bytes{device!~"lo|cali.*|tunl0|vxlan"} * 0.9 * 8)
for: 10m
labels:
severity: warning
7.4 告警分级与抑制
|
级别 |
触发条件 |
响应时间 |
通知方式 |
|
P1 Warning |
性能下降、资源预警 |
30分钟 |
钉钉+邮件 |
|
P2 Info |
趋势异常、容量预测 |
2小时 |
邮件+看板 |
告警抑制规则（Alertmanager）：
inhibit_rules:
# P0告警抑制同节点P1/P2告警
- source_match:
severity: critical
target_match:
severity: warning
equal: ['instance', 'alertname']
# 节点宕机抑制该节点所有Pod告警
- source_match:
alertname: NodeDown
target_match_re:
alertname: 'Pod.*'
equal: ['instance']
8. Alertmanager告警通知
8.1 Alertmanager部署
apiVersion: apps/v1
kind: Deployment
metadata:
name: alertmanager
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: alertmanager
template:
metadata:
labels:
app: alertmanager
spec:
containers:
- name: alertmanager
image: harbor.local/monitoring/alertmanager:v0.26.0
args:
- --config.file=/etc/alertmanager/alertmanager.yml
- --storage.path=/alertmanager
- --web.external-url=http://alertmanager:9093
ports:
- containerPort: 9093
volumeMounts:
- name: config
mountPath: /etc/alertmanager
- name: storage
mountPath: /alertmanager
volumes:
- name: config
configMap:
name: alertmanager-config
- name: storage
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: alertmanager
namespace: monitoring
spec:
selector:
app: alertmanager
ports:
- port: 9093
targetPort: 9093
8.2 告警路由配置
apiVersion: v1
kind: ConfigMap
metadata:
name: alertmanager-config
namespace: monitoring
data:
alertmanager.yml: |
global:
resolve_timeout: 5m
# 企业微信webhook
wechat_api_url: 'https://qyapi.weixin.qq.com/cgi-bin/'
wechat_api_secret: '<企业微信Secret>'
wechat_api_corp_id: '<企业ID>'
route:
group_by: ['alertname', 'cluster', 'instance']
group_wait: 10s
group_interval: 5m
repeat_interval: 4h
receiver: 'default'
routes:
# P0告警立即通知
- match:
severity: critical
receiver: 'critical-webhook'
group_wait: 0s
repeat_interval: 1h
# P1告警
- match:
severity: warning
receiver: 'warning-dingtalk'
repeat_interval: 4h
# P2告警
- match:
severity: info
receiver: 'info-email'
repeat_interval: 12h
receivers:
- name: 'default'
webhook_configs:
- url: 'http://alertmanager-webhook.monitoring.svc.cluster.local:8080/webhook'
- name: 'critical-webhook'
webhook_configs:
- url: 'http://alertmanager-webhook.monitoring.svc.cluster.local:8080/critical'
send_resolved: true
- name: 'warning-dingtalk'
webhook_configs:
- url: 'http://alertmanager-webhook.monitoring.svc.cluster.local:8080/dingtalk'
send_resolved: true
- name: 'info-email'
email_configs:
- to: 'ops@company.com'
from: 'alert@company.com'
smarthost: 'smtp.company.com:587'
auth_username: 'alert@company.com'
auth_password: '<邮箱密码>'
send_resolved: true
inhibit_rules:
- source_match:
severity: critical
target_match:
severity: warning
equal: ['instance']
8.3 钉钉/企业微信通知
部署Webhook转发服务（支持钉钉和企业微信）：
#!/usr/bin/env python3
# alertmanager-webhook.py
from flask import Flask, request
import requests
import json
app = Flask(__name__)
DINGTALK_WEBHOOK = "https://oapi.dingtalk.com/robot/send?access_token=<token>"
WECHAT_WEBHOOK = "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=<key>"
def send_dingtalk(alert):
severity = alert['labels'].get('severity', 'unknown')
emoji = "🔴" if severity == "critical" else "��" if severity == "warning" else "🔵"
content = f"""{emoji} **{alert['annotations'].get('summary', '告警')}**
> 级别: {severity}
> 节点: {alert['labels'].get('instance', 'N/A')}
> 描述: {alert['annotations'].get('description', 'N/A')}
> 时间: {alert['startsAt']}"""
data = {
"msgtype": "markdown",
"markdown": {"title": "K8s告警", "text": content}
}
requests.post(DINGTALK_WEBHOOK, json=data)
def send_wechat(alert):
severity = alert['labels'].get('severity', 'unknown')
content = f"[{severity.upper()}] {alert['annotations'].get('summary', '告警')}\n节点: {alert['labels'].get('instance', 'N/A')}\n描述: {alert['annotations'].get('description', 'N/A')}"
data = {"msgtype": "text", "text": {"content": content}}
requests.post(WECHAT_WEBHOOK, json=data)
@app.route('/webhook', methods=['POST'])
def webhook():
data = request.json
for alert in data.get('alerts', []):
if alert['status'] == 'firing':
send_dingtalk(alert)
send_wechat(alert)
return "ok"
@app.route('/critical', methods=['POST'])
def critical():
data = request.json
for alert in data.get('alerts', []):
if alert['status'] == 'firing':
send_dingtalk(alert)
send_wechat(alert)
return "ok"
@app.route('/dingtalk', methods=['POST'])
def dingtalk():
data = request.json
for alert in data.get('alerts', []):
if alert['status'] == 'firing':
send_dingtalk(alert)
return "ok"
if __name__ == '__main__':
app.run(host='0.0.0.0', port=8080)
8.4 告警聚合与抑制
# 同一告警分组聚合（避免告警风暴）
route:
group_by: ['alertname', 'cluster']
group_wait: 30s # 首次告警等待30s，聚合同组告警
group_interval: 5m # 同组新告警间隔5分钟
repeat_interval: 4h # 重复告警间隔4小时
# 静默规则（维护期间静默）
# 通过Alertmanager API创建
# curl -X POST http://alertmanager:9093/api/v2/silences -d '{
# "matchers": [{"name":"severity","value":"warning","isRegex":false}],
# "startsAt": "2026-08-18T22:00:00.000Z",
# "endsAt": "2026-08-19T02:00:00.000Z",
# "createdBy": "ops",
# "comment": "夜间维护窗口"
# }'
9. Grafana可视化看板
9.1 Grafana部署与配置
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
containers:
- name: grafana
image: harbor.local/monitoring/grafana:10.2.0
ports:
- containerPort: 3000
env:
- name: GF_SECURITY_ADMIN_PASSWORD
value: "Grafana@2026"
- name: GF_USERS_ALLOW_SIGN_UP
value: "false"
volumeMounts:
- name: storage
mountPath: /var/lib/grafana
- name: datasources
mountPath: /etc/grafana/provisioning/datasources
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: "2"
memory: 2Gi
volumes:
- name: storage
persistentVolumeClaim:
claimName: grafana-pvc
- name: datasources
configMap:
name: grafana-datasources
---
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
namespace: monitoring
data:
datasources.yml: |
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus.monitoring.svc.cluster.local:9090
isDefault: true
- name: Loki
type: loki
access: proxy
url: http://loki.monitoring.svc.cluster.local:3100
9.2 运行时总览看板
关键面板配置：
|
面板 |
查询语句 |
图表类型 |
|
containerd状态 |
up{job="containerd"} |
Stat |
|
kubelet状态 |
up{job="kubelet"} |
Stat |
|
集群CPU使用率 |
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) |
Gauge |
|
集群内存使用率 |
100 * (1 - (sum(node_memory_MemAvailable_bytes) / sum(node_memory_MemTotal_bytes))) |
Gauge |
|
集群磁盘使用率 |
100 - (100 * sum(node_filesystem_avail_bytes{mountpoint="/"}) / sum(node_filesystem_size_bytes{mountpoint="/"})) |
Gauge |
|
Pod总数 |
sum(kube_pod_status_phase{phase="Running"}) |
Stat |
|
异常Pod数 |
`sum(kube_pod_status_phase{phase=~"Failed |
Pending |
|
运行时重启数 |
increase(kube_pod_container_status_restarts_total[1h]) |
Table |
9.3 趋势分析看板
|
面板 |
查询语句 |
图表类型 |
|
内存趋势（7天） |
100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) |
Time series |
|
网络入流量趋势 |
`rate(node_network_receive_bytes_total{device!~"lo |
.*cali.* |
|
网络出流量趋势 |
`rate(node_network_transmit_bytes_total{device!~"lo |
.*cali.* |
|
conntrack趋势 |
node_nf_conntrack_entries / node_nf_conntrack_entries_limit * 100 |
Time series |
|
TCP连接数趋势 |
node_netstat_Tcp_CurrEstab |
Time series |
|
磁盘IO趋势 |
rate(node_disk_io_time_seconds_total[5m]) * 100 |
Time series |
|
同比对比 |
当前值 / 上周同期值 |
Time series |
9.4 异常告警看板
|
面板 |
内容 |
|
告警趋势（24h） |
按级别统计告警数 |
|
TOP10告警源 |
按instance/alertname统计 |
|
告警响应时长 |
从firing到resolved的时间差 |
|
告警静默列表 |
当前生效的silence |
|
P0告警历史 |
critical级别告警记录 |
10. 趋势预测与容量预警
10.1 资源使用趋势预测
# CPU使用率线性回归预测（1小时后）
predict_linear(
(100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100))[6h:],
3600
)
# 内存使用预测（2小时后）
predict_linear(
(100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)))[6h:],
7200
)
# 基于7天数据的趋势预测
predict_linear(
(sum by(instance) (rate(node_cpu_seconds_total{mode!="idle"}[1h])))[7d:],
86400
)
10.2 磁盘容量预警
# 节点磁盘监控脚本（配合cron定时执行）
#!/bin/bash
THRESHOLD=80
for node in $(kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}'); do
USAGE=$(ssh root@${node} "df -h / | awk 'NR==2{print \$5}' | tr -d '%'")
if [ ${USAGE} -gt ${THRESHOLD} ]; then
# 计算按当前增速耗尽时间
REMAIN=$(ssh root@${node} "df -B1 / | awk 'NR==2{print \$4}'")
GROWTH=$(ssh root@${node} "cat /tmp/disk_growth_rate")
HOURS_LEFT=$((REMAIN / GROWTH / 3600))
echo "WARN: ${node} 磁盘使用率${USAGE}%, 预计${HOURS_LEFT}小时后耗尽"
# 发送告警...
fi
done
Prometheus磁盘预测告警：
- alert: DiskPredictionCritical
expr: |
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay",mountpoint="/"}[6h], 86400) < 0
for: 30m
labels:
severity: critical
annotations:
summary: "磁盘预计24小时内写满"
description: "节点{{ $labels.instance }} 根分区按当前增速预计24小时内耗尽，请立即清理"
- alert: DiskPredictionWarning
expr: |
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay",mountpoint="/"}[24h], 259200) < 0
for: 1h
labels:
severity: warning
annotations:
summary: "磁盘预计3天内写满"
10.3 连接数趋势预警
# conntrack表满预测（30分钟内超过95%）
predict_linear(node_nf_conntrack_entries[1h], 1800) > node_nf_conntrack_entries_limit * 0.95
# TCP连接数异常增长（1小时内增长超过50%）
(
node_netstat_Tcp_CurrEstab
-
node_netstat_Tcp_CurrEstab offset 1h
)
/
node_netstat_Tcp_CurrEstab offset 1h > 0.5
# 容器网络连接数趋势
sum by(pod) (rate(container_network_tcp_connections_total[5m]))
10.4 自动化容量扩缩容建议
基于趋势数据生成扩容建议：
#!/bin/bash
# 容量评估脚本
echo "========== K8s集群容量评估报告 =========="
echo "生成时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo ""
# 1. CPU容量
CPU_TOTAL=$(kubectl get nodes -o jsonpath='{.items[*].status.capacity.cpu}' | tr ' ' '\n' | awk '{sum+=$1}END{print sum}')
CPU_USED=$(kubectl top nodes --no-headers | awk '{sum+=$3}END{print sum}' | tr -d '%')
echo "[CPU] 总核数: ${CPU_TOTAL}, 已用: ${CPU_USED}%"
# 2. 内存容量
MEM_TOTAL=$(kubectl get nodes -o jsonpath='{.items[*].status.capacity.memory}' | tr ' ' '\n' | sed 's/Ki//' | awk '{sum+=$1}END{print sum/1024/1024 "Gi"}')
MEM_USED=$(kubectl top nodes --no-headers | awk '{sum+=$5}END{print sum}' | tr -d '%')
echo "[内存] 总量: ${MEM_TOTAL}, 已用: ${MEM_USED}%"
# 3. Pod容量
POD_TOTAL=$(kubectl get nodes -o jsonpath='{.items[*].status.capacity.pods}' | tr ' ' '\n' | awk '{sum+=$1}END{print sum}')
POD_USED=$(kubectl get pods -A --no-headers | wc -l)
echo "[Pod] 总容量: ${POD_TOTAL}, 已用: ${POD_USED} ($((POD_USED*100/POD_TOTAL))%)"
# 4. 扩容建议
echo ""
echo "========== 扩容建议 =========="
if [ ${CPU_USED} -gt 70 ]; then
echo "⚠ CPU使用率超过70%，建议新增Worker节点"
fi
if [ ${MEM_USED} -gt 70 ]; then
echo "⚠ 内存使用率超过70%，建议新增Worker节点"
fi
if [ $((POD_USED*100/POD_TOTAL)) -gt 75 ]; then
echo "⚠ Pod容量使用率超过75%，建议新增Worker节点"
fi
echo "✅ 容量评估完成"
11. 应急响应流程
11.1 告警分级响应矩阵
|
级别 |
告警类型 |
响应时间 |
处理人 |
升级条件 |
|
P1 |
CPU/内存>90%、磁盘预测24h满、OOM、PLEG异常 |
30分钟 |
值班SRE |
1小时未解决→技术负责人 |
|
P2 |
CPU/内存>80%、TCP重传高、调度失败、趋势异常 |
2小时 |
SRE团队 |
4小时未解决→技术负责人 |
|
P3 |
容量预测、日志异常、性能基线偏离 |
工作日 |
SRE团队 |
- |
11.2 运行时异常应急处理
containerd停止应急：
# 1. 确认状态
systemctl status containerd
crictl info
# 2. 查看日志
journalctl -u containerd -n 200 --no-pager
# 3. 尝试重启
systemctl restart containerd
# 4. 如重启失败，检查配置
containerd config dump > /dev/null
# 5. 检查残留进程
ps aux | grep containerd
ss -lntp | grep containerd.sock
# 6. 紧急恢复（清理残留后重启）
rm -f /run/containerd/containerd.sock
systemctl restart containerd
# 7. 验证恢复
crictl ps
kubectl get nodes
节点NotReady应急：
# 1. 确认节点状态
kubectl get nodes
kubectl describe node <node-name>
# 2. 检查kubelet
systemctl status kubelet
journalctl -u kubelet -n 100
# 3. 检查containerd
systemctl status containerd
crictl info
# 4. 检查资源
free -h
df -h
dmesg | tail -50
# 5. 紧急驱逐（如节点无法恢复）
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force
# 6. 恢复后重新调度
kubectl uncordon <node-name>
11.3 自动化自愈脚本
#!/bin/bash
# k8s-node-auto-heal.sh
# 节点自动健康检查与自愈
LOGFILE="/var/log/k8s-heal.log"
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> ${LOGFILE}
}
# 1. containerd健康检查
if ! systemctl is-active containerd > /dev/null; then
log "containerd未运行，尝试重启..."
systemctl restart containerd
sleep 5
if systemctl is-active containerd > /dev/null; then
log "containerd重启成功"
else
log "containerd重启失败，需人工介入"
# 发送告警...
fi
fi
# 2. kubelet健康检查
if ! systemctl is-active kubelet > /dev/null; then
log "kubelet未运行，尝试重启..."
systemctl restart kubelet
fi
# 3. 磁盘空间检查
DISK_USAGE=$(df / | awk 'NR==2{print $5}' | tr -d '%')
if [ ${DISK_USAGE} -gt 90 ]; then
log "磁盘使用率${DISK_USAGE}%，执行清理..."
# 清理containerd未使用镜像
crictl rmi --prune
# 清理journal日志
journalctl --vacuum-size=500M
# 清理/tmp
find /tmp -type f -atime +7 -delete
fi
# 4. conntrack检查
CT_COUNT=$(sysctl -n net.netfilter.nf_conntrack_count)
CT_MAX=$(sysctl -n net.netfilter.nf_conntrack_max)
if [ $((CT_COUNT * 100 / CT_MAX)) -gt 90 ]; then
log "conntrack使用率过高，调大上限..."
sysctl -w net.netfilter.nf_conntrack_max=2097152
fi
# 5. OOM Pod检查
OOM_PODS=$(crictl ps -a | grep OOMKilled | wc -l)
if [ ${OOM_PODS} -gt 0 ]; then
log "检测到${OOM_PODS}个OOMKilled容器"
fi
配置定时执行：
# 每5分钟执行一次
chmod +x /opt/k8s-node-auto-heal.sh
(crontab -l 2>/dev/null; echo "*/5 * * * * /opt/k8s-node-auto-heal.sh") | crontab -
11.4 事后复盘与规则优化
复盘模板：
【告警复盘报告】
告警名称：
发生时间：
恢复时间：
持续时长：
影响范围：
根本原因：
处理过程：
改进措施：
1. 短期（立即执行）：
2. 中期（1周内）：
3. 长期（1月内）：
告警规则优化：
- 阈值调整：
- 新增告警：
- 移除误报告警：
更多推荐
4
0- 0
已为社区贡献75条内容
所有评论(0)