返利机器人灰度发布：Kubernetes Argo Rollout 基于 Prometheus 指标自动回滚

大家好，我是 微赚淘客系统3.0 的研发者省赚客！

微赚淘客系统3.0的返利机器人服务（rebate-bot）每日处理数百万条用户返利通知与订单追踪任务。为保障高可用性，我们采用 Argo Rollouts 实现渐进式灰度发布，并结合 Prometheus 监控指标 实现异常自动回滚。本文将通过完整 YAML 配置与 Java 代码示例，展示如何基于业务错误率、延迟等指标触发安全回滚。

1. 返利机器人服务暴露 Prometheus 指标

首先在 juwatech.cn.bot.RebateBotApplication 中集成 Micrometer：

// juwatech.cn.bot.metrics.BotMetricsService.java
@Component
public class BotMetricsService {

    private final Counter errorCounter;
    private final Timer latencyTimer;

    public BotMetricsService(MeterRegistry registry) {
        this.errorCounter = Counter.builder("juwatech_cn_bot_order_process_errors_total")
                .description("返利机器人订单处理失败次数")
                .register(registry);
        this.latencyTimer = Timer.builder("juwatech_cn_bot_order_process_duration_seconds")
                .description("订单处理耗时")
                .register(registry);
    }

    public <T> T recordLatency(Supplier<T> action) {
        return latencyTimer.recordCallable(action);
    }

    public void markError() {
        errorCounter.increment();
    }
}

在订单处理逻辑中埋点：

// juwatech.cn.bot.service.OrderProcessService.java
@Service
public class OrderProcessService {

    @Autowired
    private BotMetricsService metrics;

    public void handleOrder(OrderEvent event) {
        try {
            metrics.recordLatency(() -> {
                // 实际处理逻辑
                processInternal(event);
                return null;
            });
        } catch (Exception e) {
            metrics.markError();
            throw e;
        }
    }
}

Spring Boot Actuator 自动暴露 /actuator/prometheus 端点。

2. Argo Rollout 配置：金丝雀 + 流量切分

定义 rollout.yaml，采用 5% → 20% → 100% 三阶段灰度：

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rebate-bot
spec:
  replicas: 20
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 180s }
        - setWeight: 20
        - pause: { duration: 300s }
        - setWeight: 100
      trafficRouting:
        istio:
          virtualService:
            name: rebate-bot-vs
            routes:
              - primary
      analysis:
        templates:
          - templateName: bot-error-rate-analysis
        startingStep: 1  # 从第二步开始分析（即5%流量后）

3. AnalysisTemplate：基于 Prometheus 指标自动回滚

创建 analysis.yaml，监控两个关键指标：

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: bot-error-rate-analysis
spec:
  metrics:
    - name: error-rate
      interval: 30s
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            sum(rate(juwatech_cn_bot_order_process_errors_total{job="rebate-bot"}[5m]))
            /
            sum(rate(juwatech_cn_bot_order_process_duration_seconds_count{job="rebate-bot"}[5m]))
            > 0.02  # 错误率超过2%
    - name: p99-latency
      interval: 30s
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            histogram_quantile(0.99, 
              rate(juwatech_cn_bot_order_process_duration_seconds_bucket{job="rebate-bot"}[5m])
            ) > 1.5  # P99延迟超过1.5秒

若任一指标连续失败达到 failureLimit，Argo Rollouts 将自动中止发布并回滚到稳定版本。

4. Istio VirtualService 配合流量切分

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: rebate-bot-vs
spec:
  hosts:
    - rebate-bot.juwatech.cn
  http:
    - route:
        - destination:
            host: rebate-bot
            subset: stable
          weight: 95
        - destination:
            host: rebate-bot
            subset: canary
          weight: 5
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: rebate-bot-dr
spec:
  host: rebate-bot
  subsets:
    - name: stable
      labels:
        app: rebate-bot
        version: v1
    - name: canary
      labels:
        app: rebate-bot
        version: v2

Argo Rollouts 会自动更新 VirtualService 的权重。

5. 触发发布与观测

使用 kubectl 启动灰度：

kubectl argo rollouts set image rebate-bot rebate-bot=juwatech.cn/rebate-bot:v2.1.0

实时观测状态：

kubectl argo rollouts get rollout rebate-bot -w

若 Prometheus 检测到新版本错误率飙升，Argo 将输出：

Status:  ✖ Degraded
Message: AnalysisRun "rebate-bot-7d8f9c4b5-2" failed: metric "error-rate" failed

并自动将流量切回 v1，同时保留失败 Pod 供排查。

6. 增强：自定义 Webhook 通知企业微信

在 AnalysisTemplate 中添加 webhook 通知：

metrics:
  - name: notify-on-failure
    provider:
      webhook:
        url: https://alert.juwatech.cn/argorollout-hook
        headers:
          - key: Authorization
            value: Bearer xxx
        jsonBody: |
          {
            "event": "RolloutFailed",
            "rollout": "{{ args.rollout }}",
            "reason": "{{ args.message }}"
          }
    failureCondition: "true"

后端接收服务（juwatech.cn.alert.ArgoRolloutWebhookController）：

@RestController
public class ArgoRolloutWebhookController {

    @PostMapping("/argorollout-hook")
    public ResponseEntity<Void> handleRolloutAlert(@RequestBody AlertPayload payload) {
        if ("RolloutFailed".equals(payload.getEvent())) {
            WeComNotifier.sendAlert(
                String.format("【发布失败】%s: %s", payload.getRollout(), payload.getReason())
            );
        }
        return ResponseEntity.ok().build();
    }

    public static class AlertPayload {
        private String event;
        private String rollout;
        private String reason;
        // getters
    }
}

通过上述架构，微赚淘客系统3.0 实现了 无人值守的安全灰度发布，在新版本引入性能退化或逻辑缺陷时，可在 2 分钟内自动回滚，最大限度保障返利服务稳定性。

本文著作权归 微赚淘客系统3.0 研发团队，转载请注明出处！