CANN容器化部署与云原生实践

CANN组织链接：https://atomgit.com/cann
CANN community仓库链接：https://atomgit.com/cann/community

一、容器化部署概述

1.1 容器化优势

容器化技术为CANN应用部署提供了标准化和可移植的解决方案。

1.1.1 主要优势

• 环境一致性：开发、测试、生产环境一致
• 快速部署：秒级启动应用
• 资源隔离：CPU、内存、GPU隔离
• 易于扩展：水平扩展方便

1.1.2 容器技术栈

• Docker：容器运行时
• Kubernetes：容器编排
• Helm：应用包管理
• Prometheus：监控告警

1.2 CANN容器化挑战

• NPU设备直通
• 驱动兼容性
• 性能优化
• 存储管理

二、Docker容器化

2.1 基础镜像构建

# CANN基础镜像
FROM ubuntu:20.04

# 设置环境变量
ENV ASCEND_HOME=/usr/local/Ascend
ENV PATH=${ASCEND_HOME}/ascend-toolkit/latest/bin:${PATH}
ENV LD_LIBRARY_PATH=${ASCEND_HOME}/ascend-toolkit/latest/lib64:${LD_LIBRARY_PATH}

# 安装基础依赖
RUN apt-get update && apt-get install -y \
    python3.8 \
    python3-pip \
    python3-dev \
    gcc \
    g++ \
    cmake \
    wget \
    && rm -rf /var/lib/apt/lists/*

# 安装CANN运行时
COPY cann-runtime_*.run /tmp/
RUN chmod +x /tmp/cann-runtime_*.run && \
    /tmp/cann-runtime_*.run --install --quiet && \
    rm /tmp/cann-runtime_*.run

# 安装Python依赖
COPY requirements.txt /tmp/
RUN pip3 install --no-cache-dir -r /tmp/requirements.txt

# 设置工作目录
WORKDIR /app

# 复制应用代码
COPY . /app/

# 设置入口点
ENTRYPOINT ["python3", "app.py"]

2.2 应用镜像构建

# CANN应用镜像
FROM cann-base:latest

# 安装应用依赖
RUN pip3 install --no-cache-dir \
    flask==2.0.0 \
    gunicorn==20.1.0 \
    numpy==1.21.0 \
    opencv-python==4.5.0

# 复制模型文件
COPY models/ /app/models/

# 复制应用代码
COPY app/ /app/app/

# 暴露端口
EXPOSE 8080

# 健康检查
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8080/health || exit 1

# 启动命令
CMD ["gunicorn", "--bind", "0.0.0.0:8080", "--workers", "4", "app:app"]

2.3 构建脚本

#!/bin/bash
# build.sh

set -e

# 镜像名称和标签
IMAGE_NAME="cann-app"
IMAGE_TAG="v1.0"

# 构建基础镜像
echo "Building base image..."
docker build -t cann-base:latest -f Dockerfile.base .

# 构建应用镜像
echo "Building application image..."
docker build -t ${IMAGE_NAME}:${IMAGE_TAG} -f Dockerfile .

# 打标签
docker tag ${IMAGE_NAME}:${IMAGE_TAG} ${IMAGE_NAME}:latest

echo "Build completed successfully!"
echo "Images:"
docker images | grep cann

三、NPU设备管理

3.1 NPU直通配置

# 启用NPU直通
#!/bin/bash

# 检查NPU设备
npu-smi info

# 配置Docker使用NPU
# 方法1：使用设备直通
docker run -it \
  --device=/dev/davinci0 \
  --device=/dev/davinci_manager \
  --device=/dev/devmm_svm \
  --device=/dev/hisi_hdc \
  -v /usr/local/Ascend:/usr/local/Ascend \
  cann-app:latest

# 方法2：使用--privileged（不推荐）
docker run -it \
  --privileged \
  -v /usr/local/Ascend:/usr/local/Ascend \
  cann-app:latest

3.2 多NPU配置

#!/bin/bash
# multi_npu_run.sh

# 可用的NPU数量
NUM_NPUS=$(npu-smi info | grep "NPU" | wc -l)

echo "Available NPUs: ${NUM_NPUS}"

# 为每个NPU启动容器
for i in $(seq 0 $((${NUM_NPUS} - 1))); do
  echo "Starting container for NPU ${i}..."

  docker run -d \
    --name cann-app-npu-${i} \
    --device=/dev/davinci${i} \
    --device=/dev/davinci_manager \
    --device=/dev/devmm_svm \
    --device=/dev/hisi_hdc \
    -v /usr/local/Ascend:/usr/local/Ascend \
    -e ASCEND_VISIBLE_DEVICES=${i} \
    -p 808${i}:8080 \
    cann-app:latest
done

echo "Started containers for all NPUs"
docker ps | grep cann-app

3.3 NPU资源限制

#!/bin/bash
# 限制NPU使用

# 使用cgroup限制NPU计算资源
docker run -it \
  --device=/dev/davinci0 \
  --device-read-bps=/dev/davinci0:1000000000 \
  --device-write-bps=/dev/davinci0:1000000000 \
  -v /usr/local/Ascend:/usr/local/Ascend \
  cann-app:latest

四、Kubernetes部署

4.1 NPU设备插件

# npu-device-plugin.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ascend-npu-device-plugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: ascend-npu-device-plugin
  template:
    metadata:
      labels:
        name: ascend-npu-device-plugin
    spec:
      hostNetwork: true
      containers:
      - name: ascend-npu-device-plugin
        image: ascend-npu/k8s-device-plugin:latest
        command: ["./npu-device-plugin"]
        args:
        - "--v=5"
        volumeMounts:
        - name: device-dir
          mountPath: /dev
        - name: sys-dir
          mountPath: /sys
        resources:
          requests:
            memory: "100Mi"
            cpu: "100m"
          limits:
            memory: "500Mi"
            cpu: "500m"
      volumes:
      - name: device-dir
        hostPath:
          path: /dev
      - name: sys-dir
        hostPath:
          path: /sys

4.2 应用部署配置

# cann-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cann-app
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: cann-app
  template:
    metadata:
      labels:
        app: cann-app
    version: v1.0
    spec:
      containers:
      - name: cann-app
        image: cann-app:v1.0
        ports:
        - containerPort: 8080
          name: http
        resources:
          requests:
            ascend.com/npu: 1
            memory: "4Gi"
            cpu: "2"
          limits:
            ascend.com/npu: 1
            memory: "8Gi"
            cpu: "4"
        env:
        - name: ASCEND_VISIBLE_DEVICES
          value: "0,1,2,3"
        volumeMounts:
        - name: cann-software
          mountPath: /usr/local/Ascend
          readOnly: true
        - name: model-storage
          mountPath: /app/models
        - name: data-storage
          mountPath: /app/data
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
      volumes:
      - name: cann-software
        hostPath:
          path: /usr/local/Ascend
      - name: model-storage
        persistentVolumeClaim:
          claimName: model-pvc
      - name: data-storage
        emptyDir: {}
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: ascend.com/npu.count
                operator: Gt
                values:
                - "0"

4.3 Service配置

# cann-app-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: cann-app
  namespace: default
  labels:
    app: cann-app
spec:
  type: ClusterIP
  ports:
  - port: 8080
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: cann-app

---
apiVersion: v1
kind: Service
metadata:
  name: cann-app-lb
  namespace: default
spec:
  type: LoadBalancer
  ports:
  - port: 8080
    targetPort: 8080
    protocol: TCP
  selector:
    app: cann-app
  sessionAffinity: ClientIP

五、Helm Chart

5.1 Chart结构

cann-app/
├── Chart.yaml
├── values.yaml
├── templates/
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── configmap.yaml
│   ├── secret.yaml
│   └── ingress.yaml
└── README.md

5.2 Chart.yaml

apiVersion: v2
name: cann-app
description: A Helm chart for CANN application
version: 1.0.0
appVersion: "1.0"
keywords:
  - cann
  - ai
  - inference
maintainers:
  - name: CANN Team
    email: support@example.com

5.3 values.yaml

replicaCount: 3

image:
  repository: cann-app
  tag: v1.0
  pullPolicy: IfNotPresent

npu:
  count: 1
  devices: "0,1,2,3"

service:
  type: ClusterIP
  port: 8080

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
  hosts:
    - host: cann-app.example.com
      paths:
      - path: /
        pathType: Prefix

resources:
  requests:
    ascend.com/npu: 1
    memory: 4Gi
    cpu: 2
  limits:
    ascend.com/npu: 1
    memory: 8Gi
    cpu: 4

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80
  targetMemoryUtilizationPercentage: 80

config:
  logLevel: info
  batchSize: 32
  maxConcurrency: 100

5.4 deployment模板

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "cann-app.fullname" . }}
  labels:
    {{- include "cann-app.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "cann-app.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "cann-app.selectorLabels" . | nindent 8 }}
    spec:
      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        imagePullPolicy: {{ .Values.image.pullPolicy }}
        ports:
        - containerPort: {{ .Values.service.port }}
        env:
        - name: ASCEND_VISIBLE_DEVICES
          value: {{ .Values.npu.devices | quote }}
        envFrom:
        - configMapRef:
            name: {{ include "cann-app.fullname" . }}-config
        resources:
          {{- toYaml .Values.resources | nindent 10 }}
        volumeMounts:
        - name: cann-software
          mountPath: /usr/local/Ascend
      volumes:
      - name: cann-software
        hostPath:
          path: /usr/local/Ascend
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: ascend.com/npu.count
                operator: Gt
                values:
                - {{ .Values.npu.count | quote }}

六、监控与日志

6.1 Prometheus监控

# prometheus-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s

    scrape_configs:
    - job_name: 'cann-app'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: cann-app
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        target_label: __address__
        replacement: $1:8080
      metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'npu_.*'
        action: keep

6.2 自定义监控指标

# app/metrics.py
from prometheus_client import Counter, Histogram, Gauge, start_http_server

# 定义指标
inference_requests = Counter(
    'inference_requests_total',
    'Total inference requests',
    ['model', 'status']
)

inference_duration = Histogram(
    'inference_duration_seconds',
    'Inference duration',
    ['model']
)

npu_utilization = Gauge(
    'npu_utilization_percent',
    'NPU utilization',
    ['device_id']
)

def setup_metrics(port=8000):
    """启动指标服务"""
    start_http_server(port)

def record_inference(model_name, duration, status):
    """记录推理指标"""
    inference_requests.labels(model=model_name, status=status).inc()
    inference_duration.labels(model=model_name).observe(duration)

def update_npu_utilization():
    """更新NPU利用率"""
    import subprocess
    import re

    try:
        result = subprocess.run(
            ['npu-smi', 'info'],
            capture_output=True,
            text=True
        )

        # 解析NPU利用率
        for line in result.stdout.split('\n'):
            if 'NPU' in line and '%' in line:
                match = re.search(r'NPU\s+(\d+).*?(\d+)%', line)
                if match:
                    device_id = match.group(1)
                    utilization = match.group(2)
                    npu_utilization.labels(device_id=device_id).set(utilization)

    except Exception as e:
        print(f"Failed to update NPU utilization: {e}")

6.3 日志收集

# fluentd-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    # CANN应用日志收集
    <source>
      @type tail
      @id cann_app_log
      path /var/log/containers/*cann-app*.log
      pos_file /var/log/fluentd-cann-app.log.pos
      tag cann.*
      <parse>
        @type json
      </parse>
    </source>

    # 过滤和解析
    <filter cann.**>
      @type parser
      key_name log
      <parse>
        @type grok
        expression %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}
      </parse>
    </filter>

    # 输出到Elasticsearch
    <match cann.**>
      @type elasticsearch
      host elasticsearch
      port 9200
      logstash_format true
      logstash_prefix cann-app
    </match>

七、持续集成/持续部署

7.1 CI/CD流水线

# .gitlab-ci.yml
stages:
  - build
  - test
  - deploy

variables:
  IMAGE_NAME: cann-app
  REGISTRY: registry.example.com

build:
  stage: build
  script:
    - docker build -t ${REGISTRY}/${IMAGE_NAME}:${CI_COMMIT_SHA} .
    - docker push ${REGISTRY}/${IMAGE_NAME}:${CI_COMMIT_SHA}
  tags:
    - docker

test:
  stage: test
  script:
    - docker run --rm --device=/dev/davinci0 ${REGISTRY}/${IMAGE_NAME}:${CI_COMMIT_SHA} pytest tests/
  tags:
    - npu

deploy:dev:
  stage: deploy
  script:
    - helm upgrade --install cann-app ./helm-chart --set image.tag=${CI_COMMIT_SHA} --namespace dev
  environment:
    name: development
  only:
    - develop

deploy:prod:
  stage: deploy
  script:
    - helm upgrade --install cann-app ./helm-chart --set image.tag=${CI_COMMIT_SHA} --namespace production
  environment:
    name: production
  when: manual
  only:
    - main

7.2 自动化测试

# tests/test_cann_app.py
import pytest
import requests
import numpy as np

class TestCANNApp:
    """CANN应用测试"""

    @pytest.fixture
    def client(self):
        """测试客户端"""
        base_url = "http://cann-app:8080"
        return base_url

    def test_health_check(self, client):
        """健康检查"""
        response = requests.get(f"{client}/health")
        assert response.status_code == 200
        assert response.json()["status"] == "healthy"

    def test_inference(self, client):
        """推理测试"""
        # 准备测试数据
        test_data = {
            "image": np.random.rand(224, 224, 3).tolist()
        }

        # 发送推理请求
        response = requests.post(
            f"{client}/predict",
            json=test_data
        )

        assert response.status_code == 200
        result = response.json()
        assert "prediction" in result
        assert "confidence" in result

    def test_performance(self, client):
        """性能测试"""
        import time

        # 准备测试数据
        test_data = {
            "image": np.random.rand(224, 224, 3).tolist()
        }

        # 测量推理时间
        start_time = time.time()
        response = requests.post(
            f"{client}/predict",
            json=test_data
        )
        latency = time.time() - start_time

        assert response.status_code == 200
        assert latency < 0.1  # 延迟小于100ms

八、最佳实践

8.1 镜像优化

# 多阶段构建优化
FROM python:3.8-slim as builder

WORKDIR /app
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

FROM cann-base:latest

COPY --from=builder /root/.local /root/.local
COPY . /app

ENV PATH=/root/.local/bin:$PATH

WORKDIR /app
CMD ["python", "app.py"]

8.2 安全配置

# security-context.yaml
apiVersion: v1
kind: Pod
metadata:
  name: cann-app
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: cann-app
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: true

8.3 资源配额

# resource-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: cann-app-quota
  namespace: default
spec:
  hard:
    requests.ascend.com/npu: "16"
    requests.cpu: "32"
    requests.memory: 64Gi
    limits.ascend.com/npu: "16"
    limits.cpu: "64"
    limits.memory: 128Gi
    persistentvolumeclaims: "10"

九、总结

CANN容器化部署结合云原生技术，为AI应用提供了灵活、可扩展的部署方案。通过合理的容器编排和资源配置，可以构建高效的AI推理服务。

关键点：

• NPU设备直通配置
• Kubernetes资源管理
• 监控日志体系
• CI/CD自动化流程

参考资料

• CANN容器化部署指南
• CANN community仓库
• Kubernetes NPU设备插件