Docker、K8s、Argo常用命令学习

愿你想念我

260人浏览 · 2025-12-26 15:02:06

愿你想念我 · 2025-12-26 15:02:06 发布

Docker

当前程序目录如下所示。

code/
└── DataMining
    ├── data
    │   ├── a_steam_data_2021_2025.csv
    │   ├── Open_Data_BR_Site_Analytics_-_User_Access_by_Month_20251225.csv
    │   ├── paid_video_streaming_services.csv
    │   ├── payment-history.csv
    │   └── user_demographics.csv
    ├── data_aggregation.py
    ├── data_analysis.py
    ├── data_clean.py
    └── data_extraction.py

当前dockerfile所在目录如下所示。

dockerfiles/
└── data_extraction_dickerfile

当前镜像构建的dockerfile如下所示。

FROM python:3.9

WORKDIR /code

# pip换源
RUN pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

# 升级pip
RUN python3 -m pip install --upgrade pip

# 安装库
RUN pip install pandas

COPY DataMining .

ENTRYPOINT ["python", "data_extraction.py"]  # 固定运行程序

CMD ["--help"]

docker创建命令则写成如下形式。

sudo docker build -f dockerfiles/data_extraction_dickerfile -t data_extraction ./code

指定要使用的dockerfile，-f dockerfiles/data_extraction_dickerfile。指定镜像名称-t data_extraction。构建镜像时，参考的目录./code，即COPY DataMining .命令执行时，从当前系统的那个目录进行。

进入镜像内部，查看构建的信息是否正确，程序是否可以运行

sudo docker run -it --rm --entrypoint /bin/bash data_extraction

直接启动镜像

sudo docker run --rm data_extraction --file_name user_demographics --extracting_number 150

这两个参数是我的python程序需要的，--file_name user_demographics --extracting_number 150。

查看docker系统信息

sudo docker system df

查看docker镜像

sudo docker images

清除docker系统中无用的东西

sudo docker system prune

保存镜像为tar文件

data-analysis data-aggregation data-clean data-extraction为已构建的镜像。

sudo docker save -o images.tar data-analysis data-aggregation data-clean data-extraction

从tar文件读取镜像。

sudo docker load -i images.tar

K8s

创建PV

我将数据存放在了i-etesdrub节点的/home/ubuntu/myPV_Dire/data目录中。下面这种PV声明方式，为相对路径，因为使用的为hostPath。如果Pod挂载的PVC使用该PV，则Pod内部的程序，可以访问Pod所部署节点的/home/ubuntu/myPV_Dire/data目录。如果不限制nodeAffinity，则Pod被挂载在其它节点时（如node1），PV声明中hostPath所指的目录/home/ubuntu/myPV_Dire/data，是其它节点下（如node1）的/home/ubuntu/myPV_Dire/data目录，而不是i-etesdrub节点的/home/ubuntu/myPV_Dire/data目录，会访问不到我的数据。

① PV使用相对存储路径

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: 25Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: standard
  hostPath:
    path: /home/ubuntu/myPV_Dire/data
    type: DirectoryOrCreate  # 添加这行，自动创建目录
  nodeAffinity:  # 节点亲和性 Pod挂载的PVC，如果使用该PV，则需要部署在那个节点上
    required:  # required = 必须满足（强制要求）
      nodeSelectorTerms:  # 节点选择器条件
      - matchExpressions:  # 匹配表达式
        - key: kubernetes.io/hostname  # 标签键 = 节点主机名
          operator: In  # 操作符 = 在列表中
          values:  # 值列表
          - i-etesdrub  # 节点名

② PV使用NFS服务器

# nfs-static-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-nfs-pv-for-big-data
  labels:
    type: nfs
spec:
  capacity:
    storage: 100Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  # 不指定 storageClassName，表示手动管理的 PV
  storageClassName: ""  # 空字符串表示不使用 StorageClass
  mountOptions:
    - hard
    - nfsvers=4.1
    - noresvport
  nfs:
    # NFS 服务器 IP
    server: 192.168.100.20
    # NFS 共享路径
    path: "/home/ubuntu/data/shared/myPV_Dire/data"

创建PVC

① PV使用相对存储路径

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bigdata
spec:
  accessModes:
    - ReadWriteOnce  # 只能被一个节点读写
  resources:
    requests:
      storage: 10Gi    # 最少需求10GB
  storageClassName: standard
  volumeName: my-pv  # 直接指定 PV 名称

② PV使用NFS时

# nfs-static-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bigdata
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi  # 最少需求10GB
  storageClassName: ""  # 必须为空字符串与NSF的PV保持一致
  # 添加 selector 确保绑定到特定 PV
  volumeName: my-nfs-pv-for-big-data

创建Job时，给Pod中的容器，挂载上PVC。

# data-extraction-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: data-extraction-job
  labels:
    app: data-extraction
    job-type: data-processing
spec:  # Job的配置
  backoffLimit: 3
  ttlSecondsAfterFinished: 3600  # Job完成后1小时自动删除
  template:
    spec:  # pod的配置
      restartPolicy: Never  # pod重启策略

      volumes:  # 添加PVC
      - name: data-volume
        persistentVolumeClaim:
          claimName: bigdata
      
      containers:  # 添加容器
      - name: data-extraction
        image: data-extraction:latest
        imagePullPolicy: Never
        args:
        - "--file_name"
        - "users_1000w"
        - "--mid_dir"
        - "test"
        - "--process_num"
        - "55555"
        # - "--sleep"
        # - "600"
        
        volumeMounts:  # 为容器挂载PVC
        - name: data-volume
          mountPath: /code/sharedata  # 挂载在容器内的目录
           
        resources:  # 资源限制
          requests:
            memory: "128Mi"    # 最小需要256MB内存
            cpu: "500m"        # 最小需要0.05个CPU核心
          limits:
            memory: "1Gi"      # 最多使用1GB内存
            cpu: "500m"

# data-clean-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: data-clean-job
  labels:
    app: data-clean
    job-type: data-processing
spec:  # Job的配置
  backoffLimit: 3
  ttlSecondsAfterFinished: 3600  # Job完成后1小时自动删除
  template:
    spec:  # pod的配置
      restartPolicy: Never  # pod重启策略

      volumes:  # 添加PVC
      - name: data-volume
        persistentVolumeClaim:
          claimName: bigdata
      
      containers:  # 添加容器
      - name: data-clean
        image: data-clean:latest
        imagePullPolicy: Never
        args:
        - "--mid_dir"
        - "test"
        # - "--sleep"
        # - "600"
        
        volumeMounts:  # 为容器挂载PVC
        - name: data-volume
          mountPath: /code/sharedata  # 挂载在容器内的目录
           
        resources:  # 资源限制
          requests:
            memory: "128Mi"    # 最小需要256MB内存
            cpu: "500m"        # 最小需要0.05个CPU核心
          limits:
            memory: "1Gi"      # 最多使用1GB内存
            cpu: "500m"

# data-aggregation-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: data-aggregation-job
  labels:
    app: data-aggregation
    job-type: data-processing
spec:  # Job的配置
  backoffLimit: 3
  ttlSecondsAfterFinished: 3600  # Job完成后1小时自动删除
  template:
    spec:  # pod的配置
      restartPolicy: Never  # pod重启策略

      volumes:  # 添加PVC
      - name: data-volume
        persistentVolumeClaim:
          claimName: bigdata
      
      containers:  # 添加容器
      - name: data-aggregation
        image: data-aggregation:latest
        imagePullPolicy: Never
        args:
        - "--dim"
        - "['gender', 'country', 'device']"
        - "--mid_dir"
        - "test"
        # - "--sleep"
        # - "600"
        
        volumeMounts:  # 为容器挂载PVC
        - name: data-volume
          mountPath: /code/sharedata  # 挂载在容器内的目录
           
        resources:  # 资源限制
          requests:
            memory: "128Mi"    # 最小需要256MB内存
            cpu: "500m"        # 最小需要0.05个CPU核心
          limits:
            memory: "1Gi"      # 最多使用1GB内存
            cpu: "500m"

# data-analysis-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: data-analysis-job
  labels:
    app: data-analysis
    job-type: data-processing
spec:  # Job的配置
  backoffLimit: 3
  ttlSecondsAfterFinished: 3600  # Job完成后1小时自动删除
  template:
    spec:  # pod的配置
      restartPolicy: Never  # pod重启策略

      volumes:  # 添加PVC
      - name: data-volume
        persistentVolumeClaim:
          claimName: bigdata
      
      containers:  # 添加容器
      - name: data-analysis
        image: data-analysis:latest
        imagePullPolicy: Never
        args:
        - "--mid_dir"
        - "test"
        # - "--sleep"
        # - "600"
        
        volumeMounts:  # 为容器挂载PVC
        - name: data-volume
          mountPath: /code/sharedata  # 挂载在容器内的目录
           
        resources:  # 资源限制
          requests:
            memory: "128Mi"    # 最小需要256MB内存
            cpu: "500m"        # 最小需要0.05个CPU核心
          limits:
            memory: "1Gi"      # 最多使用1GB内存
            cpu: "500m"

pod运行时，进入pod内部，查看内部是否正确。data-extraction-job-c2856为pod名。

sudo kubectl exec -it data-extraction-job-c2856 -- /bin/bash

查看pod中程序的输出，data-extraction-job-fv227为pod名。

sudo kubectl logs data-extraction-job-fv227

Argo

提交任务

sudo argo submit workflow_yamls/workflow_bigdata.yaml

查看任务列表

sudo argo list

查看任务data-processing-kb22p的详细信息

sudo argo get data-processing-kb22p

删除所有工作流任务

sudo argo delete --all

创建一个工作流
首先构建yaml文件

# 无传参数
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: data-processing-
  labels:
    app: data-processing
    workflow-type: bigdata-processing
spec:
  entrypoint: data-pipeline  # 程序入口
  
  # 工作流完成后1小时自动删除
  ttlStrategy:
    secondsAfterCompletion: 3600
  
  volumes:
  - name: data-volume
    persistentVolumeClaim:
      claimName: bigdata
  
  templates:
  - name: data-pipeline
    dag:
      tasks:
      - name: extraction
        template: data-extraction
        
      - name: clean
        template: data-clean
        depends: extraction
        
      - name: aggregation
        template: data-aggregation
        depends: clean
        
      - name: analysis
        template: data-analysis
        depends: aggregation
  
  - name: data-extraction
    retryStrategy:  # 重试策略
      limit: 3
      retryPolicy: "Always"  # 总是重试
    activeDeadlineSeconds: 1800  # 30分钟超时则退出
    container:
      image: data-extraction:latest
      imagePullPolicy: IfNotPresent
      command: ["python", "data_extraction.py"]
      args:
      - "--file_name"
      - "users_1000w"
      - "--mid_dir"
      - "test"
      - "--process_num"
      - "55555"
      volumeMounts:
      - name: data-volume
        mountPath: /code/sharedata
      resources:
        requests:
          memory: "128Mi"
          cpu: "500m"
        limits:
          memory: "1Gi"
          cpu: "500m"
  
  - name: data-clean
    retryStrategy:
      limit: 3
    activeDeadlineSeconds: 1800
    container:
      image: data-clean:latest
      imagePullPolicy: IfNotPresent
      command: ["python", "data_clean.py"]
      args:
      - "--mid_dir"
      - "test"
      volumeMounts:
      - name: data-volume
        mountPath: /code/sharedata
      resources:
        requests:
          memory: "128Mi"
          cpu: "500m"
        limits:
          memory: "2Gi"
          cpu: "500m"
  
  - name: data-aggregation
    retryStrategy:
      limit: 3
    activeDeadlineSeconds: 1800
    container:
      image: data-aggregation:latest
      imagePullPolicy: IfNotPresent
      command: ["python", "data_aggregation.py"]
      args:
      - "--dim"
      - "['gender', 'country', 'device']"
      - "--mid_dir"
      - "test"
      volumeMounts:
      - name: data-volume
        mountPath: /code/sharedata
      resources:
        requests:
          memory: "128Mi"
          cpu: "500m"
        limits:
          memory: "1Gi"
          cpu: "500m"
  
  - name: data-analysis
    retryStrategy:
      limit: 3
    activeDeadlineSeconds: 1800
    container:
      image: data-analysis:latest
      imagePullPolicy: IfNotPresent
      command: ["python", "data_analysis.py"]
      args:
      - "--mid_dir"
      - "test"
      volumeMounts:
      - name: data-volume
        mountPath: /code/sharedata
      resources:
        requests:
          memory: "128Mi"
          cpu: "500m"
        limits:
          memory: "1Gi"
          cpu: "500m"

# 有传参数
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: data-processing-
  labels:
    app: data-processing
    workflow-type: bigdata-processing
spec:
  # 添加工作流级别参数
  arguments:
    parameters:
    - name: file_name
    - name: process_num
    - name: dim
    - name: cpu_limit
    - name: mid_dir
  
  entrypoint: data-pipeline  # 程序入口
  
  ttlStrategy:
    secondsAfterCompletion: 3600  # 完成一小时后，清理该工作流
  
  volumes:
  - name: data-volume
    persistentVolumeClaim:
      claimName: bigdata
  
  templates:
  - name: data-pipeline
    inputs:
      parameters:
      - name: file_name
      - name: process_num
      - name: dim
      - name: cpu_limit
      - name: mid_dir
    dag:
      tasks:
      - name: extraction
        template: data-extraction
        arguments:
          parameters:
          - name: file_name
            value: "{{inputs.parameters.file_name}}"
          - name: process_num
            value: "{{inputs.parameters.process_num}}"
          - name: cpu_limit
            value: "{{inputs.parameters.cpu_limit}}"
          - name: mid_dir
            value: "{{inputs.parameters.mid_dir}}"
        
      - name: clean
        template: data-clean
        depends: extraction
        arguments:
          parameters:
          - name: cpu_limit
            value: "{{inputs.parameters.cpu_limit}}"
          - name: mid_dir
            value: "{{inputs.parameters.mid_dir}}"
        
      - name: aggregation
        template: data-aggregation
        depends: clean
        arguments:
          parameters:
          - name: dim
            value: "{{inputs.parameters.dim}}"
          - name: cpu_limit
            value: "{{inputs.parameters.cpu_limit}}"
          - name: mid_dir
            value: "{{inputs.parameters.mid_dir}}"
        
      - name: analysis
        template: data-analysis
        depends: aggregation
        arguments:
          parameters:
          - name: cpu_limit
            value: "{{inputs.parameters.cpu_limit}}"
          - name: mid_dir
            value: "{{inputs.parameters.mid_dir}}"
  
  - name: data-extraction
    inputs:
      parameters:
      - name: file_name
      - name: process_num
      - name: cpu_limit
      - name: mid_dir
    retryStrategy:
      limit: 3
    activeDeadlineSeconds: 18000  # 该任务点的最大运行时间，单位为秒
    container:
      image: data-extraction:latest
      imagePullPolicy: IfNotPresent
      command: ["python", "data_extraction.py"]
      args:
      - "--file_name"
      - "{{inputs.parameters.file_name}}"
      - "--process_num"
      - "{{inputs.parameters.process_num}}"
      - "--mid_dir"
      - "{{inputs.parameters.mid_dir}}"
      volumeMounts:
      - name: data-volume
        mountPath: /code/sharedata
    podSpecPatch: |
      containers:
      - name: main
        resources:
          requests:
            memory: "5Gi"
            cpu: "{{inputs.parameters.cpu_limit}}"
          limits:
            memory: "5Gi"
            cpu: "{{inputs.parameters.cpu_limit}}"
  
  - name: data-clean
    inputs:
      parameters:
      - name: cpu_limit
      - name: mid_dir
    retryStrategy:
      limit: 3
    activeDeadlineSeconds: 18000
    container:
      image: data-clean:latest
      imagePullPolicy: IfNotPresent
      command: ["python", "data_clean.py"]
      args:
      - "--mid_dir"
      - "{{inputs.parameters.mid_dir}}"
      volumeMounts:
      - name: data-volume
        mountPath: /code/sharedata
    podSpecPatch: |
      containers:
      - name: main
        resources:
          requests:
            memory: "6Gi"
            cpu: "{{inputs.parameters.cpu_limit}}"
          limits:
            memory: "6Gi"
            cpu: "{{inputs.parameters.cpu_limit}}"
  
  - name: data-aggregation
    inputs:
      parameters:
      - name: dim
      - name: cpu_limit
      - name: mid_dir
    retryStrategy:
      limit: 3
    activeDeadlineSeconds: 18000
    container:
      image: data-aggregation:latest
      imagePullPolicy: IfNotPresent
      command: ["python", "data_aggregation.py"]
      args:
      - "--dim"
      - "{{inputs.parameters.dim}}"
      - "--mid_dir"
      - "{{inputs.parameters.mid_dir}}"
      volumeMounts:
      - name: data-volume
        mountPath: /code/sharedata
    podSpecPatch: |
      containers:
      - name: main
        resources:
          requests:
            memory: "5Gi"
            cpu: "{{inputs.parameters.cpu_limit}}"
          limits:
            memory: "5Gi"
            cpu: "{{inputs.parameters.cpu_limit}}"
  
  - name: data-analysis
    inputs:
      parameters:
      - name: cpu_limit
      - name: mid_dir
    retryStrategy:
      limit: 3
    activeDeadlineSeconds: 18000
    container:
      image: data-analysis:latest
      imagePullPolicy: IfNotPresent
      command: ["python", "data_analysis.py"]
      args:
      - "--mid_dir"
      - "{{inputs.parameters.mid_dir}}"
      volumeMounts:
      - name: data-volume
        mountPath: /code/sharedata
    podSpecPatch: |
      containers:
      - name: main
        resources:
          requests:
            memory: "3Gi"
            cpu: "{{inputs.parameters.cpu_limit}}"
          limits:
            memory: "3Gi"
            cpu: "{{inputs.parameters.cpu_limit}}"

无参数版启动命令，workflow_bigdata.yaml为文件名。

sudo argo submit workflow_bigdata.yaml

有参数版启动命令，workflow_bigdata_parameter.yaml为文件名。

sudo argo submit workflow_bigdata_parameter.yaml -p file_name=users_1000w -p process_num=6000 -p dim="['gender', 'country', 'device']" -p cpu_limit=1000m -p mid_dir=dir3

亚马逊云科技技术品牌专区

更多推荐

基于STM32的全屋环境智能监测与预警终端设计

亚马逊云科技技术品牌专区

Keras：面向人类的深度学习库

Keras3支持多后端（TensorFlow、PyTorch、JAX），通过设置环境变量切换，兼容指定版本。pip安装后可用Sequential API快速构建CNN模型，如MNIST分类示例，实

亚马逊云科技技术品牌专区

AI行业日报｜2026-09-25：5个热点事件

本期行业动态聚焦于端侧算力重构、自主智能体工作流、开发工具经济模型、网络隔离架构设计与公共部门合规审查。以下结合一线技术场景进行结构化梳理，供 AI 研究员与软件工程团队参考。 PrismML 正将其轻量级语言模型适配至搭载高通处理器的智能眼镜终端 1 。该平台的核心战略指向开源权重模型的本地化运行，旨在最大化挖掘穿戴设备现有算力的利用效率 1 。对于嵌入式与边缘计算开发者而言，此类微型模型向低功