k8s jenkins主从设置

完颜振江

492人浏览 · 2023-05-22 11:16:38

完颜振江 · 2023-05-22 11:16:38 发布

系统管理--系统配置--节点管理--Configure Clouds--配置集群

Kubernetes

名称：kubernetes-prod
Kubernetes 地址：https://kubernetes.default.svc.cluster.local
Use Jenkins Proxy：不勾选
Kubernetes 服务证书 key：空
禁用 HTTPS 证书检查：不勾选
Kubernetes 命名空间：空
凭据：无
WebSocket：不勾选
Direct Connection：不勾选
Jenkins 地址：http://jenkins.dayi-devops.svc.cluster.local:8080/jenkins
Jenkins 通道：空
Connection Timeout：30
Read Timeout：60
容器数量：20
Pod Labels：
1. Pod Label：
  1. 键：jenkins
  2. 值：jnlp
2. 连接 Kubernetes API 的最大连接数：32
3. Seconds to wait for pod to be running：600
Pod Templates
1. Pod Template
  1. 名称：jnlp-slave
  2. 命名空间：prod
  3. 标签列表：jnlp-slave
  4. 用法：只允许运行绑定到这台机器的Job
  5. 父级的 Pod 模板名称：为空
容器列表
1. Container Template
  1. 名称：jnlp
  2. Docker 镜像：wanyan.cn-hangzhou.cr.aliyuncs.com/yyh-prod/jenkins-jnlp:v2
  3. 总是拉取镜像：不勾选
  4. 工作目录：/home/jenkins
  5. 运行的命令：jenkins-agent
  6. 命令参数：为空
  7. 分配伪终端：勾选
2. Environment Variables：（默认配置不操作）
3. 环境变量
  1. 卷（Host Path Volume）
    1. 主机路径：/var/run/docker.sock
    2. 挂载路径：/var/run/docker.sock
  2. Host Path Volume
    1. 主机路径：/usr/bin/docker
    2. 挂载路径：/usr/bin/docker
  3. Host Path Volume
    1. 主机路径：/etc/localtime
    2. 挂载路径：/etc/localtime
  4. Persistent Volume Claim
    1. 申明值：webapps-data
    2. 只读：不勾选
    3. 挂载路径：/data/webapps
  5. 注解
    1. Concurrency Limit：空
    2. Pod Retention：Default
    3. 代理的空闲存活时间（分）:空
    4. Pod 寿命（秒）：空
    5. 连接 Jenkins 的超时时间（秒）：1000
    6. Raw YAML for the Pod：空
    7. Yaml merge strategy：Override
    8. Show raw yaml in console：勾选
4. 拉取镜像的 Secret
  1. Image Pull Secret
    1. 名称：aliregistry-secret
    2. Service Account：jenkins-prod
    3. Run As User ID：0 (root用户启动)
    4. Run As Group ID：为空
    5. Supplemental Groups：为空
    6. Host Network：不勾选
    7. 节点选择器：为空
  2. 工作空间卷：Persistent Volume Claim Workspace Volume
    1. 声明值：jenkins-jnlp-local
    2. 只读：不勾选
  3. 节点属性
    1. 工具位置：不勾选

jenkins前端应用配置

参数化构建过程
1. Git参数
  1. 名称：Branch
  2. 描述：选择发布的分支
  3. 参数类型：分支或标签
  4. 默认值：master
2. 选项参数
  1. 名称：Namespace
  2. 选项：prod
  3. 描述：选择发布环境
3. 选项参数
  1. 名称：deploy_env
  2. 选项：deploy
  3. 描述：deploy发布新代码

Pipeline script

脚本

// 项目
// 需要修改前端项目部署的目录
def project_webdir = "channelcenter"
// 需要修改对应服务的git地址
def git_address = "http://gitlab.wanyan.com/web/channel-center-web.git"
// 认证
def git_auth = "gitlab-creds" //git login auth


pipeline {
    agent { label 'jnlp-slave' }

    parameters {
        gitParameter branch: '', branchFilter: '.*', defaultValue: 'master', description: '选择发布的分支', name: 'Branch', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH_TAG' //PT_BRANCH_TAG获取分支和TAG
        //gitParameter branch: '', branchFilter: '.*', defaultValue: 'master', description: '选择发布的分支', name: 'Branch', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH' //PT_BRANCH只获取分支
        choice (choices: ['prod'], description: '选择发布环境', name: 'Namespace')
        choice choices: ['deploy'], description: '''deploy发布新代码''', name: 'deploy_env'
    }
    
    stages {
        stage('拉取代码'){
            steps {
                //build quietPeriod: 3, job: 'yyh_devops'
                checkout([$class: 'GitSCM', 
                branches: [[name: "${params.Branch}"]], 
                doGenerateSubmoduleConfigurations: false, 
                extensions: [], submoduleCfg: [], 
                userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]
                ])
            }
        }

        stage('代码编译'){
           when { environment name: 'deploy_env', value: 'deploy' }
           steps {
             sh """
                cnpm install --unsafe-perm --registry=https://registry.npm.taobao.org
                cnpm run build:prod
                pwd
                """ 
           }
        }

        stage('部署') {
           when { environment name: 'deploy_env', value: 'deploy' }
           steps {
             sh """
                rsync -avz --delete dist/ /data/webapps/${project_webdir}
                pwd
                """ 
           } 
        }

    }
}

jenkins后端应用配置

参数化构建过程
1. Git参数
  1. 名称：Branch
  2. 描述：选择发布的分支
  3. 参数类型：分支或标签
  4. 默认值：master
2. 选项参数
  1. 名称：Namespace
  2. 选项：prod
  3. 描述：选择发布环境
3. 选项参数
  1. 名称：deploy_env
  2. 选项：deploy or rollback
  3. 描述：deploy发布新代码rollback回滚
4. 字符参数
  1. 名称：version
  2. 默认值：0
  3. 描述：选择回滚版本号
  4. 清除空白字符：不勾选

Pipeline script

pipeline {
    agent { label 'jnlp-slave' }

    environment {
        // 公共
        registry = "wanyan.cn-hangzhou.cr.aliyuncs.com"
        // 项目
        project = "yyh-prod"
        app_name = "${JOB_NAME}"
        workdir = "/home/dayiops/${JOB_NAME}"
        image_name = "${registry}/${project}/${app_name}:${BUILD_NUMBER}"
        // 需要修改对应服务的端口号
        app_port = "8890"
        // 需要修改对应服务的git地址
        git_address = "http://gitlab.wanyan.com/basic-service/auth-center.git"
        // 需要修改微服务对应的gitlab群组
        git_groups = "basic-service"
        // 回滚镜像的版本
        rollback_image_name = "${registry}/${project}/${app_name}:${version}"
        // 认证
        docker_registry_auth = "jenkins-aliregistry-creds"  //Harbor login auth
        git_auth = "gitlab-creds" //git login auth

    }

    parameters {
        gitParameter branch: '', branchFilter: '.*', defaultValue: 'master', description: '选择发布的分支', name: 'Branch', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH_TAG' //PT_BRANCH_TAG获取分支和TAG
        //gitParameter branch: '', branchFilter: '.*', defaultValue: 'master', description: '选择发布的分支', name: 'Branch', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH' //PT_BRANCH只获取分支
        //choice (choices: ['1', '3', '5', '7'], description: '副本数', name: 'ReplicaCount')
        choice (choices: ['prod'], description: '选择发布环境', name: 'Namespace')
        choice choices: ['deploy', 'rollback'], description: '''deploy发布新代码rollback回滚''', name: 'deploy_env'
        string defaultValue: '0', description: '选择回滚版本号', name: 'version', trim: false
    }
    
    stages {

            stage('拉取代码'){
                steps {
                    build quietPeriod: 3, job: 'yyh_devops'
                    checkout([$class: 'GitSCM', 
                    branches: [[name: "${params.Branch}"]], 
                    doGenerateSubmoduleConfigurations: false, 
                    extensions: [], submoduleCfg: [], 
                    userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]
                    ])
                }
            }
    
            stage('代码编译'){
                when { environment name: 'deploy_env', value: 'deploy' }
                steps {
                  sh """
                     mvn clean package -Dmaven.test.skip=true -U
                     """ 
                }
            }
    
            stage('构建镜像'){
                when { environment name: 'deploy_env', value: 'deploy' }
                steps {
                    dir("${WORKSPACE}/${JOB_NAME}") {
                     withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
                     sh """
                       echo '
                         FROM ${registry}/yyh/centos-jdk:8-jre
                         LABEL author='yhh'
                         
                         ENV PROJECT="${JOB_NAME}"
                         
                         #ENV JAVA_OPTS="-Dspring.profiles.active=production -server -Xms1024M -Xmx1024M "
                         ENV TZ=Asia/Shanghai
                         ENV LANG=en_US.UTF-8
     
                         RUN mkdir /home/dayiops/${JOB_NAME} -p
                         
                         WORKDIR /home/dayiops/${JOB_NAME}
                         
                         ADD target/${JOB_NAME}.jar ${workdir}/
                         
                         EXPOSE ${app_port}/tcp
                         
                         ENTRYPOINT ["java","-Dspring.profiles.active=k8s${Namespace}","-Dmaven.wagon.http.ssl.insecure=true","-Dmaven.wagon.http.ssl.allowall=true","-server","-Xms1024M","-Xmx1024M","-XX:+HeapDumpOnOutOfMemoryError","-jar","${JOB_NAME}.jar"]
                       ' > Dockerfile
                       docker login -u ${username} -p '${password}' ${registry}
                       docker build -t ${image_name} .
                       docker push ${image_name}
                     """
                     }
                    }
                } 
            }
    
            stage('部署到K8S平台'){
                when { environment name: 'deploy_env', value: 'deploy' }
                steps {
                  dir("$WORKSPACE/../yyh_devops/${git_groups}/${JOB_NAME}") {
                     sh """
                       sed -i 's#{APP_NAME}#${JOB_NAME}#g' k8s-deployment.yaml
                       sed -i 's#{APP_PORT}#${app_port}#g' k8s-deployment.yaml
                       sed -i 's#{IMAGE_NAME}#${image_name}#' k8s-deployment.yaml
                       sed -i 's#{NAME_SPACE}#${Namespace}#' k8s-deployment.yaml
                       sed -i 's#{ADD_ENV_LABEL}#${Namespace}#' k8s-deployment.yaml
                       kubectl apply -f k8s-deployment.yaml
                     """
                     //kubernetesDeploy configs: 'k8s-deployment.yaml'
                   }
                }
            }
  
            stage("服务启动检查"){
                when { environment name: 'deploy_env', value: 'deploy' }
                steps {
                    sleep 63
                    timeout(time: 31, unit: 'SECONDS') {
                        waitUntil {
                            script {
                                def podstatus = sh (
                                    returnStdout: true,
                                    //script: "kubectl get deployment -n test | grep auth-center-api | awk \'{print \$1}\'"
                                    script: "kubectl get replicasets -n ${Namespace} |grep ${JOB_NAME} | awk \'{if (\$2 >=1 && \$4 == 0) print \"podnotready\"}\'"
                                )
                                def notrun_podname = sh (
                                    returnStdout: true,
                                    //script: "kubectl get deployment -n test | grep ${JOB_NAME} | awk \'{print \$1}\'"
                                    script: "kubectl get pod -n ${Namespace} |grep ${JOB_NAME} |awk \'{if (\$2 == \"0/1\") print \$1}\'"
                                )
                                podstatus = podstatus.trim()
                                notrun_podname = notrun_podname.trim()
                                echo   "********  ${JOB_NAME}服务启动状态为  ${podstatus} ********" 
                                if( podstatus == "podnotready" ) {
                                    //echo "${JOB_NAME} 服务启动失败 重新检测服务运行态中..."  
                                    echo "服务启动状态检查中..."
                                    sleep 10
                                    return false
                                } else {
                                    echo "********  ${JOB_NAME} 服务启动成功...  ********"
                                    return true
                                }
                            }
                       }
                    }
                }
            }
    
            stage('回滚指定的镜像'){
                when { environment name: 'deploy_env', value: 'rollback' }
                steps {
                    dir("$WORKSPACE/../yyh_devops/${git_groups}/${JOB_NAME}") {
                      sh """
                      sed -i 's#{APP_NAME}#${JOB_NAME}#g' k8s-deployment.yaml
                      sed -i 's#{APP_PORT}#${app_port}#g' k8s-deployment.yaml
                      sed -i 's#{IMAGE_NAME}#${rollback_image_name}#' k8s-deployment.yaml
                      sed -i 's#{NAME_SPACE}#${Namespace}#' k8s-deployment.yaml
                      sed -i 's#{ADD_ENV_LABEL}#${Namespace}#' k8s-deployment.yaml
                      kubectl apply -f k8s-deployment.yaml
                      """
                      //kubernetesDeploy configs: 'k8s-deployment.yaml'
                    }
                }
            } 
    }
}

后端应用Dockerfile模板

# cat Dockerfile

                     FROM dayi-registry.cn-hangzhou.cr.aliyuncs.com/yyh/centos-jdk:8-jre
                     LABEL author=yhh

                     ENV PROJECT="gateway"

                     #ENV JAVA_OPTS="-Dspring.profiles.active=production -server -Xms1024M -Xmx1024M "
                     ENV TZ=Asia/Shanghai

                     RUN mkdir /home/dayiops/gateway -p

                     WORKDIR /home/dayiops/gateway

                     ADD target/gateway.jar /home/dayiops/gateway/

                     EXPOSE 8081/tcp

                     ENTRYPOINT ["java","-Dspring.profiles.active=k8sprod","-server","-Xms2048M","-Xmx2048M","-XX:+HeapDumpOnOutOfMemoryError","-jar","gateway.jar"]

gitlab k8s模板

#kubernetes/yyh-devops/BC/connector-api/k8s-deployment.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: {APP_NAME}
  namespace: {NAME_SPACE}
  labels:
    app: {APP_NAME}
    env: {ADD_ENV_LABEL}
spec:
  ports:
  - name: http
    port: {APP_PORT}
    protocol: TCP
    targetPort: {APP_PORT}
  selector:
    app: {APP_NAME}
    env: {ADD_ENV_LABEL}
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {APP_NAME}
  namespace: {NAME_SPACE}
  labels:
    app: {APP_NAME}
    env: {ADD_ENV_LABEL}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {APP_NAME}
      env: {ADD_ENV_LABEL}
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: {APP_NAME}
        env: {ADD_ENV_LABEL}
    spec:
      imagePullSecrets:
      - name: aliregistry-secret
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - {APP_NAME}
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - env:
        - name: TZ
          value: Asia/Shanghai
        - name: LANG
          value: en_US.UTF-8
        image: {IMAGE_NAME}
        imagePullPolicy: IfNotPresent
        name: {APP_NAME}
        ports:
        - name: http
          containerPort: {APP_PORT}
          protocol: TCP
        readinessProbe:
          failureThreshold: 2
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: {APP_PORT}
          timeoutSeconds: 2
        livenessProbe:
          failureThreshold: 2
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: {APP_PORT}
          timeoutSeconds: 2          
        resources:
          limits:
            cpu: 1000m
            memory: 1024Mi
          requests:
            cpu: 200m
            memory: 256Mi
        volumeMounts:
        - mountPath: /data/logs
          name: logs
        - mountPath: /etc/localtime
          name: localtime
          readOnly: true
      dnsPolicy: ClusterFirstWithHostNet
      restartPolicy: Always
      securityContext:
        fsGroup: 2049
        runAsGroup: 2049
        runAsUser: 2049
      volumes:
      - emptyDir: {}
        name: logs
      - hostPath:
          path: /etc/localtime
          type: File
        name: localtime

依赖任务配置（每次后端发布都会执行这个任务）

名称：yyh_devops

Pipeline yyh_devops

Pipeline script

node('jnlp-slave') {
    stage('Git Clone') {
        git credentialsId: 'gitlab-creds', url: 'http://gitlab.taeteadata.com/kubernetes/yyh-devops-prod.git'
    }
}

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub