因为还没有完成控制台开发，我们k8s应用的发布目前只能直接在jenkins中实现了。

1. 基础镜像

按层次制作基础镜像

centos -> centos-sshd -> centos74-jre[678] -> centos74-jre[678]-[tomcat6|tomcat7|tomcat8|springboot]

发布应用时，根据jre\middleware参数，选择基础镜像

2. 资源申请参数说明

必填

project

项目的名称，纯数字前要加'p'

app

应用名称，以项目名称作为前缀，app在k8s集群是唯一的，作为应用的标识

domain

域名，无填'null'

sessionSticky

会话保持，类似nginx的ip_hash，'true'或’false’

replicate

副本数，即节点数

javaVersion

jre版本,支持jre6\jre7\jre8

middleware

中间件，支持tomcat6\tomcat7\tomcat8\springboot

rwFileList

Tomcat项目文件如需要写权限，多个用逗号分隔，无填’null’

codeRepo

git地址

warDir

war包相对路径，如’target’

mvnCommand

mvn打包命令

sbStartCommand

Springboot项目启动文件指定profile的命令

dingToken

如需要发dingding群通知，提供token，无则’null’

可选（运维有默认配置）

cpuReq

Cpu初始分配核数目

cpuLmt

Cpu分配核数目上限

memReq

内存初始分配大小（M）

memLmt

内存分配大小上限（M），一般和memReq相同

jvmSize

Jvm大小，默认’null’，公式根据memReq自动计算

directMemSize

对外内存最大大小，默认’null’，公式根据memReq自动计算

mvnVersion

Mvn版本

mvnJavaVersion

Mvn环境的jdk版本

imageVersion

基础镜像版本（研发不需了解）

3. jenkins pipeline

所有变量抽出至environment下，便于配置，也便于后续自研平台调用；

整个过程分为checkout、mvn、build、deploy、post action几个部分；

checkout：拉取代码

mvn：调用mvn命令打包

build：执行make_dockerfile.sh，生成dockerfile，调用docker命令build镜像、push镜像

deploy：执行make_deploymentfile.sh，make_ingressfile.sh，调用kubectl命令创建deployment、ingress（含service）

post action：发送钉钉通知完整jenkinsfile如下

pipeline {
    environment {
        project = 'myproj'
        app = 'myproc-myapp'
        domain = 'mydomain.ali.com'
        replicate = '4'
        sessionSticky = 'false'
        javaVersion = 'jre8'
        middleware = 'springboot'
        rwFileList = 'null'
        codeRepo = 'http://git.mydomain.com/usergroup/myapp.git'
        warDir = 'target'
        mvnCommand = 'clean install -DskipTests=true -Ptest'
        sbStartCommand = '-Dspring.profiles.active=test'
        dingToken = '8826e85c02c796aa5aea0a39d292e1111111111111111111111'
        
        cpuReq = '0.2'
        cpuLmt = '2'
        memReq = '1024'
        memLmt = '1024'
        jvmSize = 'null'
        directMemSize = 'null'
        mvnVersion = 'maven-3.6.0'
        mvnJavaVersion = 'jdk1.8.0_221'
        imageVersion = 'v1'

    }
    
    agent any
    
    options {
        timeout(time: 10, unit: 'MINUTES') 
    }
    
    stages {
        
        stage('checkout') {
            steps {
                script {
                    def codeRepoCred
                    def reg1 = /.+git\.mydomain\.com.+/
                    if ("${codeRepo}".matches(reg1)) {
                        codeRepoCred = 'mycred'
                    }
                    checkout([$class: 'GitSCM', branches: [[name: '$tag']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${codeRepoCred}", url: "${codeRepo}"]]])
                }
            }
        }
        stage('mvn') {
            environment {
                PATH = "/usr/local/java/${mvnJavaVersion}/bin:$PATH"
            }
            steps {
                sh "/usr/local/maven/${mvnVersion}/bin/mvn -f pom.xml ${mvnCommand}"
            }
        }
        stage('build') {
            steps {
                sh "/data1/template/make_dockerfile.sh ${WORKSPACE} ${javaVersion} ${middleware} ${cpuLmt} ${memLmt} ${rwFileList} ${imageVersion} ${sbStartCommand} ${jvmSize} ${directMemSize}"
                script {
                    tagnew = "${tag}".replaceAll("/","_");
                    sh "/usr/bin/docker build -f ${WORKSPACE}/deploy/dockerfile -t harbor.mydomain.com/$project/${app}:${tagnew} ${WORKSPACE}/${warDir}"
                    sh "/usr/bin/docker push harbor.mydomain.com/$project/${app}:${tagnew}"
                    sh "/usr/bin/docker rmi harbor.mydomain.com/$project/${app}:${tagnew}"
                }
            }
        }
        stage('deploy') {
            steps {
                script {
                    tagnew = "${tag}".replaceAll("/","_");
                    sh "/data1/template/make_deploymentfile.sh ${WORKSPACE} ${app} ${project} ${replicate} ${BUILD_NUMBER} ${cpuReq} ${cpuLmt} ${memReq} ${memLmt} ${tagnew}"
                    sh "/usr/bin/kubectl apply -f ${WORKSPACE}/deploy/deployment.yaml"
                    sh "/usr/bin/kubectl delete ingress ${app} > /dev/null 2>&1 || true"
                    sh "/usr/bin/kubectl delete service ${app} > /dev/null 2>&1 || true"
                    if ("${domain}" != "null") {
                        sh "/data1/template/make_ingressfile.sh ${WORKSPACE} ${app} ${domain} ${sessionSticky}"
                        sh "/usr/bin/kubectl apply -f ${WORKSPACE}/deploy/ingress.yaml"
                    }
                    sh "/usr/bin/kubectl rollout status deployment/${app}"
                }

            }
        }
    }
    post {
		success {
			dingTalk accessToken:"https://oapi.dingtalk.com/robot/send?access_token=${dingToken}", 
			imageUrl:'', 
			jenkinsUrl:'http://10.40.16.212/', 
			message:' deploy success', 
			notifyPeople:''
		}
		failure {
			dingTalk accessToken:'https://oapi.dingtalk.com/robot/send?access_token=${dingToken}', 
			imageUrl:'', 
			jenkinsUrl:'http://10.40.16.212/', 
			message:' deploy failed', 
			notifyPeople:''
		}
    }
}

4. shell与模板文件

jenkins服务器上的模板相关文件：*.template

make_*.sh传递pipeline中的变量，根据template文件生成各个项目特定的dockerfile、deployment.yaml、ingress.yaml

dockerfile.template 

将应用启动需要的参数，输出至用户.bash_profile中，生成环境变量；后续脚本直接调用环境变量

FROM harbor.mydomain.com/baseimage/imageTag
LABEL type="appimage"
COPY --chown=appuser:appuser *.war /apps/deploy/
RUN echo "export CPU_LIMIT=cpuLmtTag" >> /home/appuser/.bash_profile \
    && echo "export MEM_LIMIT=memLmtTag" >> /home/appuser/.bash_profile \
    && echo "export JVM_SIZE=jvmSizeTag" >> /home/appuser/.bash_profile \
    && echo "export DIRECT_MEM_SIZE=directMemSizeTag" >> /home/appuser/.bash_profile \
    && echo "export PERM_NAME=permNameTag" >> /home/appuser/.bash_profile \
    && echo "export FILE_LIST=rwFileListTag" >> /home/appuser/.bash_profile \
    && echo "export SB_START_COMMAND=sbStartCommandTag" >> /home/appuser/.bash_profile \
    && echo -e "LIBSYSCONFCPUS=\${CPU_LIMIT}" >> /home/appuser/.bash_profile \                
    && echo -e "if [ \${LIBSYSCONFCPUS} -lt 2 ]; then LIBSYSCONFCPUS=2; fi" >> /home/appuser/.bash_profile \
    && echo "export LIBSYSCONFCPUS" >> /home/appuser/.bash_profile \      
    && echo -e "export LD_PRELOAD=/usr/local/lib/libsysconfcpus.so:\${LD_PRELOAD}" >> /home/appuser/.bash_profile
CMD ["/apps/script/pod_start.sh"]

deployment.template

apiVersion: apps/v1
kind: Deployment
metadata:
  name: appTag
  labels:
    app: appTag
spec:
  replicas: replicateTag
  selector:
    matchLabels:
      app: appTag
  template:
    metadata:
      labels:
        app: appTag
        project: projectTag
        buildnum: "buildnumTag"
    spec:
      dnsPolicy: Default
      containers:
      - name: appTag
        image: harbor.mydomain.com/projectTag/appTag:tagTag
        imagePullPolicy: Always
        ports:
        - containerPort: 9999
        securityContext:
          privileged: false
        resources:
          requests:
            cpu: cpuReqTag
            memory: memReqTagMi
            ephemeral-storage: 2Gi
          limits:
            cpu: cpuLmtTag
            memory: memLmtTagMi
            ephemeral-storage: 5Gi
        readinessProbe:
          #httpGet:
          #  path: /healthProbe.jsp
          tcpSocket:
            port: 9114
          initialDelaySeconds: 30
          periodSeconds: 3
        livenessProbe:
          #httpGet:
          #  path: /healthProbe.jsp
          tcpSocket:
            port: 9114
          initialDelaySeconds: 120
          periodSeconds: 3
        lifecycle:
          preStop:
            exec:
              command: ["/apps/script/pod_pre_stop.sh"]
        volumeMounts:
        - mountPath: /apps/heapdump
          name: dump-vol
      volumes:
      - name: dump-vol
        hostPath:
          path: /var/lib/docker/dumpvol
          type: Directory

ingress.template

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: appTag
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: domainTag
    http:
      paths:
      - path: /
        backend:
          serviceName: appTag
          servicePort: http
---
kind: Service
apiVersion: v1
metadata:
  name: appTag
  annotations:
    traefik.ingress.kubernetes.io/affinity: "sessionStickyTag"
    traefik.ingress.kubernetes.io/session-cookie-name: "lbcookie"
spec:
  clusterIP: None
  selector:
    app: appTag
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 9999

make_dockerfile.sh

#!/bin/bash

workspace=$1
jre=$2
middleware=$3
cpulmt=$4
memlmt=$5
rwfilelist=$6
imageVersion=$7
sbstartCommand=$8
jvmSize=$9
directMemSize=${10}

if [ $# -ne 10 ]; then
    echo "Usage: $0 [workspace] [jre] [middlewre] [cpulmt] [memlmt] [rwfilelist] [imageVersion] [sbstartCommand] [jvmSize] [directMemSize]"
    exit 1
fi

if [ $9 == "null" ]; then
    if [ $memlmt -le 3072 ]; then
        jvmSize=`expr $memlmt / 2`
    else
        jvmSize=`expr $memlmt - 1500`
    fi
fi

if [ ${10} == "null" ]; then
    if [ $memlmt -le 2048 ]; then
        directMemSize=256
    else
        directMemSize=512
    fi
fi

if [ $jre == "jre6" || $jre == "jre7" ]; then
    permName="PermSize"
else
    permName="MetaspaceSize"
fi

image="centos74-"${jre}"-"${middleware}":"${imageVersion}

mydockerfile=$workspace/deploy/dockerfile
mkdir -p $workspace/deploy
cp /data1/template/dockerfile.template $mydockerfile

/usr/bin/sed -i "s/imageTag/${image}/g" $mydockerfile
/usr/bin/sed -i "s/cpuLmtTag/${cpulmt}/g" $mydockerfile
/usr/bin/sed -i "s/memLmtTag/${memlmt}/g" $mydockerfile
/usr/bin/sed -i "s/rwFileListTag/${rwfilelist}/g" $mydockerfile
/usr/bin/sed -i "s/sbStartCommandTag/${sbstartCommand}/g" $mydockerfile
/usr/bin/sed -i "s/jvmSizeTag/${jvmSize}/g" $mydockerfile
/usr/bin/sed -i "s/directMemSizeTag/${directMemSize}/g" $mydockerfile
/usr/bin/sed -i "s/permNameTag/${permName}/g" $mydockerfile

make_deploymentfile.sh 

#!/bin/bash

workspace=$1
app=$2
project=$3
replicate=$4
buildnum=$5
cpuReq=$6
cpuLmt=$7
memReq=$8
memLmt=$9
tag=${10}

if [ $# -ne 10 ]; then
    echo "Usage: $0 [workspace] [app] [project] [replicate] [buildnum] [cpuReq] [cpuLmt] [memReq] [memLmt] [tag]"
    exit 1
fi

mydeploymentfile=$workspace/deploy/deployment.yaml
cp /data1/template/deployment.template $mydeploymentfile

/usr/bin/sed -i "s/appTag/${app}/g" $mydeploymentfile
/usr/bin/sed -i "s/projectTag/${project}/g" $mydeploymentfile
/usr/bin/sed -i "s/replicateTag/${replicate}/g" $mydeploymentfile
/usr/bin/sed -i "s/buildnumTag/${buildnum}/g" $mydeploymentfile
/usr/bin/sed -i "s/cpuReqTag/${cpuReq}/g" $mydeploymentfile
/usr/bin/sed -i "s/cpuLmtTag/${cpuLmt}/g" $mydeploymentfile
/usr/bin/sed -i "s/memReqTag/${memReq}/g" $mydeploymentfile
/usr/bin/sed -i "s/memLmtTag/${memLmt}/g" $mydeploymentfile
/usr/bin/sed -i "s/tagTag/${tag}/g" $mydeploymentfile

make_ingressfile.sh

#!/bin/bash

workspace=$1
app=$2
domain=$3
sessionSticky=$4

if [ $# -ne 4 ]; then
    echo "Usage: $0 [workspace] [app] [domain] [sessionSticky]"
    exit 1
fi

myingressfile=$workspace/deploy/ingress.yaml
cp /data1/template/ingress.template $myingressfile

/usr/bin/sed -i "s/appTag/${app}/g" $myingressfile
/usr/bin/sed -i "s/domainTag/${domain}/g" $myingressfile
/usr/bin/sed -i "s/sessionStickyTag/${sessionSticky}/g" $myingressfile

 

# jenkins
Logo
K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 |  韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号​前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。​长夜有穷，真水无香。领略过了 K8s“身在江

avatar K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub

avatar K8S/Kubernetes

k8s 火了！

2020，上云之年，产品云端化成为一种趋势。在一线城市，很多公司都已经构建了自己的私有云环境，比如阿里云、网易云、华为云等。而Kubernetes 作为基于容器编排领域的王者，具备扩展...

avatar K8S/Kubernetes