记录一次调研jenkins实现多环境k8s服务自动化发布(简单版)

Jenkins实现多环境K8S应用发布

阳光下、为谁回眸

881人浏览 · 2023-04-11 15:35:31

阳光下、为谁回眸 · 2023-04-11 15:35:31 发布

前提：启动服务

下面内容会涉及自动生成K8S的yaml代码(py撰写) 代码内容下文中不做体现只提供思路
代码区分前后端服务且会根据需求生成Ingress deployment svc 等
大佬勿喷仅作调研实现

# 启动服务
# devops服务器
nohup python3 /opt/scripts/autodeploy_py/WebApi.py &

现状

环境	jenkins	slave	k8s	节点名称
dev	192.168.1.48	192.168.1.26	192.168.1.156	devk8s_context
uat	192.168.1.48	172.16.1.15(公网IP1)	172.16.1.16	uatk8s_context
fat	192.168.1.48	172.16.2.6(公网IP2)	172.16.2.17	fatk8s_context
lpt	192.168.1.48	172.16.1.32(公网IP3)	172.16.1.19	lptk8s_context
prod	192.168.1.48	10.0.2.42(公网IP4)	10.0.1.16	prodk8s_context

方案一、docker jenkins安装部署版本[2.222.4]

1、安装jenkins

# 获取镜像
docker pull *****.*****.com/base/jenkins
# 启动容器
docker run -u root  -d   -p 8080:8080   -p 50000:50000   -v /data/jenkins_data:/var/jenkins_home   -v /var/run/docker.sock:/var/run/docker.sock   --name jenkins_demo  *****.*****.com/base/jenkins
# 服务器请求
curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d '{"version":"5c86d41508abddfc8540db99794353579ac0eecb","appname":"***-frontend","environment":"uat","branch":"dev","k8sversion":"1.18"}' | tail -5

2、创建文件夹及job(忽略)

3、配置job

变量名称	参数化	默认值	可填
APPNAME	文本参数	无	必填
environment	选项参数	dev uat fat prd	必填
version	文本参数	无	必填
k8sversion	选项参数	1.12 1.18	必填

jenkins job配置如下：

# 生成配置文件
curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d {\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$environment\"，\"k8sversion\":\"$k8sversion\"}
# 指定对接apiserver
kubectl config set-cluster dev-k8s --server https://192.168.1.156:6443  --certificate-authority=/opt/k8s_config/dev/dev-cluster.ca
# 添加用户 需要指定crt，key文件，上一步有获取
kubectl config set-credentials kubernetes-admin     --client-certificate=/opt/k8s_config/dev/dev.crt     --client-key=/opt/k8s_config/dev/dev.key
# 指定一个上下文 
kubectl config set-context dev --cluster=dev-k8s  --namespace=app --user=kubernetes-admin 
#激活这个上下文
kubectl config use-context dev
# kubetctl发布
kubectl apply -f $APPNAME-$environment.yaml

方案二、本地化 jenkins安装部署版本[2.277.4] [√]

1、安装jenkins

# 获取安装yum
sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
# 安装jenkins
yum install jenkins
# 启动jenkins
systemctl start jenkins
# 重启及停止
systemctl restart jenkins
systemctl stop jenkins

2、创建文件夹及job(忽略)

3、配置job

变量名称	参数化	默认值	可填
APPNAME	文本参数	无	必填
environment	选项参数	dev uat fat prd	必填
version	文本参数	无	必填
k8sversion	选项参数	1.12 1.18	必填

jenkins job配置如下：此处为shell脚本调用

# 配置文件
APPNAME=`echo $JOB_NAME|awk -F '/' '{print $NF}'`
ENV=`echo $JOB_NAME|awk -F '/' '{print $1}'`
PYTHONAPI=http://devops.******.com/autodeploy/deploy
if [ ` curl -i -X POST $PYTHONAPI  -H 'Content-type':'application/json' -d \{\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$ENV\",\"branch\":\"$branch\",\"k8sversion\":\"$k8sversion\"\} | tail -5|python -c "import sys, json; print(json.load(sys.stdin)['code'])" ` -eq "200" ] ;then
	echo "生成配置文件"
    # 生成配置文件
    curl -i -X POST $PYTHONAPI -H 'Content-type':'application/json' -d \{\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$ENV\",\"branch\":\"$branch\",\"k8sversion\":\"$k8sversion\"\} | tail -5|python -c "import sys, json; print(json.load(sys.stdin)['data'])"> $version.yaml
	# apply yaml
    if [ $ENV == "develop" ]; then
		cp $version.yaml $version-fort.yaml
        sed -i "s/$APPNAME/$APPNAME-fort/g" $version-fort.yaml
        sed -i "s#$ENV/$APPNAME-fort#$ENV/$APPNAME#g" $version-fort.yaml
        kubectl apply -f $version-fort.yaml
    else
    	kubectl apply -f $version.yaml
    fi
else
	echo ` curl -i -X POST $PYTHONAPI -H 'Content-type':'application/json' -d \{\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$ENV\",\"branch\":\"$branch\",\"k8sversion\":\"$k8sversion\"\}`
	exit 1
fi
echo "结束"

可以使用jenkins插件 Configuration Slicing
Tied Label Slicer: 配置节点信息
Execute shell slicer：配置脚本信息

复制job 可以使用Python脚本操作核心代码如下:
# copy_job(“复制项”，“黏贴项”)    
jenkinsenkins.Jenkins(Jenkins_server_url, Jenkins_user, Jenkins_passwd).copy_job("fat/single-spa/report-web","fat/single-spa/demo")

目前使用jenkins[本地化部署]

地址:http://192.168.1.251:8080/
用户名:admin
密码:admin

目录结构

目录	用途
/data/yaml	存放yaml生成文件
/opt/k8s_config	存放各个环境的k8s对接信息证书等
/data/yaml/deploy.py	具体代码文件

方案一、K8S多环境调用

安装Kubectl

# 将kubectl和apiserver的版本尽量保持一致 防止出现版本问题 地址如下
https://dl.k8s.io/v1.12.3/kubernetes-client-linux-amd64.tar.gz

如果用户没有证书则执行：

# 生成签发证书的策略ca-config.json
{
  "signing": {
    "default": {
      "expiry": "438000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "438000h"
      }
    }
  }
}

# 生成ca自签名请求 cat ca-csr.json
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "HangZhou",
      "L": "XS",
      "O": "k8s",
      "OU": "System"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}

# 执行下面命令 会生成证书及私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
# 如果命令不存在执行：
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

# 创建admin用户配置
cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin



KUBE_CONFIG="/root/.kube/config"
KUBE_APISERVER="https://172.16.1.34:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials cluster-admin \
  --client-certificate=./admin.pem \
  --client-key=./admin-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
  --cluster=kubernetes \
  --user=cluster-admin \
  --kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

获取master的相关信息

# 到k8s集群master服务器 执行：cat ~/.kube/config
# 将
-cluster
-client-certificate-data
-client-key-data
# 这三个所对应的value进行转化

echo '${替换内容}' | base64 -d
echo '' | base64 -d > dev-cluster.ca
echo '' | base64 -d > dev.crt
echo '' | base64 -d > dev.key
#并记录server的值：https://192.168.1.156:6443
# 将三个保存成文件 分别为 (命名可以随便修改)
-cluster --> dev-cluster.ca
-client-certificate-data  --> dev.crt
-client-key-data --> dev.key

DEV环境执行命令：

# 将master的/root/.kube文件复制到client节点
# 指定指定的k8s的链接地址
kubectl config set-cluster dev-k8s --server https://192.168.1.156:6443  --certificate-authority=/opt/k8s_config/dev/dev-cluster.ca

# 添加用户 需要指定crt key文件 上一步有获取
kubectl config set-credentials kubernetes-admin     --client-certificate=/opt/k8s_config/dev/dev.crt     --client-key=/opt/k8s_config/dev/dev.key

# 指定一个上下文
kubectl config set-context dev --cluster=dev-k8s  --namespace=app --user=kubernetes-admin 

#激活这个上下文
kubectl config use-context dev
# 已测试可以正常使用 下面命令有输出则正常
kubectl version

问题现状

1、docker容器部署的jenkins 无法调用kubectl 如果可调用多环境的apiserver调用的key ca crt 要做挂载

2、目前dev环境kubectl调用远端apiserver实现但uat fat prd 的k8s的server均为内网地址本地无法调用尝试更改但怕影响环境

3、代码和jenkins及kubectl目前耦合度较强

方案二、jenkins_slave方式

创建从节点

# 1、服务器时间同步
yum -y install ntp ntpdate
ntpdate 0.asia.pool.ntp.org && hwclock --systohc
# 2、创建节点
登录Jenkins服务器-点击系统管理-选择Manage Nodes(节点管理)-点击新建节点-点击ok
# 3、配置节点
名称:dev_master_kubesphere_192.168.1.11 
描述:测试
执行器数量:4
远程工作目录:/home
标签:Kubesphere
用法;只允许运行绑定到这台机器的job
启动方式:Launch agents via SSH
主机:192.168.1.11
Credentials: root(后面会说明)
Host Key Verification Strategy:Non verifying verification Stategy
可用性:尽量保持代理在线
#4、服务器创建公私钥
ssh-keygen -t rsa
jenkins添加私钥 id_rsa
添加凭据:SSH Username with private key 私钥复制即可
=======================查看状态节点添加完成=======================

创建Job

General中需要注意一点:
限制项目运行节点
标签表达式
dev_master_kubesphere_192.168.1.11

执行shell(后期优化逻辑)

if [ ` curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d \{\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$environment\",\"branch\":\"$branch\"\,\"k8sversion\":\"$k8sversion\"\} | tail -5|python -c "import sys, json; print(json.load(sys.stdin)['code'])" ` -eq "200" ] ;then
	echo "生成配置文件"
    # 生成配置文件
    curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d \{\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$environment\",\"branch\":\"$branch\",\"k8sversion\":\"$k8sversion\"\} | tail -5|python -c "import sys, json; print(json.load(sys.stdin)['data'])"> $APPNAME-$environment-$BUILD_NUMBER.yaml
	# apply yaml
	kubectl apply -f $APPNAME-$environment-$BUILD_NUMBER.yaml
else
	echo ` curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d \{\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$environment\",\"branch\":\"$branch\",\"k8sversion\":\"$k8sversion\"\} | tail -5|python -c "import sys, json; print(json.load(sys.stdin)['msg'])" `
fi
echo "结束"

接口文档

序号	环境	URL
1	dev环境	htpp://192.168.1.251:8077/deploy

简要描述：
        跟据参数调用数据库生成yaml并保存成文件
			请求URL：
                    htpp://192.168.1.251:8077/deploy
			请求方式：
                    POST
			heardes：
                    Content-Type:application/json

参数：

参数名称	必选	类型	说明
appname	是	String	pod名称(镜像名称)
environment	是	String	环境参数
version	是	String	版本参数
branch	是	String	代码版本
k8sversion	是	String	k8s版本

请求示例(以下实际参数为示例)：

linux请求：

# 查看接口返回
curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d '{"version":"5c86d41508abddfc8540db99794353579ac0eecb","appname":"***-frontend","environment":"uat","branch":"dev","k8sversion":"1.18"}' | tail -5
# 获取data信息
curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d '{"version":"5c86d41508abddfc8540db99794353579ac0eecb","appname":"***-frontend","environment":"uat","branch":"dev","k8sversion":"1.18"}' | tail -5|python -c "import sys, json; print(json.load(sys.stdin)['data'])"> ***-frontend.yaml

python请求：

url = "http://127.0.0.1:8077/deploy"
data = {"version" :"7f06e1599e6d78c80302d8032b0d36b8a37f6410", "appname":"******", "environment":"prd","branch":"dev","k8sversion":"1.18"}
data = json.dumps(data)
res = requests.post(url, data=data)
print(res.text)

postman参数：

header: application/json
{    
		"version" :"7f06e1599e6d78c80302d8032b0d36b8a37f6410",       				         "appname":"******", 
		"environment":"prd",
		"branch":"dev",
		"k8sversion":"1.18"
}

返回示例(以下实际参数为示例)：

正常
{
  "code": "200", 
  "data": "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  annotations:\n    deployment.kubernetes.io/revision: '1'\n  generation: 3\n  labels:\n    app: voc-work-weixin\n    branch: dev\n    version: eced3eadff71665b10231cc7f4010148e5635dfe\n  name: voc-work-weixin-eced3eadff71665b10231cc7f4010148e5635dfe\n  namespace: app\nspec:\n  progressDeadlineSeconds: 600\n  replicas: 1\n  revisionHistoryLimit: 2\n  selector:\n    matchLabels:\n      app: voc-work-weixin\n      branch: dev\n      version: eced3eadff71665b10231cc7f4010148e5635dfe\n  template:\n    metadata:\n      annotations:\n        prometheus.io/path: api/prometheus\n        prometheus.io/port: '8080'\n        prometheus.io/scrape: 'true'\n      creationTimestamp: null\n      labels:\n        app: voc-work-weixin\n        branch: dev\n        version: eced3eadff71665b10231cc7f4010148e5635dfe\n    spec:\n      containers:\n      - env: []\n        image: harbor.vocustcloud.com/dev/uat/voc-work-weixin:eced3eadff71665b10231cc7f4010148e5635dfe\n        imagePullPolicy: IfNotPresent\n        lifecycle:\n          postStart:\n            exec:\n              command:\n              - sh\n              - -c\n              - echo `uname -a | awk '{print $2}'` > /opt/voc-work-weixin/logs/podName\n          preStop:\n            exec:\n              command:\n              - /opt/consul/bin/consul\n              - leave\n        livenessProbe:\n          failureThreshold: 3\n          httpGet:\n            path: /\n            port: 8080\n            scheme: HTTP\n          initialDelaySeconds: 120\n          periodSeconds: 60\n          successThreshold: 1\n          timeoutSeconds: 5\n        name: voc-work-weixin\n        ports:\n        - containerPort: 8080\n          name: service-http\n          protocol: TCP\n        resources:\n          limits:\n            cpu: 1000m\n            memory: 2048Mi\n          requests:\n            cpu: 50m\n            memory: 128Mi\n        terminationMessagePath: /dev/termination-log\n        terminationMessagePolicy: File\n        volumeMounts:\n        - mountPath: /opt/ctrip/AppData/\n          name: appdata-dir\n        - mountPath: /opt/voc-work-weixin/logs\n          name: log-dir\n      securityContext: {}\n      terminationGracePeriodSeconds: 30\n      volumes:\n      - hostPath:\n          path: /opt/ctrip/AppData/\n          type: ''\n        name: appdata-dir\n      - hostPath:\n          path: /opt/logs/voc-work-weixin/eced3eadff71665b10231cc7f4010148e5635dfe/\n          type: ''\n        name: log-dir\n---\napiVersion: extensions/v1beta1\nkind: Ingress\nmetadata:\n  annotations:\n    kubernetes.io/ingress.class: nginx\n  generation: 1\n  name: voc-work-weixin-ingress\n  namespace: app\nspec:\n  rules:\n  - host: voc-work-weixin.uat.vocustcloud.com\n    http:\n      paths:\n      - backend:\n          serviceName: voc-work-weixin-service\n          servicePort: 8080\nstatus:\n  loadBalancer: {}\n---\napiVersion: v1\nkind: Service\nmetadata:\n  labels:\n    app: voc-work-weixin\n    service: voc-work-weixin-service\n  name: voc-work-weixin-service\n  namespace: app\nspec:\n  ports:\n  - name: service-http\n    port: 8080\n    protocol: TCP\n    targetPort: 8080\n  selector:\n    app: voc-work-weixin\n  sessionAffinity: None\n  type: ClusterIP\n---\n", 
  "msg": "OK"
}

请求参数格式问题
{
  "code": "405", 
  "datas": {请求参数}, 
  "msg": "json decode error 请检查参数格式是否正确"
}

env格式错误导致转dict失败检查env格式
{
  "code": "405", 
  "datas": {}, 
  "msg": "文件写入异常"
}

数据库连接失败
{
  "code": "503", 
  "datas": {}, 
  "msg": "database connection error:"
}

sql执行错误
{
  "code": "503", 
  "datas": {}, 
  "msg": "SQL excute Error:"
}

数据库

-- 地址：192.168.1.157
-- 库:`cloudci_k8s` 表:`ci_deployment`
-- 用户名密码：root/root
-- 建表语句：
CREATE TABLE `ci_deployment` (
  `id` int(5) NOT NULL AUTO_INCREMENT,
  `appname` varchar(120) NOT NULL COMMENT '服务名称',
  `namespace` varchar(20) NOT NULL DEFAULT 'app' COMMENT '命名空间',
  `environment` varchar(20) DEFAULT NULL COMMENT '环境分支 区分各个环境',
  `branch` varchar(20) NOT NULL COMMENT '代码分支',
  `location` int(1) DEFAULT '1' COMMENT '1 后端 0 前端',
  `podnum` int(2) NOT NULL DEFAULT '1' COMMENT 'pod数量',
  `imageurl` varchar(120) NOT NULL COMMENT '镜像地址',
  `env` mediumtext NOT NULL COMMENT 'env变量',
  `memory_requests` varchar(5) NOT NULL DEFAULT '512' COMMENT '内存值 单位Mi',
  `cpu_requests` varchar(5) NOT NULL DEFAULT '250' COMMENT 'CPU值  单位m',
  `memory_limits` varchar(5) NOT NULL DEFAULT '2048' COMMENT '内存值 单位Mi',
  `cpu_limits` varchar(5) NOT NULL DEFAULT '1000' COMMENT 'CPU值  单位m',
  `flag` int(1) NOT NULL DEFAULT '1' COMMENT '1启动 0未启用',
  `deployment` int(1) NOT NULL DEFAULT '1' COMMENT 'deployment 1启动 0未启用',
  `ingress` int(1) NOT NULL DEFAULT '1' COMMENT 'ingress 1启动 0未启用',
  `service` int(1) NOT NULL DEFAULT '1' COMMENT 'service 1启动 0未启用',
  PRIMARY KEY (`id`),
  UNIQUE KEY `appname` (`appname`,`environment`,`branch`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8