Kubernetes 平台实现 Jenkins的 Master-Slave 分布式构建方案（下）
文章目录一、K8S上安装Jenkins-Master1. 创建NFS client provisioner2. 安装 Jenkins-Master3. Jenkins控制台配置二、Jenkins与Kubernetes整合1. 实现Jenkins与K8s整合2. 构建Jenkins-Slave自定义镜像3. Jenkins-slave流水线项目测试三、基于kubernetes平台微服务的部署1. 拉
·
文章目录
一、K8S上安装Jenkins-Master
1. 创建NFS client provisioner
nfs-client-provisioner 是一个Kubernetes的简易 NFS 的外部 provisioner，本身不提供 NFS，需要现有的 NFS 服务器提供存储。
上传 nfs-client-provisioner 构建文件
修改deployment.yaml，使用之前配置NFS服务器和目录
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: lizhenliang/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 192.168.8.12
- name: NFS_PATH
value: /opt/nfs/jenkins/
volumes:
- name: nfs-client-root
nfs:
server: 192.168.8.12
path: /opt/nfs/jenkins/
构建nfs-client-provisioner的pod资源
cd nfs-client/
kubectl create -f . #注意后面有个小数点
查看pod是否创建成功
kubectl get pods
2. 安装 Jenkins-Master
上传Jenkins-Master构建文件
-
在StatefulSet.yaml文件，声明了利用nfs-client-provisioner进行Jenkins-Master文件存储
-
Service发布方法采用NodePort，会随机产生节点访问端口
创建kube-ops的namespace
把Jenkins-Master的pod放到kube-ops下
kubectl create namespace kube-ops
构建Jenkins-Master的pod资源
kubectl create -f . #小数点
查看pod是否创建成功，需要耐心等待创建成功
kubectl get pods -n kube-ops
查看Pod运行在那个Node上，并使用自动分发的端口访问
kubectl describe pods -n kube-ops
kubectl get service -n kube-ops
访问地址为node节点的IP:
http://192.168.8.12:30474
找到密钥并复制到web页面中登录
cat /opt/nfs/jenkins/kube-ops-jenkins-home-jenkins-0-pvc-6319468f-6ccf-4781-af84-df992eb7f96b/secrets/initialAdminPassword
Kubernetes中Jenkins部署完成！！！
3. Jenkins控制台配置
设置插件下载地址
cd /opt/nfs/jenkins/kube-ops-jenkins-home-jenkins-0-pvc-6319468f-6ccf-4781-af84-df992eb7f96b/updates/
sed -i 's/http:\/\/updates.jenkins- ci.org\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json && sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json
Manage Plugins点击Advanced，把Update Site改为国内插件下载地址
https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json
安装所需插件
Chinese
Git
Pipeline
Extended Choice Parameter
Kubernetes
二、Jenkins与Kubernetes整合
1. 实现Jenkins与K8s整合
系统管理->系统配置->云->新建云->Kubernetes
#kubernetes 地址采用了kube的服务器发现：
https://kubernetes.default.svc.cluster.local namespace填kube-ops
#点击Test Connection，如果出现 Connection test successful 的提示信息证明 Jenkins 已经可以和 Kubernetes 系统正常通信
#Jenkins URL地址：
http://jenkins.kube-ops.svc.cluster.local:8080
2. 构建Jenkins-Slave自定义镜像
- Jenkins-Master在构建Job的时候，Kubernetes会创建Jenkins-Slave的Pod来完成Job的构建
- 我们选择运行Jenkins-Slave的镜像为官方推荐镜像：
jenkins/jnlp-slave:latest，但是这个镜像里面并没有Maven 环境，为了方便使用，我们需要自定义一个新的镜像：
Dockerfile文件内容如下：
FROM jenkins/jnlp-slave:latest
MAINTAINER itcast
# 切换到 root 账户进行操作
USER root
# 安装 maven
COPY apache-maven-3.6.2-bin.tar.gz .
RUN tar -zxf apache-maven-3.6.2-bin.tar.gz && \
mv apache-maven-3.6.2 /usr/local && \
rm -f apache-maven-3.6.2-bin.tar.gz && \
ln -s /usr/local/apache-maven-3.6.2/bin/mvn /usr/bin/mvn && \
ln -s /usr/local/apache-maven-3.6.2 /usr/local/apache-maven && \
mkdir -p /usr/local/apache-maven/repo
COPY settings.xml /usr/local/apache-maven/conf/settings.xml
USER jenkins
构建镜像
docker build -t jenkins-slave-maven:latest .
镜像打标签，上传harbor仓库
docker login -u admin -p Harbor12345 192.168.8.20:85
docker tag jenkins-slave-maven:latest 192.168.8.20:85/library/jenkins-slave-maven:latest
docker push 192.168.8.20:85/library/jenkins-slave-maven:latest
3. Jenkins-slave流水线项目测试
创建流水线项目
添加凭证
使用流水线语法中的片段生成器生成拉取代码
checkout([$class: 'GitSCM', branches: [[name: '*/master']], extensions: [], userRemoteConfigs: [[credentialsId: 'da4091ce-712f-42fa-a3a8-d8aad6a166c1', url: 'http://192.168.8.18:82/gl/tensquare_back.git']]])
编写Pipeline，从GItlab拉取代码（使用http方式）
def git_address = "http://192.168.8.18:82/gl/tensquare_back.git"
def git_auth = "da4091ce-712f-42fa-a3a8-d8aad6a166c1"
//创建一个Pod的模板，label为jenkins-slave
podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [
containerTemplate(
name: 'jnlp',
image: "192.168.8.20:85/library/jenkins-slave-maven:latest"
)
]
)
{
//引用jenkins-slave的pod模块来构建Jenkins-Slave的pod
node("jenkins-slave"){
stage('拉取代码'){
checkout([$class: 'GitSCM', branches: [[name: '*/master']], extensions: [], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
}
}
}
可以看到构建的时候，创建了slave节点，构建是由slave进行的；
而当项目构建完成以后，这个slave节点就消失了！
三、基于kubernetes平台微服务的部署
1. 拉取代码，创建镜像
创建流水线项目
创建NFS共享目录，让所有Jenkins-Slave构建指向NFS的Maven的共享仓库目录
vim /etc/exports
#添加内容：
/opt/nfs/jenkins *(rw,no_root_squash)
/opt/nfs/maven *(rw,no_root_squash)
systemctl restart nfs
编写构建Pipeline
tensquare_eureka_server@10086,tensquare_zuul@10020,tensquare_admin_service@9001,tensquare_gathering@9002
注册中心,服务网关,认证中心,活动微服务
继续添加字符串参数
配置harbor凭证
def git_address = "http://192.168.8.18:82/gl/tensquare_back.git"
def git_auth = "da4091ce-712f-42fa-a3a8-d8aad6a166c1"
//构建版本的名称
def tag = "latest"
//Harbor私服地址
def harbor_url = "192.168.8.20:85"
//Harbor的项目名称
def harbor_project_name = "tensquare"
//Harbor的凭证
def harbor_auth = "c77a32ac-33a1-4855-947c-b0c4d788c555"
podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [
containerTemplate(
name: 'jnlp',
image: "192.168.8.20:85/library/jenkins-slave-maven:latest"
),
containerTemplate(
name: 'docker',
image: "docker:stable",
ttyEnabled: true,
command: 'cat'
),
],
volumes: [
hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
nfsVolume(mountPath: '/usr/local/apache-maven/repo', serverAddress: '192.168.8.12' , serverPath: '/opt/nfs/maven'),
],
)
{
node("jenkins-slave"){
// 第一步
stage('pull code'){
checkout([$class: 'GitSCM', branches: [[name: '*/master']], extensions: [], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
}
// 第二步
stage('make public sub project'){
//编译并安装公共工程
sh "mvn -f tensquare_common clean install"
}
// 第三步
stage('make image'){
//把选择的项目信息转为数组
def selectedProjects = "${project_name}".split(',')
for(int i=0;i<selectedProjects.size();i++){
//取出每个项目的名称和端口
def currentProject = selectedProjects[i];
//项目名称
def currentProjectName = currentProject.split('@')[0]
//项目启动端口
def currentProjectPort = currentProject.split('@')[1]
//定义镜像名称
def imageName = "${currentProjectName}:${tag}"
//编译，构建本地镜像
sh "mvn -f ${currentProjectName} clean package dockerfile:build"
container('docker') {
//给镜像打标签
sh "docker tag ${imageName} ${harbor_url}/${harbor_project_name}/${imageName}"
//登录Harbor，并上传镜像
withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'password', usernameVariable: 'username')])
{
//登录
sh "docker login -u ${username} -p ${password} ${harbor_url}"
//上传镜像
sh "docker push ${harbor_url}/${harbor_project_name}/${imageName}"
}
//删除本地镜像
sh "docker rmi -f ${imageName}"
sh "docker rmi -f ${harbor_url}/${harbor_project_name}/${imageName}"
}
}
}
}
}
在构建过程会发现无法创建仓库目录，是因为NFS共享目录权限不足，需更改权限
mkdir -p /opt/nfs/maven
chmod -R 777 /opt/nfs/maven
#Docker命令执行权限问题(node节点)
chmod 777 /var/run/docker.sock
进行构建测试
eureka服务构建成功，并且可以看到构建成功以后，slave节点就自动被释放掉了！
2. 配置eureka服务
安装插件Kubernetes Continuous Deploy
添加k8s凭证
b05cdd9c-3df5-4f2a-a5a7-a55eb18e9bfa
编写pipeline脚本
def git_address = "http://192.168.8.18:82/gl/tensquare_back.git"
def git_auth = "da4091ce-712f-42fa-a3a8-d8aad6a166c1"
//构建版本的名称
def tag = "latest"
//Harbor私服地址
def harbor_url = "192.168.8.20:85"
//Harbor的项目名称
def harbor_project_name = "tensquare"
//Harbor的凭证
def harbor_auth = "c77a32ac-33a1-4855-947c-b0c4d788c555"
//k8s的凭证
def k8s_auth="b05cdd9c-3df5-4f2a-a5a7-a55eb18e9bfa"
//定义k8s-barbor的凭证
def secret_name="registry-auth-secret"
podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [
containerTemplate(
name: 'jnlp',
image: "192.168.8.20:85/library/jenkins-slave-maven:latest"
),
containerTemplate(
name: 'docker',
image: "docker:stable",
ttyEnabled: true,
command: 'cat'
),
],
volumes: [
hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
nfsVolume(mountPath: '/usr/local/apache-maven/repo', serverAddress: '192.168.8.12' , serverPath: '/opt/nfs/maven'),
],
)
{
node("jenkins-slave"){
// 第一步
stage('pull code'){
checkout([$class: 'GitSCM', branches: [[name: '*/master']], extensions: [], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
}
// 第二步
stage('make public sub project'){
//编译并安装公共工程
sh "mvn -f tensquare_common clean install"
}
// 第三步
stage('make image'){
//把选择的项目信息转为数组
def selectedProjects = "${project_name}".split(',')
for(int i=0;i<selectedProjects.size();i++){
//取出每个项目的名称和端口
def currentProject = selectedProjects[i];
//项目名称
def currentProjectName = currentProject.split('@')[0]
//项目启动端口
def currentProjectPort = currentProject.split('@')[1]
//定义镜像名称
def imageName = "${currentProjectName}:${tag}"
//编译，构建本地镜像
sh "mvn -f ${currentProjectName} clean package dockerfile:build"
container('docker') {
//给镜像打标签
sh "docker tag ${imageName} ${harbor_url}/${harbor_project_name}/${imageName}"
//登录Harbor，并上传镜像
withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'password', usernameVariable: 'username')])
{
//登录
sh "docker login -u ${username} -p ${password} ${harbor_url}"
//上传镜像
sh "docker push ${harbor_url}/${harbor_project_name}/${imageName}"
}
//删除本地镜像
sh "docker rmi -f ${imageName}"
sh "docker rmi -f ${harbor_url}/${harbor_project_name}/${imageName}"
}
def deploy_image_name = "${harbor_url}/${harbor_project_name}/${imageName}"
//部署到K8S
sh """
sed -i 's#\$IMAGE_NAME#${deploy_image_name}#' ${currentProjectName}/deploy.yml
sed -i 's#\$SECRET_NAME#${secret_name}#' ${currentProjectName}/deploy.yml
"""
kubernetesDeploy configs: "${currentProjectName}/deploy.yml", kubeconfigId: "${k8s_auth}"
}
}
}
}
eureka目录下创建deploy.yml文件
---
apiVersion: v1
kind: Service
metadata:
name: eureka
labels:
app: eureka
spec:
type: NodePort
ports:
- port: 10086
name: eureka
targetPort: 10086
selector:
app: eureka
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: eureka
spec:
serviceName: "eureka"
replicas: 2
selector:
matchLabels:
app: eureka
template:
metadata:
labels:
app: eureka
spec:
imagePullSecrets:
- name: $SECRET_NAME
containers:
- name: eureka
image: $IMAGE_NAME
ports:
- containerPort: 10086
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: EUREKA_SERVER
value: "http://eureka-0.eureka:10086/eureka/,http://eureka- 1.eureka:10086/eureka/"
- name: EUREKA_INSTANCE_HOSTNAME
value: ${MY_POD_NAME}.eureka
podManagementPolicy: "Parallel"
更改application.yml配置文件
server:
port: ${PORT:10086}
spring:
application:
name: eureka
eureka:
server:
# 续期时间，即扫描失效服务的间隔时间（缺省为60*1000ms）
eviction-interval-timer-in-ms: 5000
enable-self-preservation: false
use-read-only-response-cache: false
client:
# eureka client间隔多久去拉取服务注册信息 默认30s
registry-fetch-interval-seconds: 5
serviceUrl:
defaultZone: ${EUREKA_SERVER:http://127.0.0.1:${server.port}/eureka/}
instance:
# 心跳间隔时间，即发送一次心跳之后，多久在发起下一次（缺省为30s）
lease-renewal-interval-in-seconds: 5
# 在收到一次心跳之后，等待下一次心跳的空档时间，大于心跳间隔即可，即服务续约到期时间（缺省为90s）
lease-expiration-duration-in-seconds: 10
instance-id: ${EUREKA_INSTANCE_HOSTNAME:${spring.application.name}}:${server.port}@${random.l ong(1000000,9999999)}
hostname: ${EUREKA_INSTANCE_HOSTNAME:${spring.application.name}}
提交修改的配置文件和yml文件
k8s访问harbor需要密钥权限
docker login -u tom -p Abcd1234 192.168.8.20:85
kubectl create secret docker-registry registry-auth-secret --docker-server=192.168.8.20:85 --docker-username=tom --docker-password=Abcd1234 -- docker-email=tom@qq.com
kubectl get secrets
kubectl get pods
kubectl get service
构建测试
访问两个node节点的30674端口
3. 配置zuul服务网关
更改配置文件中的集群地址
defaultZone: http://eureka-0.eureka:10086/eureka/,http://eureka-1.eureka:10086/eureka/ # Eureka访问地址
创建配置文件deploy.yml
---
apiVersion: v1
kind: Service
metadata:
name: zuul
labels:
app: zuul
spec:
type: NodePort
ports:
- port: 10020
name: zuul
targetPort: 10020
selector:
app: zuul
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: zuul
spec:
serviceName: "zuul"
replicas: 2
selector:
matchLabels:
app: zuul
template:
metadata:
labels:
app: zuul
spec:
imagePullSecrets:
- name: $SECRET_NAME
containers:
- name: zuul
image: $IMAGE_NAME
ports:
- containerPort: 10020
podManagementPolicy: "Parallel"
手动上传父工程依赖到NFS的Maven共享仓库目录中
构建网关服务并查看结果
4. 配置admin权限管理中心
创建deploy.yml文件
---
apiVersion: v1
kind: Service
metadata:
name: admin
labels:
app: admin
spec:
type: NodePort
ports:
- port: 9001
name: admin
targetPort: 9001
selector:
app: admin
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: admin
spec:
serviceName: "admin"
replicas: 2
selector:
matchLabels:
app: admin
template:
metadata:
labels:
app: admin
spec:
imagePullSecrets:
- name: $SECRET_NAME
containers:
- name: admin
image: $IMAGE_NAME
ports:
- containerPort: 9001
podManagementPolicy: "Parallel"
更改配置文件中的eureka集群地址和数据库地址
defaultZone: http://eureka-0.eureka:10086/eureka/,http://eureka-1.eureka:10086/eureka/ # Eureka访问地址
更改完成以后提交代码并构建
5. 配置gathering活动微服务
---
apiVersion: v1
kind: Service
metadata:
name: gathering
labels:
app: gathering
spec:
type: NodePort
ports:
- port: 9002
name: gathering
targetPort: 9002
selector:
app: gathering
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: gathering
spec:
serviceName: "gathering"
replicas: 2
selector:
matchLabels:
app: gathering
template:
metadata:
labels:
app: gathering
spec:
imagePullSecrets:
- name: $SECRET_NAME
containers:
- name: gathering
image: $IMAGE_NAME
ports:
- containerPort: 9002
podManagementPolicy: "Parallel"
6. 使用postman访问数据库
访问的话要先经过网关服务，所以访问的端口号是zuul的31619
开源、云原生的融合云平台
更多推荐
0
0- 0
已为社区贡献3条内容
所有评论(0)