基于Kubernetes/K8S构建Jenkins持续集成平台(一)
①服务高可用:当 Jenkins Master 出现故障时,Kubernetes 会自动创建一个新的 Jenkins Master容器,并且将 Volume 分配给新创建的容器,保证数据不丢失,从而达到集群服务高可用。②动态伸缩,合理使用资源:每次运行 Job 时,会自动创建一个 Jenkins Slave,Job 完成后,Slave 自动注销并删除容器,资源自动释放,而且 Kubernetes
目录
Kubernetes实现Master-Slave分布式构建方案
Kubernates+Docker+Jenkins持续集成架构图
Kubernates+Docker+Jenkins持续集成方案好处
Kubernetes实现Master-Slave分布式构建方案
传统Jenkins的Master-Slave方案的缺陷
Master节点发生单点故障时,整个流程都不可用了
每个 Slave节点的配置环境不一样,来完成不同语言的编译打包等操作,但是这些差异化的配置导致管理起来非常不方便,维护起来也是比较费劲
资源分配不均衡,有的 Slave节点要运行的job出现排队等待,而有的Slave节点处于空闲状态
资源浪费,每台 Slave节点可能是实体机或者VM,当Slave节点处于空闲状态时,也不会完全释放掉资源
以上种种问题,我们可以引入Kubernates来解决!
Kubernates+Docker+Jenkins持续集成架构图
大致工作流程:手动/自动构建 -> Jenkins 调度 K8S API ->动态生成 Jenkins Slave pod -> Slave pod 拉取 Git 代码/编译/打包镜像 ->推送到镜像仓库 Harbor -> Slave 工作完成,Pod 自动销毁 ->部署到测试或生产 Kubernetes平台。(完全自动化,无需人工干预)
Kubernates+Docker+Jenkins持续集成方案好处
①服务高可用:当 Jenkins Master 出现故障时,Kubernetes 会自动创建一个新的 Jenkins Master容器,并且将 Volume 分配给新创建的容器,保证数据不丢失,从而达到集群服务高可用。
②动态伸缩,合理使用资源:每次运行 Job 时,会自动创建一个 Jenkins Slave,Job 完成后,Slave 自动注销并删除容器,资源自动释放,而且 Kubernetes 会根据每个资源的使用情况,动态分配Slave 到空闲的节点上创建,降低出现因某节点资源利用率高,还排队等待在该节点的情况。
③扩展性好:当 Kubernetes 集群的资源严重不足而导致 Job 排队等待时,可以很容易的添加一个Kubernetes Node 到集群中,从而实现扩展。
环境说明
主机名称 | IP地址 | 安装的软件 |
---|---|---|
代码托管服务器 | 192.168.37.103:85 | Gitlab-12.3.0 |
docker仓库服务器 | 192.168.37.106:85 | harbor-1.9.2、nfs1.3.0 |
k8s-master | 192.168.37.100 | k8s-1.20.11 |
k8s-node01 | 192.168.37.125 | kubelet、kubeproxy、Docker-20.10.17, |
k8s-node02 | 192.168.37.105 | kubelet、kubeproxy、Docker-20.10.16 |
所有k8s节点harbor仓库配置
[root@master01 ~]# vim /etc/docker/daemon.json
{
"insecure-registries": ["192.158.37.106:85"],
"registry-mirrors": ["https://xgftnkh8.mirror.aliyuncs.com"]
}
NFS简介
NFS(Network File System),它最大的功能就是可以通过网络,让不同的机器、不同的操作系统可以共享彼此的文件。我们可以利用NFS共享Jenkins运行的配置文件、Maven的仓库依赖文件等
nfs安装,nfs服务器安装在192.168.37.106上
其他服务均要有nfs
[root@master01 ~]# rpm -q nfs-utils rpcbind
nfs-utils-1.3.0-0.48.el7.x86_64
rpcbind-0.2.0-42.el7.x86_64
创建共享目录
[root@harbor ~]# mkdir -p /opt/nfs/jenkins
[root@harbor /opt/nfs/jenkins]# chmod 777 /opt/nfs/jenkins/
[root@harbor /opt/nfs/jenkins]# ll -d /opt/nfs/jenkins/
drwxrwxrwx 2 root root 6 Jun 23 03:59 /opt/nfs/jenkins
[root@harbor /opt/nfs/jenkins]# vim /etc/exports #编写nfs共享配置
/opt/nfs/jenkins 192.168.37.0/24(rw,sync,no_root_squash)
启动服务
[root@harbor /opt/nfs/jenkins]# systemctl start rpcbind nfs
[root@harbor /opt/nfs/jenkins]# systemctl enable rpcbind nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
查看NFS共享目录
[root@node01 ~]# showmount -e 192.168.37.106
Export list for 192.168.37.106:
/opt/nfs/jenkins 192.168.37.0/24
[root@node02 ~]# showmount -e 192.168.37.106
Export list for 192.168.37.106:
/opt/nfs/jenkins 192.168.37.0/24
在Kubernetes安装Jenkins-Master
由于用的是k8s1.20.0+版本,所以要关闭自连接功能
[root@master01 /etc/kubernetes/manifests]# vim kube-apiserver.yaml
spec:
containers:
- command:
- kube-apiserver
- --feature-gates=RemoveSelfLink=false #添加这一行
- --advertise-address=192.168.37.100
重新刷新apiserver
[root@master01 /etc/kubernetes/manifests]# kubectl apply -f kube-apiserver.yaml
pod/kube-apiserver created
可能导致如下情况
在重启一下apiserver即可
[root@master01 /etc/kubernetes/manifests]# kubectl delete pod -n kube-system kube-apiserver
pod "kube-apiserver" deleted
[root@master01 /etc/kubernetes/manifests]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-74ff55c5b-shjsx 1/1 Running 5 30d
coredns-74ff55c5b-spxjj 1/1 Running 5 30d
etcd-master01 1/1 Running 5 30d
kube-apiserver-master01 1/1 Running 0 3m20s
kube-controller-manager-master01 1/1 Running 7 30d
kube-flannel-ds-dh6zz 1/1 Running 6 30d
kube-flannel-ds-js6b7 1/1 Running 5 30d
kube-flannel-ds-wm4l5 1/1 Running 3 30d
kube-proxy-66l26 1/1 Running 5 30d
kube-proxy-9h855 1/1 Running 3 30d
kube-proxy-x54xx 1/1 Running 5 30d
kube-scheduler-master01 1/1 Running 6 30d
创建nfs-client-rbac
[root@master01 /opt/jenkins]# vim nfs-client-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
---
#创建集群角色
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: nfs-client-provisioner-clusterrole
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
---
#集群角色绑定
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: nfs-client-provisioner-clusterrolebinding
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-clusterrole
apiGroup: rbac.authorization.k8s.io
创建rbac资源
[root@master01 /opt/jenkins]# kubectl apply -f nfs-client-rbac.yaml
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/nfs-client-provisioner-clusterrolebinding created
创建NFS client provisioner
nfs-client-provisioner 是一个Kubernetes的简易NFS的外部provisioner,本身不提供NFS,需要现有的NFS服务器提供存储。
[root@master01 /opt/jenkins]# vim nfs-client-provisioner.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
selector:
matchLabels:
app: nfs-client-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: nfs-storage
- name: NFS_SERVER
value: 192.168.37.106
- name: NFS_PATH
value: /opt/nfs/jenkins
volumes:
- name: nfs-client-root
nfs:
server: 192.168.37.106
path: /opt/nfs/jenkins
[root@master01 /opt/jenkins]# kubectl apply -f nfs-client-provisioner.yaml
deployment.apps/nfs-client-provisioner created
[root@master01 /opt/jenkins]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-6775f9c6f4-xhq8c 1/1 Running 0 27s
创建 nfs-client-storageclass
[root@master01 /opt/jenkins]# vim nfs-client-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-client-storageclass
provisioner: nfs-storage
parameters:
archiveOnDelete: "true"
创建资源,
[root@master01 /opt/jenkins]# kubectl apply -f nfs-client-storageclass.yaml
storageclass.storage.k8s.io/nfs-client-storageclass created
[root@master01 /opt/jenkins]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-client-storageclass nfs-storage Delete Immediate false 23s
创建成功
安 装 Jenkins-Master
编写,Jenkins和master绑定的yaml
[root@master01 /opt/jenkins/master]# vim jenkins-master-rbac.yaml
apiVersion: v1
kind: Namespace
metadata:
name: kube-ops
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: kube-ops
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins
namespace: kube-ops
rules:
- apiGroups: ["extensions", "apps"]
resources: ["deployments"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: jenkins
namespace: kube-ops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkinsClusterRole
namespace: kube-ops
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: jenkinsClusterRuleBinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkinsClusterRole
subjects:
- kind: ServiceAccount
name: jenkins
namespace: kube-ops
[root@master01 /opt/jenkins/master]# kubectl apply -f jenkins-master-rbac.yaml
namespace/kube-ops created
serviceaccount/jenkins created
role.rbac.authorization.k8s.io/jenkins created
rolebinding.rbac.authorization.k8s.io/jenkins created
clusterrole.rbac.authorization.k8s.io/jenkinsClusterRole created
rolebinding.rbac.authorization.k8s.io/jenkinsClusterRuleBinding created
执行jenkins-master- StatefulSet(jenkins镜像)
在StatefulSet.yaml文件,声明了利用nfs-client-provisioner进行Jenkins-Master文件存储
volumeClaimTemplates:
- metadata:
name: jenkins-home
spec:
storageClassName: nfs-client-storageclass
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 2Gi
Service发布方法采用NodePort,会随机产生节点访问端口
spec:
selector:
app: jenkins
type: NodePort
ports:
- name: web
port: 8080
targetPort: web
- name: agent
port: 50000
targetPort: agent
具体yaml如下
[root@master01 /opt/jenkins/master]# vim jenkins-master-sts.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: jenkins
labels:
name: jenkins
namespace: kube-ops
spec:
serviceName: jenkins
selector:
matchLabels:
app: jenkins
replicas: 1
updateStrategy:
type: RollingUpdate
template:
metadata:
name: jenkins
labels:
app: jenkins
spec:
terminationGracePeriodSeconds: 10
serviceAccountName: jenkins
containers:
- name: jenkins
image: jenkins/jenkins:lts-alpine
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 0.5
memory: 500Mi
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
livenessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
securityContext:
fsGroup: 1000
volumeClaimTemplates:
- metadata:
name: jenkins-home
spec:
storageClassName: nfs-client-storageclass
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 2Gi
---
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: kube-ops
labels:
app: jenkins
spec:
selector:
app: jenkins
type: NodePort
ports:
- name: web
port: 8080
targetPort: web
- name: agent
port: 50000
targetPort: agent
[root@master01 /opt/jenkins/master]# kubectl apply -f jenkins-master-sts.yaml
statefulset.apps/jenkins created
service/jenkins created
[root@master01 /opt/jenkins/master]# kubectl get pods -n kube-ops
NAME READY STATUS RESTARTS AGE
jenkins-0 1/1 Running 0 2m4s
查看pv、pvc、svc
[root@master01 /opt/jenkins/master]# kubectl get pv,pvc,svc -n kube-ops
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-2ce93b2b-0f90-41a1-9369-2309f2eb073d 2Gi RWO Delete Bound kube-ops/jenkins-home-jenkins-0 nfs-client-storageclass 4m36s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/jenkins-home-jenkins-0 Bound pvc-2ce93b2b-0f90-41a1-9369-2309f2eb073d 2Gi RWO nfs-client-storageclass 4m36s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/jenkins NodePort 10.96.65.114 <none> 8080:32668/TCP,50000:30016/TCP 4m36s
此时nfs的服务器上会有Jenkins的工作目录
此时去浏览器输入http://192.168.37.125:32668/ (node1和node2都可以)
密码就是Jenkins的工作目录下的
[root@harbor /opt/nfs/jenkins/kube-ops-jenkins-home-jenkins-0-pvc-2ce93b2b-0f90-41a1-9369-2309f2eb073d/secrets]# cat initialAdminPassword ......
此时进入Jenkins的页面
如果要配置其他源的话,在Jenkins工作目录下的update\default.json修改
sed -i 's#http://www.google.com#https://www.baidu.com#g' default.json
sed -i 's#https://updates.jenkins.io/download#http://mirrors.tuna.tsinghua.edu.cn/jenkins#g' /updates/default.json
即可
安装插件
Manage Jenkins -> Manage Plugins ->点击 Availale ->安装以下几个插件
Localization: chinese Git Pipeline Extended choice Parameter #复选框插件 Kubernetes
重启一下,直接在浏览器restart
Jenkins与Kubernetes整合
系统管理->系统配置->云->新建云->Kubernetes
k8s的地址
[root@master01 /opt/jenkins/master]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-6775f9c6f4-xhq8c 1/1 Running 0 52m
[root@master01 /opt/jenkins/master]# kubectl exec -it nfs-client-provisioner-6775f9c6f4-xhq8c sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
/ # exit
Jenkins地址
[root@master01 /opt/jenkins/master]# kubectl get pods,svc -n kube-ops
NAME READY STATUS RESTARTS AGE
pod/jenkins-0 1/1 Running 0 42m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/jenkins NodePort 10.96.65.114 <none> 8080:32668/TCP,50000:30016/TCP 42m
[root@master01 /opt/jenkins/master]# kubectl exec -n kube-ops -it jenkins-0 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ $ cat /etc/resolv.conf
nameserver 10.96.0.10
search kube-ops.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
/ $
应用保存
此时Jenkins和k8s关联起来
构建Jenkins-slave自定义镜像
Jenkins -Master在构建Job 的时候,Kubernetes 会创建Jenkins-8lave 的 Pod 来完成Job的构建。我们选择运行uJenkins-8lave 的镜像为官方推荐镜像:jenkins/jnlp-slave:latest,但是这个镜像里面并没有Maven环境,为了方便使用,我们需要自定义一个新的镜像。
1)可在任意有Docker 的服务器中创建/opt/jenkins-slave:目录、并上传 apache-naven=3.6.2-bin.tar.gz、settinge:xml文件
[root@harbor /opt]# mkdir jenkins-slave
[root@harbor /opt]# cd jenkins
vim Dockerfile
FROM jenkins/jnlp-slave:latest
MAINTAINER Jenkins-Slave <xiaobin>
#切换到 root 账户进行操作
USER root
#安装 maven
ADD apache-maven-3.6.2-bin.tar.gz /usr/local/
RUN ln -s /usr/local/apache-maven-3.6.2/bin/mvn /usr/bin/mvn && \
ln -s /usr/local/apache-maven-3.6.2 /usr/local/apache-maven && \
mkdir -p /usr/local/apache-maven/repo
ADD settings.xml /usr/local/apache-maven/conf/settings.xml
USER jenkins
settings.xml
修改的代码
<localRepository>/usr/local/apache-maven/repo</localRepository>
<mirror>
<id>central</id>
<mirrorOf>central</mirrorOf>
<name>aliyun maven</name>
<url>https://maven.aliyun.com/repository/public</url>
</mirror>
构建镜像
[root@harbor /opt/jenkins-slave]# docker build -t jenkins-slave-maven:latest .
上传到harbor仓库(前提要有信任凭证)
[root@harbor /opt/jenkins-slave]# vim /etc/docker/daemon.json
"insecure-registries": ["192.168.37.106:85"],
上传镜像到harbor仓库
登录harbor(要有权限的 admin权限或者你的用户设置了高的权限)
[root@harbor ~]# docker login http://192.168.37.106:85
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
首先重新打标签
[root@harbor ~]# docker tag jenkins-slave-maven:latest 192.168.37.106:85/library/jenkins-slave-maven:latest
[root@harbor ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
192.168.37.106:85/library/jenkins-slave-maven latest 011525ced837 8 minutes ago 478MB
jenkins-slave-maven latest 011525ced837 8 minutes ago 478MB
上传到harbor
[root@harbor ~]# docker push 192.168.37.106:85/library/jenkins-slave-maven:latest
The push refers to repository [192.168.37.106:85/library/jenkins-slave-maven]
1c3ad25c6d80: Pushed
c9975ce81f34: Pushed
597d2bb4099f: Pushed
d5af25323a5c: Pushed
0053d86c4d81: Pushed
4cb0baa801b1: Pushed
3f01ba93adcb: Pushed
3dee86c3d230: Pushed
5ccb7c9ecca8: Pushed
8a510d97a0f7: Pushed
3f948fda930d: Pushed
cf7a8ba4ff71: Pushed
4e006334a6fd: Pushed
latest: digest: sha256:ec9b7103064a97873ac8fd4958b84e66d7ef414c9c9346eaf59bce70d8d71115 size: 3042
测试Jenkins-Slave是否可以创建
在Jenkins创建gitlab凭证
创建一个Jenkins流水线项目
新建任务-→>任务名称(test_jenkins_slave)流水线->确定
流水线->定义选择Pipeline script ->脚本如下
def git_address = "http://192.168.80.10:82/devops_group/tensquare_back.qit"
def git_auth = "7074f5d2-7fc9-4dfb-93f5-70edacb10a34"
//创建一个Pod的模板,label为jenkins-slave,cloud为之前添加的云节点kubernetes,Jenkins_Slave容器模板name必须为jnlp
流水线脚本
def git_url = "http://192.168.37.103:85/devops_group/tensquare_back.git"
def git_auth = "72a48f14-72c7-444f-a471-2d482e85d808"
podTemplate(cloud: 'kubernetes', label: 'Jenkins-slave',containers: [
containerTemplate(
image: '192.168.37.106:85/library/jenkins-slave-maven:latest',
name: 'jnlp')])
{
node('Jenkins-slave') {
stage('拉取代码') {
//切换成变量,字符串符号使用双引号
checkout([$class: 'GitSCM',
branches: [[name: "*/${branch}"]],
extensions: [],
userRemoteConfigs: [[credentialsId: "${git_auth}",
url: "${git_url}"]]])
}
}
}
应用保存
构建过程中,会有
等待执行完成,自动被回收
更多推荐
所有评论(0)