在k8s上面运行jenkins

自定义jenkins镜像jenkins镜像生成过程中安装插件新的改变功能快捷键合理的创建标题，有助于目录的生成如何改变文本的样式插入链接与图片如何插入一段漂亮的代码片生成一个适合你的列表创建一个表格设定内容居中、居左、居右SmartyPants创建一个自定义列表如何创建一个注脚注释也是必不可少的KaTeX数学公式新的甘特图功能，丰富你的文章UML 图表FLowchart流程图导出与导入导出导入je

月夜楓

1358人浏览 · 2020-07-10 16:46:38

月夜楓 · 2020-07-10 16:46:38 发布

自定义jenkins镜像

使用官方的基础镜像
在k8s上面拉起jenkins
jenkins-slave
在jenkins上面配置jenkins：
- 异常处理：
- - 错误信息

使用官方的基础镜像

先写一个总的，使用官方提供的jenkins master的基础镜像，构建自己的dockerfile，如下：

FROM jenkins/jenkins:2.256
LABEL maintainer=caoyong1
USER root
WORKDIR /usr/local/
COPY src/ .
ENV JENKINS_UC=https://updates.jenkins.io \
    JENKINS_UC_DOWNLOAD=http://mirror.xmission.com/jenkins \
    JENKINS_OPTS="-Dhudson.model.UpdateCenter.updateCenterUrl=https://cdn.jsdelivr.net/gh/jenkins-zh/update-center-mirror/tsinghua/current/update-center.json"
RUN echo ""> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian stretch main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian stretch main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian stretch-updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian stretch-updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian-security stretch/updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian-security stretch/updates main contrib non-free" >> /etc/apt/sources.list && \
    cat /etc/resolv.conf && apt-get update && apt-get install -y --allow-unauthenticated --assume-yes apt-utils lsb-core bridge-utils traceroute ca-certificates curl netbase wget vim git curl unzip net-tools iputils-ping && ln -sf /usr/local/Asia/Shanghai /etc/localtime && sed -i "20i set -e" /usr/local/bin/install-plugins.sh && mkdir -p /usr/share/jenkins/ref/plugins/ && mv /usr/local/plugins.txt /usr/share/jenkins/ref/plugins/ && /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins/plugins.txt && date
EXPOSE 8080
EXPOSE 50000
ENTRYPOINT ["/sbin/tini","--","/usr/local/bin/jenkins.sh"]

引入工具集

引入jdk、nodejs、maven、sonarqube等
在dockerfile中是这个位置：
在这里插入图片描述
目录结构是这样的：

src/
├── Asia
├── jdk
│   ├── jdk-11.0.7
│   └── jdk1.8.0_181
├── maven
│   └── 3.6.3
├── node
│   └── node-v12
├── plugins.txt
└── sonar
    └── sonar-scanner-4.4

jenkins镜像生成过程中安装插件

需要在jenkins 插件的官网的镜像上面，找到插件id，将需要安装的插件的id以及版本号写入到插件的列表文件plugins.txt中，如下：

cloudbees-folder:6.14
antisamy-markup-formatter:2.1
build-timeout:1.20
credentials-binding:1.23
timestamper:1.11.5
ws-cleanup:0.38
ant:1.11
gradle:1.36
pipeline-stage-view:2.15
pipeline-github-lib:1.0
github-branch-source:2.9.0
workflow-aggregator:2.6
git:4.3.0
subversion:2.13.1
ssh-slaves:1.31.2
matrix-auth:2.6.2
mailer:1.32
email-ext:2.75
ldap:1.24
pam-auth:1.6
localization-zh-cn:1.0.18
generic-webhook-trigger:1.67
pipeline-utility-steps:2.6.1
kubernetes:1.27.1
kubernetes-credentials:0.7.0
kubernetes-pipeline-devops-steps:1.6
kubernetes-cli:1.8.3
kubernetes-client-api:4.9.2-2
nodejs:1.3.7
sonar:2.12

将插件的更新网址和下载地址进行环境变量的配置，如上面的dockerfile中：

ENV JENKINS_UC=https://updates.jenkins.io \
    JENKINS_UC_DOWNLOAD=http://mirror.xmission.com/jenkins \
    JENKINS_OPTS="-Dhudson.model.UpdateCenter.updateCenterUrl=https://cdn.jsdelivr.net/gh/jenkins-zh/update-center-mirror/tsinghua/current/update-center.json"

使用debian源

国外的源下载忒慢了，更新为阿里源：

RUN echo ""> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian stretch main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian stretch main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian stretch-updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian stretch-updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian-security stretch/updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian-security stretch/updates main contrib non-free" >> /etc/apt/sources.list && \

安装必要的工具软件，并且将插件安装在镜像构建的时候创建完成：

RUN apt-get update && apt-get install -y --allow-unauthenticated --assume-yes apt-utils lsb-core bridge-utils traceroute ca-certificates curl netbase wget vim git curl unzip net-tools iputils-ping && ln -sf /usr/local/Asia/Shanghai /etc/localtime && sed -i "20i set -e" /usr/local/bin/install-plugins.sh && mkdir -p /usr/share/jenkins/ref/plugins/ && mv /usr/local/plugins.txt /usr/share/jenkins/ref/plugins/ && /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins/plugins.txt && date

在k8s上面拉起jenkins

master.yaml如下：

apiVersion: v1
kind: Namespace
metadata:
  name: devops
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: master-deploy
  namespace: devops
  labels:
    app: jenkins-deploy
spec:
  serviceName: master
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
      jenkins: master
  template:
    metadata:
      labels:
        app: jenkins
        jenkins: master
    spec:
      containers:
      - name: master
        image: 10.110.152.173/jenkins/master:v1.0.0.3
        imagePullPolicy:  Always
        ports:
          - containerPort: 8080
            name: http-port
          - containerPort: 50000
            name: jnlp-port
        env:
        - name: JAVA_OPTS
          value: -Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=true
        volumeMounts:
        - name: jenkins-home
          mountPath: /var/jenkins_home
        - name: maven-cache
          mountPath: /usr/local/.repository
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: master-home
      - name: maven-cache
        persistentVolumeClaim:
          claimName: maven-cache
---
apiVersion: v1
kind: Service
metadata:
  name: ui
  namespace: devops
spec:
  selector:
    app: jenkins
    jenkins: master
  ports:
  - name: http-port
    protocol: TCP
    port: 80
    targetPort: 8080
    nodePort: 32456
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  name: master
  namespace: devops
spec:
  selector:
    app: jenkins
    jenkins: master
  ports:
  - name: jnlp-port
    port: 80
    targetPort: 50000
    protocol: TCP

jenkins-slave

构建jenkins-slave镜像，添加必要的工具集，如：docker-cli,kubectl,pip(python)等
如下：

FROM jenkins/jnlp-slave:4.3-4
LABEL maintainer=caoyong1
WORKDIR /usr/local/
COPY src/ .
USER root
RUN rm -rf /etc/apt/sources.list.d/stretch-backports.list && echo ""> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian stretch main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian stretch main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian stretch-updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian stretch-updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian-security stretch/updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian-security stretch/updates main contrib non-free" >> /etc/apt/sources.list
ARG DEBIAN_FRONTEND=noninteractive
RUN ln -sf /usr/local/Asia/Shanghai /etc/localtime && uname -a && cat /etc/debian_version &&  dpkg -i containerd.io_1.2.13-2_amd64.deb && dpkg -i docker-ce-cli_19.03.12_3-0_debian-stretch_amd64.deb &&  echo "10.110.149.169  harbor.aipo.lenovo.com" >> /etc/hosts && apt-get update && python -V && apt-get install python-pip -y && pip install nose coverage nosexcover pylint&& pip install s3cmd && cp /usr/local/.s3cfg /root/ && dpkg -i kubectl_1.18.6-00_amd64_339673592303e280949d52b2f4e842ad365529a5d656dd9f6483517651e2ded8.deb && mkdir -p /root/.kube/ && touch /root/.kube/config && chown $(id -u):$(id -g) /root/.kube/config &&  cat /usr/local/admin.conf >> /root/.kube/config && kubectl version && date
ENTRYPOINT ["jenkins-agent"]

src目录结构如下：

src
├── admin.conf
├── .s3cfg
├── Asia
├── containerd.io_1.2.13-2_amd64.deb
├── docker-ce-cli_19.03.12_3-0_debian-stretch_amd64.deb
├── jdk
│   ├── jdk-11.0.7
│   └── jdk1.8.0_181
├── kubectl_1.18.6-00_amd64_339673592303e280949d52b2f4e842ad365529a5d656dd9f6483517651e2ded8.deb
├── maven
│   └── 3.6.3
├── node
│   └── node-v12
├── plugins.txt
└── sonar
    └── sonar-scanner-4.4

在jenkins上面配置jenkins：

jenkins-k8s plugin配置
设置两个label
在这里插入图片描述
添加jenkins slave镜像：

在这里插入图片描述
当然，也有实践中，使用initContainer来进行工具集的添加，那样更具有扩展性，本文主要是为了减少每次启动时候，都要启动initContainer工具镜像的消耗，所以，在slave上面预先安装了不少工具集。

异常处理：

发现启动job的过程中，不能启动slave pod的情况，原因可能是devops（默认账号没有创建pod的权限），将账号的权限（角色）设置为最大：cluster-admin，可以解决该问题。

错误信息

2021-10-12 07:24:32.295+0000 [id=31]    INFO    hudson.slaves.NodeProvisioner#lambda$update$6: slave-6n6gq provisioning successfully completed. We have now 2 computer(s)
Disconnected computer slave-6n6gq
2021-10-12 07:24:32.326+0000 [id=554]   WARNING o.c.j.p.k.KubernetesSlave#_terminate: Slave pod slave-6n6gq was not deleted due to retention policy On Failure.
2021-10-12 07:24:32.326+0000 [id=554]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer slave-6n6gq
2021-10-12 07:24:42.263+0000 [id=35]    INFO    o.c.j.p.k.KubernetesCloud#provision: Excess workload after pending Kubernetes agents: 1
2021-10-12 07:24:42.263+0000 [id=35]    INFO    o.c.j.p.k.KubernetesCloud#provision: Template for label null: slave
2021-10-12 07:24:52.272+0000 [id=554]   WARNING o.c.j.p.k.KubernetesLauncher#launch: Error in provisioning; agent=KubernetesSlave name: slave-scj1d, template=PodTemplate{inheritFrom='', name='slave', namespace='devops', hostNetwork=false, slaveConnectTimeout=180, label='jenkins-slave', nodeSelector='', nodeUsageMode=NORMAL, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock], org.csanchez.jenkins.plugins.kubernetes.volumes.PersistentVolumeClaim@a77c69f], containers=[ContainerTemplate{name='jnlp', image='10.110.147.181:89/jenkins/slave:v4.3.4.13', workingDir='/home/jenkins/agent', command='', args='${computer.jnlpmac} ${computer.name}', ttyEnabled=true, resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', envVars=[KeyValueEnvVar [getValue()=http://ui.devops.svc.cluster.local:80, getKey()=JENKINS_URL], KeyValueEnvVar [getValue()=/etc/kubernetes/ssl, getKey()=KUBE_CONFIG]], livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@5e7fa1ad}]}
java.lang.NullPointerException
        at org.csanchez.jenkins.plugins.kubernetes.PodUtils.cancelQueueItemFor(PodUtils.java:76)
        at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:138)
        at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:294)
        at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
        at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2021-10-12 07:24:52.272+0000 [id=554]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent slave-scj1d

kubectl create clusterrolebinding devops-permissive-binding --clusterrole=cluster-admin  --serviceaccount=devops:default

本文引用：
https://cloud.tencent.com/developer/article/1590856
https://blog.csdn.net/u014249394/article/details/90272768
https://github.com/jenkin/docker/blob/master/README.md

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub