安装部署nfs

1. nfs服务器创建目录

[yeqiang@harbor ~]$ sudo mkdir /home/nfs/jenkins -p

2. 设置其他用户可以写入该目录

[yeqiang@harbor ~]$ sudo chmod o+w /home/nfs/jenkins/ -R

3. 配置/etc/exports

/home/nfs/jenkins 192.168.1.167(rw,sync,no_subtree_check,no_root_squash)
/home/nfs/jenkins/agent 192.168.1.167(rw,sync,no_subtree_check,no_root_squash)

4. 启动nfs服务，配置开启自启动

[yeqiang@harbor ~]$ sudo systemctl enable nfs-server --now
Created symlink /etc/systemd/system/multi-user.target.wants/nfs-server.service → /usr/lib/systemd/system/nfs-server.service.

5. 检查服务及端口(111/tcp)状态

[yeqiang@harbor ~]$ netstat -lnetp | grep 111
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      0          19646      -                   
tcp6       0      0 :::111                  :::*                    LISTEN      0          19650      -                   
[yeqiang@harbor ~]$ sudo exportfs -v
/home/nfs/jenkins
		192.168.1.167(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/home/nfs/jenkins/agent
		192.168.1.167(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)

创建k8s存储

1. 创建StorageClass，配置文件nfs-storege-class.yml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: nfs-storage
  annotations:
    storageclass.kubernetes.io/is-default-class: 'false'
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

执行创建

[yeqiang@harbor k8s]$ kubectl apply -f nfs-storege-class.yml 
storageclass.storage.k8s.io/nfs-storage created
[yeqiang@harbor k8s]$ kubectl get storageclass
NAME                      PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
nfs-storage               kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  6s

2. 创建PersistantVolume，配置文件jenkins-pv.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: "nfs-storage"
  nfs:
    # 根据实际共享目录修改
    path: /home/nfs/jenkins
    # 根据实际 nfs服务器地址修改
    server: 192.168.1.167

执行创建

[yeqiang@harbor k8s]$ kubectl apply -f jenkins-pv.yml 
persistentvolume/jenkins-pv created
[yeqiang@harbor k8s]$ kubectl get pv 
NAME                          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                               STORAGECLASS    REASON   AGE
jenkins-pv                    10Gi       RWO            Recycle          Available                                       nfs-storage              6s

helm部署jenkins

1. helm查看jenkins信息

[yeqiang@harbor k8s]$ helm repo add jenkins https://charts.jenkins.io
"jenkins" has been added to your repositories
[yeqiang@harbor k8s]$ helm search repo jenkins
NAME           	CHART VERSION	APP VERSION	DESCRIPTION                                       
jenkins/jenkins	2.15.1       	lts        	Open source continuous integration server. It s...
stable/jenkins 	2.5.4        	lts        	DEPRECATED - Open source continuous integration...

注意stable/jenkins已经标注过时了，我们选择jenkins/jenkins

2. 创建namespace

[yeqiang@harbor k8s]$ kubectl create ns jenkins
namespace/jenkins created

3. helm安装

[yeqiang@harbor k8s]$ helm install jenkins jenkins/jenkins -n jenkins --set persistence.storageClass="nfs-storage"
NAME: jenkins
LAST DEPLOYED: Thu Oct 29 10:25:54 2020
NAMESPACE: jenkins
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:
  printf $(kubectl get secret --namespace jenkins jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
2. Get the Jenkins URL to visit by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace jenkins -l "app.kubernetes.io/component=jenkins-master" -l "app.kubernetes.io/instance=jenkins" -o jsonpath="{.items[0].metadata.name}")
  echo http://127.0.0.1:8080
  kubectl --namespace jenkins port-forward $POD_NAME 8080:8080

3. Login with the password from step 1 and the username: admin

4. Use Jenkins Configuration as Code by specifying configScripts in your values.yaml file, see documentation: http:///configuration-as-code and examples: https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos

For more information on running Jenkins on Kubernetes, visit:
https://cloud.google.com/solutions/jenkins-on-container-engine
For more information about Jenkins Configuration as Code, visit:
https://jenkins.io/projects/jcasc/

参数说明：

persistence.storageClass 赋值为刚才创建的StorageClass名称nfs-sotrage

4. 故障排除

[yeqiang@harbor k8s]$ kubectl get pods -n jenkins
NAME                       READY   STATUS    RESTARTS   AGE
jenkins-58978c6d6d-dszqz   1/2     Running   2          6m8s
[yeqiang@harbor k8s]$ kubectl describe pod jenkins-58978c6d6d-dszqz -n jenkins
Name:         jenkins-58978c6d6d-dszqz
Namespace:    jenkins
Priority:     0
Node:         192.168.1.167/192.168.1.167
Start Time:   Thu, 29 Oct 2020 10:25:55 +0800
Labels:       app.kubernetes.io/component=jenkins-master
              app.kubernetes.io/instance=jenkins
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=jenkins
              pod-template-hash=58978c6d6d
Annotations:  checksum/config: 9823a20535eb4e992b6741805b204c7c82d33080079887578e27c24328df7b29
Status:       Running
IP:           172.20.0.108
IPs:
  IP:           172.20.0.108
Controlled By:  ReplicaSet/jenkins-58978c6d6d
Init Containers:
  copy-default-config:
    Container ID:  docker://e29d8d6285ddebaf565a206213da47aac723ae79a883ff6108aafd7108a5b1fd
    Image:         jenkins/jenkins:lts
    Image ID:      docker-pullable://jenkins/jenkins@sha256:aa1ec2a6a106608a286dc06c334053fced1df87a7cb307853ca1cf8fa48ba09b
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      /var/jenkins_config/apply_config.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 29 Oct 2020 10:25:58 +0800
      Finished:     Thu, 29 Oct 2020 10:26:40 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:     50m
      memory:  256Mi
    Environment:
      ADMIN_PASSWORD:  <set to the key 'jenkins-admin-password' in secret 'jenkins'>  Optional: false
      ADMIN_USER:      <set to the key 'jenkins-admin-user' in secret 'jenkins'>      Optional: false
    Mounts:
      /tmp from tmp (rw)
      /usr/share/jenkins/ref/plugins from plugins (rw)
      /var/jenkins_config from jenkins-config (rw)
      /var/jenkins_home from jenkins-home (rw)
      /var/jenkins_plugins from plugin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-krt5l (ro)
Containers:
  jenkins:
    Container ID:  docker://13cb74bdc25782b5bf7258fc4c369544571866c3a029dbc26f9ae489fb7b742b
    Image:         jenkins/jenkins:lts
    Image ID:      docker-pullable://jenkins/jenkins@sha256:aa1ec2a6a106608a286dc06c334053fced1df87a7cb307853ca1cf8fa48ba09b
    Ports:         8080/TCP, 50000/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      --argumentsRealm.passwd.$(ADMIN_USER)=$(ADMIN_PASSWORD)
      --argumentsRealm.roles.$(ADMIN_USER)=admin
      --httpPort=8080
    State:          Running
      Started:      Thu, 29 Oct 2020 10:31:24 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Thu, 29 Oct 2020 10:29:01 +0800
      Finished:     Thu, 29 Oct 2020 10:31:18 +0800
    Ready:          False
    Restart Count:  2
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      50m
      memory:   256Mi
    Liveness:   http-get http://:http/login delay=90s timeout=5s period=10s #success=1 #failure=5
    Readiness:  http-get http://:http/login delay=60s timeout=5s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:                  jenkins-58978c6d6d-dszqz (v1:metadata.name)
      JAVA_OPTS:                 -Dcasc.reload.token=$(POD_NAME)
      JENKINS_OPTS:              
      JENKINS_SLAVE_AGENT_PORT:  50000
      ADMIN_PASSWORD:            <set to the key 'jenkins-admin-password' in secret 'jenkins'>  Optional: false
      ADMIN_USER:                <set to the key 'jenkins-admin-user' in secret 'jenkins'>      Optional: false
      CASC_JENKINS_CONFIG:       /var/jenkins_home/casc_configs
    Mounts:
      /tmp from tmp (rw)
      /usr/share/jenkins/ref/plugins/ from plugin-dir (rw)
      /var/jenkins_config from jenkins-config (ro)
      /var/jenkins_home from jenkins-home (rw)
      /var/jenkins_home/casc_configs from sc-config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-krt5l (ro)
  jenkins-sc-config:
    Container ID:   docker://a33d84511d702b29e1d1b206793666bb360f071e0f7577a24f660c8a0d4eef1b
    Image:          kiwigrid/k8s-sidecar:0.1.193
    Image ID:       docker-pullable://kiwigrid/k8s-sidecar@sha256:170069ff097679f8fafcf01510d1fe05260af683b949fd2b05866e18ca4d9c74
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 29 Oct 2020 10:26:48 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:           jenkins-58978c6d6d-dszqz (v1:metadata.name)
      LABEL:              jenkins-jenkins-config
      FOLDER:             /var/jenkins_home/casc_configs
      NAMESPACE:          jenkins
      REQ_URL:            http://localhost:8080/reload-configuration-as-code/?casc-reload-token=$(POD_NAME)
      REQ_METHOD:         POST
      REQ_RETRY_CONNECT:  10
    Mounts:
      /var/jenkins_home from jenkins-home (rw)
      /var/jenkins_home/casc_configs from sc-config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-krt5l (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  plugins:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  jenkins-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      jenkins
    Optional:  false
  plugin-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  jenkins-home:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  jenkins
    ReadOnly:   false
  sc-config-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  jenkins-token-krt5l:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  jenkins-token-krt5l
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                   Message
  ----     ------     ----                   ----                   -------
  Normal   Scheduled  6m10s                  default-scheduler      Successfully assigned jenkins/jenkins-58978c6d6d-dszqz to 192.168.1.167
  Normal   Pulling    6m9s                   kubelet, 192.168.1.167  Pulling image "jenkins/jenkins:lts"
  Normal   Pulled     6m8s                   kubelet, 192.168.1.167  Successfully pulled image "jenkins/jenkins:lts"
  Normal   Created    6m7s                   kubelet, 192.168.1.167  Created container copy-default-config
  Normal   Started    6m7s                   kubelet, 192.168.1.167  Started container copy-default-config
  Normal   Pulled     5m18s                  kubelet, 192.168.1.167  Successfully pulled image "jenkins/jenkins:lts"
  Normal   Created    5m18s                  kubelet, 192.168.1.167  Created container jenkins
  Normal   Started    5m17s                  kubelet, 192.168.1.167  Started container jenkins
  Normal   Pulled     5m17s                  kubelet, 192.168.1.167  Container image "kiwigrid/k8s-sidecar:0.1.193" already present on machine
  Normal   Created    5m17s                  kubelet, 192.168.1.167  Created container jenkins-sc-config
  Normal   Started    5m17s                  kubelet, 192.168.1.167  Started container jenkins-sc-config
  Warning  Unhealthy  3m11s (x7 over 4m11s)  kubelet, 192.168.1.167  Readiness probe failed: Get http://172.20.0.108:8080/login: dial tcp 172.20.0.108:8080: connect: connection refused
  Normal   Pulling    3m7s (x2 over 5m24s)   kubelet, 192.168.1.167  Pulling image "jenkins/jenkins:lts"
  Normal   Killing    3m7s                   kubelet, 192.168.1.167  Container jenkins failed liveness probe, will be restarted
  Warning  Unhealthy  67s (x8 over 3m47s)    kubelet, 192.168.1.167  Liveness probe failed: Get http://172.20.0.108:8080/login: dial tcp 172.20.0.108:8080: connect: connection refused

Liveness probe failed：

把deployments里面的jenkins配置项目failureThreshold改为10

原因：jenkins默认引用的国外更新源，初始化时间较长，pod还未就绪就被k8s杀死了。

可能应为网络问题，pod可能还会反创建。大约十几分钟后。可以正常访问

5. 访问Jenkins

查看访问地址

[yeqiang@harbor k8s]$ kubectl get svc -n jenkins
NAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)     AGE
jenkins         ClusterIP   10.68.62.85   <none>        8080/TCP    26m
jenkins-agent   ClusterIP   10.68.46.25   <none>        50000/TCP   26m

查看登陆用户及密码

[yeqiang@harbor k8s]$ kubectl edit secret jenkins -n jenkins
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  jenkins-admin-password: clM5bXgxMUNvQw==
  jenkins-admin-user: YWRtaW4=
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: jenkins
    meta.helm.sh/release-namespace: jenkins
  creationTimestamp: "2020-10-29T02:25:54Z"
  labels:
    app.kubernetes.io/component: jenkins-master
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-2.15.1
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:jenkins-admin-password: {}
        f:jenkins-admin-user: {}
      f:metadata:
        f:annotations:
          .: {}
          f:meta.helm.sh/release-name: {}
          f:meta.helm.sh/release-namespace: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/component: {}
          f:app.kubernetes.io/instance: {}
          f:app.kubernetes.io/managed-by: {}
          f:app.kubernetes.io/name: {}
          f:helm.sh/chart: {}
      f:type: {}
    manager: Go-http-client
    operation: Update
    time: "2020-10-29T02:25:54Z"
  name: jenkins
  namespace: jenkins
  resourceVersion: "2557784"
  selfLink: /api/v1/namespaces/jenkins/secrets/jenkins
  uid: e8eb14b7-2e7c-42b4-83b5-39889042e72b
type: Opaque
[yeqiang@harbor k8s]$ echo 'YWRtaW4=' | base64 -d
admin
[yeqiang@harbor k8s]$ echo 'clM5bXgxMUNvQw==' | base64 -d
rS9mx11CoC

得到用户admin，密码rS9mx11CoC

登陆系统

创建jenkins PipeLine构建部署项目

1. 查看jenkins cloud 配置

2. 编写pipeline

def label = "jenkins-slave"
podTemplate(
    label: label, 
    cloud: 'kubernetes', 
    containers: [containerTemplate(alwaysPullImage: true, args: 'cat', command: '/bin/sh -c', image: 'maven:3.6.3-openjdk-8', livenessProbe: containerLivenessProbe(execArgs: '', failureThreshold: 0, initialDelaySeconds: 0, periodSeconds: 0, successThreshold: 0, timeoutSeconds: 0), name: 'maven', resourceLimitCpu: '', resourceLimitMemory: '', resourceRequestCpu: '', resourceRequestMemory: '', ttyEnabled: true, workingDir: '/home/jenkins/agent')],     
    workspaceVolume: nfsWorkspaceVolume(serverAddress: '192.168.1.167', serverPath: '/home/nfs/jenkins/agent', readOnly: 'false')
) {
    node(label) {
        stage('git') {
            git 'https://github.com/hknarutofk/spring-demo.git'
        }
        
        stage('mvn'){
	    container('maven'){	        
		sh '''
cat > /usr/share/maven/conf/settings.xml << EOF
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
	  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <localRepository>/root/.m2</localRepository>
  <mirrors>
    <mirror>
      <id>alimaven</id>
      <name>aliyun maven</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
      <mirrorOf>central</mirrorOf>        
    </mirror>
  </mirrors>
</settings>
EOF
'''		
        
		sh "mvn clean package -Dmaven.test.skip=true"
	    }            
        }
    }
}

故障排除：

Could not resolve host

放弃使用镜像maven:3.3.9-jdk-8-alpine，采用maven:3.6.3-openjdk-8。原因：maven:3.3.9-jdk-8-alpine对/etc/resolv.conf解析有故障

# k8s # jenkins
Logo
K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

cover

kubernetes(k8s)安装教程_安装kubernetes

avatar K8S/Kubernetes
cover

k8s集群部署（sealos）

avatar K8S/Kubernetes
cover

Day97：云上攻防-云原生篇&Kubernetes&K8s安全&API&Kubelet未授权访问&容器执行

avatar K8S/Kubernetes