CI/CD常用服务部署

Jenkins && Github

docker-compose文件

version: "3"

services:
  jenkins:
    container_name: jenkins
    image: jenkins:latest
    restart: always
    user: root
    ports:
      - 8080:8080
      - 50000:50000
    volumes:
      - /data/jenkins/jenkins_home:/var/jenkins_home
      - /etc/localtime:/etc/localtime
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G

  gitlab:
    container_name: gitlab
    image: gitlab/gitlab-ce:latest
    restart: always
    environment:
      TZ: 'Asia/Shanghai'
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://192.168.2.10'
    ports:
      - 33022:22/tcp
      - 80:80
      - 443:443
    volumes:
      - /data/gitlab/config:/etc/gitlab
      - /data/gitlab/data:/var/opt/gitlab
      - /data/gitlab/logs:/var/log/gitlab
      - /etc/localtime:/etc/localtime
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G

Harbor镜像仓库

前置条件: 已安装docker-compose

下载地址

https://github.com/goharbor/harbor/releases

tar -zxf harbor-offline-installer-v2.1.0.tgz
mv harbor /usr/local/
cp harbor.yml.tmpl  harbor.yml

## 编辑配置
vim harbor.yml
hostname: 192.168.2.10
# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 85
# https related config
#https:
  # https port for harbor, default is 443
  #  port: 443
  # The path of cert and key files for nginx
  # certificate: /your/certificate/path
  # private_key: /your/private/key/path
# 修改data目录
data_volume: /data/harbor/data
# 修改log目录
	location: /data/harbor/log/harbor
    

## 安装
./install.sh

访问: ip:85
默认账户密码：admin/Harbor12345

配置项目,用户
docker登录私有仓库

docker login --username=wangshui898 http://192.168.2.10:85

启动关闭
进入目录下

docker-compose up -d 启动
docker-compose stop 停止
docker-compose restart 重新启动

上传镜像

配置daemon.json,把镜像仓库地址设置为可信地址

vi /etc/docker/daemon.json
{
"registry-mirrors": ["https://gsm39obv.mirror.aliyuncs.com"],
"insecure-registries": ["192.168.2.10:85"]
}

重启docker生效

systemctl restart docker

Jenkins添加Kubenetes集群

生成所需K8S集群证书

admin帐号请求证书json文件

cat >> admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "hangzhou",
      "L": "XS",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

生成admin证书

  • 前置条件: 已有k8s证书生成工具,有集群ca相关配置和公钥私钥(说白了就是部署集群时候用的那套工具以及ca配置和证书)
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
ls
admin.csr  admin-csr.json  admin-key.pem  admin.pem

生成pfx格式证书

生成pfx格式证书,并下载到电脑

openssl pkcs12 -export -out ./jenkins-admin.pfx -inkey ./admin-key.pem -in ./admin.pem -passout pass:pfx_passwd
ls jenkins-admin.pfx 
jenkins-admin.pfx

注意: -passout 参数后的pass:pfx_passwd ,pfx_passwd是fpx文件格式的密码,上传jenkins时候需要

证书上传到jenkins

在这里插入图片描述

在凭据管理中选择Certificate类型–>Upload PKCS#12 certificate证书,并根据提示上传之前的jenkins-admin.pfx

注意填写密码,点击Change Password,填入刚刚设置的pfx_passwd

配置jenkins云节点,添加kubenetes集群

插件: kubenetes

方法一:

系统管理-> 系统配置 -> cloud;点击跳转连接a separate configuration page,会跳转到k8s集群配置页面

方法二:

系统管理-> 节点管理 -> Configure Clouds

选择 Add a new cloud -> kubenetes,点击Kubernetes Cloud details…进行配置

  1. 集群名称随意填写
  2. Kubernetes 地址填写apiserver对外访问地址
  3. Kubernetes 服务证书 key复制ca.pem公钥内容
  4. 凭据选择上传的pfx证书

在这里插入图片描述

点击连接测试,如出现success(老版本)或者Connected to Kubernetes v1.18.10,表示连接k8s集群成功

如果jenkins部署在k8s集群内部,则Kubernetes地址只需要填写k8s集群名称即可,可通过kubectl get svc获取集群名称

制作jenkins-slave镜像

获取agent.jar

也叫slave.jar,实际jenkins中的remoting

官网地址

https://www.jenkins.io/zh/projects/remoting/

jar包下载地址

https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/

获取jenkins-slave

git地址

https://github.com/jenkinsci/docker-inbound-agent/blob/master/jenkins-agent

编辑Dockerfile

这里只加了jdk环境和maven环境,可以根据实际情况添加

ls -1
apache-maven-3.6.3-bin.tar.gz
Dockerfile
jdk-8u251-linux-x64.tar.gz
jenkins-agent
remoting-3.40.1.jar
remoting-4.6.jar
settings.xml
kubectl

FROM centos:7

RUN yum install git crul wget openssl openssl-devel\
	&& mkdir -p /usr/share/jenkins

ADD  jdk-8u251-linux-x64.tar.gz /usr/local/
ADD  apache-maven-3.6.3-bin.tar.gz /usr/local/
COPY settings.xml /usr/local/apache-maven-3.6.3/conf/settings.xml
COPY jenkins-agent /usr/local/bin/jenkins-slave
COPY remoting-3.40.1.jar /usr/share/jenkins/agent.jar
COPY kubectl /sbin/kubectl

RUN chmod 755 /usr/share/jenkins/agent.jar \
        && chmod +x /usr/local/bin/jenkins-slave

USER root

WORKDIR /home/jenkins

ENV M3_HOME=/usr/local/apache-maven-3.6.3
ENV JAVA_HOME=/usr/local/jdk1.8.0_251
ENV PATH=${PATH}:${JAVA_HOME}/bin:${M3_HOME}/bin

ENTRYPOINT ["jenkins-slave"]
#ENTRYPOINT ["/bin/ping"]
#CMD ["www.baidu.com"]

remoting.jar具体重命名和所放目录,要参照jenkins-slave脚本的执行命令参数来修改

使用jenkins部署项目到K8S

生成kube-admin.kubeconfig文件

插件: Kubernetes Continuous Deploy

生成admin用户的kubeconfig文件

ls admin*
admin.csr  admin-csr.json  admin-key.pem  admin.pem
ls -1 /opt/k8s/ssl/ca* 
ca-key.pem	ca.pem

# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/k8s/ssl/ca.pem \
--embed-certs=true \
--server=https://192.168.2.100:6443 \
--kubeconfig=kube-admin.kubeconfig

# 设置客户端认证参数-设置证书信息
kubectl config set-credentials admin \
--client-certificate=/data/TLS/k8s/admin.pem \
--client-key=/data/TLS/k8s/admin-key.pem \
--embed-certs=true \
--kubeconfig=kube-admin.kubeconfig

# 设置上下文参数-绑定账号和管理的集群
kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=admin \
--kubeconfig=kube-admin.kubeconfig

# 设置默认上下文
kubectl config use-context kubernetes \
--kubeconfig=kube-admin.kubeconfig

–embed-certs=true表示把证书内容写入到文件中

将kube-admin.kubeconfig保存到jenkins

安装好Kubernetes Continuous Deploy插件后,凭据管理中选择Kubernetes configuration (kubeconfig)–>Enter directily,将kube-admin.kubeconfig内容拷贝到输入框,保存.

jenkins发布到K8S逻辑说明

jenkins生成动态jenkins-slave节点,里面有kubectl命令,通过凭据保存的kube-admin.kubeconfig证书文件,在节点内保存到/root/.kube/config,动态节点即可操作集群,通过项目代码内的deployment.yaml发布到集群

配置文件示例

deployment.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: NS_NAME 
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: APP_NAME
  namespace: NS_NAME
  labels:
    app: APP_NAME
spec:
  replicas: RS_NUM
  selector:
    matchLabels:
      app: APP_NAME
  template:
    metadata:
      labels:
        app: APP_NAME
    spec:
      imagePullSecrets:
      - name: SECRET_NAME
      containers:
      - image: IMAGE_NAME
        name: APP_NAME        
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: APP_NAME
  name: APP_NAME-service
  namespace: NS_NAME
spec:
  ports:
  - port: 8091
    protocol: TCP
    targetPort: 8080
  selector:
    app: APP_NAME

Jenkinsfile

// 插件: pipeline / git / Git Parameter / Kubernetes / Kubernetes Continuous Deploy

// 项目信息
def project_name = "apps"
def app_name = "book"
// 镜像信息
def harbor_url = "192.168.2.10:85"
def harbor_auth = "715b0f10-d7ae-4a8a-bcc7-d5a188f5ba23"
def image_tag = "${BUILD_NUMBER}"
def image_name = "${harbor_url}/${project_name}/${app_name}:${image_tag}"
// Git信息
def git_url = "http://192.168.2.10/ipanel/book.git"
def git_auth = "715b0f10-d7ae-4a8a-bcc7-d5a188f5ba23"
// k8s集群信息
def k8s_auth = ""
def k8s_kubeconfig_auth = "a37784ad-f7f2-49df-96f5-f1dc9b5ebd6b"
def Secret_Name = "jenkins-slave-secret"
pipeline {
    agent {
        kubernetes {
            cloud 'kubernetes'
            defaultContainer 'jnlp'
            yaml """
                apiVersion: v1
                kind: Pod
                spec:
                  containers:
                  - name: jnlp
                    image: ${harbor_url}/library/jenkins-slave:v1
                    imagePullPolicy: Always
                    volumeMounts:
                    - mountPath: /etc/resolv.conf
                      name: resolv
                      readOnly: false
                    - mountPath: /var/run/docker.sock
                      name: docker-sock
                      readOnly: false
                    - mountPath: /data/maven/repos
                      name: maven-date
                      readOnly: false
                    - mountPath: /usr/bin/docker
                      name: docker-bin
                      readOnly: false
                    - mountPath: /home/jenkins/agent
                      name: agent-home
                      readOnly: false
                  nodeSelector:
                    kubernetes.io/os: linux
                  restartPolicy: Never
                  
                  volumes:
                  - hostPath:
                      path: /usr/bin/docker
                    name: docker-bin
                  - hostPath:
                      path: /etc/resolv.conf
                    name: resolv
                  - hostPath:
                      path: /var/run/docker.sock
                    name: docker-sock
                  - emptyDir:
                    name: agent-home
                  - name: maven-date
                    nfs:
                      path: /data/NFS/maven
                      readOnly: false
                      server: 192.168.2.10
                """
            
        }
    }
    parameters {
    gitParameter name: 'Branch', 
                 type: 'PT_BRANCH',
                 branchFilter: '.*',
                 defaultValue: 'master',
                 selectedValue: 'NONE',
                 sortMode: 'NONE',
				 description: 'Select your branch or tag.'
    choice(name: 'ReplicasNum', choices: ['1', '3', '5'], description: '请选择副本数')
	choice(name: 'NameSpace', choices: ['default', 'dev', 'apps'], description: '请选择命名空间')
    }
    stages {
        stage('源码拉取') {
            steps {
                container('jnlp') {
                    checkout([$class: 'GitSCM', 
                    branches: [[name: "${Branch}"]], 
                    doGenerateSubmoduleConfigurations: false, 
                    extensions: [], submoduleCfg: [], 
                    userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_url}"]]])
                    sh 'ls'
                }
            }
        }
        stage('代码构建') {
            steps {
                container('jnlp') {
                    sh "mvn clean package -Dmaven.test.skip=true"
                }
            }
        }
        stage('镜像构建') {
            steps {
                container('jnlp') {
                    withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'passwd', usernameVariable: 'user')]) {
                        sh """
						echo "Docker Login"
                        docker login -u "$user" -p "$passwd" "${harbor_url}"
                        echo "Add Dockfile"
                        echo '
                        	FROM ${harbor_url}/devops/centos-jdk1.8:v1
    						COPY  target/*.jar /home/app/book.jar
    						USER root
    						WORKDIR /home/app
    						EXPOSE 8080
    						ENTRYPOINT ["java","-jar","book.jar"]                                
                            ' > Dockerfile
                        echo "Docker Build"
                        docker build -t ${image_name} .
                        echo "Docker push"
                        docker push ${image_name}
                        echo "Docker rm images"
                        docker rmi ${image_name}
                        """
                   }
                }
            }
        }
        stage('发布到K8S') {
            steps {
                container('jnlp') {
                withCredentials([kubeconfigContent(credentialsId: "${k8s_kubeconfig_auth}", variable: 'KUBECONFIG_CONTENT')]) {
					withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'passwd', usernameVariable: 'user')]) {
						sh """
                        mkdir /root/.kube
                        echo "${KUBECONFIG_CONTENT}" > /root/.kube/config
                        sed -i 's#IMAGE_NAME#${image_name}#g' deployment.yaml
                        sed -i 's#APP_NAME#${app_name}#g' deployment.yaml
                        sed -i 's#NS_NAME#${params.NameSpace}#g' deployment.yaml
                        sed -i 's#RS_NUM#${params.ReplicasNum}#g' deployment.yaml
						sed -i 's#SECRET_NAME#${Secret_Name}#g' deployment.yaml
                        cat deployment.yaml
						kubectl create secret docker-registry "${Secret_Name}" --docker-server="${harbor_url}" --docker-username="$user" --docker-password="$passwd" -n "${params.NameSpace}" |true
                        kubectl apply -f deployment.yaml --record
						kubectl get node
                        kubectl get deploy -o wide -n "${params.NameSpace}"
                        kubectl get pod -o wide -n "${params.NameSpace}"
						kubectl rollout history deploy/${app_name} -n "${params.NameSpace}"
						"""
						}                    
                    }
                }
            }
        }
        
    }
}

如果使用NFS挂载方式,node节点需要安装nfs-utils

# jenkins # kubernetes
Logo
K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 |  韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号​前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。​长夜有穷，真水无香。领略过了 K8s“身在江

avatar K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub

avatar K8S/Kubernetes

k8s 火了！

2020，上云之年，产品云端化成为一种趋势。在一线城市，很多公司都已经构建了自己的私有云环境，比如阿里云、网易云、华为云等。而Kubernetes 作为基于容器编排领域的王者，具备扩展...

avatar K8S/Kubernetes