kubeadm部署k8s以及CICD环境

1、关闭防火墙2、关闭selinux3、关闭swapswapoff -a在阿里云https://developer.aliyun.com/mirror/Debian / Ubuntu安装：apt-get update && apt-get install -y apt-transport-httpscurl https://mirrors.aliyun.com/kubernetes/apt/doc

北海漁

430人浏览 · 2022-07-06 17:55:28

北海漁 · 2022-07-06 17:55:28 发布

1、关闭防火墙
2、关闭selinux
3、关闭swap
swapoff -a

在阿里云
https://developer.aliyun.com/mirror/

Debian / Ubuntu安装：
   apt-get update && apt-get install -y apt-transport-https

   curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -

   cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
   deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
   EOF

   apt-get update

   apt-get install -y kubelet kubeadm kubectl

CentOS / RHEL / Fedora安装：
   cat <<EOF > /etc/yum.repos.d/kubernetes.repo
   [kubernetes]
   name=Kubernetes
   baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
   enabled=1
   gpgcheck=1
   repo_gpgcheck=1
   gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
   EOF

   setenforce 0

   yum install -y kubelet kubeadm kubectl

   systemctl enable kubelet && systemctl start kubelet

修改docker的从group driver：
编辑/etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"]
}

以flannel为网络通讯插件为例（网络ip），阿里云镜像源为基础，初始化kubeadm：
   kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.23.3 \
   --control-plane-endpoint master.dreaminglifes.com --apiserver-advertise-address 172.23.134.12 \
               --pod-network-cidr 10.244.0.0/16 --token-ttl 0 --upload-certs

进行一下操作

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

下载网络插件
wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

之后运行yml启动flannel（calico要复杂一些）

进行将节点加入到集群中，在node1 node2执行一下命令
## kubeadm join master.dreaminglifes.com:6443 --token ipk0rj.z22icm7v0uh16kf1 \
   --discovery-token-ca-cert-hash sha256:3606050e29db1d855fe77998caf26496f752b526bc89b439a0977722d5330779 \
   --control-plane

kubeadm join master.dreaminglifes.com:6443 --token o4239d.5q22d40tblh3oxy8 \
   --discovery-token-ca-cert-hash sha256:a3474faddfc61f688ec149d13e8e4bff80a99c4bcb976f266848f916a13b640e \
   --control-plane --certificate-key 0df35a397df35101fcc2e476f23b268c5c5d890c8f91e92fabea1abb25163657

sysctl -a | grep bridge
编辑vim /etc/sysctl.d/kubernetes.conf（所有节点）
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=0
fs.file-max=52706963
fs.nr_open=52706963

master节点设置
把master node设为可以安装其它负载。如下：

kubectl taint nodes --all node-role.kubernetes.io/master-

如果安装错误，可以使用reset
kubeadm reset

删除家目录下的./hube下的所有，以及/etc/kubernetes下的所有

CI/CD部署

Gitlab部署

使用北京外国语学院的镜像库
wget https://mirrors.bfsu.edu.cn/gitlab-ce/yum/el7/gitlab-ce-13.7.1-ce.0.el7.x86_64.rpm --no-check0certificate

配置文件是 /etc/gitlab/gitlab.rb

       配置项：external_url 'http://gitlab.dreaminglifes.com'

       /var/opt/gitlab
       /var/log/gitlab

彻底完美卸载Gitlab

一、停止gitlabsudo gitlab-ctl stop

二、卸载gitlabsudo rpm -e gitlab-ce

三、查看gitlab进程

kill -9 4473

再次查看gitlab进程是否存在

四、删除gitlab文件

find / -name gitlab|xargs rm -rf 删除所有包含gitlab的文件及目录

删除gitlab-ctl uninstall时自动在root下备份的配置文件

备份与恢复

gitlab-backup create
gitlab-rake gitlab:backup:create

恢复：
gitlab-rake gitlab:backup:restore BACKUP=/PATH/TO/BACKUP_FILE
gitlab-backup restore BACKUP=/PATH/TO/BACKUP_FILE

在恢复和备份之前都需要停止gitlab服务

Jenkins部署

wget https://mirrors.bfsu.edu.cn/jenkins/redhat-stable/jenkins-2.319.3-1.1.noarch.rpm --no-check-certificate

安装JDK11
yum install java-11-openjdk

安装jenkins
yum install -y ./jenkins-2.319.3-1.1.noarch.rpm

修改端口
vim /etc/sysconfig/jenkins

修改配置文件
cd /var/lib/jenkins/

sed -i 's@https://updates.jenkins.io/download/@https://mirrors.bfsu.edu.cn/jenkins/@g' updates/default.json

sed -i 's@https://updates.jenkins.io/update-center.json@https://mirrors.bfsu.edu.cn/jenkins/updates/current/update-center.json@i' hudson.model.UpdateCenter.xml

重启jenkins
systemctl restart Jenkins

查看密码
cat /var/lib/jenkins/secrets/initialAdminPassword

kenkins的pipeline基础

pipeline{
agent any
tools {
maven 'mvn-3.6.3'
}
triggers{
#cron("H * * * *")
#pollSCM("H * * * *")
upstream(upstreamProjects: 'job-y',threshold: hudson.model.Result.SUCCESS)
}
stages{
stage("Checkout"){
steps{
git branch:'master',url:'http://gitlab.dreaminglifes.com/sunjinyan/spring-boot-helloworld.git'
}
}
stage("Build"){
steps{
sh 'mvn clean package'
}
}
}
}

pipeline{
agent any
tools {
maven 'mvn-3.6.3'
}
stages{
stage("Checkout"){
steps{
git branch:'master',url:'http://gitlab.dreaminglifes.com/sunjinyan/spring-boot-helloworld.git'
}
}
stage("Build"){
steps{
sh 'mvn clean package'
}
}
}
post{
failure{
updateGitlabCommitStatus name: 'build', state: 'failed'
}
success{
updateGitlabCommitStatus name: 'build', state: 'success'
}
}
options{
gitLabConnection('gitlab')
}
}

将gitlab和Jenkins结合

1、生成ssh密钥对，将公钥保存于gitlab用户账号之上，以使得git或Jenkins等客户端能使用匹配的私钥认证到gitlab服务上
在Jenkins servers上以Jenkins用户的身份生成ssh密钥
usermod -s /bin/bash jenkins
su - jenkins
ssh-keygen -t ed25519 或者
ssh-keygen -t rsa -P '' -C '13683234298@163.com'

然后复制rsa.pub内容到gitlab上的用户账号配置属性中
当前用户的settings下的ssh key

然后还需要把私钥部署到Jenkins系统内

Dashboard-》凭据-》系统-》全局凭据 (unrestricted) 此时选择的凭据类型为SSH username with private key

2、在Gitlab上创建Access Token
user account -> setting -> access token
CL-4w7_1KtuERRBxL42f (只显示一次)

3、在Jenkins上授权启动/project端点以创建gitlab链接
   manage plugins安装gitlab插件

   manage Jenkins configure system
   在此时配置gitlab的时候，添加凭据，此时选择的凭据类型为Gitlab API token

4、配置Jenkins项目可经由Gitlab上的事件触发

   新建一个gitlab项目，类型为pipeline
   接下来GitLab Connection选择已经添加好的用户
   在选择构建触发器时候，选择Build when a change is pushed to GitLab. GitLab webhook URL: http://node1.dreaminglifes.com:8088/project/spring-boot-helloworld

   之后点击高级，点击generate生成一个令牌，填写入secret token中
   2d9e6b976a0f99e69c251d4253af113d

5、之后再将上述secret token 写入到gitlab对应的项目下的settings下的webhook中
   url：http://node1.dreaminglifes.com:8088/project/spring-boot-helloworld（Jenkins的pipeline中git配置后的url）


6、写入pipeline
pipeline{
agent any
tools {
maven 'mvn-3.6.3'
}
stages{
stage("Checkout"){
steps{
git branch:'master',url:'http://gitlab.dreaminglifes.com/sunjinyan/spring-boot-helloworld.git'
}
}
stage("Build"){
steps{
sh 'mvn clean package'
}
}
}
post{
failure{
updateGitlabCommitStatus name: 'build', state: 'failed'
}
success{
updateGitlabCommitStatus name: 'build', state: 'success'
}
}
options{
gitLabConnection('gitlab')
}
}
pipeline{
agent any
tools {
maven 'mvn-3.6.3'
}
   triggers{
       gitlab(triggerOnPush:true,triggerOnMergeRequest:true,triggerOpenMergeRequestOnPush:'never',triggerOnNoteRequest:true,branchFilterType:'All',secretToken:'1dhoifrnasn19asd0')
   }
options{
gitLabConnection('gitlab-sunjinyan')
}
stages{
stage("Checkout"){
steps{
git branch:'master',url:'http://gitlab.dreaminglifes.com/sunjinyan/spring-boot-helloworld.git'
}
}
stage("Build"){
steps{
sh 'mvn clean package'
}
}
}
post{
failure{
updateGitlabCommitStatus name: 'build', state: 'failed'
}
success{
updateGitlabCommitStatus name: 'build', state: 'success'
}
}
}
7、在gitlab上进行push event 测试

在pipeline中使用withCredentials()来引用凭证
SSH密钥
withCredentials([sshUserPrivateKey(credentialsId:'<credentials-id>',keyFileVariable:'MYKEYFILE',passphraseVariable:'PASSPHRASE',usernameVariable:'USERNAME')])

参数化pipeline配置

parameters类型定义

pipeline{
agent any
   parameters{
       booleanParam(name:"isPublishVersion",defaultValue:false,description:"默认不发布")
       choice(name:'whichBranch',choices['origin/release-bufix','origin/release','origin/master'],description:'构建分支选择，默认origin/release-bufix')
       choice(name:'deployEnvironment',choices:['dev','test','staging'],description:'请选择部署环境')
   }
tools {
maven 'mvn-3.6.3'
}
   triggers{
       gitlab(triggerOnPush:true,triggerOnMergeRequest:true,triggerOpenMergeRequestOnPush:'never',triggerOnNoteRequest:true,branchFilterType:'All',secretToken:'1dhoifrnasn19asd0')
   }
options{
gitLabConnection('gitlab-sunjinyan')
}
stages{
       stage("buildFramework"){
           steps{
               echo "GIT_LOCAL_BRANCH:${params.whichBranch}"
               echo "isPublishVersion:${params.isPublishVersion}"
               echo "deployEnvironment:${params.deployEnvironment}"
           }
       }
stage("Checkout"){
steps{
git branch:'master',url:'http://gitlab.dreaminglifes.com/sunjinyan/spring-boot-helloworld.git'
}
}
stage("Build"){
steps{
sh 'mvn clean package'
}
}
}
post{
failure{
updateGitlabCommitStatus name: 'build', state: 'failed'
}
success{
updateGitlabCommitStatus name: 'build', state: 'success'
}
}
}

input类型定义
pipeline{
agent any
   stages{
       stage("Example"){
           steps{
               script{
                   def userInput = input(submitterParameter:"approver",id:"approver",message:"provide your approval to proceed",parameters:[string(defaultValue:'approved',description:'Please provide the message why your are approving',name:'remarks')])
                   echo "Remarks: $(userInput['remarks'])"
                   echo 'it was ${userInput.approver who approved this job}'
               }
           }
       }
   }
}

多分枝pipeline流水线操作
1、添加组件
GitLab Branch Source

2、令牌为gitlab个人令牌

3、创建多分支流水线pipeline

4、之后在git项目中添加一个Jenkinsfile就会自动触发

when指令实例
pipeline{
   agent any

   stages{
       stage("Example Build"){
           steps{
               echo "hello world"
           }
       }

       stage("Example Deploy"){
           when{
               allOf{
                   branch 'production'
                   environment name:'DEPLOY_TO',value:'producte'
               }
           }

           steps{
               echo 'Deploying'
           }
       }
   }

}

pipeline{
   agent any

   stages{
       stage("Example Build"){
           steps{
               echo "hello world"
           }
       }

       stage("Example Deploy"){
           when{
               expression {BRANCH_NAME== ~/(production|staging)}
               anyOf{
                   branch 'production'
                   environment name:'DEPLOY_TO',value:'producte'
                   environment name:'DEPLOY_TO',value:'staging'
               }
           }

           steps{
               echo 'Deploying'
           }
       }
   }

}

邮箱通知
pipeline{
   agent any

   stages{
       stage("Example Build"){
           steps{
               echo "hello world"
           }
       }

       stage("Example Deploy"){
           when{
               expression {BRANCH_NAME== ~/(production|staging)}
               anyOf{
                   branch 'production'
                   environment name:'DEPLOY_TO',value:'producte'
                   environment name:'DEPLOY_TO',value:'staging'
               }
           }

           steps{
               echo 'Deploying'
           }
       }
   }
   post{
       always{
           mail to: '13683234298@163.com',
           subject: 'status of pipeline:${currentBuild.fullDisplayName}',
           body: "${env.BUILD_URL} has result ${currentBuild.result}"
       }
   }

}

sonarqube代码检测

Jenkins集群搭建

master与agent代理模式

在主节点上选择manage node clouds 然后选择新建节点

填写节点名

然后填写详细信息，以及主从工作方式

远程工作目录不要使用Jenkins默认的/var/lib/jenkins/,可以使用/appdata/jenkins，保证Jenkins有该目录的读写权限，与原工作目录隔离，是为了避免与原来的agent服务冲突

用法选择use this node as mach as possible (尽可能的使用该节点)

启动方法选择launch agents via ssh

主机填写主机名做好，不要固定ip

选择添加凭据阶段：
   类型选择：username with password

   用户名和密码：填写agent节点的用户名、密码

从节点安装Jenkins：

修改配置文件
cd /var/lib/jenkins/

sed -i 's@https://updates.jenkins.io/download/@https://mirrors.bfsu.edu.cn/jenkins/@g' updates/default.json

sed -i 's@https://updates.jenkins.io/update-center.json@https://mirrors.bfsu.edu.cn/jenkins/updates/current/update-center.json@i' hudson.model.UpdateCenter.xml

修改Jenkins用户的密码以及shell

usermod -s /bin/bash jenkins

passwd jenkins

在主节点生成一个ssh 密钥
su - jenkins

ssh-keygen

然后把密钥公钥传给agent节点 ssh-copy-id -i .ssh/id_rsa.pub jenkins@172.23.134.12(用户和agent服务器ip)

然后用ssh 链接agent节点172.23.134.12

查看 .ssh/id_rsa 私钥信息，并在主机点生成一个凭证，将agent节点的私钥信息放到主节点的凭证配置中

类型还是SSH Username with private key

jenkins与docker集成

1、在节点上安装docker环境

2、还要为jenkins安装docker插件以及docker pipeline插件，以集成Jenkins和docker

3、将Jenkins用户加入到docker组，以确保docker命令创建容器usermod -a -G docker

4、创建pipeline
pipeline{
agent {
docker { image 'openjdk:13-alpine'}
}
stages{
   state("Test"){
   steps{
           sh 'java --version'
   }
   }
}
}

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub