基于k8s的综合的web服务器构建
centos7,k8s,docker,prometheus,nfs,JumpServer,harbor,ansible,Jenkins等。1.更加深入的了解了k8s的各个功能（service，pv，pvc，ingress）等。1.配置静态ip地址和设置主机名和关闭selinux和firewalld。k8s设计的时候为了能提升性能，默认是不允许使用交换分区的。模拟企业里的k8s生产环境,,构建一个高
·
目录
一.在三台k8s机器上安装部署好k8s，一台作为master，两台node
二，部署nfs服务，让所有的web业务pod都取访问，通过pv，pvc和卷挂载实现
三.启动nginx和MySQL的pod，采用HPA技术，cpu使用率高时进行水平扩缩,使用ab进行压力测试。
启动开启了HPA功能的nginx的部署控制器，启动nginx的pod
四.使用ingress给web业务做基于域名的负载均衡，基于url的负载均衡的实现
第3大步骤： 启用ingress 关联ingress controller 和service
在nfs服务器上进行测试，需要在/etc/hosts文件里添加域名解析记录
五.使用探针对web业务pod进行监控, 一旦出现问题马上重启, 增强业务pod的可靠性。
六.构建CI/CD环境, k8smaster上安装部署Jenkins,一台机器上安装部署harbor仓库。
在master或者被监听的机器上操作（安装相应的exporter，常规的是node_exporter）：
九.安装部署firewalld防火墙和jumpserver堡垒机，对web集群进行保护。
十.安装部署ansible机器，写好主机清单,便于日后的自动化运维。
1.建立免密通道 在ansible主机上生成密钥对，上传公钥到所有服务器的root用户家目录下
项目架构图：
项目环境:
centos7,k8s,docker,prometheus,nfs,JumpServer,harbor,ansible,Jenkins等
项目描述：
模拟企业里的k8s生产环境,,构建一个高可用高性能的系统,
项目步骤：
ip规划：
k8smaster：192.168.220.100
k8snode1：192.168.220.101
k8snode2：192.168.220.102
nfs：192.168.220.103
harbor：192.168.220.104
Prometheus：192.168.220.105
jumpserver：192.168.220.106
firewalld： 192.168.220.107
ansible：192.168.220.108
一.在三台k8s机器上安装部署好k8s，一台作为master，两台node
安装部署k8s
###下面操作每台机器都要操作，建议xshell上开启输入到所有会话
1.配置静态ip地址和设置主机名和关闭selinux和firewalld
hostnamectl set-hostname master && bash
hostnamectl set-hostname node1 && bash
hostnamectl set-hostname node2 && bash#关闭firewalld防火墙服务，并且设置开机不要启动
service firewalld stop
systemctl disable firewalld
#临时关闭selinux
setenforce 0
#永久关闭selinux
sed -i '/^SELINUX=/ s/enforcing/disabled/' /etc/selinux/config添加域名解析
[root@master ~]# vim /etc/hosts
[root@master ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.220.100 master
192.168.220.101 node1
192.168.220.102 node2 ##### 注意！！！下面为三台机器都要操作；
关闭交换分区
k8s设计的时候为了能提升性能，默认是不允许使用交换分区的。
[root@master ~]# swapoff -a 临时关闭
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
[root@master ~]# vim /etc/fstab
加注释：
#/dev/mapper/centos-swap swap swap defaults 0 0 加载网桥和地址地址转发功能（实现容器之间通信）：
[root@master ~]# modprobe br_netfilter
[root@master ~]# modprobe overlay
#如果文件不存在，tee 会创建它；如果文件已经存在，tee 会覆盖它的内容
[root@master ~]# cat << EOF | tee /etc/modules-load.d/k8s.conf
br_netfilter
overlay
EOF查看内核模板是否成功
#lsmod是Linux系统中用于列出已加载内核模块（Kernel Modules）的命令
[root@master ~]# lsmod |grep -e br_netfilter -e overlay
overlay 91659 0
br_netfilter 22256 0
bridge 151336 1 br_netfilter
[root@master ~]# cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF 更新和配置软件源
# 添加阿里云yum源
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
# 重新生成yum元数据缓存
yum clean all && yum makecache
# 安装基础软件包
yum install -y vim wget
# 配置阿里云Docker yum仓库源
yum install -y yum-utils
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
配置ipvs功能：（实现负载均衡的软件）：
# 安装ipset和ipvsadm
yum install -y ipset ipvsadm
# 添加需要加载的模块写入脚本文件，保证在节点重启后能自动加载所需模块
cat <<EOF > /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
# 为脚本文件添加执行权限
chmod +x /etc/sysconfig/modules/ipvs.modules
# 执行脚本文件
/bin/bash /etc/sysconfig/modules/ipvs.modules
# 查看对应的模块是否加载成功
lsmod | grep -e ip_vs -e nf_conntrack_ipv4
nf_conntrack_ipv4 15053 24
nf_defrag_ipv4 12729 1 nf_conntrack_ipv4
ip_vs_sh 12688 0
ip_vs_wrr 12697 0
ip_vs_rr 12600 105
ip_vs 145497 111 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack 139264 10 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_nat_masquerade_ipv6,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack
配置时间同步：
# 启用chronyd服务
systemctl start chronyd && systemctl enable chronyd
# 设置时区
timedatectl set-timezone Asia/Shanghaidocker的配置
yum install -y docker-ce-20.10.24-3.el7 docker-ce-cli-20.10.24-3.el7 containerd.io
[root@master ~]# systemctl start docker
[root@master ~]# systemctl enable docker
# 创建文件夹
mkdir -p /etc/docker
# 编辑配置
cat > /etc/docker/daemon.json <<EOF
{
"registry-mirrors": [
"https://youraddr.mirror.aliyuncs.com",
"http://hub-mirror.c.163.com",
"https://reg-mirror.qiniu.com",
"https://docker.mirrors.ustc.edu.cn"
],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
systemctl daemon-reload && systemctl restart docker
[root@master ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[root@master ~]# getenforce
Disabled
[root@master ~]# swap
-bash: swap: 未找到命令
[root@master ~]# cat /proc/s
sched_debug self/ stat sysrq-trigger
schedstat slabinfo swaps sysvipc/
scsi/ softirqs sys/
[root@master ~]# cat /proc/swaps
Filename Type Size Used Prioritynode节点加入集群：
下面步骤三台机器都要做
配置k8s集群环境：
# 配置组件源
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
# 构建本地yum缓存
yum makecache
# 安装
yum install -y kubeadm-1.23.17-0 kubelet-1.23.17-0 kubectl-1.23.17-0 --disableexcludes=kubernetes
[root@master ~]# cat <<EOF > /etc/sysconfig/kubelet
KUBELET_CGROUP_ARGS="--cgroup-driver=systemd"
KUBE_PROXY_MODE="ipvs"
EOF
# 启动并设置开机自启
systemctl enable --now kubelet
master节点初始化：
#只需要master上面做
kubeadm init \
--kubernetes-version=v1.23.17 \
--pod-network-cidr=10.224.0.0/16 \
--service-cidr=10.96.0.0/12 \
--apiserver-advertise-address=192.168.220.100 \
--image-repository=registry.aliyuncs.com/google_containers
（不能有空格）
结果：
kubeadm join 192.168.220.100:6443 --token v7a9n5.ppfursc2nbica1fg \
--discovery-token-ca-cert-hash sha256:6a4863a28201e03ee1b8083fc6fc08b6f7f39b44899b9c8bd6b627ab044b77ea
journalctl -u kubelet 看错误
在node机器上：
kubeadm join 192.168.220.100:6443 --token v7a9n5.ppfursc2nbica1fg --discovery-token-ca-cert-hash sha256:6a4863a28201e03ee1b8083fc6fc08b6f7f39b44899b9c8bd6b627ab044b77ea
master：
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady control-plane,master 22m v1.23.17
node1 NotReady <none> 50s v1.23.17
node2 NotReady <none> 44s v1.23.17
给node节点打上标签
# 在master上执行
kubectl label node node1 node-role.kubernetes.io/worker=worker
kubectl label node node2 node-role.kubernetes.io/worker=worker
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady control-plane,master 28m v1.23.17
node1 NotReady worker 6m3s v1.23.17
node2 NotReady worker 5m57s v1.23.17
kubeadm reset 可以清除初始化的操作安装Calico网络插件：
# master执行
wget --no-check-certificate https://docs.projectcalico.org/archive/v3.25/manifests/calico.yaml #用的wget下载
calico 是k8s里的一个网络组件，用于实现master和node节点之间的通信的，底层使用overlay网络
# master执行
kubectl apply -f https://docs.projectcalico.org/archive/v3.25/manifests/calico.yaml # k8s 1.23适用此版本
[root@master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 53m v1.23.17
node1 Ready worker 30m v1.23.17
node2 Ready worker 30m v1.23.17
k8s配置ipvs：
master上：
kubectl edit configmap kube-proxy -n kube-system
mode: "ipvs"
# 删除所有kube-proxy pod使之重启
kubectl delete pods -n kube-system -l k8s-app=kube-proxy
看k8s容器（pod）：
kubectl get pod -n kube-system
kubectl get pod 是查看有哪些pod在运行 --》docker ps
-n kube-system 是查看kube-system命名空间里运行的pod namespace
kube-system 是k8s控制平面的pod所在的命名空间
pod 是运行容器的单元
以pod治理pod
住在kube-system 命名空间里的pod是控制平面的pod
kubectl get ns = kubectl get namespace 查看有哪些命名空间
NAME STATUS AGE
default Active 78m #是创建的普通的pod运行的命名空间
kube-node-lease Active 78m
kube-public Active 78m
kube-system Active 78m #是管理相关的命名空间二，部署nfs服务，让所有的web业务pod都取访问，通过pv，pvc和卷挂载实现
注意点：
nfs服务器上：
#关闭firewalld防火墙服务，并且设置开机不要启动
service firewalld stop
systemctl disable firewalld
#临时关闭selinux
setenforce 0
#永久关闭selinux
sed -i '/^SELINUX=/ s/enforcing/disabled/' /etc/selinux/config
1.搭建好nfs服务器
# 在nfs服务器和k8s集群上安装nfs
[root@nfs ~]# yum install nfs-utils -y
[root@master ~]# yum install nfs-utils -y
[root@node1 ~]# yum install nfs-utils -y
[root@node2 ~]# yum install nfs-utils -y
2.设置共享目录
[root@nfs ~]# vim /etc/exports
[root@nfs ~]# cat /etc/exports
/web 192.168.220.0/24(rw,no_root_squash,sync)
[root@nfs ~]# mkdir /web
[root@nfs ~]# cd /web
[root@nfs web]# echo "tiantianming" >index.html
[root@nfs web]# ls
index.html
[root@localhost web]# exportfs -rv #刷新nfs服务
exporting 192.168.220.0/24:/web
#重启服务并且设置开机启动
[root@nfs web]# systemctl restart nfs && systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.3.创建pv使用nfs服务器上的共享目录
[root@master ~]# mkdir /pv
[root@master ~]# cd /pv/
[root@master pv]# vim nfs-pv.yml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-web
labels:
type: pv-web
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
storageClassName: nfs # pv对应的名字
nfs:
path: "/web" # nfs共享的目录
server: 192.168.220.103 # nfs服务器的ip地址
readOnly: false # 访问模式
[root@master pv]# kubectl apply -f nfs-pv.yml
persistentvolume/pv-web created
[root@master pv]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv-web 10Gi RWX Retain Available nfs 12s
# 创建pvc使用pv
[root@master pv]# vim nfs-pvc.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-web
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: nfs #使用nfs类型的pv
[root@master pv]# kubectl apply -f nfs-pvc.yml
persistentvolumeclaim/pvc-web created
[root@master pv]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-web Bound pv-web 10Gi RWX nfs 13s
#创建pod使用pvc
[root@master pv]# vim nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
volumes:
- name: sc-pv-storage-nfs
persistentVolumeClaim:
claimName: pvc-web
containers:
- name: sc-pv-container-nfs
image: nginx
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: sc-pv-storage-nfs
[root@master pv]# kubectl apply -f nginx-deployment.yaml
deployment.apps/nginx-deployment created
[root@master pv]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-794d8c5666-dsxkq 1/1 Running 0 17m 10.224.166.130 node1 <none> <none>
nginx-deployment-794d8c5666-fsctm 1/1 Running 0 12m 10.224.104.4 node2 <none> <none>
nginx-deployment-794d8c5666-spkzs 1/1 Running 0 12m 10.224.104.3 node2 <none> <none>测试：
[root@master pv]# curl 10.224.166.130
tiantianming
修改下nfs服务器上的index.html的内容
[root@nfs web]# vim index.html
[root@nfs web]# cat index.html
tiantianming
welcome to hangzhou!!!
访问也变了，表示已经成功！
[root@master pv]# curl 10.224.166.130
tiantianming
welcome to hangzhou!!!
三.启动nginx和MySQL的pod，采用HPA技术，cpu使用率高时进行水平扩缩,使用ab进行压力测试。
k8s部署mysql pod：
1.编写yaml文件，包括了deployment、service
[root@master ~]# mkdir /mysql
[root@master ~]# cd /mysql/
[root@master mysql]# vim mysql.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: mysql
name: mysql
spec:
replicas: 1
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- image: mysql:latest
name: mysql
imagePullPolicy: IfNotPresent
env:
- name: MYSQL_ROOT_PASSWORD
value: "123456" #mysql的密码
ports:
- containerPort: 3306
---
apiVersion: v1
kind: Service
metadata:
labels:
app: svc-mysql
name: svc-mysql
spec:
selector:
app: mysql
type: NodePort
ports:
- port: 3306
protocol: TCP
targetPort: 3306
nodePort: 30007
2.部署
[root@master mysql]# kubectl apply -f mysql.yaml
deployment.apps/mysql created
service/svc-mysql created
[root@master mysql]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 23h
php-apache ClusterIP 10.96.134.145 <none> 80/TCP 21h
svc-mysql NodePort 10.109.190.20 <none> 3306:30007/TCP 9s
[root@master mysql]# kubectl get pod
NAME READY STATUS RESTARTS AGE
mysql-597ff9595d-tzqzl 0/1 ContainerCreating 0 27s
nginx-deployment-794d8c5666-dsxkq 1/1 Running 1 (15m ago) 22h
nginx-deployment-794d8c5666-fsctm 1/1 Running 1 (15m ago) 22h
nginx-deployment-794d8c5666-spkzs 1/1 Running 1 (15m ago) 22h
php-apache-7b9f758896-2q44p 1/1 Running 1 (15m ago) 21h
[root@master mysql]# kubectl exec -it mysql-597ff9595d-tzqzl -- bash
root@mysql-597ff9595d-tzqzl:/# mysql -uroot -p123456 #容器内部进入mysql
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.27 MySQL Community Server - GPL
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
安装metrics
下载配置文件
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
args:
# // 新增下面两行参数
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname
# 替换image
image: registry.aliyuncs.com/google_containers/metrics-server:v0.6.0
imagePullPolicy: IfNotPresent
部署:
kubectl apply -f components.yaml
[root@master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-6949477b58-tbkl8 1/1 Running 1 7h10m
calico-node-4t8kx 1/1 Running 1 7h10m
calico-node-6lbdw 1/1 Running 1 7h10m
calico-node-p6ghl 1/1 Running 1 7h10m
coredns-7f89b7bc75-dxc9v 1/1 Running 1 7h15m
coredns-7f89b7bc75-kw7ph 1/1 Running 1 7h15m
etcd-master 1/1 Running 1 7h15m
kube-apiserver-master 1/1 Running 2 7h15m
kube-controller-manager-master 1/1 Running 1 7h15m
kube-proxy-87ptg 1/1 Running 1 7h15m
kube-proxy-8gbsd 1/1 Running 1 7h15m
kube-proxy-x4fbj 1/1 Running 1 7h15m
kube-scheduler-master 1/1 Running 1 7h15m
metrics-server-7787b94d94-jt9sc 1/1 Running 0 47s
[root@master hpa]# kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
master 129m 6% 1111Mi 64%
node1 61m 3% 608Mi 35%
node2 59m 2% 689Mi 40%
[root@master hpa]# 启动开启了HPA功能的nginx的部署控制器，启动nginx的pod
[root@master hpa]# vim nginx-hpa.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ab-nginx
spec:
selector:
matchLabels:
run: ab-nginx
template:
metadata:
labels:
run: ab-nginx
spec:
#nodeName: node-2 取消指定
containers:
- name: ab-nginx
image: nginx
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
resources:
limits:
cpu: 100m
requests:
cpu: 50m
---
apiVersion: v1
kind: Service
metadata:
name: ab-nginx-svc
labels:
run: ab-nginx-svc
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
nodePort: 31000
selector:
run: ab-nginx
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: ab-nginx
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ab-nginx
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 50
[root@master hpa]#
创建具有hpa功能的nginx的pod
[root@master hpa]# kubectl apply -f nginx-hpa.yaml
deployment.apps/ab-nginx created
service/ab-nginx-svc created
horizontalpodautoscaler.autoscaling/ab-nginx created
[root@master hpa]#
查看启动hpa和pod、deployment、service的情况
[root@master hpa]# kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
ab-nginx 1/1 1 1 55s
[root@master hpa]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
ab-nginx Deployment/ab-nginx 0%/50% 1 10 1 58s
php-apache Deployment/php-apache 0%/50% 1 10 1 20d
[root@master hpa]# kubectl get pod
NAME READY STATUS RESTARTS AGE
ab-nginx-5f4c4b9558-xbxjb 1/1 Running 0 63s
configmap-demo-pod 1/1 Running 31 2d23h
[root@master hpa]#
[root@master hpa]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ab-nginx-svc NodePort 10.107.155.209 <none> 80:31000/TCP 2m26s
访问宿主机的31000端口
http://192.168.220.100:31000/
测试nginx pod是否启动成功下面在nfs上用ab工具测试
安装http-tools工具得到ab软件
[root@nfs-server ~]# yum install httpd-tools -y
模拟访问
[root@nfs-server ~]# ab -n 1000 -c50 http://192.168.220.100:31000/index.html
root@master hpa]# kubectl get hpa --watch
增加并发数和请求总数
[root@nfs-server ~]# ab -n 5000 -c100 http://192.168.203.128:31000/index.html
[root@nfs-server ~]# ab -n 10000 -c200 http://192.168.203.128:31000/index.html
[root@nfs-server ~]# ab -n 20000 -c400 http://192.168.203.128:31000/index.html
[root@master hpa]# kubectl describe pod ab-nginx-5f4c4b9558-shtt5
Warning OutOfmemory 98s kubelet Node didn't have enough resource: memory, requested: 268435456, used: 3584032768, capacity: 3848888320
[root@master hpa]#
原因是node-2节点没有足够的内存去启动新的pod了
四.使用ingress给web业务做基于域名的负载均衡，基于url的负载均衡的实现
第1大步骤： 安装ingress controller
使用旧版本ingress controller v1.1完成
准备工作：需要提前上传下面的这些镜像和yaml文件到k8s集群里的linux系统里，建议存放到master节点上，然后再scp到node节点上
[root@master .kube]# mkdir /ingress
[root@master .kube]# cd /ingress/
[root@master ingress]#
ingress-controller-deploy.yaml 是部署ingress controller使用的yaml文件
ingress-nginx-controllerv1.1.0.tar.gz ingress-nginx-controller镜像
kube-webhook-certgen-v1.1.0.tar.gz kube-webhook-certgen镜像
#kube-webhook-certgen镜像主要用于生成Kubernetes集群中用于Webhook的证书。
#kube-webhook-certgen镜像生成的证书，可以确保Webhook服务在Kubernetes集群中的安全通信和身份验证
ingress.yaml 创建ingress的配置文件
nginx-svc-3.yaml 创建service3 和相关pod
nginx-svc-4.yaml 创建service4 和相关pod
1.将镜像scp到所有的node节点服务器上
[root@master ingress]# scp ingress-nginx-controllerv1.1.0.tar.gz node1:/root
ingress-nginx-controllerv1.1.0.tar.gz 100% 276MB 42.7MB/s 00:06
[root@master ingress]# scp ingress-nginx-controllerv1.1.0.tar.gz node2:/root
ingress-nginx-controllerv1.1.0.tar.gz 100% 276MB 45.7MB/s 00:06
[root@master ingress]# scp kube-webhook-certgen-v1.1.0.tar.gz node2:/root
kube-webhook-certgen-v1.1.0.tar.gz 100% 47MB 40.5MB/s 00:01
[root@master ingress]# scp kube-webhook-certgen-v1.1.0.tar.gz node1:/root
kube-webhook-certgen-v1.1.0.tar.gz 100% 47MB 47.1MB/s 00:00
[root@master ingress]#
2.导入镜像，在所有的节点服务器（node-1和node-2）上进行
[root@k8smaster ingress]# docker load -i ingress-nginx-controllerv1.1.0.tar.gz
[root@k8smaster ingress]# docker load -i kube-webhook-certgen-v1.1.0.tar.gz
[root@k8snode2 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller v1.1.0 ae1a7201ec95 16 months ago 285MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen v1.1.1 c41e9fcadf5a 17 months ago 47.7MB
[root@k8smaster new]#
3.使用ingress-controller-deploy.yaml 文件去启动ingress controller
[root@k8smaster 4-4]# kubectl apply -f ingress-controller-deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
[root@k8smaster 4-4]#
查看ingress controller的相关命名空间
[root@k8smaster 4-4]# kubectl get ns
NAME STATUS AGE
default Active 11d
ingress-nginx Active 52s
kube-node-lease Active 11d
kube-public Active 11d
kube-system Active 11d
[root@k8smaster 4-4]#
查看ingress controller的相关service
[root@k8smaster 4-4]# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.99.160.10 <none> 80:30092/TCP,443:30263/TCP 91s
ingress-nginx-controller-admission ClusterIP 10.99.138.23 <none> 443/TCP 91s
[root@k8smaster 4-4]#
查看ingress controller的相关pod
[root@k8smaster 4-4]# kubectl get pod -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-k69t2 0/1 Completed 0 119s
ingress-nginx-admission-patch-zsrk8 0/1 Completed 1 119s
ingress-nginx-controller-6c8ffbbfcf-bt94p 1/1 Running 0 119s
ingress-nginx-controller-6c8ffbbfcf-d49kx 1/1 Running 0 119s
[root@k8smaster 4-4]#
第2大步骤： 创建pod和暴露pod的服务
[root@master url]# cat sc-nginx-svc-3.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: sc-nginx-deploy-3
labels:
app: sc-nginx-feng-3
spec:
replicas: 3
selector:
matchLabels:
app: sc-nginx-feng-3
template:
metadata:
labels:
app: sc-nginx-feng-3
spec:
containers:
- name: sc-nginx-feng-3
image: nginx
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: sc-nginx-svc-3
labels:
app: sc-nginx-svc-3
spec:
selector:
app: sc-nginx-feng-3
ports:
- name: name-of-service-port
protocol: TCP
port: 80
targetPort: 80
[root@master url]#
[root@master url]# cat sc-nginx-svc-4.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: sc-nginx-deploy-4
labels:
app: sc-nginx-feng-4
spec:
replicas: 3
selector:
matchLabels:
app: sc-nginx-feng-4
template:
metadata:
labels:
app: sc-nginx-feng-4
spec:
containers:
- name: sc-nginx-feng-4
image: nginx
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: sc-nginx-svc-4
labels:
app: sc-nginx-svc-4
spec:
selector:
app: sc-nginx-feng-4
ports:
- name: name-of-service-port
protocol: TCP
port: 80
targetPort: 80
[root@master url]#
[root@master lb-url]# kubectl apply -f sc-nginx-svc-3.yaml
deployment.apps/sc-nginx-deploy-3 created
service/sc-nginx-svc-3 created
[root@master lb-url]# kubectl apply -f sc-nginx-svc-4.yaml
deployment.apps/sc-nginx-deploy-4 created
service/sc-nginx-svc-4 created
第3大步骤： 启用ingress 关联ingress controller 和service
[root@master url]# cat ingress-url.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: simple-url-lb-example
annotations:
kubernets.io/ingress.class: nginx
spec:
ingressClassName: nginx
rules:
- host: www.guan.com
http:
paths:
- path: /tian1
pathType: Prefix
backend:
service:
name: sc-nginx-svc-3 #必须与上面的service名字相同
port:
number: 80
- path: /tian2
pathType: Prefix
backend:
service:
name: sc-nginx-svc-4
port:
number: 80
[root@master ingress]# kubectl apply -f sc-ingress-url.yaml
ingress.networking.k8s.io/simple-fanout-example created
[root@master ingress]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
simple-fanout-example nginx www.guan.com 192.168.220.101,192.168.220.102 80 29s在nfs服务器上进行测试，需要在/etc/hosts文件里添加域名解析记录
[root@nfs-server ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.220.101 www.guan.com #两个node节点上都添加
192.168.220.102 www.guan.com
测试发现不能找到页面，到底是ingress controller的问题还是我们后端的pod的问题
进入service4，service3 对应的一个pod里，新建tian1和tian2文件夹以及index.html网页文件
[root@master ingress]# kubectl exec -it sc-nginx-deploy-3-5c4b975ffc-6x4kc -- bash
root@sc-nginx-deploy-3-5c4b975ffc-6x4kc:/# cd /usr/share/nginx/html/
root@sc-nginx-deploy-3-5c4b975ffc-6x4kc:/usr/share/nginx/html# ls
50x.html index.html
root@sc-nginx-deploy-3-5c4b975ffc-6x4kc:/usr/share/nginx/html# mkdir tian1
root@sc-nginx-deploy-3-5c4b975ffc-6x4kc:/usr/share/nginx/html# echo "tiantianming" > tian1/index.html
root@sc-nginx-deploy-3-5c4b975ffc-6x4kc:/usr/share/nginx/html# ls
50x.html index.html tian1
service4对应的一样创建文件夹tian2
[root@master ingress]# kubectl exec -it sc-nginx-deploy-4-7d4b5c487f-2sdvf -- bash
root@sc-nginx-deploy-4-7d4b5c487f-2sdvf:/# cd /usr/share/nginx/html/
root@sc-nginx-deploy-4-7d4b5c487f-2sdvf:/usr/share/nginx/html# ls
50x.html index.html
root@sc-nginx-deploy-4-7d4b5c487f-2sdvf:/usr/share/nginx/html# mkdir tian2
root@sc-nginx-deploy-4-7d4b5c487f-2sdvf:/usr/share/nginx/html# echo "tiantianming2222" > tian2/index.html
再次在nfs服务器上测试，多测试几次，因为service 背后的ipvs的调度算法是轮询的，所以建议每个pod都建立对应的文件夹
curl www.guan.com/tian1/index.html
curl www.guan.com/tian2/index.html
效果如下：
[root@nfs ~]# curl www.guan.com/tian1/index.html
tiantianming
[root@nfs ~]# curl www.guan.com/tian2/index.html
tiantianming2222五.使用探针对web业务pod进行监控, 一旦出现问题马上重启, 增强业务pod的可靠性。
livenessProbe:
exec:
command:
- ls
- /tmp
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
exec:
command:
- ls
- /tmp
initialDelaySeconds: 5
periodSeconds: 5
startupProbe:
httpGet:
path: /
port: 8000
failureThreshold: 30
periodSeconds: 10
[root@k8smaster probe]# vim my-web.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: myweb
name: myweb
spec:
replicas: 3
selector:
matchLabels:
app: myweb
template:
metadata:
labels:
app: myweb
spec:
containers:
- name: myweb
image: nginx
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8000
resources:
limits:
cpu: 300m
requests:
cpu: 100m
livenessProbe:
exec:
command:
- ls
- /tmp
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
exec:
command:
- ls
- /tmp
initialDelaySeconds: 5
periodSeconds: 5
startupProbe:
httpGet:
path: /
port: 8000
failureThreshold: 30
periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
labels:
app: myweb-svc
name: myweb-svc
spec:
selector:
app: myweb
type: NodePort
ports:
- port: 8000
protocol: TCP
targetPort: 8000
nodePort: 30001
[root@k8smaster probe]# kubectl apply -f my-web.yaml
deployment.apps/myweb created
service/myweb-svc created
[root@master probe]# kubectl get pod |grep -i myweb
myweb-7df8f89d75-2c9v6 0/1 Running 0 69s
myweb-7df8f89d75-cf82r 0/1 Running 0 69s
myweb-7df8f89d75-fmbpn 0/1 Running 0 69s
[root@k8smaster probe]# kubectl describe pod myweb-6b89fb9c7b-4cdh9
。。。
Liveness: exec [ls /tmp] delay=5s timeout=1s period=5s #success=1 #failure=3
Readiness: exec [ls /tmp] delay=5s timeout=1s period=5s #success=1 #failure=3
Startup: http-get http://:8000/ delay=0s timeout=1s period=10s #success=1 #failure=30
。。。六.构建CI/CD环境, k8smaster上安装部署Jenkins,一台机器上安装部署harbor仓库。
安装jenkins：
# Jenkins部署到k8s里
# 1.安装git软件
[root@k8smaster jenkins]# yum install git -y
# 2.下载相关的yaml文件
[root@k8smaster jenkins]# git clone https://github.com/scriptcamp/kubernetes-jenkins
[root@k8smaster jenkins]# ls
kubernetes-jenkins
[root@k8smaster jenkins]# cd kubernetes-jenkins/
[root@k8smaster kubernetes-jenkins]# ls
deployment.yaml namespace.yaml README.md serviceAccount.yaml service.yaml volume.yaml
# 3.创建命名空间
[root@k8smaster kubernetes-jenkins]# cat namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: devops-tools
[root@k8smaster kubernetes-jenkins]# kubectl apply -f namespace.yaml
namespace/devops-tools created
[root@k8smaster kubernetes-jenkins]# kubectl get ns
NAME STATUS AGE
default Active 22h
devops-tools Active 19s
ingress-nginx Active 139m
kube-node-lease Active 22h
kube-public Active 22h
kube-system Active 22h
# 4.创建服务账号，集群角色，绑定
[root@k8smaster kubernetes-jenkins]# cat serviceAccount.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: jenkins-admin
rules:
- apiGroups: [""]
resources: ["*"]
verbs: ["*"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins-admin
namespace: devops-tools
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jenkins-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins-admin
subjects:
- kind: ServiceAccount
name: jenkins-admin
[root@k8smaster kubernetes-jenkins]# kubectl apply -f serviceAccount.yaml
clusterrole.rbac.authorization.k8s.io/jenkins-admin created
serviceaccount/jenkins-admin created
clusterrolebinding.rbac.authorization.k8s.io/jenkins-admin created
# 5.创建卷，用来存放数据
[root@k8smaster kubernetes-jenkins]# cat volume.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: jenkins-pv-volume
labels:
type: local
spec:
storageClassName: local-storage
claimRef:
name: jenkins-pv-claim
namespace: devops-tools
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
local:
path: /mnt
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node1 # 需要修改为k8s里的node节点的名字
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-pv-claim
namespace: devops-tools
spec:
storageClassName: local-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
[root@k8smaster kubernetes-jenkins]# kubectl apply -f volume.yaml
storageclass.storage.k8s.io/local-storage created
persistentvolume/jenkins-pv-volume created
persistentvolumeclaim/jenkins-pv-claim created
[root@k8smaster kubernetes-jenkins]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
jenkins-pv-volume 10Gi RWO Retain Bound devops-tools/jenkins-pv-claim local-storage 33s
pv-web 10Gi RWX Retain Bound default/pvc-web nfs 21h
[root@k8smaster kubernetes-jenkins]# kubectl describe pv jenkins-pv-volume
Name: jenkins-pv-volume
Labels: type=local
Annotations: <none>
Finalizers: [kubernetes.io/pv-protection]
StorageClass: local-storage
Status: Bound
Claim: devops-tools/jenkins-pv-claim
Reclaim Policy: Retain
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity:
Required Terms:
Term 0: kubernetes.io/hostname in [k8snode1]
Message:
Source:
Type: LocalVolume (a persistent volume backed by local storage on a node)
Path: /mnt
Events: <none>
# 6.部署Jenkins
[root@k8smaster kubernetes-jenkins]# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
namespace: devops-tools
spec:
replicas: 1
selector:
matchLabels:
app: jenkins-server
template:
metadata:
labels:
app: jenkins-server
spec:
securityContext:
fsGroup: 1000
runAsUser: 1000
serviceAccountName: jenkins-admin
containers:
- name: jenkins
image: jenkins/jenkins:lts
imagePullPolicy: IfNotPresent
resources:
limits:
memory: "2Gi"
cpu: "1000m"
requests:
memory: "500Mi"
cpu: "500m"
ports:
- name: httpport
containerPort: 8080
- name: jnlpport
containerPort: 50000
livenessProbe:
httpGet:
path: "/login"
port: 8080
initialDelaySeconds: 90
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 5
readinessProbe:
httpGet:
path: "/login"
port: 8080
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: jenkins-data
mountPath: /var/jenkins_home
volumes:
- name: jenkins-data
persistentVolumeClaim:
claimName: jenkins-pv-claim
[root@k8smaster kubernetes-jenkins]# kubectl apply -f deployment.yaml
deployment.apps/jenkins created
[root@k8smaster kubernetes-jenkins]# kubectl get deploy -n devops-tools
NAME READY UP-TO-DATE AVAILABLE AGE
jenkins 1/1 1 1 5m36s
[root@k8smaster kubernetes-jenkins]# kubectl get pod -n devops-tools
NAME READY STATUS RESTARTS AGE
jenkins-7fdc8dd5fd-bg66q 1/1 Running 0 19s
# 7.启动服务发布Jenkins的pod
[root@k8smaster kubernetes-jenkins]# cat service.yaml
apiVersion: v1
kind: Service
metadata:
name: jenkins-service
namespace: devops-tools
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /
prometheus.io/port: '8080'
spec:
selector:
app: jenkins-server
type: NodePort
ports:
- port: 8080
targetPort: 8080
nodePort: 32000
[root@k8smaster kubernetes-jenkins]# kubectl apply -f service.yaml
service/jenkins-service created
[root@k8smaster kubernetes-jenkins]# kubectl get svc -n devops-tools
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins-service NodePort 10.104.76.252 <none> 8080:32000/TCP 24s
# 8.在Windows机器上访问Jenkins，宿主机ip+端口号
http://192.168.220.100:32000/
# 9.进入pod里获取登录的密码
[root@master kubernetes-jenkins]# kubectl exec -it jenkins-b96f7764f-qkzvd -n devops-tools -- bash
jenkins@jenkins-b96f7764f-qkzvd:/$ cat /var/jenkins_home/secrets/initialAdminPassword
557fc27bdf4149bb824b3c6e21a7f823
# 修改密码
部署harbor：
# 前提是安装好 docker 和 docker compose
# 1.配置阿里云的repo源
yum install -y yum-utils
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 2.安装docker服务
yum install docker-ce-20.10.6 -y
# 启动docker，设置开机自启
systemctl start docker && systemctl enable docker.service
# 3.查看docker版本，docker compose版本
[root@harbor ~]# docker version
[root@harbor ~]# docker compose version
# 5.安装 harbor，到 harbor 官网或者 github 下载harbor源码包，上传到本地。
wget https://github.com/goharbor/harbor/releases/download/v2.8.3/harbor-offline-installer-v2.8.3.tgz
[root@localhost ~]# ls
anaconda-ks.cfg harbor-offline-installer-v2.8.3.tgz
# 6.解压
[root@localhost ~]# tar xf harbor-offline-installer-v2.8.3.tgz
[root@harbor ~]# ls
anaconda-ks.cfg harbor harbor-offline-installer-v2.8.3.tgz
[root@harbor ~]# cd harbor
[root@harbor harbor]# ls
common.sh harbor.yml.tmpl LICENSE
harbor.v2.8.3.tar.gz install.sh prepare
# 7.修改配置文件
[root@harbor harbor]# vim harbor.yml.tmpl
# Configuration file of Harbor
# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: 192.168.220.104 #修改为主机ip地址
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 5001 #可以进行修改端口
# https related config
#https:
# https port for harbor, default is 443
# port: 443
# The path of cert and key files for nginx
#certificate: /your/certificate/path
#private_key: /your/private/key/path
# # Uncomment following will enable tls communication between all harbor components
# internal_tls:
# # set enabled to true means internal tls is enabled
# enabled: true
# # put your cert and key files on dir
# dir: /etc/harbor/tls/internal
# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433
# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345 #登录密码
# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433
# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345 #登录密码，可以修改
# Harbor DB configuration
database:
# The password for the root user of Harbor DB. Change this before any production use.
password: root123 #这是 Harbor 数据库的 root 用户的密码
# The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
max_idle_conns: 100 #这是空闲连接池中的最大连接数。
# The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
# Note: the default number of connections is 1024 for postgres of harbor.
max_open_conns: 900 #这是到数据库的最大打开连接数。
# 8.执行部署脚本
[root@harbor harbor]# ./install.sh
看harbor相关容器运行情况：
[root@harbor harbor]# docker compose ps |grep harbor
harbor-core goharbor/harbor-core:v2.8.3 "/harbor/entrypoint.…" core About a minute ago Up About a minute (healthy)
harbor-db goharbor/harbor-db:v2.8.3 "/docker-entrypoint.…" postgresql About a minute ago Up About a minute (healthy)
harbor-jobservice goharbor/harbor-jobservice:v2.8.3 "/harbor/entrypoint.…" jobservice About a minute ago Up About a minute (healthy)
harbor-log goharbor/harbor-log:v2.8.3 "/bin/sh -c /usr/loc…" log About a minute ago Up About a minute (healthy) 127.0.0.1:1514->10514/tcp
harbor-portal goharbor/harbor-portal:v2.8.3 "nginx -g 'daemon of…" portal About a minute ago Up About a minute (healthy)
nginx goharbor/nginx-photon:v2.8.3 "nginx -g 'daemon of…" proxy About a minute ago Up About a minute (healthy) 0.0.0.0:5001->8080/tcp, :::5001->8080/tcp
redis goharbor/redis-photon:v2.8.3 "redis-server /etc/r…" redis About a minute ago Up About a minute (healthy)
registry goharbor/registry-photon:v2.8.3 "/home/harbor/entryp…" registry About a minute ago Up About a minute (healthy)
registryctl goharbor/harbor-registryctl:v2.8.3 "/home/harbor/start.…" registryctl About a minute ago Up About a minute (healthy)
# 9.测试登录
http://192.168.220.104:5001/
问题：prepare base dir is set to /root/harbor
no config file: /root/harbor/harbor.yml
解决方法：
[root@harbor harbor]# mv harbor.yml.tmpl harbor.yml
# 账号：admin
# 密码：Harbor12345
登录后：
在harbor里创建一个项目k8s-harbor
并且新建一个用户 guan 密码是Gxx123456
授权k8s-harbor这个项目允许guan这个用户去访问，授予项目管理员权限
#10.实现pod集群都用这个harbor仓库；
master机器：
[root@master ~]# vim /etc/docker/daemon.json
{
"registry-mirrors": ["https://ruk1gp3w.mirror.aliyuncs.com"],
"insecure-registries" : ["192.168.220.104:5001"]
}
然后重启docker
[root@master ~]# systemctl daemon-reload
[root@master ~]# systemctl restart docker
[root@master ~]# journalctl -xe
出现 "3月 14 14:49:02 master systemd[1]: Failed to start Docker Application Container Engine.
配置文件不标准。测试harbor的上传和拉取
在原来安装harbor的宿主机上，重新启动harbor相关的容器
[root@harbor harbor]# cd /root
[root@harbor ~]# cd harbor
[root@harbor harbor]# docker compose up -d
pod集群机器上拉取一个镜像或者dockerfile制作一个镜像，修改镜像的名字
[root@master ~]# docker tag nginx:latest 192.168.220.104:5001/k8s-harbor/nginx:latest
[root@master ~]# docker images
192.168.220.104:5001/k8s-harbor/nginx latest 605c77e624dd 2 years ago 141MB
。。。）
本机上传
首先登陆私有仓库
登录使用guan这个用户，密码是Gxx123456
[root@master ~]# docker login 192.168.220.104:5001
Username: guan
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
上传到仓库
[root@master ~]# docker push 192.168.220.104:5001/k8s-harbor/nginx:latest
在浏览器检测已经收到
在nfs机器上拉取pod机器上传的镜像：
# 1.配置阿里云的repo源
yum install -y yum-utils
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 2.安装docker服务
yum install docker-ce-20.10.6 -y
# 启动docker，设置开机自启
systemctl start docker && systemctl enable docker.service
root@nfs ~]# vim /etc/docker/daemon.json
{
"registry-mirrors": ["https://ruk1gp3w.mirror.aliyuncs.com"],
"insecure-registries" : ["192.168.203.128:80"]
}
然后重启docker
[root@nfs ~]# systemctl daemon-reload
[root@nfs ~]# systemctl restart docker
[root@nfs ~]# docker login 192.168.220.104:5001
Username: guan
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
拉取成功：
[root@nfs ~]# docker pull 192.168.220.104:5001/k8s-harbor/nginx:latest
latest: Pulling from k8s-harbor/nginx
a2abf6c4d29d: Pull complete
a9edb18cadd1: Pull complete
589b7251471a: Pull complete
186b1aaa4aa6: Pull complete
b4df32aa5a72: Pull complete
a0bcbecc962e: Pull complete
Digest: sha256:ee89b00528ff4f02f2405e4ee221743ebc3f8e8dd0bfd5c4c20a2fa2aaa7ede3
Status: Downloaded newer image for 192.168.220.104:5001/k8s-harbor/nginx:latest
192.168.220.104:5001/k8s-harbor/nginx:latest
[root@nfs ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
192.168.220.104:5001/k8s-harbor/nginx latest 605c77e624dd 2 years ago 141MB七，使用dashboard对整个集群资源进行掌控
[root@master ~]# mkdir dashboard
[root@master ~]# cd dashboard/
[root@master dashboard]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.0/aio/deploy/recommended.yaml
[root@master dashboard]# ls
recommended.yaml
修改配置文件，因为访问用户需要RBAC鉴权才能访问资源。
[root@master dashboard]# vim recommended.yaml
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort #指定类型
ports:
- port: 443
targetPort: 8443
nodePort: 30088 #指定宿主机端口号
selector:
k8s-app: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: serviceaccount-cluster-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: system:serviceaccount:kubernetes-dashboard:kubernetes-dashboard
查看是否启动dashboard的pod
[root@master dashboard]# kubectl get pod --all-namespaces|grep dashboard
kubernetes-dashboard dashboard-metrics-scraper-799d786dbf-lltbf 1/1 Running 0 39s
kubernetes-dashboard kubernetes-dashboard-546cbc58cd-p4xlr 1/1 Running 0 39s
查看服务是否创建
[root@master dashboard]# kubectl get svc --all-namespaces|grep dash
kubernetes-dashboard dashboard-metrics-scraper ClusterIP 10.98.46.11 <none> 8000/TCP 66s
kubernetes-dashboard kubernetes-dashboard NodePort 10.109.239.147 <none> 443:30088/TCP 66s
在浏览器里访问，使用https协议去访问
https://192.168.220.100:30088/
点击继续访问
https://192.168.220.100:30088/#/login
出现一个登录画图，需要输入token
获取dashboard 的secret的名字
[root@master dashboard]# kubectl get secret -n kubernetes-dashboard|grep dashboard-token
kubernetes-dashboard-token-pnt2v kubernetes.io/service-account-token 3 6m6s
获取secret里的token
[root@master dashboard]# kubectl describe secret kubernetes-dashboard-token-pnt2v -n kubernetes-dashboard
token: eyJhbGciOiJSUzI1NiIsImtpZCI6ImhvV1g5cTQ1Q2F1N1A5RGxCQnhrTkVkeFNmczgtRG5WNlFMNWJ4SzcyaTQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi1wbnQydiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjBkYmYwMGFkLWU1YTktNDc0Ny05YWZiLWM0MDk2N2RmY2I1MCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.l1U9GljbDpv3OpQcqijj10YaqymqLC18pY1Ut6-UPzNiY8sSyKvpnC9_6aCMFLz-mXV2x17TWmvOME5mK8DO0pV8QH3JJcsS-XCkn3RBxHygZfVYqcpSnoPibCA6QhaCCchSYDQ9a6fmriIztySgsXCtNV4Rfow49l5pkafTYLllV3dXp5SbxsL3TL46IOXw2uQ0iG0JD5QYj4pMfe_rZiwDQNwriaVqLb84K88AglDh3uniPg8XuYWs_nDIy3pztdwQOjWFLCy8NsQ1TGftZg6HRXD9pon2W8QeUj3vhKvA1B8L1MdzSfGpLPIojjHVLHB9C6aCnI3HqrjjvmrKjA
然后在浏览器上就可以访问资源了
八，安装部署Prometheus+grafana：
安装Prometheus
[root@prometheus ~]# ls
anaconda-ks.cfg
grafana-enterprise-9.1.2-1.x86_64.rpm
mysqld_exporter-0.12.1.linux-amd64 (1).tar.gz
prometheus-2.43.0.linux-amd64.tar.gz
[root@prometheus ~]# mkdir /prom
[root@prometheus ~]# mv prometheus-2.43.0.linux-amd64.tar.gz /prom/prometheus-2.43.0.linux-amd64.tar.gz
[root@prometheus ~]# cd /prom
[root@prometheus prom]# ls
prometheus-2.43.0.linux-amd64.tar.gz
[root@prometheus prom]# tar xf prometheus-2.43.0.linux-amd64.tar.gz
[root@prometheus prom]# ls
prometheus-2.43.0.linux-amd64
[root@prometheus prom]# mv prometheus-2.43.0.linux-amd64 prometheus
[root@prometheus prom]# ls
prometheus prometheus-2.43.0.linux-amd64.tar.gz
[root@prometheus prometheus]# PATH=/prom/prometheus:$PATH
[root@prometheus prometheus]# vim /etc/profile
添加到末尾：PATH=/prom/prometheus:$PATH
[root@prometheus prometheus]# nohup prometheus --config.file=/prom/prometheus/prometheus.yml & #在后台运行
[1] 2137
[root@prometheus prometheus]# nohup: 忽略输入并把输出追加到"nohup.out"
看进程：
[root@prometheus prometheus]# ps aux |grep prome
root 2137 0.4 2.3 798956 44252 pts/0 Sl 12:38 0:00prometheus --config.file=/prom/prometheus/prometheus.yml
看端口：
[root@prometheus prometheus]# netstat -anplut |grep prom
tcp6 0 0 :::9090 :::* LISTEN 2137/prometheus
tcp6 0 0 ::1:48882 ::1:9090 ESTABLISHED 2137/prometheus
tcp6 0 0 ::1:9090 ::1:48882 ESTABLISHED 2137/prometheus
关闭防火墙：
[root@prometheus prometheus]# service firewalld stop
Redirecting to /bin/systemctl stop firewalld.service
[root@prometheus prometheus]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
把Prometheus做成一个服务进行管理，
[root@prometheus prometheus]# vim /usr/lib/systemd/system/prometheus.service
[Unit]
Description=prometheus
[Service]
ExecStart=/prom/prometheus/prometheus --config.file=/prom/promethe
us/prometheus.yml
ExecReload=/bin/kill -HUP $MAINPID
killMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
[root@prometheus prometheus]# systemctl daemon-reload #重新加载systemd相关的服务
浏览器访问ip+9090端口
杀死Prometheus进程 ：
[root@prometheus prometheus]# ps aux|grep prom
root 2137 0.0 2.9 799212 54400 pts/0 Sl 12:38 0:00prometheus --config.file=/prom/prometheus/prometheus.yml
root 2346 0.0 0.0 112824 972 pts/0 S+ 12:59 0:00 grep --color=auto prom
[root@prometheus prometheus]# kill -9 2137
在master或者被监听的机器上操作（安装相应的exporter，常规的是node_exporter）：
[root@master ~]# tar xf node_exporter-1.4.0-rc.0.linux-amd64.tar.gz
[root@master ~]# mv node_exporter-1.4.0-rc.0.linux-amd64 /node_exporter
[root@master ~]# cd /node_exporter/
[root@master node_exporter]# ls
LICENSE node_exporter NOTICE
修改环境变量
[root@master node_exporter]# PATH=/node_exporter/:$PATH
[root@master node_exporter]# vim /root/.bashrc
PATH=/node_exporter/:$PATH
[root@prometheus /]# nohup node_exporter --web.listen-address 0.0.0.0:8090 &
[1] 4844
[root@prometheus /]# nohup: 忽略输入并把输出追加到"nohup.out"
[root@prometheus /]# ps aux |grep node
root 4844 0.0 0.7 716544 13104 pts/0 Sl 13:55 0:00 node_exporter --web.listen-address 0.0.0.0:8090
关闭防火墙：
[root@prometheus /]# service firewalld stop
Redirecting to /bin/systemctl stop firewalld.service
[root@prometheus /]# systemctl disable firewalld
查看selinux的状态：
[root@prometheus /]# getenforce
关闭selinux：
#临时关闭selinux
setenforce 0
#永久关闭selinux
sed -i '/^SELINUX=/ s/enforcing/disabled/' /etc/selinux/config
浏览器访问：ip+8090
设置node_exporter开机启动
[root@prometheus /]# vim /etc/rc.local
nohup /node_exporter/node_exporter/node_exporter --web.listen-address 0.0.0.0:8090 &
[root@prometheus node_exporter]# chmod +x /etc/rc.d/rc.local
在Prometheus里添加我们在哪些机器里安装了exporter程序，就可以去pull了
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "master"
static_configs:
- targets: ["192.168.220.100:8090"]
（。。。）
~
[root@prometheus prometheus]# service prometheus restart
Redirecting to /bin/systemctl restart prometheus.service安装部署Grafana和Prometheus安装在一起：
wget https://d1.grafana.com/enterprise/release/grafana-enterprise-9.1.2-1.x86_64.rpm
[root@prometheus ~]# yum install grafana-enterprise-9.1.2-1.x86_64.rpm -y
启动
[root@prometheus ~]# service grafana-server start
Starting grafana-server (via systemctl): [ 确定 ]
[root@prometheus ~]# ps aux |grep grafana
grafana 5115 2.1 3.6 1129728 68768 ? Ssl 14:28 0:00 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-server.pid --packaging=rpm cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning
root 5124 0.0 0.0 112824 976 pts/0 S+ 14:28 0:00 grep --color=auto grafana
[root@prometheus ~]# netstat -anlput |grep grafana
tcp 0 0 192.168.220.165:54852 34.120.177.193:443 ESTABLISHED 5115/grafana-server
tcp 0 0 192.168.220.165:46122 185.199.109.133:443 ESTABLISHED 5115/grafana-server
tcp6 0 0 :::3000 :::* LISTEN 5115/grafana-server
浏览器访问 ip+3000端口
默认用户admin
密码：admin
添加数据源：
管理——》数据源-》添加Prometheus -》http://192.168.220.105:9090
添加模板：1860
仪表盘-》 导入
监听的是master的使用情况九.安装部署firewalld防火墙和jumpserver堡垒机，对web集群进行保护。
jumpserver 的安装部署
准备一台 2核4G （最低）且可以访问互联网的 64 位 Linux 主机；
以 root 用户执行如下命令一键安装 JumpServer。curl -sSL https://resource.fit2cloud.com/jumpserver/jumpserver/releases/latest/download/quick_start.sh | bash
部署firewall服务器，对内网进行保护
ens33的wan口ip为外面window的ip网段，连接外网
# 关闭虚拟机，增加一块网卡（ens36）用来连内网
vim /etc/sysconfig/network-scripts/ifcfg-ens36
关闭dhcp
添加：
IPADDR=192.168.220.107
GATEWAY=192.168.220.106 #jump server的ip地址
NETMASK=255.255.255.0
DNS1=114.114.114.114
而且内网的机器都需要把网关设置为firewalld的ip
GATEWAY=192.168.220.108
#内网的机器出去都需要通过这台firewalld机器# 编写脚本实现SNAT_DNAT功能
[root@firewalld ~]# cat snat_dnat.sh
#!/bin/bash
# 开启路由功能
echo 1 >/proc/sys/net/ipv4/ip_forward
# stop firewall
systemctl stop firewalld
systemctl disable firewalld
# clear iptables rule
iptables -F
iptables -t nat -F
# enable snat
iptables -t nat -A POSTROUTING -s 192.168.220.0/24 -o ens33 -j MASQUERADE
#内网来的192.168.220.0网段过来的ip地址全部伪装（替换）为ens33接口的公网ip地址,好处就是不需要考虑ens33接口的ip地址是多少，你是哪个ip地址，我就伪装成哪个ip地址
iptables -t filter -P INPUT ACCEPT #默认是ACCPET
# web服务器上操作，开启相应的端口，默认为drop，实现对k8s集群的保护。
[root@k8smaster ~]# cat open.sh
#!/bin/bash
# open ssh
iptables -t filter -A INPUT -p tcp --dport 22 -j ACCEPT
# 打开 dns
iptables -t filter -A INPUT -p udp --dport 53 -s 192.168.220.0/24 -j ACCEPT 可以dns解析来自于内网段的机器ip
# 打开 dhcp
iptables -t filter -A INPUT -p udp --dport 67 -j ACCEPT
# 打开 http/https
iptables -t filter -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -t filter -A INPUT -p tcp --dport 443 -j ACCEPT
# 打开mysql端口
iptables -t filter -A INPUT -p tcp --dport 3306 -j ACCEPT
# 默认为drop
iptables -t filter -P INPUT DROP
十.安装部署ansible机器，写好主机清单,便于日后的自动化运维。
1.建立免密通道 在ansible主机上生成密钥对，上传公钥到所有服务器的root用户家目录下
# 所有服务器上开启ssh服务 ，开放22号端口，允许root用户登录
[root@ansible ~]# yum install -y epel-release
[root@ansible ~]# yum install ansible
[root@ansible ~]# ssh-keygen
[root@ansible ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.220.100
[root@ansible ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.220.101
[root@ansible ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.220.102
[root@ansible ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.220.103
（。。。）2.编写主机清单
[root@ansible ansible]# vim hosts
[master]
192.168.220.100
[node]
192.168.220.101
192.168.220.102
[nfs]
192.168.220.103
[harbor]
192.168.220.104
[prometheus]
192.168.220.105
[jumpserver]
192.168.220.1063.测试
[root@ansible ansible]# ansible all -m shell -a "ip add"项目心得：
1.更加深入的了解了k8s的各个功能（service，pv，pvc，ingress）等。
2.更加了解开发和运维的关系。
3.对负载均衡和高可用， 自动扩缩有了认识。
4.对各个服务(Prometheus, nfs等) 深入了解
K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容
更多推荐
12
0- 0
已为社区贡献1条内容
所有评论(0)