K8S部署
KubernetesKubernetes 是用于自动部署，扩展和管理容器化应用程序的开源系统。 它将组成应用程序的容器组合成逻辑单元，以便于管理和服务发现。Kubernetes 源自Google 15 年生产环境的运维经验，同时凝聚了社区的最佳创意和实践。 星际尺度 Google 每周运行数十亿个容器，Kubernetes 基于与之相同的原则来设计，能够在不扩张运维团队的情况下进行规模扩展。 处处
·
https://kubernetes.io/zh/一、环境
节点CPU核数必须是：>= 2核，否则k8s无法启动
DNS网络：最好设置为本地网络连通的DNS,否则网络不通，无法下载一些镜像
linux内核： linux内核必须是 4 版本以上，因此必须把linux核心进行升级
1、服务器说明
| 主机名 | 操作系统版本 | ip | docker version | k8s | 备注 |
| master01 | CentOS Linux release 7.6.1810 (Core) | 192.168.23.160 | 20.10.12 | V1.22.2 | master主机 |
| node01 | CentOS Linux release 7.6.1810 (Core) | 192.168.23.161 | 20.10.12 | V1.22.2 | node节点 |
| node02 | CentOS Linux release 7.6.1810 (Core) | 192.168.23.162 | 20.10.12 | V1.22.2 | node节点 |
2、其他说明
查看系统版本 cat /etc/redhat-release
查看系统内核 uname -r
VMware Workstation版本 Pro 16.2.1 build-18811642
3、给节点设置主机名称
[root@k8s-master01 ~]# hostnamectl set-hostname k8s-master01
[root@k8s-node01 ~]# hostnamectl set-hostname k8s-node01
[root@k8s-node02 ~]# hostnamectl set-hostname k8s-node02二、服务器基础配置
所有节点
1、配置IP HOST映射关系
cat >> /etc/hosts <<EOF
192.168.23.160 k8s-master01
192.168.23.161 k8s-node01
192.168.23.162 k8s-node02
EOF
2、安装依赖
yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git iproute lrzsz bash-completion tree bridge-utils unzip bind-utils gcc4、关闭防火墙
systemctl stop firewalld && systemctl disable firewalld5、禁用swap
swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab6、关闭selinux
setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config7、升级Linux内核（内核大于4，忽略）
查看内核命令
uname -r下载内核
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm安装内核
yum --enablerepo=elrepo-kernel install -y kernel-lt设置开机从新内核启动
查看已安装内核命令 rpm -qa | grep -i kernel 或 rpm -qa kernel 需要重启才可以看到最新安装版本
设置默认启动内核
grub2-set-default 'kernel-lt-5.4.170-1.el7.elrepo.x86_64'三、Docker安装
所有节点都需要安装docker
1、安装依赖
yum install -y yum-utils device-mapper-persistent-data lvm22、设置Docker源
配置一个稳定（stable）的仓库，仓库配置会保存到/etc/yum.repos.d/docker-ce.repo文件中
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo3、安装Docker CE
docker安装版本查看
yum list docker-ce --showduplicates | sort -r更新与安装Docker
yum update -y && yum install docker-ce4、命令补全
安装bash-completion
yum -y install bash-completion加载bash-completion
source /etc/profile.d/bash_completion.sh5、配置daemon.json文件
由于Docker Hub的服务器在国外，下载镜像会比较慢，可以配置镜像加速器。主要的加速器有：Docker官方提供的中国registry mirror、阿里云加速器、DaoCloud 加速器，本文以阿里加速器配置为例
创建目录
mkdir /etc/docker配置daemon.json文件
cat > /etc/docker/daemon.json <<EOF
{"exec-opts": ["native.cgroupdriver=systemd"],"log-driver": "json-file","log-opts": {"max-size": "100m"},"registry-mirrors": ["https://98zyayk8.mirror.aliyuncs.com"]}
EOF
说明"registry-mirrors": ["https://98zyayk8.mirror.aliyuncs.com"]是配置镜像加速器
6、启动Docker
systemctl daemon-reload && systemctl restart docker && systemctl enable docker7、查看Docker
docker --version四、K8S安装
1、调整内核参数
cat > kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF将优化内核文件拷贝到/etc/sysctl.d/文件夹下，这样优化文件开机的时候能够被调用
cp kubernetes.conf /etc/sysctl.d/kubernetes.conf2、创建日志配置
创建日志目录
mkdir /var/log/journal创建配置文件存放目录
mkdir /etc/systemd/journald.conf.dcat > /etc/systemd/journald.conf.d/99-prophet.conf <<EOF
[Journal]
Storage=persistent
Compress=yes
SyncIntervalSec=5m
RateLimitInterval=30s
RateLimitBurst=1000
SystemMaxUse=10G
SystemMaxFileSize=200M
MaxRetentionSec=2week
ForwardToSyslog=no
EOF说明
#持久化保存到磁盘
Storage=persistent
#压缩历史日志
Compress=yes
SyncIntervalSec=5m
RateLimitInterval=30s
RateLimitBurst=1000
#最大占用空间
SystemMaxUse=10G
#单个日志文件大小
SystemMaxFileSize=200M
#日志保存时间2周
MaxRetentionSec=2week
#不将日志转发Syslog
ForwardToSyslog=no
重启systemd journald的配置
systemctl restart systemd-journald3、打开文件数调整 (可忽略，不执行)
echo "* soft nofile 65536" >> /etc/security/limits.confecho "* hard nofile 65536" >> /etc/security/limits.conf4、kube-proxy 开启 ipvs 前置条件
modprobe br_netfiltercat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF
使用lsmod命令查看这些文件是否被引导
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack5、设置kubernetes源
安装kubernetes的时候，需要安装kubelet, kubeadm等包，但k8s官网给的yum源是packages.cloud.google.com，国内访问不了，此时我们可以使用阿里云的yum仓库镜像
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
说明
[] 中括号中的是repository id，唯一，用来标识不同仓库
name 仓库名称，自定义
baseurl 仓库地址
enable 是否启用该仓库，默认为1表示启用
gpgcheck 是否验证从该仓库获得程序包的合法性，1为验证
repo_gpgcheck 是否验证元数据的合法性 元数据就是程序包列表，1为验证
gpgkey=URL 数字签名的公钥文件所在位置，如果gpgcheck值为1，此处就需要指定gpgkey文件的位置，如果gpgcheck值为0就不需要此项了
6、安装kubelet、kubeadm和kubectl
版本查看
yum list kubelet --showduplicates | sort -r 安装
yum install -y kubeadm-1.22.2 kubelet-1.22.2 kubectl-1.22.2若不指定版本直接运行‘yum install -y kubelet kubeadm kubectl’则默认安装最新版
说明
- kubelet 运行在集群所有节点上，用于启动Pod和容器等对象的工具
- kubeadm 用于初始化集群，启动集群的命令工具
- kubectl 用于和集群通信的命令行，通过kubectl可以部署和管理应用，查看各种资源，创建、删除和更新各种组件
启动 kubelet并设置开机启动
systemctl enable kubelet && systemctl start kubeletkubelet命令补全
echo "source <(kubectl completion bash)" >> ~/.bash_profile
source .bash_profile 7、下载镜像
镜像下载的脚本
Kubernetes几乎所有的安装组件和Docker镜像都放在goolge自己的网站上,直接访问可能会有网络问题，这里的解决办法是从阿里云镜像仓库下载镜像，拉取到本地以后改回默认的镜像tag
新建脚本image.sh
#!/bin/bash
url=registry.cn-hangzhou.aliyuncs.com/google_containers
version=v1.22.2
images=(`kubeadm config images list --kubernetes-version=$version|awk -F '/' '{print $2}'`)
for imagename in ${images[@]} ; do
docker pull $url/$imagename
docker tag $url/$imagename k8s.gcr.io/$imagename
docker rmi -f $url/$imagename
done
url为阿里云镜像仓库地址，version为安装的kubernetes版本
下载镜像
行脚本image.sh，下载指定版本的镜像，运行脚本前先赋权
[root@k8s-master01 ~]# chmod u+x image.sh
[root@k8s-master01 ~]# ./image.sh
[root@k8s-master01 ~]# docker images拷贝到node01节点
[root@k8s-master01 ~]# scp -r image.sh root@k8s-node01:/root/执行脚本
[root@k8s-node01 ~]# ./image.sh拷贝到node02节点
[root@k8s-master01 ~]# scp -r image.sh root@k8s-node02:/root/执行脚本
[root@k8s-node02 ~]# ./image.sh8、初始化Master
[root@k8s-master01 ~]# kubeadm init --kubernetes-version=1.22.2 \--apiserver-advertise-address=192.168.23.160 \--image-repository registry.aliyuncs.com/google_containers \--service-cidr=10.1.0.0/16 \--pod-network-cidr=10.244.0.0/16 | tee kubeadm-init.log命令解析：
- --pod-network-cidr: 定义pod网段为：10.244.0.0/16
- --kubernetes-version=1.22.2 版本设置
- --apiserver-advertise-address指定master的interface
- --apiserver-advertise-address：master主机内网IP地址
- --image-repository：指定阿里云镜像仓库地址。由于kubeadm 默认从官网http://k8s.grc.io下载所需镜像，国内无法访问，因此需要通过–image-repository指定阿里云镜像仓库地址。（国外主机可以不用设定）
加载环境变量
[root@k8s-master01 ~]# mkdir -p $HOME/.kube
[root@k8s-master01 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master01 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config9、安装pod网络
[root@k8s-master01 ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml10、node节点加入集群
在所有node节点执行上文master节点部署成功后生成的信息
查看信息
[root@k8s-master01 ~]# cat kubeadm-init.log
node01
[root@k8s-node01 ~]# kubeadm join 192.168.23.160:6443 --token p8x62i.rc2czn80qfr4en8s \
> --discovery-token-ca-cert-hash sha256:b0c8d888ed199961940490f510021e6b0dc30a750b42ffab26b45742c37e9716node02
[root@k8s-node02 ~]# kubeadm join 192.168.23.160:6443 --token p8x62i.rc2czn80qfr4en8s \
> --discovery-token-ca-cert-hash sha256:b0c8d888ed199961940490f510021e6b0dc30a750b42ffab26b45742c37e9716master01查看集群
[root@k8s-master01 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01 Ready control-plane,master 15m v1.22.2
k8s-node01 Ready <none> 5m46s v1.22.2
k8s-node02 Ready <none> 5m42s v1.22.2
[root@k8s-master01 ~]#
查询工作空间中pod容器的详细信息
[root@k8s-master01 ~]# kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-7f6cbbb7b8-47hqw 1/1 Running 0 16m 10.244.0.2 k8s-master01 <none> <none>
coredns-7f6cbbb7b8-cns6k 1/1 Running 0 16m 10.244.0.3 k8s-master01 <none> <none>
etcd-k8s-master01 1/1 Running 0 16m 192.168.23.160 k8s-master01 <none> <none>
kube-apiserver-k8s-master01 1/1 Running 0 16m 192.168.23.160 k8s-master01 <none> <none>
kube-controller-manager-k8s-master01 1/1 Running 0 16m 192.168.23.160 k8s-master01 <none> <none>
kube-flannel-ds-684pr 1/1 Running 0 6m52s 192.168.23.161 k8s-node01 <none> <none>
kube-flannel-ds-l2mk7 1/1 Running 0 6m48s 192.168.23.162 k8s-node02 <none> <none>
kube-flannel-ds-z5mvl 1/1 Running 0 8m4s 192.168.23.160 k8s-master01 <none> <none>
kube-proxy-2th5p 1/1 Running 0 6m52s 192.168.23.161 k8s-node01 <none> <none>
kube-proxy-56jxv 1/1 Running 0 16m 192.168.23.160 k8s-master01 <none> <none>
kube-proxy-6c9jb 1/1 Running 0 6m48s 192.168.23.162 k8s-node02 <none> <none>
kube-scheduler-k8s-master01 1/1 Running 0 16m 192.168.23.160 k8s-master01 <none> <none>
[root@k8s-master01 ~]#
五、Dashboard安装
Web 界面 (Dashboard) | Kubernetes
浏览器访问https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml
1、新建recommended.yaml
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: v1
kind: Namespace
metadata:
name: kubernetes-dashboard
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30001
selector:
k8s-app: kubernetes-dashboard
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kubernetes-dashboard
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-csrf
namespace: kubernetes-dashboard
type: Opaque
data:
csrf: ""
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-key-holder
namespace: kubernetes-dashboard
type: Opaque
---
kind: ConfigMap
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-settings
namespace: kubernetes-dashboard
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
rules:
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster", "dashboard-metrics-scraper"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
verbs: ["get"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
rules:
# Allow Metrics Scraper to get metrics from the Metrics server
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
containers:
- name: kubernetes-dashboard
image: kubernetesui/dashboard:v2.4.0
imagePullPolicy: Always
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
- --namespace=kubernetes-dashboard
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
nodeSelector:
"kubernetes.io/os": linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
ports:
- port: 8000
targetPort: 8000
selector:
k8s-app: dashboard-metrics-scraper
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: dashboard-metrics-scraper
template:
metadata:
labels:
k8s-app: dashboard-metrics-scraper
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: dashboard-metrics-scraper
image: kubernetesui/metrics-scraper:v1.0.7
ports:
- containerPort: 8000
protocol: TCP
livenessProbe:
httpGet:
scheme: HTTP
path: /
port: 8000
initialDelaySeconds: 30
timeoutSeconds: 30
volumeMounts:
- mountPath: /tmp
name: tmp-volume
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
serviceAccountName: kubernetes-dashboard
nodeSelector:
"kubernetes.io/os": linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
volumes:
- name: tmp-volume
emptyDir: {}
2、修改配置recommended.yaml
3、部署Dashboard
[root@k8s-master01 ~]# kubectl apply -f recommended.yaml4、查看状态查看
[root@k8s-master01 ~]# kubectl get all -n kubernetes-dashboard
5、使用token认证进行登录
创建serviceAccount，假如用户名为dashboard-admin
kubectl create serviceaccount dashboard-admin -n kube-system通过clusterrolebinding绑定到内置的cluster-admin角色上
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin令牌查看
kubectl describe secrets -n kube-system dashboard-admin
访问
https://192.168.23.160:30001/#/login
参考
K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容
更多推荐
1
0- 0
已为社区贡献1条内容
所有评论(0)