【kubernetes】使用kubeadm快速搭建k8s集群学习
专题：Docker&KubernetesDocker&Kubernetes更多内容请点击 我的博客 查看，欢迎来访。系统初始化配置部署Kubernetes(k8s)时，为什么要关闭swap、selinux、防火墙？Swap会导致docker的运行不正常，性能下降，是个bug，但是后来关闭swap就解决了，就变成了通用方案，后续可能修复了(我没关注)，基本上默认关闭了就OK，内存开大
·
Docker&Kubernetes
更多内容请点击 我的博客 查看，欢迎来访。
系统初始化配置
部署Kubernetes(k8s)时，为什么要关闭swap、selinux、防火墙？
- Swap会导致docker的运行不正常，性能下降，是个bug，但是后来关闭swap就解决了，就变成了通用方案，后续可能修复了(我没关注)，基本上默认关闭了就OK，内存开大点儿不太会oom，本来容器也可以限制内存的使用量，控制一下就好。
- Selinux是内核级别的一个安全模块，通过安全上下文的方式控制应用服务的权限，是应用和操作系统之间的一道ACL，但不是所有的程序都会去适配这个模块，不适配的话开着也不起作用，何况还有规则影响到正常的服务。比如权限等等相关的问题。
- 防火墙看你说的是那种，如果你说的是iptables的话 ，那样应该是开着的，不应该关。k8s通过iptables做pod间流量的转发和端口映射，如果你说的防火墙是firewalld那就应该关闭，因为firewalld和iptables属于前后两代方案，互斥。当然，如果k8s没用iptables，比如用了其他的方式那就不需要用到自然可以关掉。
配置主机名和hosts
配置所有节点的主机名
# 192.168.99.100 Master节点
[root@localhost ~]# hostnamectl set-hostname k8s-master
[root@localhost ~]# su
[root@k8s-master ~]#
# 192.168.99.201 Node1节点
[root@localhost ~]# hostnamectl set-hostname k8s-node1
[root@localhost ~]# su
[root@k8s-node1 ~]#
# 192.168.99.202 Node2节点
[root@localhost ~]# hostnamectl set-hostname k8s-node2
[root@localhost ~]# su
[root@k8s-node2 ~]#
配置hosts，以Master节点为例，其他一样
[root@k8s-master ~]# vim /etc/hosts
[root@k8s-master ~]# cat /etc/hosts | grep k8s
192.168.99.100 k8s-master
192.168.99.201 k8s-node1
192.168.99.202 k8s-node2
# 之后ping主机名就可以通了
[root@k8s-master ~]# ping k8s-node2
关闭防火墙
所有节点都需要关闭
# 关闭防火墙
[root@k8s-master ~]# systemctl stop firewalld
# 禁止开机启动
[root@k8s-master ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
关闭SELinux
关闭selinux：linux下的一个安全机制，主要是对文件系统访问做一个权限控制，这个权限控制会影响到kubernetes中的一个组件kuberlete的安装，因为这个组件的安装会访问本地的文件操作系统。
所有节点都需要操作
# 先临时关闭，永久关闭需要重启主机
[root@k8s-master ~]# setenforce 0
# 永久关闭
[root@k8s-master ~]# sed -i 's/enforcing/disabled/' /etc/selinux/config
[root@k8s-master ~]# cat /etc/selinux/config | grep ^SELINUX=
SELINUX=disabled
关闭Swap
禁止swap分区：swap分区的作用是当物理内存不足时，利用swap分区做数据交换，但是在kubernetes中完全不支持swap分区，所以必须禁止掉，或者创建系统的时候就不创建它。
所有节点都需要关闭
# 临时关闭
[root@k8s-master ~]# swapoff -a
# 永久关闭
[root@k8s-master ~]# cat /etc/fstab | grep swap
/dev/mapper/centos-swap swap swap defaults 0 0
[root@k8s-master ~]# vim /etc/fstab
[root@k8s-master ~]# cat /etc/fstab | grep swap
# /dev/mapper/centos-swap swap swap defaults 0 0
将桥接的IPv4流量传递到iptables的链
有一些ipv4的流量不能走iptables链，（linux内核的一个过滤器，每个流量都会经过他，然后再匹配是否可进入当前应用进程去处理），导致流量丢失。
# k8s.conf文件原来不存在，需要自己创建的
[root@k8s-master ~]# cat > /etc/sysctl.d/k8s.conf << EOF
> net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-call-iptables = 1
> EOF
[root@k8s-master ~]# sysctl --system
* Applying /usr/lib/sysctl.d/00-system.conf ...
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/k8s.conf ...
* Applying /etc/sysctl.conf ...
所有节点安装Docker、kubeadm、kubelet、kubectl
安装安装Doctor、kubeadm【引导集群的客户端工具】、kubelet【kubernetes中管理容器】
安装Docker（实际版本18.06.3）
# 配置Docker官方源并将下载的源文件存放于/etc/yum.repos.d/下
[root@k8s-master ~]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
# 安装Docker-CE，先测试最新版，如果指定的版本号错误，会安装最新版，先测试下行不行
[root@k8s-master ~]# yum install docker-ce-19.03.9-3.el7 -y
# 实际需要安装
[root@k8s-master ~]# yum install docker-ce-18.06.3.ce-3.el7
# 启动Docker
[root@k8s-master ~]# systemctl start docker
[root@k8s-master ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
# 查看Docker版本
[root@k8s-master ~]# docker --version
Docker version 19.03.9, build 9d988398e7
安装kubeadm，kubelet和kubectl（实际版本1.18.2）
ubernetes默认CRI（容器运行时）为Docker，因此先安装Docker。
# 设置源
[root@k8s-master ~]# cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
# 开始安装
[root@k8s-master ~]# yum install kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2 -y
# 提示xxx.rpm公钥尚未安装
# 跳过公钥检查
[root@k8s-master ~]# yum install kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2 --nogpgcheck -y
已安装:
kubeadm.x86_64 0:1.18.2-0 kubectl.x86_64 0:1.18.2-0 kubelet.x86_64 0:1.18.2-0
作为依赖被安装:
conntrack-tools.x86_64 0:1.4.4-7.el7 cri-tools.x86_64 0:1.13.0-0
kubernetes-cni.x86_64 0:0.7.5-0 libnetfilter_cthelper.x86_64 0:1.0.0-11.el7
libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2
socat.x86_64 0:1.7.3.2-2.el7
完毕！
[root@k8s-master ~]# whereis kubeadm
kubeadm: /usr/bin/kubeadm
[root@k8s-master ~]# whereis kubectl
kubectl: /usr/bin/kubectl
[root@k8s-master ~]# whereis kubelet
kubelet: /usr/bin/kubelet
# 此时此刻不需要启动kubelet，因为最后kubeadm会一键部署，仅仅设置开机启动即可
[root@k8s-master ~]# systemctl enable kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
部署Kubernetes Master
参数说明
kubeadm init命令参数说明
--apiserver-advertise-address string 设置 apiserver 绑定的 IP.
--apiserver-bind-port int32 设置apiserver 监听的端口. (默认 6443)
--apiserver-cert-extra-sans strings api证书中指定额外的Subject Alternative Names (SANs) 可以是IP 也可以是DNS名称。 证书是和SAN绑定的。
--cert-dir string 证书存放的目录 (默认 "/etc/kubernetes/pki")
--certificate-key string kubeadm-cert secret 中 用于加密 control-plane 证书的key
--config string kubeadm 配置文件的路径.
--cri-socket string CRI socket 文件路径，如果为空 kubeadm 将自动发现相关的socket文件; 只有当机器中存在多个 CRI socket 或者 存在非标准 CRI socket 时才指定.
--dry-run 测试，并不真正执行;输出运行后的结果.
--feature-gates string 指定启用哪些额外的feature 使用 key=value 对的形式。
-h, --help 帮助文档
--ignore-preflight-errors strings 忽略前置检查错误，被忽略的错误将被显示为警告. 例子: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
--image-repository string 选择拉取 control plane images 的镜像repo (default "k8s.gcr.io")
--kubernetes-version string 选择K8S版本. (default "stable-1")
--node-name string 指定node的名称，默认使用 node 的 hostname.
--pod-network-cidr string 指定 pod 的网络， control plane 会自动将 网络发布到其他节点的node，让其上启动的容器使用此网络
--service-cidr string 指定service 的IP 范围. (default "10.96.0.0/12")
--service-dns-domain string 指定 service 的 dns 后缀, e.g. "myorg.internal". (default "cluster.local")
--skip-certificate-key-print 不打印 control-plane 用于加密证书的key.
--skip-phases strings 跳过指定的阶段（phase）
--skip-token-print 不打印 kubeadm init 生成的 default bootstrap token
--token string 指定 node 和control plane 之间，简历双向认证的token ，格式为 [a-z0-9]{6}\.[a-z0-9]{16} - e.g. abcdef.0123456789abcdef
--token-ttl duration token 自动删除的时间间隔。 (e.g. 1s, 2m, 3h). 如果设置为 '0', token 永不过期 (default 24h0m0s)
--upload-certs 上传 control-plane 证书到 kubeadm-certs Secret.
在Master节点kubeadm部署
kubeadm init初始化命令
由于测试修改了kubernetes-version，但是阿里云没有v1.18.3 not found，仍使用v1.18.2
[root@k8s-master ~]# kubeadm init \
--apiserver-advertise-address=192.168.99.100 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.18.2 \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.244.0.0/16
由于默认拉取镜像地址k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址。
初始化报错处理
W0523 16:43:11.953762 3851 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.2
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
[ERROR CRI]: container runtime is not running: output: Client:
Debug Mode: false
Server:
ERROR: Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.39
errors pretty printing info
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
client version 1.40 is too new
Docker版本 19.03.9 过高，考虑升级k8s版本，从 kubernetes 1.18.2 -> 1.18.3，仍然不行，在降级Docker
[root@k8s-master ~]# yum update kubelet-1.18.3 kubeadm-1.18.3 kubectl-1.18.3 --nogpgcheck -y
更新完毕:
kubeadm.x86_64 0:1.18.3-0 kubectl.x86_64 0:1.18.3-0 kubelet.x86_64 0:1.18.3-0
完毕！
docker 19.03.9降级18.06.3安装冲突
# 降级Docker版本
[root@k8s-master ~]# yum list docker-ce --showduplicates | sort -r
已加载插件：fastestmirror
已安装的软件包
可安装的软件包
* updates: mirror.lzu.edu.cn
Loading mirror speeds from cached hostfile
* extras: mirror.lzu.edu.cn
docker-ce.x86_64 3:19.03.9-3.el7 docker-ce-stable
docker-ce.x86_64 3:19.03.8-3.el7 docker-ce-stable
docker-ce.x86_64 3:19.03.7-3.el7 docker-ce-stable
docker-ce.x86_64 3:19.03.6-3.el7 docker-ce-stable
docker-ce.x86_64 3:19.03.5-3.el7 docker-ce-stable
docker-ce.x86_64 3:19.03.4-3.el7 docker-ce-stable
docker-ce.x86_64 3:19.03.3-3.el7 docker-ce-stable
docker-ce.x86_64 3:19.03.2-3.el7 docker-ce-stable
docker-ce.x86_64 3:19.03.1-3.el7 docker-ce-stable
docker-ce.x86_64 3:19.03.0-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.9-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.9-3.el7 @docker-ce-stable
docker-ce.x86_64 3:18.09.8-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.7-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.6-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.5-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.4-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.3-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.2-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.1-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.0-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.3.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.2.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.1.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.0.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 18.03.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.12.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.12.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.09.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.09.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.3.ce-1.el7 docker-ce-stable
docker-ce.x86_64 17.03.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.0.ce-1.el7.centos docker-ce-stable
* base: mirror.lzu.edu.cn
[root@k8s-master ~]# yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine
[root@k8s-master ~]# yum install docker-ce-18.06.3.ce-3.el7
# 安装报错
Transaction check error:
file /usr/bin/docker from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.9-3.el7.x86_64
file /usr/share/bash-completion/completions/docker from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.9-3.el7.x86_64
file /usr/share/fish/vendor_completions.d/docker.fish from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.9-3.el7.x86_64
file /usr/share/man/man1/docker-attach.1.gz from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.9-3.el7.x86_64
file /usr/share/man/man1/docker-checkpoint-create.1.gz from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.9-3.el7.x86_64
file /usr/share/man/man1/docker-checkpoint-ls.1.gz from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.9-3.el7.x86_64
# 。。。。
# 可以通过find / -name docker查找缓存包，如果找不到直接下载即可。
# 用rpm手动安装并添加–replacefiles 参数
[root@k8s-master ~]# rpm -ivh /var/cache/yum/x86_64/7/docker-ce-stable/packages/docker-ce-18.06.3.ce-3.el7.x86_64.rpm --replacefiles
错误：依赖检测失败：
container-selinux >= 2.9 被 docker-ce-18.06.3.ce-3.el7.x86_64 需要
# 需要安装container-selinux
[root@k8s-master ~]# yum install container-selinux -y
已安装:
container-selinux.noarch 2:2.119.1-1.c57a6f9.el7
完毕！
# 再次使用rpm安装docker
[root@k8s-master ~]# rpm -ivh /var/cache/yum/x86_64/7/docker-ce-stable/packages/docker-ce-18.06.3.ce-3.el7.x86_64.rpm --replacefiles
[root@k8s-master ~]# docker version
Client:
Version: 18.06.3-ce
API version: 1.38
# 启动docker
[root@k8s-master ~]# systemctl start docker
[root@k8s-master ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
cgroupfs警告问题（所有节点修改）
[警告IsDockerSystemdCheck]：检测到“cgroupfs”作为Docker cgroup驱动程序。 推荐的驱动程序是“systemd”。
[root@k8s-node1 ~]# vim /etc/docker/daemon.json
[root@k8s-node1 ~]# cat /etc/docker/daemon.json
{
"exec-opts":["native.cgroupdriver=systemd"]
}
[root@k8s-node1 ~]# systemctl restart docker
CPU核心数修改
由于是VMware安装的，所以所有节点修改 处理器内核数量大于2
阿里云v1.18.3 not found
[root@k8s-master ~]# kubeadm init \
--apiserver-advertise-address=192.168.99.100 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.18.3 \
--service-cidr=10.1.0.0/16\
--pod-network-cidr=10.244.0.0/16
rror execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image registry.aliyuncs.com/google_containers/kube-apiserver:v1.18.3: output: Error response from daemon: manifest for registry.aliyuncs.com/google_containers/kube-apiserver:v1.18.3 not found
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.aliyuncs.com/google_containers/kube-controller-manager:v1.18.3: output: Error response from daemon: manifest for registry.aliyuncs.com/google_containers/kube-controller-manager:v1.18.3 not found
, error: exit status 1
# 所有节点上都进行降级安装，即 1.18.3 -> 1.18.2
[root@k8s-master ~]# yum downgrade kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2 --nogpgcheck -y
kubeadm init初始化日志
# 1、运行一系列预检代码来检查系统的状态；
# 大部分的检查只会抛出一个警告，也有一部分会抛出异常错误从而导致工作流推出
# （比如没有关闭swap或者没有安装docker）。官方给出一个参数–ignore-preflight-errors=...
W0523 18:08:32.999414 2448 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
# 2、生成一个用来认证k8s组件间调用的自签名的CA（Certificate Authority，证书授权）；
# 这个证书也可以通过–cert-dir（默认是/etc/kubernetets/pki）的方式传入，那么这一步就会跳过。
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.1.0.1 192.168.99.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.99.100 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.99.100 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
# 3、把kubelet、controller-manager和scheduler等组件的配置文件写到/etc/kubernets/目录，
# 这几个组件会使用这些配置文件来连接API-server的服务；
# 除了上面几个配置文件，还会生成一个管理相关的admin.conf文件。
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
# 4、创建一些 静态pod 的配置文件了，包括API-server、controller-manager和scheduler。
# 假如没有提供外部etcd，还会另外生成一个etcd的静态Pod配置文件。
# 这些静态pod会被写入/etc/kubernetes/manifests，kubelet进程会监控这个目录，从而创建相关的pod。
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0523 18:09:33.958611 2448 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0523 18:09:33.960089 2448 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
# 5、如果上一步比较顺利，这个时候k8s的控制面进程（api-server、controller-manager、scheduler）就全都起来了。
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 16.002340 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
# 6、给当前的节点（Master节点）打label和taints，从而防止其他的负载在这个节点运行。
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
# 7、生成token，其他节点如果想加入当前节点（Master）所在的k8s集群，会用到这个token。
[bootstrap-token] Using token: mfouno.1nacdahemjebmzgb
# 8、进行一些允许节点以 Bootstrap Tokens) 和 TLS bootstrapping 方式加入集群的必要的操作
# 设置RBAC规则，同时创建一个用于节点加入集群的ConfigMap（包含了加入集群需要的所有信息）。
# 让Bootstrap Tokens可以访问CSR签名的API。
# 给新的CSR请求配置自动认证机制。
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
# 9、通过API-server安装DNS服务器（1.11版本后默认为CoreDNS，早期版本默认为kube-dns）和kube-proxy插件。
# 这里需要注意的是，DNS服务器只有在安装了CNI（flannel或calico）之后才会真正部署，否则会处于挂起（pending）状态。
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.99.100:6443 --token mfouno.1nacdahemjebmzgb \
--discovery-token-ca-cert-hash sha256:b333ae5c7fc2888d4a416816b1c977866277caf467e1a0f18e585c857d5d17ed
开始使用集群前创建配置文件
直接复制上面的一段即可
[root@k8s-master ~]# mkdir -p $HOME/.kube
[root@k8s-master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
使用kubectl获取节点
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master NotReady master 67m v1.18.2
上面的状态为NotReady，因为他在等待网络的加入。
安装Pod网络插件（CNI）
flannel解释
Kubernetes-基于flannel的集群网络
Kubernetes网络分析之Flannel
flannel is a network fabric for containers, designed for Kubernetes（GitHub）
应用kube-flannel.yml文件
Master节点，Node节点加入后自动下载
# 应用在线的kube-flannel.yml文件
[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
# 查看命名空间kube-system中的pod变化
[root@k8s-master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-5dl9p 0/1 Pending 0 74m
coredns-7ff77c879f-x5cmh 0/1 Pending 0 74m
etcd-k8s-master 1/1 Running 0 74m
kube-apiserver-k8s-master 1/1 Running 0 74m
kube-controller-manager-k8s-master 1/1 Running 0 74m
kube-flannel-ds-amd64-d7mbx 0/1 Init:0/1 0 55s
kube-proxy-jndns 1/1 Running 0 74m
kube-scheduler-k8s-master 1/1 Running 0 74m
# 节点只有master，因为还没有其他节点加入进来
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master NotReady master 75m v1.18.2
[root@k8s-master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-5dl9p 0/1 Pending 0 80m
coredns-7ff77c879f-x5cmh 0/1 Pending 0 80m
etcd-k8s-master 1/1 Running 0 80m
kube-apiserver-k8s-master 1/1 Running 0 80m
kube-controller-manager-k8s-master 1/1 Running 0 80m
kube-flannel-ds-amd64-d7mbx 0/1 Init:ImagePullBackOff 0 6m18s
kube-proxy-jndns 1/1 Running 0 80m
kube-scheduler-k8s-master 1/1 Running 0 80m
# 查看所有pod的状态
[root@k8s-master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-5dl9p 1/1 Running 0 3h5m
coredns-7ff77c879f-x5cmh 1/1 Running 0 3h5m
etcd-k8s-master 1/1 Running 0 3h5m
kube-apiserver-k8s-master 1/1 Running 0 3h5m
kube-controller-manager-k8s-master 1/1 Running 0 3h5m
kube-flannel-ds-amd64-d7mbx 1/1 Running 0 111m
kube-proxy-jndns 1/1 Running 0 3h5m
kube-scheduler-k8s-master 1/1 Running 0 3h5m
kube-flannel下载慢处理
如果下载太慢，将 https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 下载到本地
修改，将image的地址修改为Docker Hub国内仓库地址，或者私有仓库。
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.12.0-amd64
例如修改为七牛云的，然后再次执行kubectl apply -f kube-flannel.yml，kube-flannel.yml为本地的文件。
containers:
- name: kube-flannel
image: quay-mirror.qiniu.com/coreos/flannel:v0.12.0-amd64
编辑该文件，然后重新应用kubectl apply -f kube-flannel.yml，即创建+更新，可以重复使用
[root@k8s-master ~]# vim kube-flannel.yml
# 重新应用更新的本地kube-flannel.yml文件
[root@k8s-master ~]# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged configured
clusterrole.rbac.authorization.k8s.io/flannel unchanged
clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged
serviceaccount/flannel unchanged
configmap/kube-flannel-cfg unchanged
daemonset.apps/kube-flannel-ds-amd64 configured # 可以看到这项更新了
daemonset.apps/kube-flannel-ds-arm64 unchanged
daemonset.apps/kube-flannel-ds-arm unchanged
daemonset.apps/kube-flannel-ds-ppc64le unchanged
daemonset.apps/kube-flannel-ds-s390x unchanged
将Node加入集群
在所有Node节点上运行，向集群添加新节点，执行在kubeadm init输出的kubeadm join命令
为Node节点初始化
Node1节点加入集群
[root@k8s-node1 ~]# kubeadm join 192.168.99.100:6443 --token mfouno.1nacdahemjebmzgb \
--discovery-token-ca-cert-hash sha256:b333ae5c7fc2888d4a416816b1c977866277caf467e1a0f18e585c857d5d17ed
# Node1日志如下：
W0523 21:21:28.246095 10953 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
Node2节点加入集群
[root@k8s-node2 ~]# kubeadm join 192.168.99.100:6443 --token mfouno.1nacdahemjebmzgb \
--discovery-token-ca-cert-hash sha256:b333ae5c7fc2888d4a416816b1c977866277caf467e1a0f18e585c857d5d17ed
# Node2日志如下
W0523 21:23:01.585236 11009 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
Master查看集群状态
查看集群中的节点信息
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 3h13m v1.18.2
k8s-node1 NotReady <none> 119s v1.18.3
k8s-node2 NotReady <none> 26s v1.18.3
# 过一段时间后，有一个节点已经加入进来了
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 3h32m v1.18.2
k8s-node1 NotReady <none> 21m v1.18.3
k8s-node2 Ready <none> 19m v1.18.3
查看命名空间kube-system下的所有pod信息
[root@k8s-master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-5dl9p 1/1 Running 0 3h21m
coredns-7ff77c879f-x5cmh 1/1 Running 0 3h21m
etcd-k8s-master 1/1 Running 0 3h21m
kube-apiserver-k8s-master 1/1 Running 0 3h21m
kube-controller-manager-k8s-master 1/1 Running 0 3h21m
kube-flannel-ds-amd64-d7mbx 1/1 Running 0 127m
kube-flannel-ds-amd64-lbqwz 0/1 Init:ImagePullBackOff 0 9m39s
kube-flannel-ds-amd64-z69q4 0/1 Init:ImagePullBackOff 0 8m6s
kube-proxy-dwpvg 1/1 Running 0 9m39s
kube-proxy-jndns 1/1 Running 0 3h21m
kube-proxy-w88h4 1/1 Running 0 8m6s
kube-scheduler-k8s-master 1/1 Running 0 3h21m
# # 过一段时间后，有一个节点已经加入进来了
[root@k8s-master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-5dl9p 1/1 Running 0 3h32m
coredns-7ff77c879f-x5cmh 1/1 Running 0 3h32m
etcd-k8s-master 1/1 Running 0 3h32m
kube-apiserver-k8s-master 1/1 Running 0 3h32m
kube-controller-manager-k8s-master 1/1 Running 0 3h32m
kube-flannel-ds-amd64-d7mbx 1/1 Running 0 138m
kube-flannel-ds-amd64-th9h9 1/1 Running 0 3m44s
kube-flannel-ds-amd64-xv27n 0/1 Init:ImagePullBackOff 0 3m44s
kube-proxy-dwpvg 1/1 Running 0 21m
kube-proxy-jndns 1/1 Running 0 3h32m
kube-proxy-w88h4 1/1 Running 0 19m
kube-scheduler-k8s-master 1/1 Running 0 3h32m
Master节点测试kubernetes集群
创建一个nginx的资源
# 创建一个nginx的pod
[root@k8s-master ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
# 创建service：暴露容器让外部访问，--type=NodePort说明创建的是NodePort类型，则通过任意一个NodeIP+Port就可以访问。
[root@k8s-master ~]# kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed
查看pod、svc信息
-o wide 选项可以查看存在哪个对应的节点
# 查看pod详情
[root@k8s-master ~]# kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/nginx-f89759699-vn4gb 0/1 ImagePullBackOff 0 32s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 3h35m
service/nginx NodePort 10.1.0.141 <none> 80:31519/TCP 25s
[root@k8s-master ~]# kubectl get pod,svc -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-f89759699-vn4gb 0/1 ImagePullBackOff 0 49s 10.244.3.2 k8s-node2 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 3h35m <none>
service/nginx NodePort 10.1.0.141 <none> 80:31519/TCP 42s app=nginx
查看pod的详情描述
状态为ImagePullBackOff或ErrImagePull时可以查看该pod描述，在排错时尤为实用
[root@k8s-master ~]# kubectl describe pod nginx-f89759699-vn4gb
Name: nginx-f89759699-vn4gb
Namespace: default
Priority: 0
Node: k8s-node2/192.168.99.202
Start Time: Sat, 23 May 2020 21:44:55 +0800
Labels: app=nginx
pod-template-hash=f89759699
Annotations: <none>
Status: Pending
IP: 10.244.3.2
IPs:
IP: 10.244.3.2
Controlled By: ReplicaSet/nginx-f89759699
Containers:
nginx:
Container ID:
Image: nginx
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-s2qmv (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-s2qmv:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-s2qmv
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m31s default-scheduler Successfully assigned default/nginx-f89759699-vn4gb to k8s-node2
Warning Failed 5m47s kubelet, k8s-node2 Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/library/nginx/manifests/latest: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fnginx%3Apull&service=registry.docker.io: net/http: TLS handshake timeout
Normal Pulling 4m21s (x4 over 6m30s) kubelet, k8s-node2 Pulling image "nginx"
Warning Failed 4m8s (x3 over 6m20s) kubelet, k8s-node2 Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: TLS handshake timeout
Warning Failed 4m8s (x4 over 6m20s) kubelet, k8s-node2 Error: ErrImagePull
Warning Failed 3m41s (x7 over 6m19s) kubelet, k8s-node2 Error: ImagePullBackOff
Normal BackOff 78s (x14 over 6m19s) kubelet, k8s-node2 Back-off pulling image "nginx"
拉取Docker镜像慢处理
所有节点都设置国内的源
[root@k8s-master ~]# vim /etc/docker/daemon.json
[root@k8s-master ~]# cat /etc/docker/daemon.json
{
"exec-opts":["native.cgroupdriver=systemd"],
"registry-mirrors": [
"https://hub-mirror.c.163.com",
"https://docker.mirrors.ustc.edu.cn"
]
}
[root@k8s-master ~]# systemctl restart docker
[root@k8s-master ~]# docker info
# ...
Registry Mirrors:
https://hub-mirror.c.163.com/
https://docker.mirrors.ustc.edu.cn/
# ...
稍等下再次进行查看
查看pod、svc信息
[root@k8s-master ~]# kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/nginx-f89759699-vn4gb 1/1 Running 0 2d22h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 3d2h
service/nginx NodePort 10.1.0.141 <none> 80:31519/TCP 2d22h
可以看到资源nginx-f89759699-vn4gb的状态由ImagePullBackOff变为Running。
访问nginx资源
现在就可以通过任意一个节点+31519端口访问到，例如http://192.168.99.100:31519/、http://192.168.99.201:31519/、http://192.168.99.202:31519/
查看该pod的详情
[root@k8s-master ~]# kubectl describe pod nginx-f89759699-vn4gb
Name: nginx-f89759699-vn4gb
Namespace: default
Priority: 0
Node: k8s-node2/192.168.99.202
Start Time: Sat, 23 May 2020 21:44:55 +0800
Labels: app=nginx
pod-template-hash=f89759699
Annotations: <none>
Status: Running
IP: 10.244.3.5
# .....
Warning FailedCreatePodSandBox 107s kubelet, k8s-node2 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1953fda080b8d84918c671364c50409e3365a34dd1f983e9b2dc4f3042b96e71" network for pod "nginx-f89759699-vn4gb": networkPlugin cni failed to set up pod "nginx-f89759699-vn4gb_default" network: open /run/flannel/subnet.env: no such file or directory
Normal Pulling 106s kubelet, k8s-node2 Pulling image "nginx"
Normal Pulled 82s kubelet, k8s-node2 Successfully pulled image "nginx"
Normal Created 82s kubelet, k8s-node2 Created container nginx
Normal Started 82s kubelet, k8s-node2 Started container nginx
查看集群节点是否正常启动了
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 3d2h v1.18.2
k8s-node1 Ready <none> 2d23h v1.18.3
k8s-node2 Ready <none> 2d23h v1.18.3
[root@k8s-master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-5dl9p 1/1 Running 3 3d2h
coredns-7ff77c879f-x5cmh 1/1 Running 3 3d2h
etcd-k8s-master 1/1 Running 3 3d2h
kube-apiserver-k8s-master 1/1 Running 4 3d2h
kube-controller-manager-k8s-master 1/1 Running 3 3d2h
kube-flannel-ds-amd64-dsf69 1/1 Running 0 30m
kube-flannel-ds-amd64-th9h9 1/1 Running 5 2d23h
kube-flannel-ds-amd64-xv27n 1/1 Running 0 2d23h
kube-proxy-dwpvg 1/1 Running 3 2d23h
kube-proxy-jndns 1/1 Running 3 3d2h
kube-proxy-w88h4 1/1 Running 4 2d23h
kube-scheduler-k8s-master 1/1 Running 4 3d2h
扩容
一个可以有多个副本，可以对该资源进行扩容
[root@k8s-master ~]# kubectl scale deployment nginx --replicas=3
deployment.apps/nginx scaled
通过--replicas=3可以指定扩容后有多少个
查看扩容后的pod变化
# 扩容前
[root@k8s-master ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-f89759699-vn4gb 1/1 Running 0 2d22h
# 扩容后
[root@k8s-master ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-f89759699-lm824 1/1 Running 0 14s
nginx-f89759699-nf9f4 0/1 ContainerCreating 0 14s
nginx-f89759699-vn4gb 1/1 Running 0 2d23h
[root@k8s-master ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-f89759699-lm824 1/1 Running 0 108s
nginx-f89759699-nf9f4 1/1 Running 0 108s
nginx-f89759699-vn4gb 1/1 Running 0 2d23h
# 可以看到每个nginx所在的node节点
[root@k8s-master ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-f89759699-lm824 1/1 Running 0 3m17s 10.244.3.6 k8s-node2 <none> <none>
nginx-f89759699-nf9f4 1/1 Running 0 3m17s 10.244.1.2 k8s-node1 <none> <none>
nginx-f89759699-vn4gb 1/1 Running 0 2d23h 10.244.3.5 k8s-node2 <none> <none>
并发性扩大了3倍，对于用户来说，只需要访问之前的端口即可。
Master部署Dashboard
部署dashboard/v1.10.1
下载修改kubernetes-dashboard.yaml文件
[root@k8s-master ~]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
如果镜像访问慢，在Docker Hub找到了kubernetes-dashboard的链接，将
containers:
- name: kubernetes-dashboard
image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1
修改为
containers:
- name: kubernetes-dashboard
image: siriuszg/kubernetes-dashboard-amd64:v1.10.1
允许外部访问端口
默认Dashboard只能集群内部访问，修改Service为NodePort类型，暴露到外部：
将
# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
修改为type:NodePort、nodePort: 30001（端口默认30000起），修改后
# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30001
selector:
k8s-app: kubernetes-dashboard
应用kubernetes-dashboard.yaml文件
[root@k8s-master ~]# kubectl apply -f kubernetes-dashboard.yaml
secret/kubernetes-dashboard-certs created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
error: error parsing kubernetes-dashboard.yaml: error converting YAML to JSON: yaml: line 12: mapping values are not allowed in this context
# 每个属性如设置值的时候，属性 冒号 : 后面 要加上空格隔开，然后再写值。检查刚才修改的内容
# 修改后再次执行
[root@k8s-master ~]# kubectl apply -f kubernetes-dashboard.yaml
secret/kubernetes-dashboard-certs unchanged
serviceaccount/kubernetes-dashboard unchanged
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal unchanged
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal unchanged
deployment.apps/kubernetes-dashboard unchanged
service/kubernetes-dashboard created
查看命名空间kube-system下的pod
默认放在了kube-system命名空间下。
如果想要快速运行，可以先在每个节点下使用docker pull siriuszg/kubernetes-dashboard-amd64:v1.10.1提前下载好镜像。然后使用kubectl apply时不会再次拉取镜像。
[root@k8s-master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-5dl9p 1/1 Running 3 3d4h
coredns-7ff77c879f-x5cmh 1/1 Running 3 3d4h
etcd-k8s-master 1/1 Running 3 3d4h
kube-apiserver-k8s-master 1/1 Running 4 3d4h
kube-controller-manager-k8s-master 1/1 Running 3 3d4h
kube-flannel-ds-amd64-dsf69 1/1 Running 0 112m
kube-flannel-ds-amd64-th9h9 1/1 Running 5 3d
kube-flannel-ds-amd64-xv27n 1/1 Running 0 3d
kube-proxy-dwpvg 1/1 Running 3 3d
kube-proxy-jndns 1/1 Running 3 3d4h
kube-proxy-w88h4 1/1 Running 4 3d
kube-scheduler-k8s-master 1/1 Running 4 3d4h
kubernetes-dashboard-694997dccd-htfqx 0/1 ContainerCreating 0 10m
# 过了一会儿，就运行起来了
[root@k8s-master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-5dl9p 1/1 Running 3 3d4h
coredns-7ff77c879f-x5cmh 1/1 Running 3 3d4h
etcd-k8s-master 1/1 Running 3 3d4h
kube-apiserver-k8s-master 1/1 Running 4 3d4h
kube-controller-manager-k8s-master 1/1 Running 3 3d4h
kube-flannel-ds-amd64-dsf69 1/1 Running 0 114m
kube-flannel-ds-amd64-th9h9 1/1 Running 5 3d
kube-flannel-ds-amd64-xv27n 1/1 Running 0 3d
kube-proxy-dwpvg 1/1 Running 3 3d
kube-proxy-jndns 1/1 Running 3 3d4h
kube-proxy-w88h4 1/1 Running 4 3d
kube-scheduler-k8s-master 1/1 Running 4 3d4h
kubernetes-dashboard-694997dccd-htfqx 1/1 Running 0 12m
# 查看运行的节点信息
[root@k8s-master ~]# kubectl get pod -n kube-system -o wide | grep kubernetes-dashboard
kubernetes-dashboard-694997dccd-htfqx 1/1 Running 0 15m 10.244.1.3 k8s-node1 <none> <none>
查看暴露的端口
[root@k8s-master ~]# kubectl get pod,svc -n kube-system
NAME READY STATUS RESTARTS AGE
pod/coredns-7ff77c879f-5dl9p 1/1 Running 3 3d4h
pod/coredns-7ff77c879f-x5cmh 1/1 Running 3 3d4h
pod/etcd-k8s-master 1/1 Running 3 3d4h
pod/kube-apiserver-k8s-master 1/1 Running 4 3d4h
pod/kube-controller-manager-k8s-master 1/1 Running 3 3d4h
pod/kube-flannel-ds-amd64-dsf69 1/1 Running 0 119m
pod/kube-flannel-ds-amd64-th9h9 1/1 Running 5 3d
pod/kube-flannel-ds-amd64-xv27n 1/1 Running 0 3d
pod/kube-proxy-dwpvg 1/1 Running 3 3d
pod/kube-proxy-jndns 1/1 Running 3 3d4h
pod/kube-proxy-w88h4 1/1 Running 4 3d
pod/kube-scheduler-k8s-master 1/1 Running 4 3d4h
pod/kubernetes-dashboard-694997dccd-htfqx 1/1 Running 0 18m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.1.0.10 <none> 53/UDP,53/TCP,9153/TCP 3d4h
service/kubernetes-dashboard NodePort 10.1.116.46 <none> 443:30001/TCP 8m31s
可以看到暴露的端口为30001。且需要使用https访问： https://192.168.99.100:30001/
在这个页面没有继续访问的按钮，直接在键盘上打thisisunsafe就可以自动进去了
https://192.168.99.100:30001/#!/login
创建用户token登录
选择令牌，创建一个管理员的身份去访问UI，创建一个面向应用的serviceaccount用户去访问
[root@k8s-master ~]# kubectl create serviceaccount dashboard-admin -n kube-system
serviceaccount/dashboard-admin created
[root@k8s-master ~]# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin created
[root@k8s-master ~]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
Name: dashboard-admin-token-n25df
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: dashboard-admin
kubernetes.io/service-account.uid: 074b8084-cc73-4a30-88e2-a59ee518fee1
Type: kubernetes.io/service-account-token
Data
====
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IlVHZDZsTGNLbTM3OEUzYzk3Q3k1dTFJVnFEZFJ6eG90cHZZMjN6RUdLVnMifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tbjI1ZGYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMDc0YjgwODQtY2M3My00YTMwLTg4ZTItYTU5ZWU1MThmZWUxIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.FNoMLtHKhmcMwFdEE2PcvBWRV4nei5T8wPwvJWSH_A_H09OZp-FIrwxotswzZbVE90-h76zXLbAghNp-kzPYWvdXYWlMUraFYnNRrRQMsrzcaHe_Ex5KiDwMmavxa0fd_1x0RQLNsIDL20gUR8LVfelKZ2Cdzy0v_xHAQjui4bW2HX6GtSmfVTcBO8PHrCwgOGNw3uBKjNxZcmhMoOc_m4Zj2lerUjpZFpmyhoGbmCytPgGywxPmWK56pK-SsDYLFFX-kSSpeDyv2GvH5MXK51aONU5P3n-jo4OngUvTjFT-FTGmSZ_oKN_YMTlsNlUYjrVHhDWmeqdYHkqGhSgV5Q
ca.crt: 1025 bytes
namespace: 11 bytes
然后将token粘贴到输入框
处理“未知服务器错误 (404)”问题
提示
未知服务器错误 (404)
the server could not find the requested resource
3 秒内重定向到上一个页面……
删除使用kubernetes-dashboard.yaml旧版创建的资源
# 通过yaml文件删除
[root@k8s-master ~]# kubectl delete -f kubernetes-dashboard.yaml
secret "kubernetes-dashboard-certs" deleted
serviceaccount "kubernetes-dashboard" deleted
role.rbac.authorization.k8s.io "kubernetes-dashboard-minimal" deleted
rolebinding.rbac.authorization.k8s.io "kubernetes-dashboard-minimal" deleted
deployment.apps "kubernetes-dashboard" deleted
service "kubernetes-dashboard" deleted
部署dashboard/v2.0.1
访问 https://github.com/kubernetes/dashboard/releases 安装新版
下载新版recommended.yaml
[root@k8s-master ~]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.1/aio/deploy/recommended.yaml
修改recommended.yaml端口暴露
增加type: NodePort和nodePort: 30001
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30001
selector:
k8s-app: kubernetes-dashboard
应用recommended.yaml 文件
[root@k8s-master ~]# kubectl apply -f recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
可以看到dashboard是运行到kubernetes-dashboard命名空间下的，所以要指定-n kubernetes-dashboard查看信息
查看kubernetes-dashboard无法启动的日志
# 查看状态
[root@k8s-master ~]# kubectl get pod -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
dashboard-metrics-scraper-6b4884c9d5-z4cqb 1/1 Running 7 15m
kubernetes-dashboard-7bfbb48676-pgjcg 0/1 CrashLoopBackOff 7 15m
# 一直处于CrashLoopBackOff，
# 查看详情
[root@k8s-master ~]# kubectl describe pod kubernetes-dashboard-7bfbb48676-pgjcg -n kubernetes-dashboard
Name: kubernetes-dashboard-7bfbb48676-pgjcg
Namespace: kubernetes-dashboard
Priority: 0
Node: k8s-node1/192.168.99.201
Start Time: Wed, 27 May 2020 20:40:51 +0800
Labels: k8s-app=kubernetes-dashboard
pod-template-hash=7bfbb48676
# ........
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned kubernetes-dashboar
Normal Pulling 2m26s (x4 over 3m20s) kubelet, k8s-node1 Pulling image "kubernetesui/dashboard:v2.
Normal Pulled 2m24s (x4 over 3m17s) kubelet, k8s-node1 Successfully pulled image "kubernetesui/d
Normal Created 2m24s (x4 over 3m17s) kubelet, k8s-node1 Created container kubernetes-dashboard
Normal Started 2m24s (x4 over 3m17s) kubelet, k8s-node1 Started container kubernetes-dashboard
Warning BackOff 115s (x9 over 3m11s) kubelet, k8s-node1 Back-off restarting failed container
# 部署到k8s-node1节点上
# 查看日志
[root@k8s-master ~]# kubectl logs kubernetes-dashboard-7bfbb48676-pgjcg -n kubernetes-dashboard
2020/05/27 12:52:00 Starting overwatch
2020/05/27 12:52:00 Using namespace: kubernetes-dashboard
2020/05/27 12:52:00 Using in-cluster config to connect to apiserver
2020/05/27 12:52:00 Using secret token for csrf signing
2020/05/27 12:52:00 Initializing csrf token from kubernetes-dashboard-csrf secret
panic: Get https://10.1.0.1:443/api/v1/namespaces/kubernetes-dashboard/secrets/kubernetes-dashboard-csrf: dial tcp 10.1.0.1:443: connect: no route to host
goroutine 1 [running]:
github.com/kubernetes/dashboard/src/app/backend/client/csrf.(*csrfTokenManager).init(0xc0004ca880)
/home/travis/build/kubernetes/dashboard/src/app/backend/client/csrf/manager.go:41 +0x446
github.com/kubernetes/dashboard/src/app/backend/client/csrf.NewCsrfTokenManager(...)
/home/travis/build/kubernetes/dashboard/src/app/backend/client/csrf/manager.go:66
github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).initCSRFKey(0xc000462100)
/home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:501 +0xc6
github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).init(0xc000462100)
/home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:469 +0x47
github.com/kubernetes/dashboard/src/app/backend/client.NewClientManager(...)
/home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:550
main.main()
/home/travis/build/kubernetes/dashboard/src/app/backend/dashboard.go:105 +0x20d
日志中kubernetes-dashboard-csrf报错处理
参考 https://www.jianshu.com/p/e359d3fe238f 的解决方案
由于 pod 不会分配到到 master 节点, 并且 kubeadm部署的 apiserver 中启用的验证方式为 Node 和 RBAC, 且关闭了 insecure-port，猜测可能是这个原因导致连接不上 apiServer , 即使是手动修改也不行 --apiserver-host 参数也不行。
重新修改 recommended.yaml 文件
[root@k8s-master ~]# vim recommended.yaml
增加nodeName: k8s-master的配置
# ...
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
nodeName: k8s-master
containers:
- name: kubernetes-dashboard
image: kubernetesui/dashboard:v2.0.1
# ...
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: dashboard-metrics-scraper
template:
metadata:
labels:
k8s-app: dashboard-metrics-scraper
annotations:
seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'
spec:
nodeName: k8s-master
containers:
- name: dashboard-metrics-scraper
image: kubernetesui/metrics-scraper:v1.0.4
# ...
上述配置将指定 kubernetes-dashboard 和 kubernetes-metrics-scraper 分配部署的节点，此处选择部署在 master 节点解决文中出现的通信问题。之前的端口暴露不变。
重新应用recommended.yaml文件
[root@k8s-master ~]# kubectl apply -f recommended.yaml
可以看到pod的名称也发生了变化，因为做了配置文件的修改。
查看kubernetes-dashboard状态运行成功
[root@k8s-master ~]# kubectl get pod -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
dashboard-metrics-scraper-6fb66f87b-gdfbp 1/1 Running 0 3m47s
kubernetes-dashboard-fb89c4d57-sz86q 1/1 Running 0 3m47s
访问 https://192.168.99.100:30001/ 在token中输入之前生成的token字符串，点击登录
即可正常进入页面了
开源、云原生的融合云平台
更多推荐
0
0- 0
已为社区贡献1条内容
所有评论(0)