k8s多节点部署
目录前提master02部署前端负载nginx搭建测试node IP指向VIP地址测试方法负载前提多节点在单节点的基础上进行资源规划名称主机名IP地址ks8-master01master20.0.0.10/24ks8-master02master0220.0.0.40/24node01node0120.0.0.20/24node02node0220.0.0.30/24nginx-masterngi
前提
多节点在单节点的基础上进行
资源规划
名称 | 主机名 | IP地址 |
---|---|---|
ks8-master01 | master | 20.0.0.10/24 |
ks8-master02 | master02 | 20.0.0.40/24 |
node01 | node01 | 20.0.0.20/24 |
node02 | node02 | 20.0.0.30/24 |
nginx-master | nginx-master | 20.0.0.40/24 |
nginx-backup | nginx-backup | 20.0.0.50/24 |
VIP地址 | 20.0.0.100/24 |
master02部署
1、将master01的配置信息输出到master02上
[root@master kubeconfig]# scp -r /opt/kubernetes/ root@20.0.0.40:/opt/
The authenticity of host '20.0.0.40 (20.0.0.40)' can't be established.
ECDSA key fingerprint is SHA256:3YC4ehaBnUbIovfc6ha74KIHn4KPTl4hXHvXgZZX4J0.
ECDSA key fingerprint is MD5:f9:36:e7:e0:99:70:ba:c0:ba:46:85:58:6f:60:2c:70.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '20.0.0.40' (ECDSA) to the list of known hosts.
root@20.0.0.40's password:
token.csv 100% 84 102.3KB/s 00:00
kube-apiserver 100% 909 1.3MB/s 00:00
kube-scheduler 100% 94 161.6KB/s 00:00
kube-controller-manager 100% 483 842.0KB/s 00:00
kube-apiserver 100% 184MB 93.6MB/s 00:01
kubectl 100% 55MB 94.6MB/s 00:00
kube-controller-manager 100% 155MB 97.4MB/s 00:01
kube-scheduler 100% 55MB 97.3MB/s 00:00
admin-key.pem 100% 1679 1.9MB/s 00:00
admin.pem 100% 1399 1.9MB/s 00:00
ca-key.pem 100% 1675 3.7MB/s 00:00
ca.pem 100% 1359 3.0MB/s 00:00
kube-proxy-key.pem 100% 1679 3.9MB/s 00:00
kube-proxy.pem 100% 1403 3.2MB/s 00:00
server-key.pem 100% 1675 3.7MB/s 00:00
server.pem 100% 1643 3.9MB/s 00:00
2、master01复制启动文件到master02中
[root@master kubeconfig]# scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@20.0.0.40:/usr/lib/systemd/system/
root@20.0.0.40's password:
kube-apiserver.service 100% 282 509.4KB/s 00:00
kube-controller-manager.service 100% 317 706.6KB/s 00:00
kube-scheduler.service 100% 281 616.3KB/s 00:00
3、master02修改kube-apiserver IP地址指向自己
[root@master02 ~]# vim /opt/kubernetes/cfg/kube-apiserver
--bind-address=20.0.0.40 \ #修改为自己主机IP地址
--advertise-address=20.0.0.40 \ #修改为自己主机IP地址
4、将master01的etcd证书复制到master02上面
[root@master kubeconfig]# scp -r /opt/etcd/ root@20.0.0.40:/opt/
root@20.0.0.40's password:
etcd 100% 481 88.9KB/s 00:00
etcd 100% 18MB 88.8MB/s 00:00
etcdctl 100% 15MB 92.6MB/s 00:00
ca-key.pem 100% 1679 899.3KB/s 00:00
ca.pem 100% 1265 490.2KB/s 00:00
server-key.pem 100% 1679 2.2MB/s 00:00
server.pem 100% 1338 1.2MB/s 00:00
5、启动apiserver、scheduler、controller-manager
[root@master02 ~]# systemctl start kube-apiserver.service
[root@master02 ~]# systemctl enable kube-apiserver.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.
root@master02 ~]# systemctl start kube-scheduler.service
[root@master02 ~]# systemctl enable kube-scheduler.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
[root@master02 ~]# systemctl start kube-controller-manager.service
[root@master02 ~]# systemctl enable kube-controller-manager.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
6、添加环境变量
[root@master02 ~]# vim /etc/profile
export PATH=$PATH:/opt/kubernetes/bin/ #增加
[root@master02 ~]# source /etc/profile #加载环境变量
7、查看群集状态
[root@master02 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
20.0.0.20 Ready <none> 64m v1.12.3
20.0.0.30 Ready <none> 14m v1.12.3
前端负载nginx搭建
nginx-master配置
1、设置nginx源
[root@nginx-master ~]# vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
2、安装nginx
[root@nginx-master ~]# yum -y install nginx
3、修改nginx配置文件代理后端master
[root@nginx-master ~]# vim /etc/nginx/nginx.conf
events {
worker_connections 1024;
}
增加下面内容
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 20.0.0.10:6443;
server 20.0.0.40:6443;
}
server {
listen 6443;
proxy_pass k8s-apiserver;
}
}
4、检查语法
[root@nginx-master ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
5、启动nginx
[root@nginx-master ~]# systemctl start nginx.service
[root@nginx-master ~]# systemctl enable nginx.service
Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.
6、安装keepalived
[root@nginx-master ~]# yum -y install keepalived
7、修改keepalived配置文件
[root@nginx-master ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
# 接收邮件地址
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
# 邮件发送地址
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/etc/nginx/nginx.sh"
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的
priority 100 # 优先级,备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间,默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
20.0.0.100/24
}
track_script {
check_nginx
}
}
8、创建nginx健康检查脚本
[root@nginx-master ~]# vim /etc/nginx/nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
systemctl stop keepalived
fi
9、给脚本加执行权限
[root@nginx-master ~]# chmod +x /etc/nginx/nginx.sh
10、启动keepalived
[root@nginx-master ~]# systemctl start keepalived.service
[root@nginx-master ~]# systemctl enable keepalived.service
nginx-backup配置
1、设置nginx源
[root@nginx-backup ~]# vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
2、安装nginx
[root@nginx-backup ~]# yum -y install nginx
3、修改nginx配置文件代理后端master
[root@nginx-master ~]# vim /etc/nginx/nginx.conf
events {
worker_connections 1024;
}
增加下面内容
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 20.0.0.10:6443;
server 20.0.0.40:6443;
}
server {
listen 6443;
proxy_pass k8s-apiserver;
}
}
4、检查语法
[root@nginx-master ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
5、启动nginx
[root@nginx-master ~]# systemctl start nginx.service
[root@nginx-master ~]# systemctl enable nginx.service
Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.
6、安装keepalived
[root@nginx-backup ~]# yum -y install keepalived
7、修改keepalived配置文件
[root@nginx-backup ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
# 接收邮件地址
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
# 邮件发送地址
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/etc/nginx/nginx.sh"
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的
priority 90 # 优先级,备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间,默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
20.0.0.100/24
}
track_script {
check_nginx
}
}
8、创建nginx健康检查脚本
[root@nginx-backup ~]# vim /etc/nginx/nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
systemctl stop keepalived
fi
9、给脚本加执行权限
[root@nginx-backup ~]# chmod +x /etc/nginx/nginx.sh
10、启动keepalived
[root@nginx-backup ~]# systemctl start keepalived.service
[root@nginx-backup ~]# systemctl enable keepalived.service
测试
测试漂移地址
1、查看IP地址
[root@nginx-master ~]# ip a
ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:fe:a2:12 brd ff:ff:ff:ff:ff:ff
inet 20.0.0.50/24 brd 20.0.0.255 scope global ens33
valid_lft forever preferred_lft forever
inet 20.0.0.100/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::edac:d705:9b30:5197/64 scope link
valid_lft forever preferred_lft forever
[root@nginx-backup ~]# ip a
ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:e9:43:b6 brd ff:ff:ff:ff:ff:ff
inet 20.0.0.60/24 brd 20.0.0.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::d283:86ee:ba5e:dcf8/64 scope link
valid_lft forever preferred_lft forever
2、关闭主节点的nginx看VIP地址是否漂移
[root@nginx-master ~]# pkill nginx
[root@nginx-master ~]# ip a
ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:fe:a2:12 brd ff:ff:ff:ff:ff:ff
inet 20.0.0.50/24 brd 20.0.0.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::edac:d705:9b30:5197/64 scope link
valid_lft forever preferred_lft forever
[root@nginx-backup ~]# ip a
ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:e9:43:b6 brd ff:ff:ff:ff:ff:ff
inet 20.0.0.60/24 brd 20.0.0.255 scope global ens33
valid_lft forever preferred_lft forever
inet 20.0.0.100/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::d283:86ee:ba5e:dcf8/64 scope link
valid_lft forever preferred_lft forever
#主节点nginx恢复正常后VIP地址会自动漂移回来,先启动nginx在启动keepalived
node IP指向VIP地址
所有node节点都进行如下设置
1、修改地址为VIP地址
vim /opt/kubernetes/cfg/bootstrap.kubeconfig
server: https://20.0.0.100:6443 #修改为VIP地址
vim /opt/kubernetes/cfg/kubelet.kubeconfig
server: https://20.0.0.100:6443 #修改为VIP地址
vim /opt/kubernetes/cfg/kube-proxy.kubeconfig
server: https://20.0.0.100:6443 #修改为VIP地址
2、重启
systemctl restart kubelet.service
systemctl restart kube-proxy.service
3、查看是否修改成功
grep 100 *
bootstrap.kubeconfig: server: https://20.0.0.100:6443
kubelet.kubeconfig: server: https://20.0.0.100:6443
kube-proxy.kubeconfig: server: https://20.0.0.100:6443
测试方法负载
[root@master k8s-cert]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
20.0.0.20 Ready <none> 5h5m v1.12.3
20.0.0.30 Ready <none> 4h15m v1.12.3
[root@nginx-backup ~]# tail -f /var/log/nginx/k8s-access.log
20.0.0.30 20.0.0.10:6443 - [21/Jan/2021:15:25:32 +0800] 200 1115
20.0.0.30 20.0.0.40:6443 - [21/Jan/2021:15:25:32 +0800] 200 1115
20.0.0.20 20.0.0.10:6443 - [21/Jan/2021:15:33:55 +0800] 200 1115
20.0.0.20 20.0.0.40:6443 - [21/Jan/2021:15:33:55 +0800] 200 1114
1、创建pod测试
[root@master k8s-cert]# kubectl run nginx --image=nginx
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx created
2、查看状态
[root@master k8s-cert]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-dbddb74b8-mdfnc 0/1 ContainerCreating 0 22s
[root@master k8s-cert]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-dbddb74b8-mdfnc 1/1 Running 0 50s
3、查看pod日志
[root@master k8s-cert]# kubectl logs nginx-dbddb74b8-mdfnc
#默认匿名用户没有权限
[root@master k8s-cert]# kubectl logs nginx-dbddb74b8-mdfnc
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-mdfnc)
#设置权限
[root@master k8s-cert]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created
#查看pod创建在哪个节点创建
[root@master k8s-cert]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
nginx-dbddb74b8-mdfnc 1/1 Running 0 4m15s 172.17.75.3 20.0.0.30 <none>
去node2上查看
[root@node2 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0040331bf8c1 nginx "/docker-entrypoint.…" 4 minutes ago Up 4 minutes k8s_nginx_nginx-dbddb74b8-mdfnc_default_c9b8648e-5bbb-11eb-b8b6-000c29d29307_0
66f042492ba0 registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0 "/pause" 5 minutes ago Up 5 minutes k8s_POD_nginx-dbddb74b8-mdfnc_default_c9b8648e-5bbb-11eb-b8b6-000c29d29307_0
43a46bd8868f centos:7
更多推荐
所有评论(0)