k8s 1.17.2高可用版本重启问题

• 开发环境平均 每10分钟重启一次。
• 测试环境6天平均 每天重启一次。
• cube环境 每小时重启一次。
• ramdisk 到目前18小时没有重启

1.测试的环境：192.168.1.135 密码：xxxx 配置：3master 8c32g etcd版本：3.2.20

1.1重启次数：

[root@portal135 ~]# kubectl get pods -n kube-system|grep "kube-.*bocloud.com"
kube-apiserver-portal134.bocloud.com            1/1     Running   2          5d23h
kube-apiserver-portal135.bocloud.com            1/1     Running   8          5d23h
kube-apiserver-portal136.bocloud.com            1/1     Running   3          5d23h
kube-controller-manager-portal134.bocloud.com   1/1     Running   2          5d23h
kube-controller-manager-portal135.bocloud.com   1/1     Running   7          5d23h
kube-controller-manager-portal136.bocloud.com   1/1     Running   5          5d23h
kube-scheduler-portal134.bocloud.com            1/1     Running   2          5d23h
kube-scheduler-portal135.bocloud.com            1/1     Running   7          5d23h
kube-scheduler-portal136.bocloud.com            1/1     Running   5          5d23h

1.2响应时间：

[root@portal134 upload]# export NODE_IPS="192.168.1.134 192.168.1.135 192.168.1.136"
[root@portal134 upload]# for ip in ${NODE_IPS}; do   ETCDCTL_API=3 etcdctl   --endpoints=https://${ip}:2379    --cacert=/etc/etcd/ssl/ca.crt   --cert=/etc/etcd/ssl/client.crt   --key=/etc/etcd/ssl/client.key   endpoint health; done
https://192.168.1.134:2379 is healthy: successfully committed proposal: took = 2.711272ms
https://192.168.1.135:2379 is healthy: successfully committed proposal: took = 2.089683ms
https://192.168.1.136:2379 is healthy: successfully committed proposal: took = 1.935061ms

2.开发环境：192.168.2.103 密码：xxxx 配置：3master 4c8g etcd版本：3.4.3

2.1重启次数：

[root@boc-108 ~]# kubectl get pods -n kube-system|grep "kube-.*.dev"
kube-apiserver-boc-103.dev                 1/1     Running            63         4d4h
kube-apiserver-boc-104.dev                 1/1     Running            67         4d4h
kube-apiserver-boc-108.dev                 1/1     Running            82         4d4h
kube-controller-manager-boc-103.dev        1/1     Running            543        4d4h
kube-controller-manager-boc-104.dev        1/1     Running            561        4d4h
kube-controller-manager-boc-108.dev        1/1     Running            558        4d4h
kube-scheduler-boc-103.dev                 1/1     Running            562        4d4h
kube-scheduler-boc-104.dev                 1/1     Running            556        4d4h
kube-scheduler-boc-108.dev                 1/1     Running            561        4d4h

2.2响应时间：

[root@boc-103 test]# export NODE_IPS="192.168.2.103 192.168.2.104 192.168.2.108"
[root@boc-103 test]# for ip in ${NODE_IPS}; do   ETCDCTL_API=3 etcdctl   --endpoints=https://${ip}:2379    --cacert=/etc/etcd/ssl/ca.crt   --cert=/etc/etcd/ssl/client.crt   --key=/etc/etcd/ssl/client.key   endpoint health; done
https://192.168.2.103:2379 is healthy: successfully committed proposal: took = 1.318040913s
https://192.168.2.104:2379 is healthy: successfully committed proposal: took = 1.557571675s
https://192.168.2.108:2379 is healthy: successfully committed proposal: took = 54.749774ms

3. 错误现象：

​ kube-controller-manager和kube-scheduler重启原因etcdserver: request timed out

E0224 03:54:36.433286       1 cronjob_controller.go:125] Failed to extract job list: etcdserver: request timed out
E0224 03:54:37.592575       1 leaderelection.go:331] error retrieving resource lock kube-system/kube-controller-manager: etcdserver: request timed out
I0224 03:54:38.047024       1 leaderelection.go:288] failed to renew lease kube-system/kube-controller-manager: timed out waiting for the condition
I0224 03:54:38.047089       1 event.go:281] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"", Name:"", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' portal135.bocloud.com_46d96c59-fa2a-43d3-aa0e-4c969a287338 stopped leading
F0224 03:54:38.047158       1 controllermanager.go:279] leaderelection lost

​ apiserver也出现多次

E0224 03:54:37.591267       1 status.go:71] apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:"etcdserver: request timed out"}

4.cube环境 10.10.5.8 密码：xxxx 配置：3master 8c16g

4.1重启次数：

[root@master ~]# kubectl get pods -n kube-system|grep .novalocal
kube-apiserver-master.novalocal            1/1     Running            1          8h
kube-apiserver-node-1.novalocal            1/1     Running            1          8h
kube-apiserver-node-2.novalocal            1/1     Running            1          8h
kube-controller-manager-master.novalocal   1/1     Running            7          8h
kube-controller-manager-node-1.novalocal   1/1     Running            8          8h
kube-controller-manager-node-2.novalocal   1/1     Running            6          8h
kube-scheduler-master.novalocal            1/1     Running            8          8h
kube-scheduler-node-1.novalocal            1/1     Running            6          8h
kube-scheduler-node-2.novalocal            1/1     Running            8          8h

4.2响应时间：

https://10.10.5.8:2379 is healthy: successfully committed proposal: took = 15.970648ms
https://10.10.5.48:2379 is healthy: successfully committed proposal: took = 13.325127ms
https://10.10.5.38:2379 is healthy: successfully committed proposal: took = 18.190006ms

5.解决办法

5.1 扩容到32g6c，但io操作依旧很慢，关机在于硬盘的读写速度。

5.2 做一个ramdisk。把内存挂载到/var/lib/etcd下当做硬盘存储etcd数据。

mkdir /var/lib/etcd && mount -t tmpfs -o size=2G tmpfs /var/lib/etcd && echo "tmpfs                    /var/lib/etcd    tmpfs   defaults,size=2G        0 0" >> /etc/fstab

5.2.1 io操作对比结果：

• ramdisk 2.3 GB/s
• 数据盘 24.0 MB/s

[root@node-139 ~]# dd bs=1M count=1000 if=/dev/zero of=/media/tmp/a.txt conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 0.457043 s, 2.3 GB/s
[root@node-139 ~]# dd bs=1M count=1000 if=/dev/zero of=/home/a.txt conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 43.7001 s, 24.0 MB/s

注：/media/tmp是挂载ramdisk的路径 

5.3 验证结果：etcd的响应速度有明显提升。

[root@node-218 ~]# for ip in ${NODE_IPS}; do   ETCDCTL_API=3 etcdctl   --endpoints=https://${ip}:2379    --cacert=/etc/etcd/ssl/ca.crt   --cert=/etc/etcd/ssl/client.crt   --key=/etc/etcd/ssl/client.key   endpoint health; done
https://192.168.2.218:2379 is healthy: successfully committed proposal: took = 11.960397ms
https://192.168.2.219:2379 is healthy: successfully committed proposal: took = 10.482817ms
https://192.168.2.220:2379 is healthy: successfully committed proposal: took = 10.786569ms
[root@node-218 ~]# kubectl get pods -n kube-system|grep .dev
kube-apiserver-node-218.dev                1/1     Running            0          10m
kube-apiserver-node-219.dev                1/1     Running            0          10m
kube-apiserver-node-220.dev                1/1     Running            0          11m
kube-controller-manager-node-218.dev       1/1     Running            1          10m
kube-controller-manager-node-219.dev       1/1     Running            1          10m
kube-controller-manager-node-220.dev       1/1     Running            1          11m
kube-scheduler-node-218.dev                1/1     Running            1          10m
kube-scheduler-node-219.dev                1/1     Running            0          11m
kube-scheduler-node-220.dev                1/1     Running            1          11m
[root@node-218 ~]# df -h
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/centos-root    28G  3.5G   25G  13% /
devtmpfs                  5.8G     0  5.8G   0% /dev
tmpfs                     5.8G     0  5.8G   0% /dev/shm
tmpfs                     5.8G   34M  5.8G   1% /run
tmpfs                     5.8G     0  5.8G   0% /sys/fs/cgroup
/dev/sda1                1014M  145M  870M  15% /boot
tmpfs                     1.2G     0  1.2G   0% /run/user/0
tmpfs                     2.0G  125M  1.9G   7% /var/lib/etcd
192.168.2.214:/opt/share   28G  2.5G   26G   9% /abcsys/upload

5.4 经过18小时后0次重启

[root@node-218 ~]# kubectl get pods -n kube-system|grep dev
kube-apiserver-node-218.dev                1/1     Running            0          18h
kube-apiserver-node-219.dev                1/1     Running            0          18h
kube-apiserver-node-220.dev                1/1     Running            0          18h
kube-controller-manager-node-218.dev       1/1     Running            1          18h
kube-controller-manager-node-219.dev       1/1     Running            1          18h
kube-controller-manager-node-220.dev       1/1     Running            1          18h
kube-scheduler-node-218.dev                1/1     Running            1          18h
kube-scheduler-node-219.dev                1/1     Running            0          18h
kube-scheduler-node-220.dev                1/1     Running            1          18h

5.5 使用ramdisk纯在的问题

由于数据是存储在内存中的，断电即丢失。

6 总结

​ 开发环境需要提升硬盘的读写速度，现在的硬盘读写熟读只有24.0 MB/s，导致etcd数据库经常出现request timeout，健康检查超时，服务重启等问题。