k8s 节点notready

登录到节点上执行journalctl -u kubelet -f 查看发现

Failed to watch directory "/sys/fs/cgroup/memory/system.slice/run-rb5b8b6479cfc40368b678478fb18571d.scope": inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-rb5b8b6479cfc40368b678478fb18571d.scope: no space left on device
Failed to watch directory "/sys/fs/cgroup/memory/system.slice": inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-rb5b8b6479cfc40368b678478fb18571d.scope: no space left on device
kubelet.go:1344] Failed to start cAdvisor inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-rb5b8b6479cfc40368b678478fb18571d.scope: no space left on device
Using node IP: ...
kubelet.service: Main process exited, code=exited, status=255/n/a
kubelet.service: Unit entered failed state.
kubelet.service: Failed with result 'exit-code'.
kubelet.service: Service hold-off time over, scheduling restart.
Stopped Kubernetes Kubelet Server.
Starting Kubernetes Kubelet Server...
Started Kubernetes Kubelet Server.

由于没有剩余空间导致kubelet一直在重启，问题在于"/sys/fs/cgroup/memory/system.slice/目录下run-*.scope"文件过多导致的。

执行

 cat /proc/sys/fs/inotify/max_user_watches # default is 8192

可以看到默认最大只能监视8192。
执行

sudo sysctl fs.inotify.max_user_watches=1048576 # increase to 1048576

修改最大值，之后再查看节点状态就是ready了。