/var/log/message 中报错 No space left on device when starting/stopping services only

怀疑是 inotify 中 watchers 用完，需要增加 watchers 数量

查看 inotify 参数

[root@k8s01 ~]# sysctl fs.inotify
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192

增加 inotify.max_user_watches

sysctl -n -w fs.inotify.max_user_watches=16384

查看已经使用的 inotyfy 数量，可以使用脚本 inotify-consumers

[root@k8s01 ~]# sh inotify-consumer 

   INOTIFY
   WATCHER
    COUNT     PID     CMD
----------------------------------------
     332     5498  /usr/bin/kubelet --address=10.2.7.200 --hostname-override=10.2.7.200 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.ku
       6      662  /usr/lib/systemd/systemd-udevd
       5     1054  /usr/sbin/NetworkManager --no-daemon
       4        1  /usr/lib/systemd/systemd --switched-root --system --deserialize 22
       4        1  /usr/lib/systemd/systemd --switched-root --system --deserialize 22
       4     1064  /usr/lib/polkit-1/polkitd --no-debug
       4     1054  /usr/sbin/NetworkManager --no-daemon
       3     1074  /usr/sbin/crond -n
       3     1064  /usr/lib/polkit-1/polkitd --no-debug
       2     1338  /usr/sbin/rsyslogd -n
       2     1053  /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
       1     5498  /usr/bin/kubelet --address=10.2.7.200 --hostname-override=10.2.7.200 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.ku
       1        1  /usr/lib/systemd/systemd --switched-root --system --deserialize 22
       0     5498  /usr/bin/kubelet --address=10.2.7.200 --hostname-override=10.2.7.200 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.ku

也可以使用下面的脚本

[root@k8s01 ~]# cat inotify_watchers.sh
set -o errexit
set -o pipefail
lsof +c 0 -n -P -u root \
| awk '/inotify$/ { gsub(/[urw]$/,"",$4); print $1" "$2" "$4 }' \
| while read name pid fd; do \
exe="$(readlink -f /proc/$pid/exe || echo n/a)"; \
fdinfo="/proc/$pid/fdinfo/$fd" ; \
count="$(grep -c inotify "$fdinfo" || true)"; \
echo "$name $exe $pid $fdinfo $count"; \
done

参考文章：

No space left on device when starting/stopping services only

How do I find out what inotify watches have been registered?