最近的docker容器经常被kill掉,k8s中该节点的pod也被驱赶,因而使用以下命令查看被kill的所有进程

dmesg | grep -i -B100 'killed process'

查看到pod被驱赶的原因:

[3899860.525793] Out of memory: Kill process 64058 (nvidia-device-p) score 999 or sacrifice child
[3899860.526961] Killed process 64058 (nvidia-device-p) total-vm:126548kB, anon-rss:2080kB, file-rss:0kB, shmem-rss:0kB

查看到docker容器被kill 的原因:

[3899859.737598] Out of memory: Kill process 27562 (jupyter-noteboo) score 1000 or sacrifice child
[3899859.738640] Killed process 27562 (jupyter-noteboo) total-vm:215864kB, anon-rss:45928kB, file-rss:0kB, shmem-rss:0kB

查看内存使用情况

tilyp@tilyp:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           251G         96G        114G        2.5M         41G        153G
Swap:            0B          0B          0B

 

Logo

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐