起因: 同事反馈 服务挂了，kuboard上查看是服务挂掉了，liveness port 异常，通过查看pod状态，发现服务被重启了。

1 pod里的java进程因为k8s主机内存不足被kill了

通过journalctl -f 查看日志 发现如下

通过journalctl 查看所有日志 找到如下

这个是宿主机内存不足。显示的进程是宿主机的进程。包含了多个pod的信息

8月 27 14:50:40 op-k8s-n003 kubelet[1736]: I0827 14:50:40.323686    1736 reconciler.go:301] Volume detached for volume "volume-1" (UniqueName: "kubernetes.io/host-path/9781685f-e
...skipping...
8月 27 21:12:32 op-k8s-n003 kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=934
8月 27 21:12:32 op-k8s-n003 kernel: java cpuset=138618e5e5982e4f2f0e68b8375424716cce030767cad1d14aa2ace09396bf8e mems_allowed=0
8月 27 21:12:32 op-k8s-n003 kernel: CPU: 2 PID: 23243 Comm: java Kdump: loaded Tainted: G               ------------ T 3.10.0-1127.8.2.el7.x86_64 #1
8月 27 21:12:32 op-k8s-n003 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
8月 27 21:12:32 op-k8s-n003 kernel: Call Trace:
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa757ffa5>] dump_stack+0x19/0x1b
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa757a8c3>] dump_header+0x90/0x229
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fc24ce>] oom_kill_process+0x25e/0x3f0
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6f33a01>] ? cpuset_mems_allowed_intersects+0x21/0x30
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fc1f2d>] ? oom_unkillable_task+0xcd/0x120
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fc1fd6>] ? find_lock_task_mm+0x56/0xc0
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fc2d26>] out_of_memory+0x4b6/0x4f0
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa757b3e0>] __alloc_pages_slowpath+0x5db/0x729
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fc91a6>] __alloc_pages_nodemask+0x436/0x450
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa7018e78>] alloc_pages_current+0x98/0x110
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fbe3d7>] __page_cache_alloc+0x97/0xb0
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fc0f90>] filemap_fault+0x270/0x420
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffc066ba4e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffc066bc4c>] xfs_filemap_fault+0x2c/0x30 [xfs]
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fedf4a>] __do_fault.isra.61+0x8a/0x100
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fee4fc>] do_read_fault.isra.63+0x4c/0x1b0
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6ff5d60>] handle_mm_fault+0xa20/0xfb0
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa7584fa6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa758d653>] __do_page_fault+0x213/0x500
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa758d975>] do_page_fault+0x35/0x90
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa7589778>] page_fault+0x28/0x30
8月 27 21:12:32 op-k8s-n003 kernel: Mem-Info:
8月 27 21:12:32 op-k8s-n003 kernel: active_anon:1776970 inactive_anon:23046 isolated_anon:0
                                      active_file:0 inactive_file:378 isolated_file:177
                                      unevictable:0 dirty:0 writeback:0 unstable:0
                                      slab_reclaimable:21428 slab_unreclaimable:35127
                                      mapped:20469 shmem:26463 pagetables:13227 bounce:0
                                      free:25804 free_pcp:114 free_cma:0
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 DMA free:15860kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB iso
8月 27 21:12:32 op-k8s-n003 kernel: lowmem_reserve[]: 0 2735 7707 7707
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 DMA32 free:44036kB min:23940kB low:29924kB high:35908kB active_anon:2509752kB inactive_anon:34620kB active_file:0kB inactive_file:0kB u
8月 27 21:12:32 op-k8s-n003 kernel: lowmem_reserve[]: 0 0 4971 4971
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 Normal free:43320kB min:43504kB low:54380kB high:65256kB active_anon:4598128kB inactive_anon:57564kB active_file:0kB inactive_file:2260
8月 27 21:12:32 op-k8s-n003 kernel: lowmem_reserve[]: 0 0 0 0
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15860kB
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 DMA32: 1268*4kB (UEM) 672*8kB (UEM) 687*16kB (UEM) 510*32kB (UEM) 85*64kB (UEM) 4*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096k
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 Normal: 2733*4kB (UEM) 1811*8kB (UE) 1146*16kB (UEM) 1*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 43788kB
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
8月 27 21:12:32 op-k8s-n003 kernel: 27157 total pagecache pages
8月 27 21:12:32 op-k8s-n003 kernel: 0 pages in swap cache
8月 27 21:12:32 op-k8s-n003 kernel: Swap cache stats: add 0, delete 0, find 0/0
8月 27 21:12:32 op-k8s-n003 kernel: Free swap  = 0kB
8月 27 21:12:32 op-k8s-n003 kernel: Total swap = 0kB
8月 27 21:12:32 op-k8s-n003 kernel: 2097022 pages RAM
8月 27 21:12:32 op-k8s-n003 kernel: 0 pages HighMem/MovableOnly
...skipping...
8月 27 21:12:32 op-k8s-n003 kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=934
8月 27 21:12:32 op-k8s-n003 kernel: java cpuset=a4dd32fa243a6c20261d4b557f36651f23401303a9af274c99ab3ce5f3f62cff mems_allowed=0
8月 27 21:12:32 op-k8s-n003 kernel: CPU: 6 PID: 28372 Comm: java Kdump: loaded Tainted: G               ------------ T 3.10.0-1127.8.2.el7.x86_64 #1
8月 27 21:12:32 op-k8s-n003 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
8月 27 21:12:32 op-k8s-n003 kernel: Call Trace:
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa757ffa5>] dump_stack+0x19/0x1b
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa757a8c3>] dump_header+0x90/0x229
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6f06ce2>] ? ktime_get_ts64+0x52/0xf0
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fc24ce>] oom_kill_process+0x25e/0x3f0
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6f33a01>] ? cpuset_mems_allowed_intersects+0x21/0x30
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fc1f2d>] ? oom_unkillable_task+0xcd/0x120
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fc1fd6>] ? find_lock_task_mm+0x56/0xc0
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fc2d26>] out_of_memory+0x4b6/0x4f0
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa757b3e0>] __alloc_pages_slowpath+0x5db/0x729
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fc91a6>] __alloc_pages_nodemask+0x436/0x450
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa7018e78>] alloc_pages_current+0x98/0x110
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fbe3d7>] __page_cache_alloc+0x97/0xb0
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fc0f90>] filemap_fault+0x270/0x420
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffc066ba4e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffc066bc4c>] xfs_filemap_fault+0x2c/0x30 [xfs]
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fedf4a>] __do_fault.isra.61+0x8a/0x100
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6fee4fc>] do_read_fault.isra.63+0x4c/0x1b0
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa6ff5d60>] handle_mm_fault+0xa20/0xfb0
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa758d653>] __do_page_fault+0x213/0x500
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa758d975>] do_page_fault+0x35/0x90
8月 27 21:12:32 op-k8s-n003 kernel:  [<ffffffffa7589778>] page_fault+0x28/0x30
8月 27 21:12:32 op-k8s-n003 kernel: Mem-Info:
8月 27 21:12:32 op-k8s-n003 kernel: active_anon:1776970 inactive_anon:23046 isolated_anon:0
                                      active_file:0 inactive_file:433 isolated_file:177
                                      unevictable:0 dirty:0 writeback:0 unstable:0
                                      slab_reclaimable:21428 slab_unreclaimable:35127
                                      mapped:20469 shmem:26463 pagetables:13227 bounce:0
                                      free:25622 free_pcp:74 free_cma:0
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 DMA free:15860kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB iso
8月 27 21:12:32 op-k8s-n003 kernel: lowmem_reserve[]: 0 2735 7707 7707
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 DMA32 free:43352kB min:23940kB low:29924kB high:35908kB active_anon:2509752kB inactive_anon:34620kB active_file:0kB inactive_file:404kB
8月 27 21:12:32 op-k8s-n003 kernel: lowmem_reserve[]: 0 0 4971 4971
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 Normal free:43276kB min:43504kB low:54380kB high:65256kB active_anon:4598128kB inactive_anon:57564kB active_file:0kB inactive_file:1328
8月 27 21:12:32 op-k8s-n003 kernel: lowmem_reserve[]: 0 0 0 0
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15860kB
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 DMA32: 1423*4kB (UEM) 681*8kB (UEM) 687*16kB (UEM) 516*32kB (UEM) 85*64kB (UEM) 4*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096k
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 Normal: 2737*4kB (UE) 1820*8kB (UEM) 1148*16kB (UEM) 9*32kB (UM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44164kB
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
8月 27 21:12:32 op-k8s-n003 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
8月 27 21:12:32 op-k8s-n003 kernel: 27090 total pagecache pages
8月 27 21:12:32 op-k8s-n003 kernel: 0 pages in swap cache
8月 27 21:12:32 op-k8s-n003 kernel: Swap cache stats: add 0, delete 0, find 0/0
8月 27 21:12:32 op-k8s-n003 kernel: Free swap  = 0kB
8月 27 21:12:32 op-k8s-n003 kernel: Total swap = 0kB
8月 27 21:12:32 op-k8s-n003 kernel: 2097022 pages RAM
8月 27 21:12:32 op-k8s-n003 kernel: 0 pages HighMem/MovableOnly
8月 27 21:12:32 op-k8s-n003 kernel: 119246 pages reserved
...skipping...
8月 27 21:14:57 op-k8s-n003 kernel: flanneld invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=-998
8月 27 21:14:57 op-k8s-n003 kernel: flanneld cpuset=c4cd2a181d3a84d359696bbeda48fb38947f8889ea9ec0c1576cc466f5f3dcea mems_allowed=0
8月 27 21:14:57 op-k8s-n003 kernel: CPU: 4 PID: 9467 Comm: flanneld Kdump: loaded Tainted: G               ------------ T 3.10.0-1127.8.2.el7.x86_64 #1
8月 27 21:14:57 op-k8s-n003 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
8月 27 21:14:57 op-k8s-n003 kernel: Call Trace:
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa757ffa5>] dump_stack+0x19/0x1b
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa757a8c3>] dump_header+0x90/0x229
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa6f06ce2>] ? ktime_get_ts64+0x52/0xf0
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa6fc24ce>] oom_kill_process+0x25e/0x3f0
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa6f33a01>] ? cpuset_mems_allowed_intersects+0x21/0x30
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa6fc1f2d>] ? oom_unkillable_task+0xcd/0x120
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa6fc1fd6>] ? find_lock_task_mm+0x56/0xc0
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa6fc2d26>] out_of_memory+0x4b6/0x4f0
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa757b3e0>] __alloc_pages_slowpath+0x5db/0x729
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa6fc91a6>] __alloc_pages_nodemask+0x436/0x450
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa7018e78>] alloc_pages_current+0x98/0x110
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa6fbe3d7>] __page_cache_alloc+0x97/0xb0
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa6fc0f90>] filemap_fault+0x270/0x420
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffc066ba4e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffc066bc4c>] xfs_filemap_fault+0x2c/0x30 [xfs]
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa6fedf4a>] __do_fault.isra.61+0x8a/0x100
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa6fee4fc>] do_read_fault.isra.63+0x4c/0x1b0
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa6ff5d60>] handle_mm_fault+0xa20/0xfb0
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa704c5e3>] ? do_sync_read+0x93/0xe0
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa758d653>] __do_page_fault+0x213/0x500
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa758d975>] do_page_fault+0x35/0x90
8月 27 21:14:57 op-k8s-n003 kernel:  [<ffffffffa7589778>] page_fault+0x28/0x30
8月 27 21:14:57 op-k8s-n003 kernel: Mem-Info:
8月 27 21:14:57 op-k8s-n003 kernel: active_anon:1796326 inactive_anon:22875 isolated_anon:0
                                      active_file:7 inactive_file:0 isolated_file:0
                                      unevictable:0 dirty:0 writeback:0 unstable:0
                                      slab_reclaimable:20345 slab_unreclaimable:33853
                                      mapped:21429 shmem:26596 pagetables:12811 bounce:0
                                      free:25491 free_pcp:0 free_cma:0
8月 27 21:14:57 op-k8s-n003 kernel: Node 0 DMA free:15860kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB iso
8月 27 21:14:57 op-k8s-n003 kernel: lowmem_reserve[]: 0 2735 7707 7707
8月 27 21:14:57 op-k8s-n003 kernel: Node 0 DMA32 free:42604kB min:23940kB low:29924kB high:35908kB active_anon:2550980kB inactive_anon:34368kB active_file:0kB inactive_file:0kB u
8月 27 21:14:57 op-k8s-n003 kernel: lowmem_reserve[]: 0 0 4971 4971
8月 27 21:14:57 op-k8s-n003 kernel: Node 0 Normal free:43500kB min:43504kB low:54380kB high:65256kB active_anon:4634324kB inactive_anon:57132kB active_file:28kB inactive_file:0kB
8月 27 21:14:57 op-k8s-n003 kernel: lowmem_reserve[]: 0 0 0 0
8月 27 21:14:57 op-k8s-n003 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15860kB
8月 27 21:14:57 op-k8s-n003 kernel: Node 0 DMA32: 1549*4kB (UEM) 603*8kB (UEM) 422*16kB (UEM) 446*32kB (UEM) 161*64kB (UEM) 2*128kB (EM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096k
8月 27 21:14:57 op-k8s-n003 kernel: Node 0 Normal: 3056*4kB (UEM) 1636*8kB (UEM) 1120*16kB (UE) 13*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 43648kB
8月 27 21:14:57 op-k8s-n003 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
8月 27 21:14:57 op-k8s-n003 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
8月 27 21:14:57 op-k8s-n003 kernel: 26603 total pagecache pages
8月 27 21:14:57 op-k8s-n003 kernel: 0 pages in swap cache
8月 27 21:14:57 op-k8s-n003 kernel: Swap cache stats: add 0, delete 0, find 0/0
8月 27 21:14:57 op-k8s-n003 kernel: Free swap  = 0kB
8月 27 21:14:57 op-k8s-n003 kernel: Total swap = 0kB
8月 27 21:14:57 op-k8s-n003 kernel: 2097022 pages RAM
8月 27 21:14:57 op-k8s-n003 kernel: 0 pages HighMem/MovableOnly
...skipping...
8月 27 21:16:22 op-k8s-n003 kernel: coredns invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=-998
8月 27 21:16:22 op-k8s-n003 kernel: coredns cpuset=2bdc14648fbfb1cec8e6fdb1600a79c2ce3d9fa2c1c3b801e31e6e423fec5560 mems_allowed=0
8月 27 21:16:22 op-k8s-n003 kernel: CPU: 1 PID: 30943 Comm: coredns Kdump: loaded Tainted: G               ------------ T 3.10.0-1127.8.2.el7.x86_64 #1
8月 27 21:16:22 op-k8s-n003 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
8月 27 21:16:22 op-k8s-n003 kernel: Call Trace:
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa757ffa5>] dump_stack+0x19/0x1b
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa757a8c3>] dump_header+0x90/0x229
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa6f06ce2>] ? ktime_get_ts64+0x52/0xf0
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa6fc24ce>] oom_kill_process+0x25e/0x3f0
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa6f33a01>] ? cpuset_mems_allowed_intersects+0x21/0x30
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa6fc1f2d>] ? oom_unkillable_task+0xcd/0x120
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa6fc1fd6>] ? find_lock_task_mm+0x56/0xc0
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa6fc2d26>] out_of_memory+0x4b6/0x4f0
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa757b3e0>] __alloc_pages_slowpath+0x5db/0x729
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa6fc91a6>] __alloc_pages_nodemask+0x436/0x450
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa7018e78>] alloc_pages_current+0x98/0x110
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa6fbe3d7>] __page_cache_alloc+0x97/0xb0
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa6fc0f90>] filemap_fault+0x270/0x420
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffc066ba4e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffc066bc4c>] xfs_filemap_fault+0x2c/0x30 [xfs]
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa6fedf4a>] __do_fault.isra.61+0x8a/0x100
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa6eca2d0>] ? hrtimer_get_res+0x50/0x50
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa6fee4fc>] do_read_fault.isra.63+0x4c/0x1b0
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa6ff5d60>] handle_mm_fault+0xa20/0xfb0
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa758d653>] __do_page_fault+0x213/0x500
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa758d975>] do_page_fault+0x35/0x90
8月 27 21:16:22 op-k8s-n003 kernel:  [<ffffffffa7589778>] page_fault+0x28/0x30
8月 27 21:16:22 op-k8s-n003 kernel: Mem-Info:
8月 27 21:16:22 op-k8s-n003 kernel: active_anon:1795773 inactive_anon:22885 isolated_anon:0
                                      active_file:0 inactive_file:223 isolated_file:178
                                      unevictable:0 dirty:5 writeback:2 unstable:0
                                      slab_reclaimable:20103 slab_unreclaimable:32821
                                      mapped:22083 shmem:26626 pagetables:12389 bounce:0
                                      free:25648 free_pcp:10 free_cma:0
8月 27 21:16:22 op-k8s-n003 kernel: Node 0 DMA free:15860kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB iso
8月 27 21:16:22 op-k8s-n003 kernel: lowmem_reserve[]: 0 2735 7707 7707
8月 27 21:16:22 op-k8s-n003 kernel: Node 0 DMA32 free:43804kB min:23940kB low:29924kB high:35908kB active_anon:2547324kB inactive_anon:34396kB active_file:0kB inactive_file:684kB
8月 27 21:16:22 op-k8s-n003 kernel: lowmem_reserve[]: 0 0 4971 4971
8月 27 21:16:22 op-k8s-n003 kernel: Node 0 Normal free:42928kB min:43504kB low:54380kB high:65256kB active_anon:4635768kB inactive_anon:57144kB active_file:604kB inactive_file:20
8月 27 21:16:22 op-k8s-n003 kernel: lowmem_reserve[]: 0 0 0 0
8月 27 21:16:22 op-k8s-n003 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15860kB
8月 27 21:16:22 op-k8s-n003 kernel: Node 0 DMA32: 1845*4kB (UEM) 646*8kB (UEM) 591*16kB (UEM) 401*32kB (UEM) 150*64kB (UEM) 1*128kB (E) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB
8月 27 21:16:22 op-k8s-n003 kernel: Node 0 Normal: 3158*4kB (UEM) 1753*8kB (UEM) 1065*16kB (UEM) 3*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 43792kB
8月 27 21:16:22 op-k8s-n003 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
8月 27 21:16:22 op-k8s-n003 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
8月 27 21:16:22 op-k8s-n003 kernel: 26739 total pagecache pages
8月 27 21:16:22 op-k8s-n003 kernel: 0 pages in swap cache
8月 27 21:16:22 op-k8s-n003 kernel: Swap cache stats: add 0, delete 0, find 0/0
8月 27 21:16:22 op-k8s-n003 kernel: Free swap  = 0kB
8月 27 21:16:22 op-k8s-n003 kernel: Total swap = 0kB
8月 27 21:16:22 op-k8s-n003 kernel: 2097022 pages RAM
8月 27 21:16:22 op-k8s-n003 kernel: 0 pages HighMem/MovableOnly
8月 27 21:16:22 op-k8s-n003 kernel: 119246 pages reserved
8月 27 21:16:22 op-k8s-n003 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
8月 27 21:16:22 op-k8s-n003 kernel: [  537]     0   537    11915     3235      29        0             0 systemd-journal
8月 27 21:16:22 op-k8s-n003 kernel: [  566]     0   566    49643       98      29        0             0 lvmetad
8月 27 21:16:22 op-k8s-n003 kernel: [  586]     0   586    11219      120      24        0         -1000 systemd-udevd
8月 27 21:16:22 op-k8s-n003 kernel: [  735]     0   735    13883      112      27        0         -1000 auditd
8月 27 21:16:22 op-k8s-n003 kernel: [  759]    81   759    14559      164      32        0          -900 dbus-daemon
8月 27 21:16:22 op-k8s-n003 kernel: [  760]     0   760     5424       85      15        0             0 irqbalance
8月 27 21:16:22 op-k8s-n003 kernel: [  761]     0   761    24922      403      42        0             0 VGAuthService
8月 27 21:16:22 op-k8s-n003 kernel: [  762]     0   762   119219      620      87        0             0 NetworkManager
8月 27 21:16:22 op-k8s-n003 kernel: [  763]     0   763    76320      362      63        0             0 vmtoolsd
8月 27 21:16:22 op-k8s-n003 kernel: [  765]   999   765   153062     1886      62        0             0 polkitd
8月 27 21:16:22 op-k8s-n003 kernel: [  770]     0   770     6629      113      18        0             0 systemd-logind
8月 27 21:16:22 op-k8s-n003 kernel: [  773]     0   773    31596      155      20        0             0 crond
8月 27 21:16:22 op-k8s-n003 kernel: [ 1041]     0  1041    56686     2054      47        0             0 rsyslogd
8月 27 21:16:22 op-k8s-n003 kernel: [ 1048]     0  1048   146609     3347     104        0             0 tuned
8月 27 21:16:22 op-k8s-n003 kernel: [ 1052]     0  1052    28231      258      59        0         -1000 sshd
8月 27 21:16:22 op-k8s-n003 kernel: [ 1058]     0  1058   709325    10271     134        0             0 containerd
8月 27 21:16:22 op-k8s-n003 kernel: [ 1059]     0  1059    28952       40      14        0             0 rhsmcertd
8月 27 21:16:22 op-k8s-n003 kernel: [ 1061]     0  1061   320645    18352     139        0          -999 dockerd
8月 27 21:16:22 op-k8s-n003 kernel: [ 1086]     0  1086    27551       33      10        0             0 agetty
8月 27 21:16:22 op-k8s-n003 kernel: [ 1736]     0  1736   497129    16141     148        0          -999 kubelet
8月 27 21:16:22 op-k8s-n003 kernel: [ 2191]     0  2191    26922      976       9        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [ 2192]     0  2192    26922      982       9        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [ 2226]     0  2226      253        1       4        0          -998 pause
8月 27 21:16:22 op-k8s-n003 kernel: [ 2231]     0  2231      253        1       4        0          -998 pause
8月 27 21:16:22 op-k8s-n003 kernel: [ 2456]     0  2456    27274     1463      10        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [ 2473]     0  2473   135460     3928      54        0          -998 flanneld
8月 27 21:16:22 op-k8s-n003 kernel: [ 2541]     0  2541    27274      976      10        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [ 2559]     0  2559    35256     5686      36        0          -999 kube-proxy
8月 27 21:16:22 op-k8s-n003 kernel: [31437]     0 31437    26922      775      11        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [31458]     0 31458      253        1       4        0          -998 pause
8月 27 21:16:22 op-k8s-n003 kernel: [24473]     0 24473    26922      234       9        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [24491]     0 24491    35953     3067      32        0          -998 coredns
8月 27 21:16:22 op-k8s-n003 kernel: [10620]     0 10620    27274      720      10        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [10648]     0 10648      253        1       4        0          -998 pause
8月 27 21:16:22 op-k8s-n003 kernel: [10732]     0 10732    26922      730      10        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [10749]     0 10749    32092     1120      22        0          1000 dex
8月 27 21:16:22 op-k8s-n003 kernel: [26997]     0 26997    26922      959       9        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [27017]     0 27017      253        1       4        0          -998 pause
8月 27 21:16:22 op-k8s-n003 kernel: [27114]     0 27114    26922      949       9        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [27131]     0 27131   177876     2605      57        0           975 filebeat
8月 27 21:16:22 op-k8s-n003 kernel: [27211]     0 27211    26922     1413      10        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [27231]     0 27231     3771       66      12        0           934 sh
8月 27 21:16:22 op-k8s-n003 kernel: [27274]     0 27274  1632971   209598     714        0           934 java
8月 27 21:16:22 op-k8s-n003 kernel: [27275]     0 27275     1940       25       9        0           934 tailf
8月 27 21:16:22 op-k8s-n003 kernel: [30153]     0 30153    26922      966       9        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [30173]     0 30173      253        1       4        0          -998 pause
8月 27 21:16:22 op-k8s-n003 kernel: [30266]     0 30266    26922      961       9        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [30284]     0 30284   164598     3374      60        0           975 filebeat
8月 27 21:16:22 op-k8s-n003 kernel: [31763]     0 31763    27274      968      10        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [31786]     0 31786      253        1       4        0          -998 pause
8月 27 21:16:22 op-k8s-n003 kernel: [31879]     0 31879    26922      941       9        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [31896]     0 31896   197102     3235      62        0           975 filebeat
8月 27 21:16:22 op-k8s-n003 kernel: [18573]     0 18573    26922      932      10        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [18594]     0 18594      253        1       4        0          -998 pause
8月 27 21:16:22 op-k8s-n003 kernel: [18687]     0 18687    27274      983      10        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [18704]     0 18704   178933     3169      60        0           975 filebeat
8月 27 21:16:22 op-k8s-n003 kernel: [18856]     0 18856    26922      975       9        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [18891]     0 18891     3771       66      12        0           934 sh
8月 27 21:16:22 op-k8s-n003 kernel: [18926]     0 18926  1666106   182079     730        0           934 java
8月 27 21:16:22 op-k8s-n003 kernel: [18927]     0 18927     1940       25       9        0           934 tailf
8月 27 21:16:22 op-k8s-n003 kernel: [19116]     0 19116    27274      958      10        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [19137]     0 19137      253        1       4        0          -998 pause
8月 27 21:16:22 op-k8s-n003 kernel: [19225]     0 19225    27274      969      10        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [19242]     0 19242   235753     3387      67        0           975 filebeat
8月 27 21:16:22 op-k8s-n003 kernel: [19411]     0 19411    26922      956      10        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [19430]     0 19430     3771       66      11        0           934 sh
8月 27 21:16:22 op-k8s-n003 kernel: [19465]     0 19465  1667180   189217     736        0           934 java
8月 27 21:16:22 op-k8s-n003 kernel: [19466]     0 19466     1940       25       9        0           934 tailf
8月 27 21:16:22 op-k8s-n003 kernel: [27091]     0 27091    26922      965       9        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [27111]     0 27111      253        1       4        0          -998 pause
8月 27 21:16:22 op-k8s-n003 kernel: [27247]     0 27247    26922      933      10        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [27264]     0 27264   122579     2436      51        0           975 filebeat
8月 27 21:16:22 op-k8s-n003 kernel: [27427]     0 27427    26922      979       9        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [27447]     0 27447     3801       79      12        0           934 sh
8月 27 21:16:22 op-k8s-n003 kernel: [27488]     0 27488  1720073   191636     841        0           934 java
8月 27 21:16:22 op-k8s-n003 kernel: [27489]     0 27489     1942       27       9        0           934 tailf
8月 27 21:16:22 op-k8s-n003 kernel: [23974]     0 23974    39331      355      79        0             0 sshd
8月 27 21:16:22 op-k8s-n003 kernel: [23977]     0 23977    28919      127      14        0             0 bash
8月 27 21:16:22 op-k8s-n003 kernel: [28968]     0 28968     1096       24       8        0             0 runsvdir
8月 27 21:16:22 op-k8s-n003 kernel: [29033]     0 29033     1058       19       7        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [29035]   994 29035    10383      906      17        0             0 redis-server
8月 27 21:16:22 op-k8s-n003 kernel: [29053]     0 29053     1094       23       8        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [29054]     0 29054     1058       18       7        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [29117]     0 29117     1094       19       8        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [29278]     0 29278     1058       19       6        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [29280]   993 29280   522437    16793      57        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [29282]   993 29282   522504     5992      45        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [29283]   993 29283   522437     4308      31        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [29284]   993 29284   522471     4554      29        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [29286]   993 29286   522589      570      31        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [29287]   993 29287     8450      538      19        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [29288]   993 29288   522552      529      26        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [29362]     0 29362     1094       19       8        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [29837]     0 29837     1058       18       7        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [29839]   995 29839     2924       73      11        0             0 gitlab-unicorn-
8月 27 21:16:22 op-k8s-n003 kernel: [29865]   995 29865   201509   100983     351        0             0 bundle
8月 27 21:16:22 op-k8s-n003 kernel: [29875]     0 29875     1094       23       8        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [29877]     0 29877     1058       19       7        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [29879]   995 29879   249807   100122     452        0             0 bundle
8月 27 21:16:22 op-k8s-n003 kernel: [29922]     0 29922     1094       18       8        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [29935]     0 29935     1058       20       6        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [29973]     0 29973     1094       23       7        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [30040]     0 30040     1058       19       7        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [30044]     0 30044     9847      260      20        0             0 nginx
8月 27 21:16:22 op-k8s-n003 kernel: [30046]   996 30046    10946     1342      21        0             0 nginx
8月 27 21:16:22 op-k8s-n003 kernel: [30047]   996 30047    10910     1297      21        0             0 nginx
8月 27 21:16:22 op-k8s-n003 kernel: [30048]   996 30048    10910     1297      21        0             0 nginx
8月 27 21:16:22 op-k8s-n003 kernel: [30049]   996 30049    10910     1297      21        0             0 nginx
8月 27 21:16:22 op-k8s-n003 kernel: [30050]   996 30050    10910     1297      21        0             0 nginx
8月 27 21:16:22 op-k8s-n003 kernel: [30051]   996 30051    10910     1297      21        0             0 nginx
8月 27 21:16:22 op-k8s-n003 kernel: [30052]   996 30052    10989     1353      21        0             0 nginx
8月 27 21:16:22 op-k8s-n003 kernel: [30053]   996 30053    10993     1379      21        0             0 nginx
8月 27 21:16:22 op-k8s-n003 kernel: [30054]   996 30054     9888      292      19        0             0 nginx
8月 27 21:16:22 op-k8s-n003 kernel: [30124]     0 30124     1094       18       8        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [30136]     0 30136     1058       18       7        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [30140]     0 30140     2922       52      11        0             0 gitlab-logrotat
8月 27 21:16:22 op-k8s-n003 kernel: [30144]     0 30144     1091       18       8        0             0 sleep
8月 27 21:16:22 op-k8s-n003 kernel: [30214]     0 30214     1094       18       8        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [30286]     0 30286     1058       17       7        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [30305]   995 30305   204080   103847     358        0             0 bundle
8月 27 21:16:22 op-k8s-n003 kernel: [30306]   995 30306   204080   104020     358        0             0 bundle
8月 27 21:16:22 op-k8s-n003 kernel: [30307]   995 30307   204599   102180     358        0             0 bundle
8月 27 21:16:22 op-k8s-n003 kernel: [30308]   995 30308   204595   102098     358        0             0 bundle
8月 27 21:16:22 op-k8s-n003 kernel: [30310]   995 30310   204080   103968     358        0             0 bundle
8月 27 21:16:22 op-k8s-n003 kernel: [30311]   995 30311   204080   103075     357        0             0 bundle
8月 27 21:16:22 op-k8s-n003 kernel: [30312]   995 30312   204076   100019     348        0             0 bundle
8月 27 21:16:22 op-k8s-n003 kernel: [30313]   995 30313   204080   105291     360        0             0 bundle
8月 27 21:16:22 op-k8s-n003 kernel: [30314]   995 30314   204076    99997     348        0             0 bundle
8月 27 21:16:22 op-k8s-n003 kernel: [30361]     0 30361     1094       23       8        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [30380]     0 30380     1058       19       7        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [30436]     0 30436     1094       23       8        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [30457]     0 30457     1058       19       7        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [30483]     0 30483     1094       22       8        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [30545]     0 30545     1058       18       7        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [30600]     0 30600     1094       22       7        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [30616]     0 30616     1058       18       7        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [30702]     0 30702     1094       22       8        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [30703]     0 30703     1058       18       7        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [30730]     0 30730     1094       23       8        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [30850]   993 30850   549883     3563      53        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [30992]   993 30992   548864     2843      49        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [31549]     0 31549     1058       17       7        0             0 runsv
8月 27 21:16:22 op-k8s-n003 kernel: [31599]     0 31599     1094       23       8        0             0 svlogd
8月 27 21:16:22 op-k8s-n003 kernel: [31790]     0 31790    26922      957       9        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [31810]     0 31810     3801       79      12        0           934 sh
8月 27 21:16:22 op-k8s-n003 kernel: [31853]     0 31853  1613925   145425     563        0           934 java
8月 27 21:16:22 op-k8s-n003 kernel: [31854]     0 31854     1942       28       8        0           934 tailf
8月 27 21:16:22 op-k8s-n003 kernel: [31986]   995 31986   115851      920      30        0             0 gitaly-wrapper
8月 27 21:16:22 op-k8s-n003 kernel: [31993]   995 31993   285391     3636      63        0             0 gitaly
8月 27 21:16:22 op-k8s-n003 kernel: [32012]   995 32012   171359     2568      47        0             0 gitlab-workhors
8月 27 21:16:22 op-k8s-n003 kernel: [32027]   992 32027    28970     1778      17        0             0 node_exporter
8月 27 21:16:22 op-k8s-n003 kernel: [32037]   995 32037   734525    14067     212        0             0 ruby
8月 27 21:16:22 op-k8s-n003 kernel: [32040]   995 32040   734529    14076     213        0             0 ruby
8月 27 21:16:22 op-k8s-n003 kernel: [32057]   995 32057   119648     7288      82        0             0 gitlab-mon
8月 27 21:16:22 op-k8s-n003 kernel: [32065]   994 32065   180610     1930      41        0             0 redis_exporter
8月 27 21:16:22 op-k8s-n003 kernel: [32082]   992 32082   335606    17466     103        0             0 prometheus
8月 27 21:16:22 op-k8s-n003 kernel: [32156]   992 32156   171826     4132      52        0             0 alertmanager
8月 27 21:16:22 op-k8s-n003 kernel: [32233]   993 32233   105076     1689      34        0             0 postgres_export
8月 27 21:16:22 op-k8s-n003 kernel: [32241]   993 32241   548957     1858      42        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [32253]   992 32253   226119     5663      63        0             0 grafana-server
8月 27 21:16:22 op-k8s-n003 kernel: [32392]   993 32392   549785     2825      49        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [32393]   993 32393   548718     1556      45        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [32394]   993 32394   549010     2193      47        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [32419]   993 32419   549174     3062      48        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [  923]   993   923   549915     3552      54        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [ 1042]   993  1042   549723     3209      51        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [ 1056]   993  1056   548858     2704      48        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [ 1070]   993  1070   549284     3426      49        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [ 1219]   993  1219   548985     3119      51        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [ 1839]   993  1839   548804     1525      46        0             0 postgres
8月 27 21:16:22 op-k8s-n003 kernel: [ 1900]     0  1900    26906      932       9        0          -999 containerd-shim
8月 27 21:16:22 op-k8s-n003 kernel: [ 1929]     0  1929     3801       79      11        0           934 sh
8月 27 21:16:22 op-k8s-n003 kernel: [ 1968]     0  1968   940856   119046     334        0           934 java
8月 27 21:16:22 op-k8s-n003 kernel: [ 1969]     0  1969     1942       27       9        0           934 tailf
8月 27 21:16:22 op-k8s-n003 kernel: [ 2516]   995  2516     7444       92      18        0             0 sh
8月 27 21:16:22 op-k8s-n003 kernel: [ 2518]   995  2518     1091       19       8        0             0 sleep
8月 27 21:16:22 op-k8s-n003 kernel: Out of memory: Kill process 27274 (java) score 1039 or sacrifice child
8月 27 21:16:22 op-k8s-n003 kernel: Killed process 27274 (java), UID 0, total-vm:6531884kB, anon-rss:838392kB, file-rss:0kB, shmem-rss:0kB
8月 27 21:16:58 op-k8s-n003 dockerd[1061]: time="2020-08-27T21:16:58.688498424+08:00" level=info msg="Container 01c685517c7ca21ce7b94abc4abaa16f6eb02c6b9b57f971f36e8b29d64789c3 f
8月 27 21:16:59 op-k8s-n003 containerd[1058]: time="2020-08-27T21:16:59.210476038+08:00" level=info msg="shim reaped" id=01c685517c7ca21ce7b94abc4abaa16f6eb02c6b9b57f971f36e8b29d
8月 27 21:16:59 op-k8s-n003 dockerd[1061]: time="2020-08-27T21:16:59.221181547+08:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type
8月 27 21:17:00 op-k8s-n003 containerd[1058]: time="2020-08-27T21:17:00.007318010+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/82a4d737238e
8月 27 21:18:01 op-k8s-n003 kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=934
8月 27 21:18:01 op-k8s-n003 kernel: java cpuset=1b55d3e3a89d5b088f490225b63d58cc45faef8305cd7f2e161176d816f64ecf mems_allowed=0

最后发现自己在k8s机子上安装了一个测试的gitlab，导致内存被吃大半，pod里的jav进程一直被kill，导致pod重启。

2 当宿主机的内存很充足的时候 还会出现oom的情况。

原因是Kubernetes 对内存资源的限制实际上是通过 cgroup 来控制的，cgroup 是容器的一组用来控制内核如何运行进程的相关属性集合。针对内存、CPU 和各种设备都有对应的 cgroup。cgroup 是具有层级的，这意味着每个 cgroup 拥有一个它可以继承属性的父亲，往上一直直到系统启动时创建的 root cgroup。关于其背后的原理可以参考：深入理解Kubernetes资源限制：内存。
这种情况在journalctl 似乎是看不到的。

解决: 把limit 内存2048提高到4096

参考博客 自己也做了如下实验。看网上博客基本一致啊。

https://cloud.tencent.com/developer/article/1495508

1 通过 kubectl 创建一个 Pod，内存限制为 123Mi。

# 创建pod
kubectl run --restart=Never --rm -it --image=ubuntu --limits='memory=123Mi' -- sh

2 重新打开一个 shell 窗口，找出刚创建的 Pod 的 uid：

# sh 是pod的name
kubectl get pods sh -o yaml | grep uid

3 查看pod所在的node 查看资源限制

# master查看pod所在的node
kubectl get pods -o wide

# node上操作
cd /sys/fs/cgroup/memory/kubepods/burstable/pod4fda8057-eb2e-11ea-bdcd-005056b7c84c/
cat memory.limit_in_bytes

其中 memory.limit_in_bytes 表示当前限制的内存额度。128974848 正好等于 123*1024*1024。

如果你查看一下这个 Pod 的 cgroup 目录，就会发现 Pod 中的每个容器都会在该目录下创建一个子 cgroup 目录：

ll /sys/fs/cgroup/memory/kubepods/burstable/pod4fda8057-eb2e-11ea-bdcd-005056b7c84c/

输出结果的前两个目录其实就是 pause 容器的 pause 进程和业务容器的 bash 进程创建的两个子 cgroup 目录。可以来证实一下，我的环境使用的容器运行时是 containerd，可以通过 crictl 工具来查看，如果你使用的是 docker，方法类似。

4 查看容器的id 并以此查看容器的pid

docker ps -a  |grep sh |grep ubuntu
docker inspect ccb356fb4731

5 查看该进程所属的 cgroup，即进程在 cgroup 树中的路径

6 进入该目录，查看内存限制

可以看到该 cgroup 的内存限制和父 cgroup 一样，而父 cgroup 其实就是 Pod 级别的 cgroup。

按照预想，一旦 Pod 消耗的内存资源超过这个限制，cgroup 就会杀死容器进程，我们来测试一下。

7. 压力测试

先在容器中安装压力测试工具：

apt update; apt install -y stress

在另一个一个 shell 窗口中执行 dmesg -Tw 命令查看系统的 Syslog。

kubectl exec -it sh /bin/bash
dmesg -Tw

回到第一个 shell 窗口进行压力测试，限制内存在 100M 以内：

stress --vm 1 --vm-bytes 100M &

执行第二次压力测试：

stress --vm 1 --vm-bytes 50M

可以看到系统通过发送 signal 9（SIGKILL） 信号杀死了第一次压力测试的进程（进程 ID 为 265）。

可以在另一个 shell 窗口中看到系统的 syslog 日志输出：

从宿主机的视角来看，PID 为 17453 的进程被 oom-killed 了，我们需要重点关注红色及上面部分日志输出：

对于刚刚创建的 Pod 而言，有好几个进程作为 OOM killer 的候选人，其中最重要的进程是 pause，用来为业务容器创建共享的 network namespace，其 oom_score_adj 值为 -998，可以确保不被杀死。oom_score_adj 值越低就越不容易被杀死。关于 Pod 的 QoS 与 OOM 值的对应关系，可以参考：Kubernetes 资源管理概述。

除了 pause 进程外，剩下的进程 oom_score_adj 值均为 993，我们可以根据 Kubernetes 官方文档中公式来验证一下：

# 其中 memoryRequest 是 pod 申请的资源，memoryCapacity 是节点的内存总量。可以看到，申请的内存越多，oom 值越低，也就越不容易被杀死。
min(max(2, 1000 - (1000 * memoryRequestBytes) / machineMemoryCapacityBytes), 999)

进程的 oom_score_adj 值可以通过以下命令来查看：

 cat /proc/17453/oom_score_adj

查看运行该 Pod 的节点内存总量：

kubectl describe nodes k3s | grep Allocatable -A 5

如果只设置了 limits，Kubernetes 会自动把 Pod 的 requests 设置成和 limits 一样。所以其他进程的 oom_score_adj 值为 1000–123*1024/2041888=938.32，这个值已经很接近 syslog 中输出的 939 了。

OOM killer 会根据进程的内存使用情况来计算 oom_score 的值，并根据 oom_score_adj 的值来进行微调。

进程的 oom_score 值可以通过以下命令来查看：

cat /proc/17453/oom_score

因为业务容器内所有进程的 oom_score_adj 值都相同，所以谁的内存使用量最多，oom_score 值就越高，也就越容易被杀死。因为第一个 stress 进程使的内存使用量最多（100M），oom_score 值最高（值为 1718），所以被杀死。

3. 总结

---

Kubernetes 通过 cgroup 和 OOM killer 来限制 Pod 的内存资源，在实际使用中我们需要小心区分 OS 级别的 OOM 和 Pod 级别的 OOM。