k8s docker memory limits oom分析、docker status MEM USAGE采样来源分析

k8s内存限制配置spec:template:spec:containers:- name: xxximage: xxxresources:limits:memory: 2Gi等同docker rundocker run -m 2G xxx准备分析环境docker run --rm -it -m 2G centos:7 /b

hkNaruto

1768人浏览 · 2022-02-11 15:06:54

hkNaruto · 2022-02-11 15:06:54 发布

k8s内存限制配置

spec:  
  template:    
    spec:
      containers:
        - name: xxx
          image: xxx
          resources:
            limits:
              memory: 2Gi

等同docker run

docker run -m 2G xxx

准备分析环境

docker run --rm -it -m 2G centos:7 /bin/bash

容器内运行

[root@ce32f3bdbef6 /]# yum install -y gcc

分析1：

采用malloc申请1G内存，查看docker status状态

a.c源码

#include<stdio.h>
#include<stdlib.h>
#include<unistd.h>

int main(){
        void *p = malloc(1024*1024*1024);
        char c = getchar();
        free(p);
        return 0;
}

编译运行

gcc a.c
./a.out

此时查看docker status

容器内查看ps aux

其中虚拟内存VSZ=1052812，实际物理内存RSS=676

容器内查看pmap

可以看到malloc申请的1G内存，总内存1G多点。

分析2：malloc后采用memset给内存赋值

a.c

#include<stdio.h>
#include<stdlib.h>
#include<unistd.h>

int main(){
        void *p = malloc(1024*1024*1024);
        memset(p, 1, 1024*1024*1024);
        char c = getchar();
        free(p);
        return 0;
}

重新编译，运行，此时docker status

ps aux

pmap

经上述分析，可以得出

1. docker status字段MEM USAGE对应容器内进程物理内存使用量RSS汇总值。

2. malloc申请的内存在没有使用是，只占用VSZ虚拟内存容量，使用该内存（写入数据）时，才占用RSS物理内存

分析3：触发oom

a.c

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main()
{
    int counter = 0;
    while (1)
    {
        void *p = malloc(1024 * 1024 * 1024);
        memset(p, 1, 1024 * 1024 * 1024);
        printf("%d GB\n", ++counter);
    }

    return 0;
}

编译，重新运行a.c

[root@ce32f3bdbef6 /]# ./a.out 
1 GB
2 GB
3 GB
Killed

注意程序内内存使用超过3GB RSS才超过2GB（原因还不清楚）并触发oom

tail -f /var/log/message

2022-02-11 15:03:01 yeqiang-PC kernel: [24965.815627] traps: tsar[30709] trap divide error ip:7fcb3587f4a8 sp:7ffcf925f4e0 error:0 in mod_partition.so[7fcb3587f000+1000]
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074747] a.out invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074749] CPU: 4 PID: 30718 Comm: a.out Tainted: P           OE     5.10.60-amd64-desktop #3
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074750] Hardware name: Micro-Star International Co., Ltd. MS-7B23/B360M MORTAR (MS-7B23), BIOS 1.60 03/25/2019
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074751] Call Trace:
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074755]  dump_stack+0x74/0x92
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074756]  dump_header+0x4f/0x1ed
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074757]  oom_kill_process.cold.36+0xb/0x10
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074759]  out_of_memory+0x1bf/0x510
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074761]  mem_cgroup_out_of_memory+0xed/0x100
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074762]  try_charge+0x696/0x6d0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074764]  ? mem_cgroup_charge+0xdf/0x270
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074765]  mem_cgroup_charge+0x84/0x270
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074766]  __add_to_page_cache_locked+0x3a5/0x420
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074767]  ? scan_shadow_nodes+0x30/0x30
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074768]  add_to_page_cache_lru+0x4f/0xd0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074769]  pagecache_get_page+0x133/0x330
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074770]  filemap_fault+0x669/0xb40
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074771]  ? try_charge+0xd2/0x6d0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074773]  ? filemap_map_pages+0x2c0/0x420
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074775]  ext4_filemap_fault+0x31/0x50
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074777]  __do_fault+0x3c/0xe0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074778]  handle_mm_fault+0x1367/0x17e0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074780]  exc_page_fault+0x285/0x510
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074782]  ? asm_exc_page_fault+0x8/0x30
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074783]  asm_exc_page_fault+0x1e/0x30
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074789] RIP: 0033:0x7f9e62350aa0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074791] Code: Unable to access opcode bytes at RIP 0x7f9e62350a76.
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074792] RSP: 002b:00007ffe34dcdf78 EFLAGS: 00010206
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074793] RAX: 00007f9d622bd010 RBX: 0000000000000000 RCX: 00007f9da17ed000
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074793] RDX: 00007f9da22bd000 RSI: 0000000000000001 RDI: 00007f9d622bd010
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074794] RBP: 00007ffe34dcdf90 R08: ffffffffffffffff R09: 0000000040000000
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074794] R10: 0000000000000022 R11: 0000000000001000 R12: 00000000004004e0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074795] R13: 00007ffe34dce070 R14: 0000000000000000 R15: 0000000000000000
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074797] memory: usage 2097152kB, limit 2097152kB, failcnt 2553944
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074797] memory+swap: usage 4194188kB, limit 4194304kB, failcnt 629
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074798] kmem: usage 8880kB, limit 9007199254740988kB, failcnt 0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074799] Memory cgroup stats for /docker/ce32f3bdbef6af8c0cf6ecc3ac9d9e2da2214dd925865e366ab401ebfd0a41be:
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] anon 2132815872
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] file 0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] kernel_stack 49152
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] percpu 0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] sock 0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] shmem 0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] file_mapped 0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] file_dirty 135168
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] file_writeback 4730880
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] anon_thp 0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] inactive_anon 2138222592
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] active_anon 0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] inactive_file 0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] active_file 57344
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] unevictable 0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] slab_reclaimable 114824
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] slab_unreclaimable 258200
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] slab 373024
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] workingset_refault_anon 1749
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] workingset_refault_file 20328
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] workingset_activate_anon 0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] workingset_activate_file 6897
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] workingset_restore_anon 0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] workingset_restore_file 6270
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] workingset_nodereclaim 0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] pgfault 18537552
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074807] pgmajfault 3696
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074808] Tasks state (memory values in pages):
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074808] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074810] [  29719]     0 29719     2961        0    69632      117             0 bash
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074811] [  13734]     0 13734     2961        0    65536      122             0 bash
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074812] [  29018]     0 29018     2961        2    65536       99             0 bash
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074813] [  30718]     0 30718  1049638   520654  8433664   525144             0 a.out
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.074814] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ce32f3bdbef6af8c0cf6ecc3ac9d9e2da2214dd925865e366ab401ebfd0a41be,mems_allowed=0,oom_memcg=/docker/ce32f3bdbef6af8c0cf6ecc3ac9d9e2da2214dd925865e366ab401ebfd0a41be,task_memcg=/docker/ce32f3bdbef6af8c0cf6ecc3ac9d9e2da2214dd925865e366ab401ebfd0a41be,task=a.out,pid=30718,uid=0
2022-02-11 15:03:10 yeqiang-PC kernel: [24975.201510] oom_reaper: reaped process 30718 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub