提示:文章写完后,目录可以自动生成,如何生成可参考右边的帮助文档


前言

  • 问题描述:docker容器采用alpine作为基础镜像,部署netty应用,在运行一段时间后出现cpu达到120-180%

一、检测步骤

1.查看容器状态

docker stats
CONTAINER ID        NAME       CPU %       MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
e29132fdbac5        netty      130.92%      235.3MiB / 500MiB     47.05%              0B / 0B             1.79MB / 0B         77

2.进入容器

docker exec -it netty /bin/bash

3.获取程序进程ID

top

4.获取进程ID下的线程cpu消耗情况

ps -mp pid -o THREAD,tid,time | sort -rn

出现问题:
ps: unrecognized option: m
BusyBox v1.29.3 (2019-01-24 07:45:07 UTC) multi-call binary.
Usage: ps [-o COL1,COL2=HEADER]
Show list of processes -o COL1,COL2=HEADER Select columns for display
解决:
alpine系统版本比较低,busybox用的是阉割版,很多命令无法提供,最好是升级alpine到v3.12,然后
安装apk add --no-cache procps即可

5.获取cpu消耗最高的线程ID

printf "%x\n" tid

6.通过jdk的工具查看进程的线程栈信息

jstack pid > t.log

出现问题:
/ # jstack 1 1:
Unable to open socket file: target process not responding or HotSpot VM not loaded
解决:
docker启动命令中添加–init参数
docker run -d --init xxx

7.通过线程ID查看线程栈信息

"worker-3-1" #54 prio=5 os_prio=0 tid=0x0000556e2c044000 nid=0x3d runnable [0x00007febf2f6d000]                              
   java.lang.Thread.State: RUNNABLE                                                                                          
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)                                                            
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)                                                         
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)                                              
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)                                                    
        - locked <0x00000000fbbdbf68> (a io.netty.channel.nio.SelectedSelectionKeySet)                                            
        - locked <0x00000000fbbea820> (a java.util.Collections$UnmodifiableSet)                                              
        - locked <0x00000000fbbdbf20> (a sun.nio.ch.EPollSelectorImpl)                                                      
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)                                                                  
        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)         
        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:814)                                                  
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)                                                           
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)                       
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)                                        
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                                 
        at java.lang.Thread.run(Thread.java:748)

在线程栈信息中并没有发现业务代码,发现都是netty底层运行信息,查看官方,升级netty到最新版本4.1.54,观察几天问题解决

Logo

权威|前沿|技术|干货|国内首个API全生命周期开发者社区

更多推荐