gitlab arm64 FT2000+ 64核服务器k8s平台部署后，内存消耗过大，报错“PumaWorkerKiller: Out of memory.
错误日志==> /var/log/gitlab/puma/puma_stdout.log <=={"timestamp":"2022-02-14T06:42:53.064Z","pid":18068,"message":"PumaWorkerKiller: Out of memory. 64 workers consuming total: 79653.6875 mb out of m
·
错误日志
==> /var/log/gitlab/puma/puma_stdout.log <==
{"timestamp":"2022-02-14T06:42:53.064Z","pid":18068,"message":"PumaWorkerKiller: Out of memory. 64 workers consuming total: 79653.6875 mb out of max: 65009.28 mb. Sending TERM to pid 20916 consuming 1227.6875 mb."}
CPU数量
# lscpu
Architecture: aarch64
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 16
NUMA node(s): 8
Model: 2
BogoMIPS: 100.00
NUMA node0 CPU(s): 0-7
NUMA node1 CPU(s): 8-15
NUMA node2 CPU(s): 16-23
NUMA node3 CPU(s): 24-31
NUMA node4 CPU(s): 32-39
NUMA node5 CPU(s): 40-47
NUMA node6 CPU(s): 48-55
NUMA node7 CPU(s): 56-63
Flags: fp asimd evtstrm crc32 cpuid
容器内情况
root@gitlab-0:~# ps aux | grep puma
root 523 0.0 0.0 2240 896 ? Ss 14:16 0:00 runsv puma
root 535 0.0 0.0 2432 896 ? S 14:16 0:00 svlogd -tt /var/log/gitlab/puma
root 3528 0.0 0.0 4992 896 ? S 14:18 0:00 tail --follow=name --retry /var/log/gitlab/sshd/current /var/log/gitlab/gitlab-rails/production.log /var/log/gitlab/gitlab-rails/grpc.log /var/log/gitlab/gitlab-rails/service_measurement.log /var/log/gitlab/gitlab-rails/application.log /var/log/gitlab/gitlab-rails/application_json.log /var/log/gitlab/gitlab-rails/auth.log /var/log/gitlab/gitlab-rails/sidekiq_client.log /var/log/gitlab/gitlab-rails/production_json.log /var/log/gitlab/gitlab-rails/api_json.log /var/log/gitlab/redis/current /var/log/gitlab/gitaly/current /var/log/gitlab/postgresql/current /var/log/gitlab/logrotate/current /var/log/gitlab/puma/current /var/log/gitlab/puma/puma_stdout.log /var/log/gitlab/puma/puma_stderr.log /var/log/gitlab/sidekiq/current /var/log/gitlab/gitlab-workhorse/current /var/log/gitlab/nginx/current /var/log/gitlab/nginx/error.log /var/log/gitlab/nginx/gitlab_access.log /var/log/gitlab/nginx/gitlab_error.log /var/log/gitlab/nginx/access.log /var/log/gitlab/gitlab-exporter/current /var/log/gitlab/redis-exporter/current /var/log/gitlab/prometheus/current /var/log/gitlab/alertmanager/current /var/log/gitlab/postgres-exporter/current /var/log/gitlab/grafana/current
git 18068 7.6 0.4 8089920 1274624 ? Ssl 14:28 1:30 puma 4.3.5.gitlab.3 (unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket,tcp://127.0.0.1:8080) [gitlab-puma-worker]
git 18273 0.0 0.4 16495424 1254080 ? Sl 14:29 0:00 puma: cluster worker 1: 18068 [gitlab-puma-worker] orker]
git 18275 0.0 0.4 16495424 1254016 ? Sl 14:29 0:00 puma: cluster worker 2: 18068 [gitlab-puma-worker] orker]
git 18279 0.0 0.4 16495424 1254272 ? Sl 14:29 0:00 puma: cluster worker 4: 18068 [gitlab-puma-worker] orker]
git 18281 0.0 0.4 16495424 1254208 ? Sl 14:29 0:00 puma: cluster worker 5: 18068 [gitlab-puma-worker] orker]
git 18283 0.0 0.4 16495424 1254144 ? Sl 14:29 0:00 puma: cluster worker 6: 18068 [gitlab-puma-worker] orker]
git 18285 0.0 0.4 16495424 1254272 ? Sl 14:29 0:00 puma: cluster worker 7: 18068 [gitlab-puma-worker] orker]
git 18293 0.0 0.4 16495424 1254272 ? Sl 14:29 0:00 puma: cluster worker 9: 18068 [gitlab-puma-worker] orker]
git 18311 0.0 0.4 16495424 1254464 ? Sl 14:29 0:00 puma: cluster worker 18: 18068 [gitlab-puma-worker] orker]
git 18312 0.0 0.4 16495424 1254464 ? Sl 14:29 0:00 puma: cluster worker 19: 18068 [gitlab-puma-worker] orker]
git 18315 0.0 0.4 16495424 1254336 ? Sl 14:29 0:00 puma: cluster worker 20: 18068 [gitlab-puma-worker] orker]
git 18316 0.0 0.4 16495424 1254144 ? Sl 14:29 0:00 puma: cluster worker 21: 18068 [gitlab-puma-worker] orker]
git 18319 0.0 0.4 16495424 1254080 ? Sl 14:29 0:00 puma: cluster worker 22: 18068 [gitlab-puma-worker] orker]
git 18325 0.0 0.4 16495424 1253760 ? Sl 14:29 0:00 puma: cluster worker 25: 18068 [gitlab-puma-worker] orker]
git 18326 0.0 0.4 16495424 1253760 ? Sl 14:29 0:00 puma: cluster worker 26: 18068 [gitlab-puma-worker] orker]
git 18332 0.0 0.4 16495424 1253760 ? Sl 14:29 0:00 puma: cluster worker 29: 18068 [gitlab-puma-worker] orker]
git 18334 0.0 0.4 16495424 1253952 ? Sl 14:29 0:00 puma: cluster worker 30: 18068 [gitlab-puma-worker] orker]
git 18336 0.0 0.4 16495424 1254208 ? Sl 14:29 0:00 puma: cluster worker 31: 18068 [gitlab-puma-worker] orker]
git 18338 0.0 0.4 16495424 1253952 ? Sl 14:29 0:00 puma: cluster worker 32: 18068 [gitlab-puma-worker] orker]
git 18344 0.0 0.4 16495424 1253760 ? Sl 14:29 0:00 puma: cluster worker 35: 18068 [gitlab-puma-worker] orker]
git 18348 0.0 0.4 16495424 1253760 ? Sl 14:29 0:00 puma: cluster worker 37: 18068 [gitlab-puma-worker] orker]
git 18349 0.0 0.4 16495424 1253504 ? Sl 14:29 0:00 puma: cluster worker 38: 18068 [gitlab-puma-worker] orker]
git 18352 0.0 0.4 16495424 1253760 ? Sl 14:29 0:00 puma: cluster worker 39: 18068 [gitlab-puma-worker] orker]
git 18353 0.0 0.4 16495424 1253632 ? Sl 14:29 0:00 puma: cluster worker 40: 18068 [gitlab-puma-worker] orker]
git 18359 0.0 0.4 16495424 1253824 ? Sl 14:29 0:00 puma: cluster worker 43: 18068 [gitlab-puma-worker] orker]
git 18362 0.0 0.4 16495424 1253632 ? Sl 14:29 0:00 puma: cluster worker 44: 18068 [gitlab-puma-worker] orker]
git 18368 0.0 0.4 16495424 1254016 ? Sl 14:29 0:00 puma: cluster worker 47: 18068 [gitlab-puma-worker] orker]
git 18371 0.0 0.4 16495424 1253760 ? Sl 14:29 0:00 puma: cluster worker 49: 18068 [gitlab-puma-worker] orker]
git 18373 0.0 0.4 16495424 1253696 ? Sl 14:29 0:00 puma: cluster worker 50: 18068 [gitlab-puma-worker] orker]
git 18375 0.0 0.4 16495424 1253632 ? Sl 14:29 0:00 puma: cluster worker 51: 18068 [gitlab-puma-worker] orker]
git 18378 0.0 0.4 16495424 1253568 ? Sl 14:29 0:00 puma: cluster worker 53: 18068 [gitlab-puma-worker] orker]
git 18380 0.0 0.4 16495424 1253504 ? Sl 14:29 0:00 puma: cluster worker 54: 18068 [gitlab-puma-worker] orker]
git 18382 0.0 0.4 16495424 1253312 ? Sl 14:29 0:00 puma: cluster worker 55: 18068 [gitlab-puma-worker] orker]
git 18387 0.0 0.4 16495424 1253696 ? Sl 14:29 0:00 puma: cluster worker 57: 18068 [gitlab-puma-worker] orker]
git 18388 0.0 0.4 16495424 1253952 ? Sl 14:29 0:00 puma: cluster worker 58: 18068 [gitlab-puma-worker] orker]
git 18389 0.0 0.4 16495424 1253952 ? Sl 14:29 0:00 puma: cluster worker 59: 18068 [gitlab-puma-worker] orker]
git 18392 0.0 0.4 16495424 1253888 ? Sl 14:29 0:00 puma: cluster worker 60: 18068 [gitlab-puma-worker] orker]
git 18396 0.0 0.4 16495424 1253824 ? Sl 14:29 0:00 puma: cluster worker 61: 18068 [gitlab-puma-worker] orker]
git 18397 0.0 0.4 16495424 1253952 ? Sl 14:29 0:00 puma: cluster worker 62: 18068 [gitlab-puma-worker] orker]
git 18399 0.0 0.4 16495424 1254144 ? Sl 14:29 0:00 puma: cluster worker 63: 18068 [gitlab-puma-worker] orker]
git 19306 0.0 0.4 16495424 1254592 ? Sl 14:29 0:00 puma: cluster worker 3: 18068 [gitlab-puma-worker] orker]
git 19351 0.0 0.4 16495424 1253888 ? Sl 14:30 0:00 puma: cluster worker 0: 18068 [gitlab-puma-worker] orker]
git 19489 0.0 0.4 16495424 1255104 ? Sl 14:31 0:00 puma: cluster worker 13: 18068 [gitlab-puma-worker] orker]
git 19757 0.0 0.4 16495424 1256384 ? Sl 14:33 0:00 puma: cluster worker 12: 18068 [gitlab-puma-worker] orker]
git 20009 0.0 0.4 16495424 1255232 ? Sl 14:33 0:00 puma: cluster worker 45: 18068 [gitlab-puma-worker] orker]
git 20101 0.0 0.4 16495424 1254464 ? Sl 14:34 0:00 puma: cluster worker 36: 18068 [gitlab-puma-worker] orker]
git 20163 0.0 0.4 16495424 1255808 ? Sl 14:34 0:00 puma: cluster worker 24: 18068 [gitlab-puma-worker] orker]
git 20209 0.0 0.4 16495424 1255744 ? Sl 14:34 0:00 puma: cluster worker 27: 18068 [gitlab-puma-worker] orker]
git 20254 0.0 0.4 16495424 1255360 ? Sl 14:35 0:00 puma: cluster worker 11: 18068 [gitlab-puma-worker] orker]
git 20295 0.0 0.4 16495424 1255232 ? Sl 14:35 0:00 puma: cluster worker 34: 18068 [gitlab-puma-worker] orker]
git 20342 0.0 0.4 16495424 1255552 ? Sl 14:35 0:00 puma: cluster worker 46: 18068 [gitlab-puma-worker] orker]
git 20388 0.0 0.4 16495424 1256064 ? Sl 14:36 0:00 puma: cluster worker 17: 18068 [gitlab-puma-worker] orker]
git 20650 0.0 0.4 16495424 1256320 ? Sl 14:38 0:00 puma: cluster worker 28: 18068 [gitlab-puma-worker] orker]
git 20742 0.0 0.4 16495424 1256192 ? Sl 14:38 0:00 puma: cluster worker 41: 18068 [gitlab-puma-worker] orker]
git 20783 0.0 0.4 16495424 1255808 ? Sl 14:39 0:00 puma: cluster worker 33: 18068 [gitlab-puma-worker] orker]
git 20828 0.1 0.4 16495424 1256384 ? Sl 14:39 0:00 puma: cluster worker 10: 18068 [gitlab-puma-worker] orker]
git 21316 0.1 0.4 16495424 1256896 ? Sl 14:43 0:00 puma: cluster worker 56: 18068 [gitlab-puma-worker] orker]
git 21405 0.1 0.4 16495424 1256256 ? Sl 14:43 0:00 puma: cluster worker 42: 18068 [gitlab-puma-worker] orker]
git 21449 0.1 0.4 16495424 1256256 ? Sl 14:44 0:00 puma: cluster worker 52: 18068 [gitlab-puma-worker] orker]
git 21491 0.2 0.4 16495424 1256256 ? Sl 14:44 0:00 puma: cluster worker 14: 18068 [gitlab-puma-worker] orker]
git 21716 0.4 0.4 16495424 1256576 ? Sl 14:46 0:00 puma: cluster worker 23: 18068 [gitlab-puma-worker] orker]
git 21758 0.5 0.4 16495424 1255296 ? Sl 14:46 0:00 puma: cluster worker 48: 18068 [gitlab-puma-worker] orker]
git 21805 0.8 0.4 16495424 1256000 ? Sl 14:46 0:00 puma: cluster worker 15: 18068 [gitlab-puma-worker] orker]
git 21845 1.4 0.4 16495424 1255808 ? Sl 14:47 0:00 puma: cluster worker 8: 18068 [gitlab-puma-worker] orker]
git 21891 8.5 0.4 16495424 1255616 ? Sl 14:47 0:00 puma: cluster worker 16: 18068 [gitlab-puma-worker] orker]解决方案：修改/etc/gitlab/gitlab.rb
puma['worker_processes'] = 4
puma['per_worker_max_memory_mb'] = 2048执行
gitlabctl reconfigure效果
==> /var/log/gitlab/puma/puma_stdout.log <==
{"timestamp":"2022-02-14T07:41:49.447Z","pid":27809,"message":"PumaWorkerKiller: Consuming 6428.625 mb with master and 4 workers."}
K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容
更多推荐
0
0- 0
已为社区贡献18条内容
所有评论(0)