gitlab 启动报错:

`==> /var/log/gitlab/prometheus/current <==
2023-12-07_05:10:36.23823 level=error ts=2023-12-07T05:10:36.238Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:335: Failed to list *v1.Node: nodes is forbidden: User \"system:serviceaccount:auser:default\" cannot list resource \"nodes\" in API group \"\" at the cluster scope"
2023-12-07_05:10:36.23845 level=error ts=2023-12-07T05:10:36.238Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:283: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:auser:default\" cannot list resource \"pods\" in API group \"\" at the cluster scope"

原因:如果 GitLab 服务器在 Kubernetes 中运行,则 Prometheus 将从群集中的节点和带注释的 Pod 收集指标,包括每个容器上的性能数据.。
但是 gitlab运行在 自定义namespace中,默认没权限访问k8s的资源,导致内置的prometheus获取k8s信息报错。

解决:关闭gitlab里内置prometheus监控k8s的功能
方式一:修改 /etc/gitlab/gitlab.rb
在这里插入图片描述

方式二:通过添加环境变量关闭

env:
- name: GITLAB_OMNIBUS_CONFIG
  value: "prometheus['monitor_kubernetes'] = false"

方式三:当然也可以给 prometheus 放开 K8S 的RBAC权限。(因为这是GitLab内置的Prometheus,实际上k8s超过了其监控范围,不建议这么配。如果是独立Prometheus比较合适)
参考:https://github.com/prometheus-operator/prometheus-operator/issues/2155

参考:
https://gitlab.com/gitlab-org/gitlab/-/issues/34405
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template

Logo

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐