gitlab prometheus Failed to list *v1.Node: nodes is forbidden: User \“system:serviceaccount:default:
原因:如果 GitLab 服务器在 Kubernetes 中运行,则 Prometheus 将从群集中的节点和带注释的 Pod 收集指标,包括每个容器上的性能数据.。解决:关闭gitlab里内置prometheus监控k8s的功能。方式一:修改 /etc/gitlab/gitlab.rb。方式二:通过添加环境变量关闭。
gitlab 启动报错:
`==> /var/log/gitlab/prometheus/current <==
2023-12-07_05:10:36.23823 level=error ts=2023-12-07T05:10:36.238Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:335: Failed to list *v1.Node: nodes is forbidden: User \"system:serviceaccount:auser:default\" cannot list resource \"nodes\" in API group \"\" at the cluster scope"
2023-12-07_05:10:36.23845 level=error ts=2023-12-07T05:10:36.238Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:283: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:auser:default\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
原因:如果 GitLab 服务器在 Kubernetes 中运行,则 Prometheus 将从群集中的节点和带注释的 Pod 收集指标,包括每个容器上的性能数据.。
但是 gitlab运行在 自定义namespace中,默认没权限访问k8s的资源,导致内置的prometheus获取k8s信息报错。
解决:关闭gitlab里内置prometheus监控k8s的功能
方式一:修改 /etc/gitlab/gitlab.rb
方式二:通过添加环境变量关闭
env:
- name: GITLAB_OMNIBUS_CONFIG
value: "prometheus['monitor_kubernetes'] = false"
方式三:当然也可以给 prometheus 放开 K8S 的RBAC权限。(因为这是GitLab内置的Prometheus,实际上k8s超过了其监控范围,不建议这么配。如果是独立Prometheus比较合适)
参考:https://github.com/prometheus-operator/prometheus-operator/issues/2155
参考:
https://gitlab.com/gitlab-org/gitlab/-/issues/34405
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template
更多推荐
所有评论(0)