devOps系列（七）grafana+prometheus监控告警
作者目前打算分享一期关于devOps系列的文章，希望对热爱学习和探索的你有所帮助。文章主要记录一些简洁、高效的运维部署指令，旨在和能够。就像运维文档或者手册一样，方便进行系统的重建、改造和优化。每篇文章独立出来，可以单独作为其中一项组件的部署和使用。本章为。
·
前言
作者目前打算分享一期关于devOps系列的文章，希望对热爱学习和探索的你有所帮助。
文章主要记录一些简洁、高效的运维部署指令，旨在 记录和能够快速地构建系统。就像运维文档或者手册一样，方便进行系统的重建、改造和优化。每篇文章独立出来，可以单独作为其中一项组件的部署和使用。
本章为 devOps系列（七）grafana+prometheus监控告警
大纲
devOps系列（六）grafana+prometheus搭建
devOps系列（七）grafana+prometheus监控告警
devOps系列（八）efk+prometheus+grafana日志监控和告警
使用 prometheus + blackbox-exporter + alertmanager 做http的接口监控和告警
安装blackbox-exporter
docker run --restart=always -d --name blackbox-exporter -p 9115:9115 prom/blackbox-exporter
好了之后 http://localhost:9115可以访问查看
修改prometheus.yml
rule_files:
- "blackbox_rules.yml"
scrape_configs:
- job_name: 'blackbox-http'
metrics_path: /probe
params:
module: [http_2xx]
file_sd_configs:
- files: ['http_check.yml']
refresh_interval: 10s
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 192.168.20.2:9115 #blackbox-exporter 所在的机器和端口
rule_files 下面添加blackbox_rules.yml
scrape_configs下面添加job
http_check.yml中添加检查接口
vi blackbox_rules.yml
groups:
- name: 服务探测
rules:
- alert: BlackboxProbeFailed
expr: probe_success == 0
for: 0m
labels:
severity: critical
team: node
annotations:
summary: Blackbox probe failed (instance {{ $labels.instance }})
description: "服务在线检查失败\n当前值= {{ $value }}\nIp = {{ $labels.ip }}\nDomain= {{ $labels.domain }}\nEnv= {{ $labels.env }}\n服务名= {{ $labels.service }}"
- alert: BlackboxProbeHttpFailure
expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400
for: 0m
labels:
severity: critical
team: node
annotations:
summary: Blackbox probe HTTP failure (instance {{ $labels.instance }})
description: "HTTP状态码不在200-399\n当前值= {{ $value }}\nIp = {{ $labels.ip }}\nDomain= {{ $labels.domain }}\nEnv= {{ $labels.env }}\n服务名= {{ $labels.service }}"
- alert: BlackboxSslCertificateWillExpireSoon
expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
for: 0m
labels:
severity: warning
annotations:
summary: Blackbox SSL certificate will expire soon (instance {{ $labels.instance }})
description: "SSL certificate expires in 30 days\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
添加接口检查配置
vi http_check.yml
- targets: [ 'https://www.jafir.top' ]
labels:
name: web
env: prod
service: api
重启prometheus，测试即可
如上示例：可以对 www.jafir.top进行检测，如果失败的话会进行相应的告警信息提示，最终通过prometheus和alertmanager来触发告警。也可以对一些重要的网站、接口等进行检测。
短信和电话 prometheus-alert全家桶里面支持
目前我们使用的aliyun的语音和短信模板 不支持code (code只提供给验证码用) 所以这边只有把prometheus-alert源码下载下来改一下即可
可以找一套linux服务器，clone源码 当前是4.9版本
git@github.com:feiyu563/PrometheusAlert.git
安装go
yum install go
目前修改地方为
PrometheusAlert/controllers/aliyun.go
添加了
TemplateKey := beego.AppConfig.DefaultString("ALY_DX_Template_Key","code")
TemplateKey := beego.AppConfig.DefaultString("ALY_DH_Template_Key","code")
全文件如下：
package controllers
import (
"PrometheusAlert/models"
"github.com/aliyun/alibaba-cloud-sdk-go/sdk/requests"
"github.com/aliyun/alibaba-cloud-sdk-go/services/dysmsapi"
"github.com/aliyun/alibaba-cloud-sdk-go/services/dyvmsapi"
"github.com/astaxie/beego"
"github.com/astaxie/beego/logs"
"strings"
)
func PostALYmessage(Messages, PhoneNumbers, logsign string) string {
open := beego.AppConfig.String("open-alydx")
if open != "1" {
logs.Info(logsign, "[alymessage]", "阿里云短信接口未配置未开启状态,请先配置open-alydx为1")
return "阿里云短信接口未配置未开启状态,请先配置open-alydx为1"
}
AccessKeyId := beego.AppConfig.String("ALY_DX_AccessKeyId")
AccessSecret := beego.AppConfig.String("ALY_DX_AccessSecret")
SignName := beego.AppConfig.String("ALY_DX_SignName")
Template := beego.AppConfig.String("ALY_DX_Template")
TemplateKey := beego.AppConfig.DefaultString("ALY_DX_Template_Key","code")
client, err := dysmsapi.NewClientWithAccessKey("cn-hangzhou", AccessKeyId, AccessSecret)
request := dysmsapi.CreateSendSmsRequest()
request.Scheme = "https"
request.PhoneNumbers = PhoneNumbers
request.SignName = SignName
request.TemplateCode = Template
request.TemplateParam = `{"`+TemplateKey+`":"` + Messages + `"}`
response, err := client.SendSms(request)
if err != nil {
logs.Error(logsign, "[alymessage]", err.Error())
}
logs.Info(logsign, "[alymessage]", response)
models.AlertToCounter.WithLabelValues("alydx").Add(1)
ChartsJson.Alydx += 1
return response.Message
}
func PostALYphonecall(Messages string, PhoneNumbers, logsign string) string {
open := beego.AppConfig.String("open-alydh")
if open != "1" {
logs.Info(logsign, "[alyphonecall]", "阿里云电话接口未配置未开启状态,请先配置open-alydh为1")
return "阿里云电话接口未配置未开启状态,请先配置open-alydh为1"
}
AccessKeyId := beego.AppConfig.String("ALY_DH_AccessKeyId")
AccessSecret := beego.AppConfig.String("ALY_DH_AccessSecret")
CalledShowNumber := beego.AppConfig.String("ALY_DX_CalledShowNumber")
TtsCode := beego.AppConfig.String("ALY_DH_TtsCode")
TemplateKey := beego.AppConfig.DefaultString("ALY_DH_Template_Key","code")
mobiles := strings.Split(PhoneNumbers, ",")
for _, m := range mobiles {
client, err := dyvmsapi.NewClientWithAccessKey("cn-hangzhou", AccessKeyId, AccessSecret)
request := dyvmsapi.CreateSingleCallByTtsRequest()
request.Scheme = "https"
request.CalledShowNumber = CalledShowNumber
request.CalledNumber = m
request.TtsCode = TtsCode
request.TtsParam = `{"`+TemplateKey+`":"` + Messages + `"}`
request.PlayTimes = requests.NewInteger(2)
response, err := client.SingleCallByTts(request)
if err != nil {
logs.Error(logsign, "[alyphonecall]", err.Error())
}
logs.Info(logsign, "[alyphonecall]", response)
}
models.AlertToCounter.WithLabelValues("alydh").Add(1)
ChartsJson.Alydh += 1
return PhoneNumbers + "Called Over."
}
后面可以在conf文件中添加使用
#阿里云短信模板key
ALY_DX_Template_Key=desc
#阿里云电话模板key
ALY_DH_Template_Key=desc
对应变量可以自定义，默认是code
比如这里是msg，我的aliyun的模板里面 就可以创建一个desc的
这里还遇到一个问题，可能是文件系统的问题，这里把Dockerfile的文件系统做了改动 FROM alpine:3.18 改为了centos7
FROM centos:7
LABEL maintainer="jikun.zhang"
RUN yum -y install epel-release && \
yum -y install tzdata && \
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
echo "Asia/Shanghai" > /etc/timezone && \
yum -y remove epel-release && \
mkdir -p /app/logs && \
yum -y install sqlite curl && \
yum clean all
HEALTHCHECK --start-period=10s --interval=20s --timeout=3s --retries=3 \
CMD curl -fs http://localhost:8080/health || exit 1
修改好源码之后重新编译
make docker
就会在本地打包一个镜像出来，如果需要上传私服可以再打个tag上传（需要docker login先）
最终alertmanager就支持aliyun的短信和语音了
docker tag feiyu563/prometheus-alert:latest harbor.jafir.top/java/feiyu563/prometheus-alert:latest
docker push harbor.jafir.top/java/feiyu563/prometheus-alert:latest
prometheus-laert配置
部署好了之后，修改app.conf文件 aliyun的accesskey secret等 （建议把数据映射到主机目录）
添加短信自定义模板
{{ range $k,$v:=.alerts }}{{if eq $v.status "resolved"}}
[Prometheus恢复信息]
{{$v.annotations.description}}
{{else}}
[Prometheus告警信息]
{{$v.annotations.description}}
{{end}}
{{ end }}
模板测试内容：
{"receiver":"prometheus-dx-ali","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"BlackboxProbeHttpFailure","env":"prod","instance":"https://www.jafir.top","job":"blackbox-http","name":"http-web","service":"web","severity":"critical","team":"node"},"annotations":{"description":"HTTP状态码不在200-399\n当前值= 502\nName= http-web\nEnv= prod\n服务名= web","summary":"Blackbox probe HTTP failure (instance https://www.jafir.top)"},"startsAt":"2023-09-08T09:13:24.324Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://prometheus:9090/graph?g0.expr=probe_http_status_code+%3C%3D+199+or+probe_http_status_code+%3E%3D+400\u0026g0.tab=1","fingerprint":"af6387fb6deb5043"}],"groupLabels":{"alertname":"BlackboxProbeHttpFailure","instance":"https://www.jafir.top"},"commonLabels":{"alertname":"BlackboxProbeHttpFailure","env":"prod","instance":"https://www.jafir.top","job":"blackbox-http","name":"http-web","service":"web","severity":"critical","team":"node"},"commonAnnotations":{"description":"HTTP状态码不在200-399\n当前值= 502\nName= http-web\nEnv= prod\n服务名= web","summary":"Blackbox probe HTTP failure (instance https://www.jafir.top)"},"externalURL":"http://alertmanager:9093","version":"4","groupKey":"{}/{alertname=\"BlackboxProbeHttpFailure\"}:{alertname=\"BlackboxProbeHttpFailure\", instance=\"https://www.jafir.top\"}","truncatedAlerts":0}
添加电话自定义模板
{{ range $k,$v:=.alerts }}{{if eq $v.status "resolved"}}恢复信息{{$v.annotations.description}}{{else}}告警信息{{$v.annotations.description}}{{end}}{{ end }}
模板测试内容
{"receiver":"prometheus-dh-ali","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"BlackboxProbeFailed","env":"prod","instance":"https://www.jafir.top","job":"blackbox-http","name":"http-web","service":"web","severity":"critical","team":"node"},"annotations":{"description":"服务在线检查失败\n当前值= 0\nName= http-web\nEnv= prod\n服务名= web","summary":"Blackbox probe failed (instance https://www.jafir.top)"},"startsAt":"2023-09-08T09:13:24.324Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://prometheus:9090/graph?g0.expr=probe_success+%3D%3D+0\u0026g0.tab=1","fingerprint":"9d044445256d0302"}],"groupLabels":{"alertname":"BlackboxProbeFailed","instance":"https://www.jafir.top"},"commonLabels":{"alertname":"BlackboxProbeFailed","env":"prod","instance":"https://www.jafir.top","job":"blackbox-http","name":"http-web","service":"web","severity":"critical","team":"node"},"commonAnnotations":{"description":"服务在线检查失败\n当前值= 0\nName= http-web\nEnv= prod\n服务名= web","summary":"Blackbox probe failed (instance https://www.jafir.top)"},"externalURL":"http://alertmanager:9093","version":"4","groupKey":"{}/{alertname=\"BlackboxProbeFailed\"}:{alertname=\"BlackboxProbeFailed\", instance=\"https://www.jafir.top\"}","truncatedAlerts":0}
测试：
修改app.conf 的defaultPhone
alertmanager配置
添加路由和receiver
路由规则自己定义，这里是把blackbox的监控检查配置了进来，发送语音和打电话
routes:
- receiver: 'web.hook.grafanaalert' # 路由到名为 "web.hook.grafanaalert" 的接收器
match:
__alert_rule_namespace_uid__: 'IrqNMj34z' # 匹配 alertname 为 "grafana" 的告警
- receiver: 'prometheus-dh-ali'
match:
alertname: "BlackboxProbeFailed"
- receiver: 'prometheus-dx-ali'
match:
alertname: "BlackboxProbeHttpFailure"
- name: 'prometheus-dx-ali'
webhook_configs:
- url: 'http://prometheus-alert-new:8080/prometheusalert?type=alydx&tpl=prometheus-dx-ali&phone=139xxxx'
send_resolved: false
- name: 'prometheus-dh-ali'
webhook_configs:
- url: 'http://prometheus-alert-new:8080/prometheusalert?type=alydh&tpl=prometheus-dh-ali&phone=139xxxx'
send_resolved: false
全文件：（这里又添加了两个receiver，发短信和wx \ 打电话和wx）
global:
resolve_timeout: 15s
route:
group_by: ['alertname','instance']
group_wait: 10s
group_interval: 10s
repeat_interval: 2m
receiver: 'web.hook.prometheusalert'
routes:
- receiver: 'web.hook.grafanaalert' # 路由到名为 "web.hook.grafanaalert" 的接收器
match:
__alert_rule_namespace_uid__: 'IrqNMj34z' # 匹配 alertname 为 "grafana" 的告警
- match:
alertname: "BlackboxProbeFailed"
receiver: 'prometheus-dh-ali-all'
- match:
alertname: "BlackboxProbeHttpFailure"
receiver: 'prometheus-dx-ali-all'
receivers:
- name: 'web.hook.prometheusalert'
webhook_configs:
- url: 'http://prometheus-alert:8080/prometheusalert?type=wx&tpl=prometheus-wx&wxurl=你的企业微信webhookurl'
- name: 'web.hook.grafanaalert'
webhook_configs:
- url: 'http://prometheus-alert:8080/prometheusalert?type=wx&tpl=grafana-wx&wxurl=你的企业微信webhookurl'
- name: 'prometheus-dx-ali'
webhook_configs:
- url: 'http://prometheus-alert-new:8080/prometheusalert?type=alydx&tpl=prometheus-dx-ali&phone=139xxxx'
send_resolved: false
- name: 'prometheus-dh-ali'
webhook_configs:
- url: 'http://prometheus-alert-new:8080/prometheusalert?type=alydh&tpl=prometheus-dh-ali&phone=139xxxx'
send_resolved: false
- name: 'prometheus-dx-ali-all'
webhook_configs:
- url: 'http://prometheus-alert-new:8080/prometheusalert?type=alydx&tpl=prometheus-dx-ali&phone=139xxxx'
- url: 'http://prometheus-alert:8080/prometheusalert?type=wx&tpl=prometheus-wx&wxurl=你的企业微信webhookurl'
send_resolved: false
- name: 'prometheus-dh-ali-all'
webhook_configs:
- url: 'http://prometheus-alert-new:8080/prometheusalert?type=alydh&tpl=prometheus-dh-ali&phone=139xxxx'
- url: 'http://prometheus-alert:8080/prometheusalert?type=wx&tpl=prometheus-wx&wxurl=你的企业微信webhookurl'
send_resolved: false
重启alertmanager
docker restart alertmanager
目前上述配置文件可以参考使用，作用主要是配置电话+短信+企业微信的通知。
一起探索未来云端世界的核心，云原生技术专区带您领略创新、高效和可扩展的云计算解决方案，引领您在数字化时代的成功之路。
更多推荐
47
0- 0
已为社区贡献3条内容
所有评论(0)