背景

我们集群是openfaas+k8s，之前已经加入了Prometheus和grafana。

然后我最近又想在集群中添加Istio,但是我安装的时候没有自带Prometheus的组件，只能自己去集成已有的。踩了很多雷，狂改三天的配置文件才终于集成进去了。

具体步骤

找到Prometheus的配置文件

我是在openfaas的namespace下找到的

#查看pod的具体信息，找到配置文件
kubectl get configmap  prometheus-config -n openfaas -o yaml

然后进行修改

#找到了之后直接查看里面的内容，应该是有 prometheus.yml 这个文件内容在里面，然后直接修改这个prometheus-config文件
kubectl edit configmap prometheus-config -n openfaas

但是 我遇见了特别吐血的事情，不知道大家有没有好的解决办法，我用到方法很笨拙

我的配置文件一打开，发现它不是yaml格式，而是作为字符串进行插入的，这就给我直接修改带来了很多麻烦

prometheus.yml: "global:\n  scrape_interval:     15s\n  evaluation_interval: 15s\n
    \ external_labels:\n      monitor: 'faas-monitor'\nrule_files:\n    - 'alert.rules.yml'\nalerting:\n
    \ alertmanagers:\n  - static_configs:\n    - targets:\n      - alertmanager:9093\nscrape_configs:\n
    \ - job_name: 'kubernetes-node-exporter'\n    kubernetes_sd_configs:\n    - role:
    node\n    relabel_configs:\n    - source_labels: [__address__]\n      regex: '(.*):10250'\n
    \     replacement: '${1}:9100'\n      target_label: __address__\n      action:
    replace\n    - source_labels: [__meta_kubernetes_node_name]\n      action: replace\n
    \     target_label: node      \n    - action: labelmap\n      regex: __meta_kubernetes_node_label_(.+)\n
    \   - source_labels: [__meta_kubernetes_node_address_InternalIP]\n      action:
    replace\n      target_label: ip\n  - job_name: 'prometheus'\n    scrape_interval:
    10s\n    static_configs:\n      - targets: ['localhost:9090']\n  - job_name: 'keda-operator-metrics'\n
    \   scrape_interval: 5s\n    kubernetes_sd_configs:\n      - role: service\n        namespaces:\n
    \         names:\n            - keda\n    relabel_configs:\n    - source_labels:
    [__meta_kubernetes_service_name]\n      action: keep\n      regex: keda-operator\n
    \   - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]\n
    \     action: keep\n      regex: keda-operator\n    - source_labels: [__meta_kubernetes_service_port_name]\n
    \     action: keep\n      regex: metrics\n    - source_labels: [__address__]\n
    \     action: replace\n      target_label: __address__\n      regex: (.+)(?::\\d+)\n
    \     replacement: $1:8080\n  - job_name: 'openfaas-pods'\n    scrape_interval:
    5s\n    honor_labels: false\n    kubernetes_sd_configs:\n      - role: pod\n        namespaces:\n
    \         names:\n            - openfaas\n    relabel_configs:\n    - action:
    labelmap\n      regex: __meta_kubernetes_pod_label_(.+)\n    - source_labels:
    [__meta_kubernetes_namespace]\n      action: replace\n      target_label: kubernetes_namespace\n
    \   - source_labels: [__meta_kubernetes_pod_name]\n      action: replace\n      target_label:
    kubernetes_pod_name\n    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]\n
    \     action: keep\n      regex: true\n    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]\n
    \     action: replace\n      regex: ([^:]+)(?::\\d+)?;(\\d+)\n      replacement:
    $1:$2\n      target_label: __address__\n  - job_name: 'kubernetes-cadvisor'\n
    \   scheme: https\n    tls_config:\n      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n
    \   bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token\n    kubernetes_sd_configs:\n
    \   - role: node\n    relabel_configs:\n    - target_label: __address__\n      replacement:
    kubernetes.default.svc:443\n    - source_labels: [__meta_kubernetes_node_name]\n
    \     regex: (.+)\n      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor\n
    \     target_label: __metrics_path__\n    - action: labelmap\n      regex: __meta_kubernetes_node_label_(.+)\n
    \   - source_labels: [__meta_kubernetes_node_name]\n      action: replace\n      target_label:
    node\n    - source_labels: [__meta_kubernetes_node_label_node]\n      action:
    replace\n      target_label: node_name\n"

于是我把这段字符串让chat老师格式化了一下，千万注意空格什么的，这个格式很容易出错，一定要很小心很仔细！

prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
      external_labels:
        monitor: 'faas-monitor'
    rule_files:
      - 'alert.rules.yml'
    alerting:
      alertmanagers:
        - static_configs:
            - targets:
                - alertmanager:9093
    scrape_configs:
      - job_name: 'prometheus'
        scrape_interval: 10s
        static_configs:
          - targets: ['localhost:9090']
      - job_name: 'envoy-stats'
        metrics_path: /stats/prometheus
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_container_port_name]
            action: keep
            regex: '.*-envoy-prom'
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: '${1}:15090'
            target_label: __address__
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: pod_name
      - job_name: 'istiod'
        scrape_interval: 5s
        kubernetes_sd_configs:
          - role: endpoints
            namespaces:
              names:
                - istio-system
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
            action: keep
            regex: istiod;http-monitoring
      - job_name: 'keda-operator-metrics'
        scrape_interval: 5s
        kubernetes_sd_configs:
          - role: service
            namespaces:
              names:
                - keda
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_name]
            action: keep
            regex: keda-operator
          - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
            action: keep
            regex: keda-operator
      - job_name: 'kubernetes-node-exporter'
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - source_labels: [__address__]
            regex: '(.*):10250'
            replacement: '${1}:9100'
            target_label: __address__
            action: replace
          - source_labels: [__meta_kubernetes_node_name]
            action: replace
            target_label: node
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - source_labels: [__meta_kubernetes_node_address_InternalIP]
            action: replace
            target_label: ip

保存之后就需要重启prometheus这个组件。但是我是直接修改的yaml文件，把原来的yaml备份，然后重新再创建一个新的yaml文件，把格式化和添加了修改之后的内容写进去，然后命名为prometheus-config.yaml，再这个新的文件的目录下执行

kubectl apply -f prometheus-config.yaml

有时候会报错，就是两者迭代出问题，这个时候可以强制执行

kubectl replace -f prometheus-config.yaml --force

要是有用就行，没用还得再换种方式

#把现在的配置文件导出来，然后在这个导出来的文件里修改。一开始就可以这样做，直接把当前的配置文件导出来，然后格式化修改，添加内容

kubectl get configmap prometheus-config -n openfaas -o yaml > current-prometheus-config.yaml

#之后再执行

kubectl apply -f current-prometheus-config.yaml

然后再执行这条命令，这条命令不论哪种修改方式都要最终执行一下

kubectl patch deployment prometheus -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"

查看新产生的Prometheus pod的状态，要是有问题查看日志

kubectl logs prometheus-796f556c87-xf49k -n openfaas

添加Istio的组件

把这一段插入配置文件就行

刚才已经给出来的，可以直接照着我的抄，我查看了Istio官网文档和chat老师的回答，一起结合。

经验之谈

不要全部听chat老师的话，如果一直报错，可以去官网找找源文档

Istio / Prometheus