在部署es集群收集日志，以及用prometheus实现监控的遇到很多的坑，大概花了两个礼拜的时间，走通了，希望可以对读者的有一定的帮助。

一、部署集群k8s, 这个需要自行部署，我选择的k8s 1.16

二、部署es集群

1、添加ECK自定义资源

kubectl apply -f all-in-one.yaml  (说明，我设置的命名空间为logging-system)
kubectl apply -f https://download.elastic.co/downloads/eck/1.1.2/all-in-one.yaml
如果有需求的，可以把这个文件下载下来进行命名空间的设置，部署完可以查看如下。

kubectl get all -n logging-system
pod/elastic-operator-0                       1/1     Running   0          18d
service/elastic-webhook-server          ClusterIP   172.30.198.157   <none>        443/TCP         18d
statefulset.apps/elastic-operator                1/1     18d

2、部署elasticsearch

存储是通过节点本地存储的方式

(1) 创建存储类

kubectl apply -f es-data-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: es-data
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain

(2) 创建PV

如果/var/lib/hawkeye/esdata文件夹不存在，需要先进行创建。

apiVersion: v1
kind: PersistentVolume
metadata:
  name: es-data-0
spec:
  capacity:
    storage: 20Gi
  accessModes:
  - ReadWriteOnce
  storageClassName: es-data
  persistentVolumeReclaimPolicy: Retain
  local:
    path: /var/lib/hawkeye/esdata
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - nodeName(需要修改)
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: es-data-1
spec:
  capacity:
    storage: 20Gi
  accessModes:
  - ReadWriteOnce
  storageClassName: es-data
  persistentVolumeReclaimPolicy: Retain
  local:
    path: /var/lib/hawkeye/esdata
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - nodeName(需要修改)
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: es-data-2
spec:
  capacity:
    storage: 20Gi
  accessModes:
  - ReadWriteOnce
  storageClassName: es-data
  persistentVolumeReclaimPolicy: Retain
  local:
    path: /var/lib/hawkeye/esdata
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - nodeName(需要修改)

(3) 创建PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: elasticsearch-data-elasticsearch-es-master-nodes-0
  namespace: logging-system
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: es-data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: elasticsearch-data-elasticsearch-es-data-nodes-0
  namespace: logging-system
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: es-data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: elasticsearch-data-elasticsearch-es-data-nodes-1
  namespace: logging-system
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: es-data

(4)创建集群密码，如果不创建，那么集群就会随机生成

apiVersion: v1
data:
  elastic: yourSecret #(需要base64加密)
kind: Secret
metadata:
  labels:
    common.k8s.elastic.co/type: elasticsearch
    eck.k8s.elastic.co/credentials: "true"
    elasticsearch.k8s.elastic.co/cluster-name: elasticsearch
  name: elasticsearch-es-elastic-user #名字不能改
  namespace: logging-system
type: Opaque

(5)创建es集群

这里创建三个节点，两个数据节点，一个管理节点，

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: logging-system
spec:
  version: 7.2.0
  image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
  nodeSets:
  - name: master-nodes
    count: 1
    config:
      node.master: true
      node.data: false
    podTemplate:
      metadata:
        namespace: logging-system
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        #volumes:
        #- name: elasticsearch-data
        #  emptyDir: {}
        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms1g -Xmx1g
          resources:
            requests:
              memory: 2Gi
            limits:
              memory: 10Gi
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
        storageClassName: es-data
  - name: data-nodes
    count: 2
    config:
      node.master: false
      node.data: true
    podTemplate:
      metadata:
        namespace: logging-system
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        #volumes:
        #- name: elasticsearch-data
        #  emptyDir: {}
        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms1g -Xmx1g
          resources:
            requests:
              memory: 2Gi
            limits:
              memory: 10Gi
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
        storageClassName: es-data

会生成如下

kubectl get all -n logging-system
pod/elasticsearch-es-data-nodes-0            1/1     Running   0          10d
pod/elasticsearch-es-data-nodes-1            1/1     Running   0          10d
pod/elasticsearch-es-master-nodes-0          1/1     Running   0          10d

service/elasticsearch-es-data-nodes     ClusterIP   None             <none>        <none>          10d
service/elasticsearch-es-http           ClusterIP   172.30.229.248   <none>        9200/TCP        10d
service/elasticsearch-es-master-nodes   ClusterIP   None             <none>        <none>          10d

kubectl get elasticsearch -n logging-system

NAME            HEALTH   NODES   VERSION   PHASE   AGE
elasticsearch   green    3       7.2.0     Ready   10d

可以查看集群状态
curl -u "elastic:yourpassword" -k https://172.30.229.248(是你集群中生成的IP,为elasticsearch-es-http的ClusterIP):9200
{
  "name" : "elasticsearch-es-data-nodes-0",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "caWatVXyQmSNDLQ7lEO7qg",
  "version" : {
    "number" : "7.2.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "508c38a",
    "build_date" : "2019-06-20T15:54:18.811730Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

获取集群密码
kubectl -n logging-system get secret elasticsearch-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode; echo

2、部署kibana

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: logging-system
spec:
  version: 7.2.0
  count: 1
  elasticsearchRef:
    name: elasticsearch
  http:
    tls:
      selfSignedCertificate:
        disabled: true

这样就可以登录到kibana中，用户名密码和es集群中的相同。

3、部署fluentbit

分别包含文件fluentbit-clusterRoleBinding.yaml、fluentbit-clusterRole.yaml、fluentbit-configmap.yaml、fluentbit-daemonset.yaml、fluentbit-serviceAccount.yaml、fluentbit-service.yaml、kustomization.yaml
文件都放在同一个文件夹下，
执行 kubectl apply -k .

fluentbit-clusterRoleBinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluentbit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluentbit-read
subjects:
- kind: ServiceAccount
  name: fluentbit
  namespace: logging-system

fluentbit-clusterRole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentbit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]

fluentbit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentbit-config
  namespace: logging-system
data:
  filter-kubernetes.conf: |
    [FILTER]
        Name record_modifier
        Match *
        Record hostname ${HOSTNAME}
  fluent-bit.conf: |
    [SERVICE]
        # Set an interval of seconds before to flush records to a destination
        Flush        5
        # Instruct Fluent Bit to run in foreground or background mode.
        Daemon       Off
        # Set the verbosity level of the service, values can be:
        Log_Level    info
        # Specify an optional 'Parsers' configuration file
        Parsers_File parsers.conf
        # Plugins_File plugins.conf
        # Enable/Disable the built-in Server for metrics
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf
  input-kubernetes.conf: |
    [INPUT]
        Name              systemd
        Tag               host.*
        Path              /var/log/journal
        DB /var/log/fluentbit/td.sys.pos
  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           kube.*
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        tls             ${TLS_ENABLE}
        tls.verify      ${TLS_VERIFY}
        HTTP_User       ${ELASTICSEARCH_USERNAME}
        HTTP_Passwd     ${ELASTICSEARCH_PASSWORD}
        # Replace_Dots    On
        Retry_Limit     False
        Index kube
        Type kube
        Buffer_Size 2M
        Include_Tag_Key On
        Tag_Key component
        Logstash_Format On
        Logstash_prefix umstor-monitor

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>.*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
        # Command      |  Decoder | Field | Optional Action
        # =============|==================|=================
        Decode_Field_As   escaped    log

    [PARSER]
        Name        docker-daemon
        Format      regex
        Regex       time="(?<time>[^ ]*)" level=(?<level>[^ ]*) msg="(?<msg>[^ ].*)"
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        syslog-rfc5424
        Format      regex
        Regex       ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        syslog-rfc3164-local
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S
        Time_Keep   On

    [PARSER]
        Name        syslog-rfc3164
        Format      regex
        Regex       /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
        Time_Key    time
        Time_Format %b %d %H:%M:%S
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name    mongodb
        Format  regex
        Regex   ^(?<time>[^ ]*)\s+(?<severity>\w)\s+(?<component>[^ ]+)\s+\[(?<context>[^\]]+)]\s+(?<message>.*?) *(?<ms>(\d+))?(:?ms)?$
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
        Time_Key time

    [PARSER]
        # http://rubular.com/r/izM6olvshn
        Name crio
        Format Regex
        Regex /^(?<time>.+)\b(?<stream>stdout|stderr)\b(?<log>.*)$/
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%N%:z
        Time_Keep   On

    [PARSER]
        Name    kube-custom
        Format  regex
        Regex   var\.log\.containers\.(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$

    [PARSER]
        Name    filter-kube-test
        Format  regex
        Regex   .*kubernetes.(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$

    [PARSER]
        # umstor for all log files
        # http://rubular.com/r/IvZVElTgNl
        Name umstor
        Format regex
        Regex ^(?<log_time>[^ ][-.\d\+:T]+[ ]*[.:\d]*)\s+(?<thread_id>\w+)\s+(?<log_level>-*\d+)\s+(?<message>.*)$
        Time_Format %Y-%m-%d %H:%M:%S.%L
        Time_Keep Off
        Time_Key log_time

    [PARSER]
        # scrub for osd
        Name umstor-scrub
        Format regex
        Regex ^(?<log_time>[^ ][-.\d\+:T]+[ ]*[.:\d]*)\s+(?<m>\w+)\s+(?<ret>-*\d+)\s+(?<message>.*)\s+(?<scrub_pg>\d+.\w+)\s+(?<scrub_status>scrub\s\w+)$
        Time_Format %Y-%m-%d %H:%M:%S.%L
        Time_Keep Off
        Time_Key log_time

    [PARSER]
        # deep-scrub for osd
        Name umstor-deep-scrub
        Format regex
        Regex ^(?<log_time>[^ ][-.\d\+:T]+[ ]*[.:\d]*)\s+(?<m>\w+)\s+(?<ret>-*\d+)\s+(?<message>.*)\s+(?<scrub_pg>\d+.\w+)\s+(?<scrub_status>deep-scrub\s\w+)$
        Time_Format %Y-%m-%d %H:%M:%S.%L
        Time_Keep Off
        Time_Key log_time

    [PARSER]
        # log warning for osd, mon
        Name umstor-log-warn
        Format regex
        Regex ^(?<log_time>[^ ][-.\d\+:T]+[ ]*[.:\d]*)\s+(?<m>\w+)\s+(?<ret>-*\d+)\s+(?<log_channel>[^ ]+)\s+\w+\s+(?<log_level>[\[WRN\]]+)\s+(?<message>.*)$
        Time_Format %Y-%m-%d %H:%M:%S.%L
        Time_Keep Off
        Time_Key log_time

    [PARSER]
        # log debug for osd, mon
        Name umstor-log-debug
        Format regex
        Regex ^(?<log_time>[^ ][-.\d\+:T]+[ ]*[.:\d]*)\s+(?<m>\w+)\s+(?<ret>-*\d+)\s+(?<log_channel>[^ ]+)\s+\w+\s+(?<log_level>[\[DBG\]]+)\s+(?<message>.*)$
        Time_Format %Y-%m-%d %H:%M:%S.%L
        Time_Keep Off
        Time_Key log_time

 fluentbit-daemonset.yaml
 apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentbit
  namespace: logging-system
  labels:
    k8s-app: fluentbit-logging
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluentbit-logging
      kubernetes.io/cluster-service: "true"
  template:
    metadata:
      labels:
        k8s-app: fluentbit-logging
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluentbit
        image: registry.umstor.io:5050/vendor/fluent-bit:1.3
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 2020
            name: http-metrics
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch-es-http"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: "elastic"
        - name: ELASTICSEARCH_PASSWORD
          value: "r00tme"
        - name: TLS_ENABLE
          value: "On"
        - name: TLS_VERIFY
          value: "Off"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluentbit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluentbit-config
        configMap:
          name: fluentbit-config
      serviceAccountName: fluentbit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"

fluentbit-serviceAccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentbit
  namespace: logging-system

fluentbit-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: fluentbit-logging
  name: fluentbit-logging
  namespace: logging-system
spec:
  clusterIP: None
  ports:
  - name: http-metrics
    port: 2020
    protocol: TCP
    targetPort: http-metrics
  type: ClusterIP
  selector:
    k8s-app: fluentbit-logging

kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: logging-system
resources:
- fluentbit-clusterRoleBinding.yaml
- fluentbit-clusterRole.yaml
- fluentbit-daemonset.yaml
- fluentbit-serviceAccount.yaml
- fluentbit-service.yaml
- fluentbit-configmap.yaml

三、prometheus-elasticsearch-exporter

1、创建deployment

		这里面有个坑，就是设置容器的参数，先添加用户名和密码，以及要关闭SSL验证

在这里插入apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch-exporter
  namespace: logging-system
  labels:
    app: elasticsearch-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch-exporter
  template:
    metadata:
      labels:
        app: elasticsearch-exporter
    spec:
      containers:
      - name: elasticsearch-exporter
        image: registry.umstor.io:5050/vendor/elasticsearch_exporter:1.1.0
        resources:
          limits:
            cpu: 300m
          requests:
            cpu: 200m
        ports:
        - containerPort: 9114
          name: https
        command:
        - /bin/elasticsearch_exporter
        - --es.all
        - --web.telemetry-path=/_prometheus/metrics
        - --es.ssl-skip-verify
        - --es.uri=https://elastic:r00tme@elasticsearch-es-http:9200
        securityContext:
          capabilities:
            drop:
            - SETPCAP
            - MKNOD
            - AUDIT_WRITE
            - CHOWN
            - NET_RAW
            - DAC_OVERRIDE
            - FOWNER
            - FSETID
            - KILL
            - SETGID
            - SETUID
            - NET_BIND_SERVICE
            - SYS_CHROOT
            - SETFCAP
          readOnlyRootFilesystem: true
        livenessProbe:
          httpGet:
            path: /healthz
            port: 9114
          initialDelaySeconds: 30
          timeoutSeconds: 10
        readinessProbe:
          httpGet:
            path: /healthz
            port: 9114
          initialDelaySeconds: 10
          timeoutSeconds: 10代码片

2、创建es对应prometheus的serviceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    es-app: es-exporter
  name: es-client-node
  namespace: logging-system
spec:
  endpoints:
  - interval: 30s
    honorLabels: true
    port: https
    path: /_prometheus/metrics
  namespaceSelector:
    matchNames:
    - logging-system
  jobLabel: es-app
  selector:
    matchLabels:
      app: elasticsearch-exporter

3、创建elasticsearch-exporter 对应的service

这也是个坑，如果不创建service ,就不会生成endpoints,

apiVersion: v1
kind: Service
metadata:
  labels:
    app: elasticsearch-exporter
  name: elasticsearch-exporter
  namespace: logging-system
spec:
  ports:
  - name: https
    port: 9114
    protocol: TCP
    targetPort: 9114
  selector:
    app: elasticsearch-exporter
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

四、es日志的自动清理

这可能是最大的坑，因为es7.2版本不兼容任何的elasticsearch-curator,所以作者自己要开发的一个版本的elasticsearch-curator,我放在github上。
对应的文件有个四个，actions.yaml, curator-cronjob.yaml, curator.yaml, kustmoization.yaml

actions.yaml
actions:
  1:
    action: delete_indices
    description: >-
      Delete metric indices older than 21 days (based on index name), for
      .monitoring-es-6-
      .monitoring-kibana-6-
      umstor-os-
      umstor-sys-
      umstor-monitor-
      umstor-internal-
      security-auditlog-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      continue_if_exception: False
      disable_action: False
      ignore_empty_list: True
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(\.monitoring-(es|kibana)-6-|umstor-(os|sys|internal|kube|monitor)-|security-auditlog-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 21
  2:
    action: close
    description: >-
      Close metric indices older than 14 days (based on index name), for
      .monitoring-es-6-
      .monitoring-kibana-6-
      umstor-os-
      umstor-sys-
      umstor-monitor-
      umstor-internal-
      security-auditlog-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      continue_if_exception: True
      disable_action: False
      ignore_empty_list: True
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(\.monitoring-(es|kibana)-6-|umstor-(os|sys|internal|kube|monitor)-|security-auditlog-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 14

curator-cronjob.yaml
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: curator
spec:
  schedule: 59 23 * * *
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: curator
            image: registry.umstor.io:5050/vendor/elasticsearch-curator:v1.0.0
            volumeMounts:
            - mountPath: /etc/curator/
              name: curator-config
              readOnly: true
            - mountPath: /var/log/curator
              name: curator-log
          restartPolicy: OnFailure
          volumes:
          - configMap:
              name: curator-config
            name: curator-config
          - hostPath:
              path: /var/log/curator
            name: curator-log

curator.yaml
client:
  hosts:
    - elasticsearch-es-http
  port: 9200
  url_prefix:
  use_ssl: True
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: True
  http_auth: elastic:r00tme
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile: /var/log/curator/curator.log
  logformat: default
  blacklist: []

kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: logging-system
resources:
- curator-cronjob.yaml
generatorOptions:
  disableNameSuffixHash: true
configMapGenerator:
- files:
  - curator.yaml
  - actions.yaml
  name: curator-config
images:
- name: registry.umstor.io:5050/vendor/elasticsearch-curator
  newTag: "v1.0.0"