需求: 想把ingress的日志都拿出来看或者采集起来，通过kubectl log -f的方式不方便看之前的日志 无法持久化

思路:将stdout和stderr的日志持久化到/var/log/nginx 再通过filebeat采集到kibana

即用filebeat镜像作为sidecar，把nginx-ingress-controller里的/var/log/nginx 的日志采集出来

个人笔记 可能会有错误

主要配置 重定向nginx日志。

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  use-forwarder-headers: "true"
  proxy-body-size: "100m"
  # 配置nginx日志重定向目标
  access-log-path: /var/log/nginx/access.log
  error-log-path: /var/log/nginx/error.log

这里如果写了不存在的目录 会报目录不存在。建议老老实实用/var/log/nginx 否则还需要用initcontainer创建对应用户的文件夹 感觉比较麻烦

完整配置

创建base.yaml 作用说明我写了注释。

base.yaml

---

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
---

apiVersion: extensions/v1beta1
kind: Deployment
#tcp:
#   9000: "devops/test-jenkins:80"
metadata:
  name: default-http-backend
  labels:
    app.kubernetes.io/name: default-http-backend
    app.kubernetes.io/part-of: ingress-nginx
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: default-http-backend
  template:
    metadata:
      labels:
        app.kubernetes.io/name: default-http-backend
        app.kubernetes.io/part-of: ingress-nginx
    spec:
#      hostAliases:
#      - ip: "192.168.1.203"
#        hostnames:
#        - "test-jenkins.ihaozhuo.com"
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissible as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.io/google_containers/defaultbackend:1.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: default-http-backend
    app.kubernetes.io/part-of: ingress-nginx
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app.kubernetes.io/name: default-http-backend
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  use-forwarder-headers: "true"
  proxy-body-size: "100m"
  # 配置nginx日志重定向目标
  access-log-path: /var/log/nginx/access.log
  error-log-path: /var/log/nginx/error.log

---
# logrotate配置文件 偷懒就写个通配，可以根据自己需要调整
apiVersion: v1
data:
  nginx.log: |
    /var/log/nginx/*.log {
        rotate 5
        maxsize 50M
        minsize 50M
        compress
        sharedscripts
        missingok
        create 0644 www-data www-data
    }

kind: ConfigMap
metadata:
  name: nginx-ingress-logrotate
  namespace: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
# 定义日志收集相关配置的一个通配configmap  如果有需要controller的日志也可以采集
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-filebeat-config
  namespace: ingress-nginx
  labels:
    config: nginx-filebeat-config
data:
  filebeat.yml: |-
    filebeat.prospectors:
    - type: log
      paths:
        - /var/log/nginx/access.log
      json.keys_under_root: true
      json.overwrite_keys: true
      tags: ["info","nginx_ingress_controller"]
    - type: log
      enabled: true
      paths:
       - /var/log/nginx/error.log
      json.keys_under_root: true
      json.overwrite_keys: true
      tags: ["error","nginx_ingress_controller"]

    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
    setup.template.enabled: false
    setup.template.settings:
      index.number_of_shards: 3
    setup.template.name: "filebeat"
    setup.template.fields: "fields.yml"
    setup.template.overwrite: false
    setup.kibana:
    output.elasticsearch:
     hosts: ["192.168.1.165:9200"]
     index: "jsonlog-%{+yyyy.MM.dd}"
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

创建deploy.yaml

deploy.yaml

---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
# hostNetwork网络
      hostNetwork: true
# 节点选择器 标签app=nginx
      nodeSelector:
        app: ingress
# 0906
      containers:
      - image: registry.cn-shanghai.aliyuncs.com/yjk-datag/filebeat:6     # 提前下载下来到私有镜像库的镜像(官方的可能会被墙)
        name: filebeat
        args: [
          "-c", "/srv/filebeat/filebeat.yml",
          "-e",
        ]
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: pod_name
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: pod_name
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
       # securityContext:
       #   runAsUser: 0
        resources:
          limits:
            cpu: 400m
            memory: 200Mi
          requests:
            cpu: 200m
            memory: 200Mi
        volumeMounts:
        - name: logrotateconfig               # 将configmap的内容放到容器本地目录
          mountPath: /etc/logrotate.d/nginx.log
          subPath: nginx.log
        - name: config               # 将configmap的内容放到容器本地目录
          mountPath: /srv/filebeat/
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: logdata
          mountPath: /var/log/nginx


      - name: nginx-ingress-controller
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
        args:
          - /nginx-ingress-controller
          - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
          - --configmap=$(POD_NAMESPACE)/nginx-configuration
          - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
          - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
          - --publish-service=$(POD_NAMESPACE)/ingress-nginx
          - --annotations-prefix=nginx.ingress.kubernetes.io
        # 设置ingress-controller日志输出地址 偷懒和nginx日志放一起了
          - --log_dir=/var/log/nginx/
          - --logtostderr=false
        securityContext:
          capabilities:
              drop:
              - ALL
              add:
              - NET_BIND_SERVICE
          # www-data -> 33
          runAsUser: 33
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        volumeMounts:
        - name: logdata               # 将configmap的内容放到容器本地目录
          mountPath: /var/log/nginx

# 0906

      volumes:
      - name: data
        emptyDir: {}
      - name: logdata         #定义logdata为EmptyDir类型挂载目录
        emptyDir: {}
      - name: config
        configMap:
          name: nginx-filebeat-config  #使用前面定义的configmap
          items:
          - key: filebeat.yml
            path: filebeat.yml
      - name: logrotateconfig
        configMap:
          name: nginx-ingress-logrotate
          items:
          - key: nginx.log
            path: nginx.log
      # 需要的话加上仓库的secret      
     # imagePullSecrets:
     # - name: aliyun-ns-test

# 0906

tcp-service.yaml 开发443 80 和额外的9000端口

---
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
    - name: proxied-tcp-9000
      port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

应用yaml

kubectl apply -f ./
kubectl get pods -w -n ingress-nginx

进入容器检查

文件都已经挂载了

kibana中采集到的日志

pod内查看如下

日志轮转方面

用filebeat6的镜像稍作修改

# 1 更换apt源

cat > /etc/apt/sources.list <<EOF
deb http://mirrors.aliyun.com/ubuntu/ xenial main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ xenial main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-proposed main restricted universe multiverse
EOF

apt updata && apt upgrade


# 2 安装轮转需要的logrotate 和 cron 定时任务
apt-get install vim logrotate cron
service cron start
# 3 修改logrotate配置 加入 su root list 原因下面会说

# 4 设置定时任务 crontab -e 加入0 */6 * * *  root logrotate  -vf /etc/logrotate.d/nginx.log

# 5 可以手动测试日志轮转
logrotate  -vf /etc/logrotate.d/nginx.log

手动轮转结果如下 文件会被压缩 不管是否达到配置里的要求。 我用filebeat采集的话 其实用cat 清空也行

找到对应的filebeat容器，进行commit，然后push成新镜像filebeat:6.2

docker commit -a "install vim logrotate crontab"  761e161026c7  registry.cn-shanghai.aliyuncs.com/yjk-datag/filebeat:6.2
docker push registry.cn-shanghai.aliyuncs.com/yjk-datag/filebeat:6.2
docker tag registry.cn-shanghai.aliyuncs.com/yjk-datag/filebeat:6.2 woshidaxu/filebeat:6.2
docker push woshidaxu/filebeat:6.2

修改过后的镜像可以下载 也可以替换deploy.yaml里filebeat的tag 方便下次重新部署 我这个镜像logrotate是3.9以下的

docker pull  woshidaxu/filebeat:6.2

遇到的问题

error: skipping "/var/log/nginx/access.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

需要在configmap中加入su root list

# nginx.log
/var/log/nginx/*.log {
    su root list
    rotate 5
    maxsize 50M
    minsize 50M
    compress
    # sharedscripts
    missingok
    create 0644 www-data www-data
}

如果自己做镜像 可以用logrotate 3.9以上版本可以用下面的配置 可以实现按小时切割 。logrotate3.9以上支持小时 分钟。ubuntu16 apt-cache madison logrotate 了一下只能3.8

/var/log/nginx/*.log {
    su root list
    size 50M
    notifempty
    copytruncate
    rotate 5
    missingok
    compress
    dateext
    dateformat .%Y%m%d-%H
    create 0644 www-data www-data
}

可参考文档 网上貌似都是这篇复制的吧？我和他差别比较多 我直接挂出来采集了。没有用hostpath

https://segmentfault.com/a/1190000022200374