elfk收集k8s日志（二）
本文介绍通过elk + filebeat方式收集k8s日志，其中filebeat以sidecar方式部署。elfk最新版本：7.6.2k8s日志收集方案3种日志收集方案：1. node上部署一个日志收集程序Daemonset方式部署日志收集程序，对本节点 /var/log 和 /var/lib/docker/containers 两个目录下的日志进行采集2. sidecar方式部署日志收集程序每个
·
本文介绍通过elk + filebeat方式收集k8s日志，其中filebeat以sidecar方式部署。elfk最新版本：7.6.2
k8s日志收集方案
- 3种日志收集方案：
1. node上部署一个日志收集程序
Daemonset方式部署日志收集程序，对本节点 /var/log 和 /var/lib/docker/containers 两个目录下的日志进行采集
2. sidecar方式部署日志收集程序
每个运行应用程序的pod中附加一个日志收集的容器，使用 emptyDir 共享日志目录让日志容器收集日志
3. 应用程序直接推送日志
常见的如 graylog 工具，直接修改代码推送日志到es，然后在graylog上展示出来
- 3种收集方案的优缺点：
| 方案 | 优点 | 缺点 |
|---|---|---|
| 1. node上部署一个日志收集程序 | 每个node仅需部署一个日志收集程序，消耗资源少，对应用无侵入 | 应用程序日志需要写到标准输出和标准错误输出，不支持多行日志 |
| 2. pod中附加一个日志收集容器 | 低耦合 | 每个pod启动一个日志收集容器，增加资源消耗 |
| 3. 应用程序直接推送日志 | 无需额外收集工具 | 侵入应用，增加应用复杂度 |
下面测试第1种方案：每个node上部署一个日志收集程序，注意elfk版本保持一致。
SideCar方式收集k8s日志
- 主机说明：
| 系统 | ip | 角色 | cpu | 内存 | hostname |
|---|---|---|---|---|---|
| CentOS 7.8 | 192.168.30.128 | master、deploy | >=2 | >=2G | master1 |
| CentOS 7.8 | 192.168.30.129 | master | >=2 | >=2G | master2 |
| CentOS 7.8 | 192.168.30.130 | node | >=2 | >=2G | node1 |
| CentOS 7.8 | 192.168.30.131 | node | >=2 | >=2G | node2 |
| CentOS 7.8 | 192.168.30.132 | node | >=2 | >=2G | node3 |
- 搭建k8s集群：
搭建过程省略，具体参考：Kubeadm方式搭建k8s集群 或 二进制方式搭建k8s集群
搭建完成后，查看集群：
kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready master 4d16h v1.14.0
master2 Ready master 4d16h v1.14.0
node1 Ready <none> 4d16h v1.14.0
node2 Ready <none> 4d16h v1.14.0
node3 Ready <none> 4d16h v1.14.0
这里为了方便，直接使用之前的k8s集群，注意删除之前实验的k8s资源对象。
- 部署es集群：
mkdir /elfk & cd /elfk
vim elasticsearch.yaml
apiVersion: v1
kind: Service
metadata:
name: elasticsearch
namespace: default
labels:
app: elasticsearch
spec:
selector:
app: elasticsearch
clusterIP: None
ports:
- name: api
port: 9200
- name: discovery
port: 9300
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: elasticsearch
namespace: default
spec:
serviceName: elasticsearch
replicas: 3
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2
resources:
limits:
cpu: 1000m
requests:
cpu: 100m
ports:
- containerPort: 9200
name: api
protocol: TCP
- containerPort: 9300
name: discovery
protocol: TCP
env:
- name: "http.host"
value: "0.0.0.0"
- name: "network.host"
value: "_eth0_"
- name: "cluster.name"
value: "es-cluster"
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: "bootstrap.memory_lock"
value: "false"
- name: "discovery.seed_hosts"
value: "elasticsearch"
- name: "cluster.initial_master_nodes"
value: "elasticsearch-0,elasticsearch-1,elasticsearch-2"
- name: "discovery.seed_resolver.timeout"
value: "10s"
- name: "discovery.zen.minimum_master_nodes"
value: "2"
- name: "ES_JAVA_OPTS"
value: "-Xms512m -Xmx512m"
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
terminationGracePeriodSeconds: 30
volumes:
- name: data
hostPath:
path: /home/elasticsearch/data #该路径为es数据存储目录，自动创建
initContainers:
- name: fix-permissions
image: busybox
command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
securityContext:
privileged: true
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
- name: increase-vm-max-map
image: busybox
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
- name: increase-fd-ulimit
image: busybox
command: ["sh", "-c", "ulimit -n 65536"]
securityContext:
privileged: true
为了方便，建议提前在所有node节点上拉取elasticsearch镜像：docker pull docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2。
kubectl apply -f elasticsearch.yaml
kubectl get pod
NAME READY STATUS RESTARTS AGE
elasticsearch-0 1/1 Running 0 110s
elasticsearch-1 1/1 Running 0 97s
elasticsearch-2 1/1 Running 0 83s
kubectl get sts
NAME READY AGE
elasticsearch 3/3 2m13s
- 部署kibana：
这里为了方便，直接使用NodePort暴露kibana端口。
vim kibana.yaml
apiVersion: v1
kind: Service
metadata:
name: kibana
namespace: default
labels:
app: kibana
spec:
selector:
app: kibana
ports:
- port: 5601
nodePort: 30080
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kibana
namespace: default
labels:
app: kibana
spec:
selector:
matchLabels:
app: kibana
template:
metadata:
labels:
app: kibana
spec:
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana-oss:7.6.2
resources:
limits:
cpu: 1000m
requests:
cpu: 100m
env:
- name: ELASTICSEARCH_HOSTS
value: "http://elasticsearch:9200"
ports:
- containerPort: 5601
kubectl apply -f kibana.yaml
kubectl get pod |grep kibana
kibana-84d7449d95-jg5nt 1/1 Running 0 29s
kubectl get deploy |grep kibana
kibana 1/1 1 1 53s
部署没问题的话，是可以正常访问kibana页面的，浏览器访问ip:30080，
接下来部署应用程序，收集日志。
- 以sidecar方式部署filebeat收集nginx日志：
vim filebeat-nginx.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: default
labels:
app: nginx
spec:
selector:
app: nginx
ports:
- port: 80
nodePort: 30090
type: NodePort
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: default
labels:
app: filebeat
data:
filebeat.yml: |-
filebeat.config:
inputs:
path: ${path.config}/inputs.d/*.yml
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.inputs:
- type: log
paths:
- /logdata/*.log
tail_files: true
fields:
pod_name: '${pod_name}'
POD_IP: '${POD_IP}'
setup.template.name: "nginx-logs"
setup.template.pattern: "nginx-logs-*"
output.elasticsearch:
hosts: ["elasticsearch:9200"]
index: "nginx-logs"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: default
spec:
replicas: 1
minReadySeconds: 15
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat-oss:7.6.2
args: [
"-c", "/etc/filebeat/filebeat.yml",
"-e",
]
env:
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: pod_name
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 200m
memory: 200Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat/
- name: data
mountPath: /usr/share/filebeat/data
- name: logdata
mountPath: /logdata
- name: nginx
image: nginx:1.17.0
ports:
- containerPort: 80
volumeMounts:
- name: logdata
mountPath: /var/log/nginx
volumes:
- name: data
emptyDir: {}
- name: logdata
emptyDir: {}
- name: config
configMap:
name: filebeat-config
items:
- key: filebeat.yml
path: filebeat.yml
kubectl apply -f filebeat-nginx.yaml
kubectl get pod |grep nginx
nginx-865f745bdd-q6xwm 2/2 Running 0 16s
kubectl describe pod nginx-865f745bdd-q6xwm
Normal Scheduled 51s default-scheduler Successfully assigned default/nginx-865f745bdd-q6xwm to node2
Normal Pulled 50s kubelet, node2 Container image "docker.elastic.co/beats/filebeat-oss:7.6.2" already present on machine
Normal Created 50s kubelet, node2 Created container filebeat
Normal Started 50s kubelet, node2 Started container filebeat
Normal Pulled 50s kubelet, node2 Container image "nginx:1.17.0" already present on machine
Normal Created 50s kubelet, node2 Created container nginx
Normal Started 50s kubelet, node2 Started container nginx
访问nginx页面以产生日志：ip:30090，
- kibana创建索引，查看nginx日志：
可以看到index就是在filebeat配置文件中指定的index——nginx-logs，添加可用的fields：log.file.path后，显示日志来源，
因为上面filebeat配置文件中收集的日志路径是/var/log/nginx/*.log，也可以只指定单个日志（具体的日志路径）并指定index。
- 以sidecar方式部署filebeat收集tomcat日志：
vim filebeat-tomcat.yaml
apiVersion: v1
kind: Service
metadata:
name: tomcat
namespace: default
labels:
app: tomcat
spec:
selector:
app: tomcat
ports:
- port: 8080
nodePort: 30100
type: NodePort
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config-tomcat
namespace: default
labels:
app: filebeat
data:
filebeat.yml: |-
filebeat.config:
inputs:
path: ${path.config}/inputs.d/*.yml
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.inputs:
- type: log
paths:
- /logdata/*.log
tail_files: true
fields:
pod_name: '${pod_name}'
POD_IP: '${POD_IP}'
setup.template.name: "tomcat-logs"
setup.template.pattern: "tomcat-logs-*"
output.elasticsearch:
hosts: ["elasticsearch:9200"]
index: "tomcat-logs"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat
namespace: default
spec:
replicas: 1
minReadySeconds: 15
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
selector:
matchLabels:
app: tomcat
template:
metadata:
labels:
app: tomcat
spec:
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat-oss:7.6.2
args: [
"-c", "/etc/filebeat/filebeat.yml",
"-e",
]
env:
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: pod_name
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 200m
memory: 200Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat/
- name: data
mountPath: /usr/share/filebeat/data
- name: logdata
mountPath: /logdata
- name: tomcat
image: tomcat:8.0.51-alpine
ports:
- containerPort: 8080
volumeMounts:
- name: logdata
mountPath: /usr/local/tomcat/logs
volumes:
- name: data
emptyDir: {}
- name: logdata
emptyDir: {}
- name: config
configMap:
name: filebeat-config-tomcat
items:
- key: filebeat.yml
path: filebeat.yml
kubectl apply -f filebeat-tomcat.yaml
kubectl get pod |grep tomcat
tomcat-5c7b6644f4-9hslh 2/2 Running 0 16s
kubectl describe pod tomcat-5c7b6644f4-9hslh
Normal Scheduled 34s default-scheduler Successfully assigned default/tomcat-5c7b6644f4-9hslh to node1
Normal Pulled 33s kubelet, node1 Container image "docker.elastic.co/beats/filebeat-oss:7.6.2" already present on machine
Normal Created 33s kubelet, node1 Created container filebeat
Normal Started 33s kubelet, node1 Started container filebeat
Normal Pulled 33s kubelet, node1 Container image "tomcat:8.0.51-alpine" already present on machine
Normal Created 33s kubelet, node1 Created container tomcat
Normal Started 33s kubelet, node1 Started container tomcat
访问tomcat页面以产生日志：ip:30100，
- kibana查看tomcat日志：
可以看到index就是在filebeat配置文件中指定的index——tomcat-logs，添加可用的fields：log.file.path后，显示日志来源，
因为上面filebeat配置文件中收集的日志路径是/usr/local/tomcat/logs/*.log，也可以只指定单个日志（具体的日志路径）并指定index。
- 总结：
上面通过sidecar方式部署filebeat收集k8s日志，比使用logagent方式部署filebeat收集k8s日志更加直接，可以指定具体路径并自定义index，在查看日志时也更加方便，缺点就是会增加资源消耗，不过资源消耗在可接受的范围内。
需要注意的是，filebeat对不同应用程序日志收集的ConfigMap名称尽量不要相同，避免冲突和误删除。
上面为了方便，filebeat收集日志是直接输出到es中的，也可以在集群中再部署一个logstash，由filebeat传输到logstash之后进行日志处理再输出到es中，这里就不演示了。
logstash.yaml（参考）：
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-config
namespace: default
data:
logstash.yml: |
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
logstash.conf: |
input {
beats {
port => 5044
}
}
filter {
#multiline {
#pattern => "^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}"
#negate => true
#what => "previous"
#}
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:logtime} %{LOGLEVEL:level}" ]
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "your-index-%{+YYYY.MM.dd}"
}
}
---
kind: Service
apiVersion: v1
metadata:
name: logstash
namespace: default
spec:
selector:
app: logstash
ports:
- protocol: TCP
port: 5044
targetPort: 5044
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: logstash
namespace: default
spec:
selector:
matchLabels:
app: logstash
replicas: 1
template:
metadata:
labels:
app: logstash
spec:
containers:
- name: logstash
image: docker.elastic.co/logstash/logstash-oss:7.6.2
ports:
- containerPort: 5044
volumeMounts:
- name: config
mountPath: /usr/share/logstash/config
- name: pipeline
mountPath: /usr/share/logstash/pipeline
volumes:
- name: config
configMap:
name: logstash-config
items:
- key: logstash.yml
path: logstash.yml
- name: pipeline
configMap:
name: logstash-config
items:
- key: logstash.conf
path: logstash.conf
kubectl get pod |grep logstash
logstash-6b475db5f6-mxxsj 1/1 Running 0 26s
kubectl logs -f logstash-6b475db5f6-mxxsj
[INFO ] 2020-05-17 04:46:22.488 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2020-05-17 04:46:22.501 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2020-05-17 04:46:22.532 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2020-05-17 04:46:22.615 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2020-05-17 04:46:22.639 [[main]<beats] Server - Starting server on port: 5044
kubectl get svc |grep logstash
logstash ClusterIP 10.96.103.207 <none> 5044/TCP 1m34s
kubectl get pod |grep nginx
nginx-865f745bdd-q6xwm 2/2 Running 0 62m
kubectl exec -it nginx-865f745bdd-q6xwm bash
[root@nginx-865f745bdd-q6xwm filebeat]# yum install -y telnet
[root@nginx-865f745bdd-q6xwm filebeat]# telnet logstash 5044
Trying 10.96.103.207...
Connected to logstash.
Escape character is '^]'.
简单测试部署没问题，filebeat容器测试端口连接也没问题，此处省略进一步配置logstash过滤处理日志的过程。
前面提过，sidecar方式部署filebeat的话，elk组件是否部署在k8s集群内无关紧要，只需要连接没问题即可。上面部署的elk组件是在k8s集群内，接下来将elk部署于k8s集群外测试日志收集。
K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容
更多推荐
0
0- 0
已为社区贡献29条内容
所有评论(0)