参考文章

## 自己的博客
https://knight.blog.csdn.net/article/details/121476695

##
https://www.cnblogs.com/dogfei/p/14679813.html

# 他人的博客
https://blog.csdn.net/weixin_38367535/article/details/121117729

## sidecar 和daemonset 收集日志
https://blog.csdn.net/m0_60244783/article/details/127027964

k8s日志文件说明

一般情况下，容器中的日志在输出到标准输出(stdout)时，会以 *-json.log 的命名方式保存在/var/lib/docker/containers目录中。

root@xx-k8s-non-prd-002:/var/lib/docker/containers# pwd
/var/lib/docker/containers
root@hwy-hn1-carsales-k8s-non-prd-002:/var/lib/docker/containers# ll
total 108
drwx------ 27 root root 4096 Dec 22 13:48 ./
drwx--x--x 14 root root 4096 Jun  4  2021 ../
drwx------  4 root root 4096 Jun  4  2021 012fec605ce7ca95aa0bd3805ca823cbfbdbb13bd4f3a4cbcecbbf786e87bf46/
drwx------  4 root root 4096 Jun  4  2021 01936bd10a9d52b94f8d0fad1cda0c61acbccad89ca21965ad55a0f91bd31f21/
drwx------  4 root root 4096 Nov 25 12:39 1a97f6689b4766a14bb247e6372d2994d374ecdf261007959559fd6eeb605b17/
drwx------  4 root root 4096 Dec 22 13:47 

这里能看到，有这么个文件： /data/docker/containers/container id/*-json.log，然后k8s默认会在 /var/log/containers 和 /var/log/pods目录中会生成这些日志文件的软连接，如下所示：

root@hwy-hn1-carsales-k8s-non-prd-002:/var/log/containers# pwd
/var/log/containers
root@hwy-hn1-carsales-k8s-non-prd-002:/var/log/containers# ll
total 68
drwxr-xr-x  2 root root   4096 Dec 22 13:48 ./
drwxrwxr-x 14 root syslog 4096 Dec 23 06:25 ../
lrwxrwxrwx  1 root root     95 Nov 24 23:30 beat-7d969db479-dr7ql_test-ns_filebeat-d4759a3b7c8479cbe6992c354561245554488f5e3c8e4e6c8ab520e2330eb02d.log -> /var/log/pods/test-ns_beat-7d969db479-dr7ql_b91c07f9-bc54-4d04-92ab-c5ad51b76521/filebeat/0.log
lrwxrwxrwx  1 root root    132 Nov 25 17:57 cceaddon-prometheus-node-exporter-wsxbk_monitoring_prometheus-node-exporter-d59cf53ea86ace10a13901d53cb486b24b647f61d473bf6ebfc05b7eefe5a5eb.log -> /var/log/pods/monitoring_cceaddon-prometheus-node-exporter-wsxbk_cf97c646-271e-4927-a0cb-c4e598eec3d7/prometheus-node-exporter/0.log
lrwxrwxrwx  1 root root    107 Jun  4  2021 console-nginx-64775877fd-q4vtl_car-stg_container-0-a66ad1381e335144f8a34bc2b2d77d35151fc20a3d190bfdb3fe650d2d666ee3.log -> /var/log/pods/car-stg_console-nginx-64775877fd-q4vtl_55d64e2c-c9b2-41a4-add2-d766545ff612/container-0/0.log
lrwxrwxrwx  1 root root    111 Jun  9  2021 coredns-59656d8687-6j4m8_kube-system_coredns-bootstrap-f1c05d3aaaeeb841925d1c88aa17de094b2f00bb01c66846a9e428ffc39ba42d.log -> /var/log/pods/kube-system_coredns-59656d8687-6j4m8_56faeca7-c75e-43f0-b0f1-8b6689d7eeae/coredns-bootstrap/0.log
lrwxrwxrwx  1 root root    101 Jun  9  2021 coredns-59656d8687-6j4m8_kube-system_coredns-d2d77af99f8ec093c2f7410440ba9ed0ef380733112306ced6009814554db9da.log -> /var/log/pods/kube-system_coredns-59656d8687-6j4m8_56faeca7-c75e-43f0-b0f1-8b6689d7eeae/coredns/0.log
lrwxrwxrwx  1 root root     97 Nov 23 16:50 css-aly-5f74c45965-pf22z_test-ns_css-aly-61e785bdf54815617168afdfea57818d013a8bb94e0a4f9cd268a07890bee58c.log -> /var/log/pods/test-ns_css-aly-5f74c45965-pf22z_c28d6615-08c9-49c4-a346-a46b1a830b7d/css-aly/0.log
lrwxrwxrwx  1 root root     99 Nov 25 12:39 css-beat-7589645bc4-tfngq_car-dev_filebeat-a59c58937ccd37551ba603514fb9856fbb54a9145b1f897df76edaf266b9bd9e.log -> /var/log/pods/car-dev_css-beat-7589645bc4-tfngq_f65e8727-c1ce-48df-a2a2-87a273d9b6b0/filebeat/0.log
lrwxrwxrwx  1 root root     99 Nov 25 12:39 css-beat-7589645bc4-tfngq_car-dev_java-aly-b41f68f797a46304d66265645d910e44ea54dc3678e2413915c654262c814c49.log -> /var/log/pods/car-dev_css-beat-7589645bc4-tfngq_f65e8727-c1ce-48df-a2a2-87a273d9b6b0/java-aly/0.log
lrwxrwxrwx  1 root root     96 Nov 25 12:39 css-beat-7589645bc4-tfngq_car-dev_nginx-1a97f6689b4766a14bb247e6372d2994d374ecdf261007959559fd6eeb605b17.log -> /var/log/pods/car-dev_css-beat-7589645bc4-tfngq_f65e8727-c1ce-48df-a2a2-87a273d9b6b0/nginx/0.log
lrwxrwxrwx  1 root root    130 Jun  4  2021 everest-csi-controller-9c4bbcb94-2w2df_kube-system_everest-csi-controller-a536b1637c424e068e7cb7fb67515393f9920bb96337c8a8d726b7945fb73388.log -> /var/log/pods/kube-system_everest-csi-controller-9c4bbcb94-2w2df_23ebd20f-6723-436d-b21b-9b6f66a33c23/everest-csi-controller/0.log

然后，会看到这个目录下存在了此宿主机上的所有容器日志，文件的命名方式为：

[podName]_[nameSpace]_[depoymentName]-[containerId].log

上面这个是deployment的命名方式，其他的会有些不同，例如：DaemonSet，StatefulSet等，不过所有的都有一个共同点，就是:

*_[nameSpace]_*.log

到这里，知道这个特性，就可以往下来看Filebeat的部署和配置了。

filebeat部署

总结： 

• 基于daemonset， 用于收集标准输出和标准错误输出的日志，以及所有容器的产生日志。
• 基于sidecar，用于收集pod中的容器的应用程序打的日志，打的日志通过emptyDir共享给sidcar服务。

基于daemonset部署日志收集进程，实现json-file类型(标准输出/dev/stdout、错误输 出/dev/stderr)日志收集，即应用程序产生的标准输出和错误输出的日志。

使用sidecar容器(一个pod多容器)收集当前pod内一个或者多个业务容器的日志(通常基于emptyDir{}实现业务容器与sidcar之间的日志共享)

下载官网的yaml并进行修改，

##
wget https://raw.githubusercontent.com/elastic/beats/master/deploy/kubernetes/filebeat-kubernetes.yaml

##

修改的内容如下:

pod cpu处建议做个限制，否则发生错误可能会占用很大cpu，另外就是删除了output部分，删除了原文的输出到ES，改为kafka 或者logstash

output.kafka:
      enabled: true
      hosts: ["10.10.11.61:9092"]
      topic: 'k8s-uat-log'
      max_message_bytes: 5242880
      partition.round_robin:
        reachable_only: true
      keep-alive: 120
      required_acks: 1