EFK 收集 K8S 容器日志


日志在生产环境中非常重要,可以帮助我们排查问题,分析业务问题,分析用户行为等。

对于单机版的集群式应用来说,如果遇到故障我们可以登录上服务器上通过查看日志文件的方式进行查看日志,但对多规模的分布式多节点应用来说,日志分散在多个节点,这显然通过登录上每个节点服务器通过命令查看日志不可取的,就需要统一的日志管理平台收集各各节点上的日志集中管理系统。ELK stack 就是一款日志收集、处理、搜索、分析的成熟的管理系统。

一、ELK简介:

ELK是三个开源软件的缩写,分别为:Elasticsearch 、 Logstash以及Kibana , 它们都是开源软件。不过现在还新增了一个Beats,它是一个轻量级的日志收集处理工具(Agent),Beats占用资源少,适合于在各个服务器上搜集日志后传输给Logstash,官方也推荐此工具,目前由于原本的ELK Stack成员中加入了 Beats 工具所以已改名为Elastic Stack。

Elasticsearch 主要负责日志存储,索引,分片,搜索等

Logstash 主要负责日志的收集,日志处理,是一个重量级的组件。随后就有Fluentd出现,相比它更易用、资源消耗更少、性能更高,在数据处理上更高效可靠,受到企业欢迎。

F:filebeat
F:Fluentd 日志收集代理工具

二、监控方案

日志管理如果手动处理,可以使用kubectl log -n 名称空间 pod名 (kubectl log -f -n 名称空间 pod名)动态查看日志。但是在集群情况下,由于节点数量多,环境复杂,pod的生命周期等特性,是手动查看日志不现实,而使用 elkstack收集日志我们有两种方案:

方案方式
方案一每个容器中集成日志收集工具
方案二每个pod中运行两个容器,一个业务主容器,一个日志收集容器
方案三直接从k8s 宿主机中获取容器日志 【官方推荐】本次测试方案

日志收集工具方案:

  • Logstash 占内存不推荐使用
  • filebeat 轻量15M左右
  • fluentd 每台节点运行一个进程,本次测试方案

三、监控原理

k8s集群中的pod产生的日志(stdout/stderr)都存放在对应node的/var/lib/docker/containers/下,使用fluentd的pod,以deamonset形式运行在各个节点之上收集所有pod的日志
efk on k8s --来源于网络

四、安装方式

k8s是一个 容器的编排平台,而在k8s之上我们运行的是pod。那么EFK这套环境也运行在k8s上面。elasticsearch是日志收集存储的,在k8s之上我们要使用存储卷来保证数据的稳定性,为了保证eslaticsearch的稳定性,我们采用分角色集群(mast/data/client)。fluentd 需要运行在每台node节点(包括master节点)之上,且每个节点只能运行一个服务,我们采用deamonset控制器控制pod。kibana只是eslasticsearch的web客户端,小规模的集群我们使用deployment运行一个副本即可。运行pod我们需要yml格式的资源清单文件,为了方便起见我们使用helm进行安装。

五、安装es

5.1 搜索helm stable仓库中的elasticsearch 版本

helm search xxx

[root@hw-apptest01-11-172 ~]# helm search stable/elasticsearch -l
NAME                         	CHART VERSION	APP VERSION	DESCRIPTION                                                 
stable/elasticsearch         	1.29.0       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.28.5       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.28.4       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.28.2       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.28.1       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.28.0       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.27.3       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.27.2       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.26.2       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.26.1       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.26.0       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.25.0       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.24.0       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.23.0       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.22.0       	6.7.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.21.3       	6.6.2      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.21.2       	6.6.1      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.21.1       	6.6.1      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.21.0       	6.6.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.20.0       	6.6.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.19.1       	6.6.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.19.0       	6.6.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.18.1       	6.6.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.18.0       	6.6.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.17.2       	6.5.4      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.17.1       	6.5.4      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.17.0       	6.5.4      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.16.0       	6.5.4      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.15.5       	6.5.4      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.15.4       	6.5.4      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.15.3       	6.5.4      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.15.2       	6.5.4      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.15.1       	6.5.3      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.15.0       	6.5.1      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.14.3       	6.5.1      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.14.2       	6.5.1      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.14.1       	6.5.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.14.0       	6.5.0      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.13.3       	6.4.3      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.13.2       	6.4.2      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.13.1       	6.4.2      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.13.0       	6.4.2      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.12.0       	6.4.2      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.11.1       	6.4.2      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.11.0       	6.4.2      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch-curator 	1.5.0        	5.5.4      	A Helm chart for Elasticsearch Curator                      
stable/elasticsearch-curator 	1.4.0        	5.5.4      	A Helm chart for Elasticsearch Curator                      
stable/elasticsearch-curator 	1.3.2        	5.5.4      	A Helm chart for Elasticsearch Curator                      
stable/elasticsearch-curator 	1.3.1        	5.5.4      	A Helm chart for Elasticsearch Curator                      
stable/elasticsearch-curator 	1.2.1        	5.5.4      	A Helm chart for Elasticsearch Curator                      
stable/elasticsearch-curator 	1.2.0        	5.5.4      	A Helm chart for Elasticsearch Curator                      
stable/elasticsearch-curator 	1.1.0        	5.5.4      	A Helm chart for Elasticsearch Curator                      
stable/elasticsearch-curator 	1.0.1        	5.5.4      	A Helm chart for Elasticsearch Curator                      
stable/elasticsearch-curator 	1.0.0        	5.5.4      	A Helm chart for Elasticsearch Curator                      
stable/elasticsearch-exporter	1.4.1        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	1.4.0        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	1.3.1        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	1.3.0        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	1.2.0        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	1.1.3        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	1.1.2        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	1.1.1        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	1.1.0        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	1.0.0        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	0.4.1        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	0.4.0        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	0.3.0        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	0.2.2        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	0.2.1        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	0.2.0        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	0.1.4        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	0.1.3        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	0.1.2        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	0.1.1        	1.0.2      	Elasticsearch stats exporter for Prometheus                 
stable/elasticsearch-exporter	0.1.0        	1.0.2      	Elasticsearch stats exporter for Prometheus                 

本次我们使用较新的版本1.28.5,使用的elasticsearch版本为6.7

5.2 获取 1.28.5 elasticsearch

[root@hw-apptest01-11-172 ~]# helm fetch stable/elasticsearch --version=1.28.5

[root@hw-apptest01-11-172 ~]# 
[root@hw-apptest01-11-172 ~]# ls
elasticsearch-1.28.5.tgz 
[root@hw-apptest01-11-172 ~]# tar -xf elasticsearch-1.28.5.tgz 
tar: elasticsearch/Chart.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/NOTES.txt: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/_helpers.tpl: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/client-auth.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/client-deployment.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/client-ingress.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/client-pdb.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/client-serviceaccount.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/client-svc.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/configmap.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/data-pdb.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/data-serviceaccount.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/data-statefulset.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/job.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/master-pdb.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/master-serviceaccount.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/master-statefulset.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/master-svc.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/podsecuritypolicy.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/role.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/rolebinding.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/tests/test-configmap.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/tests/test.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/.helmignore: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/README.md: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/ci/expose-transport-port-on-service-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/ci/extrainitcontainers-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/ci/plugin-initcontainer-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/ci/updatestrategy-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
[root@hw-apptest01-11-172 ~]# cd elasticsearch/
[root@hw-apptest01-11-172 elasticsearch]# ls
Chart.yaml  ci  README.md  templates  values.yaml

5.3 修改values.yaml文件

[root@hw-apptest01-11-172 elasticsearch]# vim values.yaml 

# 修改数据卷,测试期间我们禁用数据卷
[root@hw-apptest01-11-172 elasticsearch]# grep -A1 -Rn persistence values.yaml 
170:  persistence:
171-    enabled: false
--
217:  persistence:
218-    enabled: false


# 安装需求修改jvm参数,由于是测试,我们测试效果即可,jvm调至最小

[root@hw-apptest01-11-172 elasticsearch]# egrep  -Rn  heapSize values.yaml 
121:  heapSize: "256m"
168:  heapSize: "256m"
215:  heapSize: "256m"

# 安装需求,修改 elasticsearch 集群每个角色的副本数。(测试可以每个角色单节点,但为了测试每个角色的机器我们不改变集群的规模)

[root@hw-apptest01-11-172 elasticsearch]# egrep -B 3  -Rn  replicas values.yaml 
109-
110-client:
111-  name: client
112:  replicas: 2
--
164-master:
165-  name: master
166-  exposeHttp: false
167:  replicas: 3
--
211-data:
212-  name: data
213-  exposeHttp: false
214:  replicas: 2

5.4 elasticsearch 集群角色

stable 官方提供的 elasticsearch 集群配置,是将elasticsearch 按角色做集群,在生产环境下,如果不修改elasticsearch节点的角色信息,在高数据量,高并发的场景下集群容易出现脑裂等问题。
默认情况下,elasticsearch集群中每个节点都有成为主节点的资格,也都存储数据,即双重角色。
由两个属性控制:node.master和node.data,默认情况下这两个属性的值都是true,及既是master 又是data
elasticsearch 集群的三个角色:

  • master 负责成为主节点,维护整个集群的状态
  • data 负责存储数据,后期提供存储和查询服务,分片
  • client 海量请求的时候可以进行负载均衡

参考: https://blog.csdn.net/psc0606/article/details/80247662

5.5 elasticsearch 集群安装

下面我们将es集群安装到我们的k8s中,helm install

# efk 是一组组件,负责我们整个k8s集群的日志管理。为了便于后期的管理,我们新建一个efk的名称空间,将efk集群运行于efk名称空间中

[root@hw-apptest01-11-172 elasticsearch]# kubectl create namespace efk
[root@hw-apptest01-11-172 elasticsearch]# kubectl get ns
NAME          STATUS   AGE
default       Active   127d
efk           Active   39d
kube-public   Active   127d
kube-system   Active   127d

# 安装efk 生产release
[root@hw-apptest01-11-172 elasticsearch]# helm install --name=elstest --namespace=efk -f values.yaml stable/elasticsearch --version=1.28.5
NAME:   elstest
LAST DEPLOYED: Wed Jun 26 21:53:35 2019
NAMESPACE: efk
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                        DATA  AGE
elstest-elasticsearch       4     0s
elstest-elasticsearch-test  1     0s

==> v1/Pod(related)
NAME                                           READY  STATUS    RESTARTS  AGE
elstest-elasticsearch-client-5f68dfbddc-9h6ds  0/1    Init:0/1  0         0s
elstest-elasticsearch-client-5f68dfbddc-lqhx2  0/1    Init:0/1  0         0s
elstest-elasticsearch-data-0                   0/1    Init:0/2  0         0s
elstest-elasticsearch-master-0                 0/1    Init:0/2  0         0s

==> v1/Service
NAME                             TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)   AGE
elstest-elasticsearch-client     ClusterIP  10.103.83.183  <none>       9200/TCP  0s
elstest-elasticsearch-discovery  ClusterIP  None           <none>       9300/TCP  0s

==> v1/ServiceAccount
NAME                          SECRETS  AGE
elstest-elasticsearch-client  1        0s
elstest-elasticsearch-data    1        0s
elstest-elasticsearch-master  1        0s

==> v1beta1/Deployment
NAME                          READY  UP-TO-DATE  AVAILABLE  AGE
elstest-elasticsearch-client  0/2    2           0          0s

==> v1beta1/StatefulSet
NAME                          READY  AGE
elstest-elasticsearch-data    0/2    0s
elstest-elasticsearch-master  0/3    0s


NOTES:
The elasticsearch cluster has been installed.

Elasticsearch can be accessed:

  * Within your cluster, at the following DNS name at port 9200:

    elstest-elasticsearch-client.efk.svc

  * From outside the cluster, run these commands in the same shell:

    export POD_NAME=$(kubectl get pods --namespace efk -l "app=elasticsearch,component=client,release=elstest" -o jsonpath="{.items[0].metadata.name}")
    echo "Visit http://127.0.0.1:9200 to use Elasticsearch"
    kubectl port-forward --namespace efk $POD_NAME 9200:9200

5.6 观察 release 的状态,pod的启动过程

pod 的变化过程
Init --> PodInitializing --> Running --> 1/1

[root@hw-apptest01-11-172 elasticsearch]# kubectl get pod -n efk -o wide -w
NAME                                            READY   STATUS     RESTARTS   AGE   IP             NODE                             NOMINATED NODE   READINESS GATES
elstest-elasticsearch-client-5f68dfbddc-9h6ds   0/1     Init:0/1   0          20s   <none>         hw-apptest02-11-174.7dtest.cn    <none>           <none>
elstest-elasticsearch-client-5f68dfbddc-lqhx2   0/1     Running    0          20s   10.244.6.113   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
elstest-elasticsearch-data-0                    0/1     Init:0/2   0          20s   <none>         hw-apptest02-11-174.7dtest.cn    <none>           <none>
elstest-elasticsearch-master-0                  0/1     Init:0/2   0          20s   <none>         hw-apptest02-11-174.7dtest.cn    <none>           <none>


elstest-elasticsearch-data-0   0/1   Init:1/2   0     31s   10.244.1.85   hw-apptest02-11-174.7dtest.cn   <none>   <none>
elstest-elasticsearch-data-0   0/1   PodInitializing   0     32s   10.244.1.85   hw-apptest02-11-174.7dtest.cn   <none>   <none>
elstest-elasticsearch-client-5f68dfbddc-9h6ds   0/1   PodInitializing   0     52s   10.244.1.86   hw-apptest02-11-174.7dtest.cn   <none>   <none>
elstest-elasticsearch-client-5f68dfbddc-9h6ds   0/1   Running   0     53s   10.244.1.86   hw-apptest02-11-174.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-0   0/1   Init:1/2   0     73s   10.244.1.87   hw-apptest02-11-174.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-0   0/1   PodInitializing   0     74s   10.244.1.87   hw-apptest02-11-174.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-0   0/1   Running   0     75s   10.244.1.87   hw-apptest02-11-174.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-0   1/1   Running   0     2m7s   10.244.1.87   hw-apptest02-11-174.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-1   0/1   Pending   0     0s    <none>   <none>   <none>   <none>
elstest-elasticsearch-master-1   0/1   Pending   0     0s    <none>   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-1   0/1   Init:0/2   0     0s    <none>   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-1   0/1   Init:1/2   0     8s    10.244.6.114   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-1   0/1   PodInitializing   0     9s    10.244.6.114   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-1   0/1   Running   0     10s   10.244.6.114   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-client-5f68dfbddc-lqhx2   1/1   Running   0     2m39s   10.244.6.113   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-1   1/1   Running   0     33s   10.244.6.114   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-2   0/1   Pending   0     0s    <none>   <none>   <none>   <none>
elstest-elasticsearch-master-2   0/1   Pending   0     0s    <none>   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-2   0/1   Init:0/2   0     0s    <none>   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-data-0   0/1   Running   0     2m41s   10.244.1.85   hw-apptest02-11-174.7dtest.cn   <none>   <none>
elstest-elasticsearch-client-5f68dfbddc-9h6ds   1/1   Running   0     2m42s   10.244.1.86   hw-apptest02-11-174.7dtest.cn   <none>   <none>
elstest-elasticsearch-data-0   1/1   Running   0     2m45s   10.244.1.85   hw-apptest02-11-174.7dtest.cn   <none>   <none>
elstest-elasticsearch-data-1   0/1   Pending   0     0s    <none>   <none>   <none>   <none>
elstest-elasticsearch-data-1   0/1   Pending   0     0s    <none>   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-data-1   0/1   Init:0/2   0     0s    <none>   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-2   0/1   Init:1/2   0     19s   10.244.6.115   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-2   0/1   PodInitializing   0     20s   10.244.6.115   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-2   0/1   Running   0     21s   10.244.6.115   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-data-1   0/1   Init:0/2   0     33s   10.244.6.116   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-data-1   0/1   Init:1/2   0     34s   10.244.6.116   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-data-1   0/1   PodInitializing   0     35s   10.244.6.116   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-data-1   0/1   Running   0     36s   10.244.6.116   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-master-2   1/1   Running   0     47s   10.244.6.115   hw-apptest03-11-1761.7dtest.cn   <none>   <none>
elstest-elasticsearch-data-1   1/1   Running   0     56s   10.244.6.116   hw-apptest03-11-1761.7dtest.cn   <none>   <none>

^C[root@hw-apptest01-11-172 elasticsearch]# kubectl get pod -n efk -o wide 
NAME                                            READY   STATUS    RESTARTS   AGE     IP             NODE                             NOMINATED NODE   READINESS GATES
elstest-elasticsearch-client-5f68dfbddc-9h6ds   1/1     Running   0          5m12s   10.244.1.86    hw-apptest02-11-174.7dtest.cn    <none>           <none>
elstest-elasticsearch-client-5f68dfbddc-lqhx2   1/1     Running   0          5m12s   10.244.6.113   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
elstest-elasticsearch-data-0                    1/1     Running   0          5m12s   10.244.1.85    hw-apptest02-11-174.7dtest.cn    <none>           <none>
elstest-elasticsearch-data-1                    1/1     Running   0          2m27s   10.244.6.116   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
elstest-elasticsearch-master-0                  1/1     Running   0          5m12s   10.244.1.87    hw-apptest02-11-174.7dtest.cn    <none>           <none>
elstest-elasticsearch-master-1                  1/1     Running   0          3m5s    10.244.6.114   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
elstest-elasticsearch-master-2                  1/1     Running   0          2m32s   10.244.6.115   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
flutest-fluentd-elasticsearch-4llr2             1/1     Running   3675       33d     10.244.0.21    hw-apptest01-11-172.7dtest.cn    <none>           <none>
flutest-fluentd-elasticsearch-j8sv4             1/1     Running   3636       33d     10.244.6.89    hw-apptest03-11-1761.7dtest.cn   <none>           <none>
flutest-fluentd-elasticsearch-xfqzm             1/1     Running   3026       33d     10.244.1.54    hw-apptest02-11-174.7dtest.cn    <none>           <none>

5.7 测试集群状态

5.7.1 获取集群 service 地址
[root@hw-apptest01-11-172 elasticsearch]# kubectl -n efk get svc 
NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
elstest-elasticsearch-client      ClusterIP   10.103.83.183   <none>        9200/TCP    5m41s
elstest-elasticsearch-discovery   ClusterIP   None            <none>        9300/TCP    5m41s
5.7.2 访问 es,查看版本等信息
[root@hw-apptest01-11-172 elasticsearch]# curl 10.103.83.183:9200
{
  "name" : "elstest-elasticsearch-client-5f68dfbddc-lqhx2",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "iM0f_xRDTHKbWokLhfiiDg",
  "version" : {
    "number" : "6.7.0",
    "build_flavor" : "oss",
    "build_type" : "docker",
    "build_hash" : "8453f77",
    "build_date" : "2019-03-21T15:32:29.844721Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
[root@hw-apptest01-11-172 elasticsearch]# curl 10.103.83.183:9200/_cat/node?v
{"error":"Incorrect HTTP method for uri [/_cat/node?v] and method [GET], allowed: [POST]","status":405}[root@hw-apptest01-11-172 elasticsearch]# curl 10.103.83.183:9200/_cat/nodes?v
ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.244.6.115           35          88   5    0.10    0.25     0.18 mi        -      elstest-elasticsearch-master-2
10.244.6.114           45          88   6    0.10    0.25     0.18 mi        -      elstest-elasticsearch-master-1
10.244.1.85            35          39   9    0.01    0.22     0.18 di        -      elstest-elasticsearch-data-0
10.244.1.87            43          39   4    0.01    0.22     0.18 mi        *      elstest-elasticsearch-master-0
10.244.1.86            35          39   7    0.01    0.22     0.18 i         -      elstest-elasticsearch-client-5f68dfbddc-9h6ds
10.244.6.116           36          88   3    0.10    0.25     0.18 di        -      elstest-elasticsearch-data-1
10.244.6.113           37          88   6    0.10    0.25     0.18 i         -      elstest-elasticsearch-client-5f68dfbddc-lqhx2
5.7.3 查看集群健康状态
[root@hw-apptest01-11-172 elasticsearch]# curl 10.103.83.183:9200/_cat/health
1561557688 14:01:28 elasticsearch green 7 2 0 0 0 0 0 0 - 100.0%
[root@hw-apptest01-11-172 elasticsearch]# curl 10.103.83.183:9200/_cat/health?v
epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1561557691 14:01:31  elasticsearch green           7         2      0   0    0    0        0             0                  -                100.0%

六、安装fluentd

6.1 查询fluentd的可用包

[root@hw-apptest01-11-172 elasticsearch]# helm search fluentd
NAME                          	CHART VERSION	APP VERSION	DESCRIPTION                                                 
kiwigrid/fluentd-elasticsearch	4.0.0        	2.5.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd                	1.10.0       	v2.4.0     	A Fluentd Elasticsearch Helm chart for Kubernetes.          
stable/fluentd-elasticsearch  	2.0.7        	2.3.2      	DEPRECATED! - A Fluentd Helm chart for Kubernetes with El...
stable/sumologic-fluentd      	1.0.0        	2.3.0      	Sumologic Log Collector                                     
stable/fluent-bit             	2.0.5        	1.1.2      	Fast and Lightweight Log/Data Forwarder for Linux, BSD an...

[root@hw-apptest01-11-172 elasticsearch]# helm search fluentd-elasticsearch -l
NAME                          	CHART VERSION	APP VERSION	DESCRIPTION                                                 
kiwigrid/fluentd-elasticsearch	4.0.0        	2.5.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	3.0.2        	2.5.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	3.0.1        	2.5.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	3.0.0        	2.5.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.12.0       	2.5.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.11.1       	2.5.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.11.0       	2.5.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.10.1       	2.5.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.10.0       	2.5.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.9.1        	2.5.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.9.0        	2.5.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.8.3        	2.5.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.8.2        	2.5.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.8.1        	2.5.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.8.0        	2.5.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.7.0        	2.4.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.6.0        	2.4.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.5.0        	2.4.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.4.0        	2.4.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.3.3        	2.4.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.3.2        	2.4.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.3.1        	2.4.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.3.0        	2.4.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.2.4        	2.4.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.2.3        	2.4.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.2.2        	2.4.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.2.1        	2.4.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.2.0        	2.4.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.1.2        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.1.1        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch	2.1.0        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	2.0.7        	2.3.2      	DEPRECATED! - A Fluentd Helm chart for Kubernetes with El...
stable/fluentd-elasticsearch  	2.0.6        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	2.0.5        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	2.0.4        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	2.0.3        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	2.0.2        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	2.0.1        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	2.0.0        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	1.5.0        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	1.4.2        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	1.4.1        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	1.4.0        	2.3.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	1.3.0        	2.3.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	1.2.0        	2.3.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	1.1.1        	2.3.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	1.1.0        	2.3.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	1.0.3        	2.3.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	1.0.2        	2.3.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	1.0.1        	2.3.1      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch  	1.0.0        	2.2.0      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...

我们看到 stable 和 kiwigrid 都有 fluentd-elasticsearch 的可用版本,我们使用稳定的版本stable的2.0.7

6.2 fluentd-elasticsearch 获取安装包

[root@hw-apptest01-11-172 ~]# tar -xf fluentd-elasticsearch-2.0.7.tgz 
tar: fluentd-elasticsearch/Chart.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/NOTES.txt: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/_helpers.tpl: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/clusterrole.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/clusterrolebinding.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/configmap.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/daemonset.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/pod-security-policy.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/role.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/rolebinding.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/service-account.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/service.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/OWNERS: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/README.md: implausibly old time stamp 1970-01-01 08:00:00
[root@hw-apptest01-11-172 ~]# cd fluentd-elasticsearch/
[root@hw-apptest01-11-172 fluentd-elasticsearch]# ls
Chart.yaml  OWNERS  README.md  templates  values.yaml

6.3 修改values配置文件

[root@hw-apptest01-11-172 fluentd-elasticsearch]# vim values.yaml 

# 修改镜像地址。默认的镜像地址为google的镜像地址,这个地址国内无法访问
[root@hw-apptest01-11-172 fluentd-elasticsearch]# grep -B2 -Rn repository values.yaml 
1-image:
2:  repository: gcr.azk8s.cn/google-containers/fluentd-elasticsearch
3:  #repository: gcr.io/google-containers/fluentd-elasticsearch

# 修改 elasticsearch 的地址端口
[root@hw-apptest01-11-172 fluentd-elasticsearch]# grep -A2 -Rn elasticsearch: values.yaml 
21:elasticsearch:
22-  host: 'elstest-elasticsearch-client.efk.svc'
23-  port: 9200

# 取消master污点
## 默认情况下,master是有一个污点的,但fluentd是为了收集日志,master的日志也需要手机,我们修改fluentd的容忍度

[root@hw-apptest01-11-172 fluentd-elasticsearch]# grep -A3 -Rn tolerations: values.yaml 
81:tolerations: 
82-   - key: node-role.kubernetes.io/master
83-     operator: Exists
84-     effect: NoSchedule

6.4 部署 fluentd ,生产release

# 部署 fluentd ,生产release
[root@hw-apptest01-11-172 fluentd-elasticsearch]# helm install name=flutest --namespace=efk -f values.yaml ../fluentd-elasticsearch-2.0.7.tgz 
Release "flutest" has been upgraded. Happy Helming!
LAST DEPLOYED: Thu Jun 27 00:06:45 2019
NAMESPACE: efk
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                           AGE
flutest-fluentd-elasticsearch  33d

==> v1/ClusterRoleBinding
NAME                           AGE
flutest-fluentd-elasticsearch  33d

==> v1/ConfigMap
NAME                           DATA  AGE
flutest-fluentd-elasticsearch  6     33d

==> v1/DaemonSet
NAME                           DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
flutest-fluentd-elasticsearch  3        3        1      0           1          <none>         33d

==> v1/Pod(related)
NAME                                 READY  STATUS       RESTARTS  AGE
flutest-fluentd-elasticsearch-67hfd  1/1    Running      7         90m
flutest-fluentd-elasticsearch-g9m47  0/1    Terminating  12        48m
flutest-fluentd-elasticsearch-qhfh8  0/1    Terminating  14        49m

==> v1/ServiceAccount
NAME                           SECRETS  AGE
flutest-fluentd-elasticsearch  1        33d


NOTES:
1. To verify that Fluentd has started, run:

  kubectl --namespace=efk get pods -l "app.kubernetes.io/name=fluentd-elasticsearch,app.kubernetes.io/instance=flutest"

THIS APPLICATION CAPTURES ALL CONSOLE OUTPUT AND FORWARDS IT TO elasticsearch . Anything that might be identifying,
including things like IP addresses, container images, and object names will NOT be anonymized.


[root@hw-apptest01-11-172 fluentd-elasticsearch]# kubectl -n efk get pod -w
NAME                                            READY   STATUS              RESTARTS   AGE
elstest-elasticsearch-client-5f68dfbddc-9h6ds   1/1     Running             0          133m
elstest-elasticsearch-client-5f68dfbddc-lqhx2   1/1     Running             0          133m
elstest-elasticsearch-data-0                    1/1     Running             0          133m
elstest-elasticsearch-data-0                    1/1     Running             0          133m
elstest-elasticsearch-data-1                    1/1     Running             0          130m
elstest-elasticsearch-master-0                  1/1     Running             0          133m
elstest-elasticsearch-master-1                  1/1     Running             0          131m
elstest-elasticsearch-master-2                  1/1     Running             0          130m
flutest-fluentd-elasticsearch-67hfd             1/1     Running             7          91m
flutest-fluentd-elasticsearch-7rph9             0/1     ContainerCreating   0          7s
flutest-fluentd-elasticsearch-rbjrx             0/1     ContainerCreating   0          7s
kibtest-kibana-7c4c7cd9df-whqtz                 1/1     Running             0          118m

^[[A



^C[root@hw-apptest01-11-172 fluentd-elasticsearch]# kubectl -n efk get pod -o wide -w
NAME                                            READY   STATUS              RESTARTS   AGE    IP             NODE                             NOMINATED NODE   READINESS GATES
elstest-elasticsearch-client-5f68dfbddc-9h6ds   1/1     Running             0          133m   10.244.1.86    hw-apptest02-11-174.7dtest.cn    <none>           <none>
elstest-elasticsearch-client-5f68dfbddc-lqhx2   1/1     Running             0          133m   10.244.6.113   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
elstest-elasticsearch-data-0                    1/1     Running             0          133m   10.244.1.85    hw-apptest02-11-174.7dtest.cn    <none>           <none>
elstest-elasticsearch-data-1                    1/1     Running             0          131m   10.244.6.116   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
elstest-elasticsearch-master-0                  1/1     Running             0          133m   10.244.1.87    hw-apptest02-11-174.7dtest.cn    <none>           <none>
elstest-elasticsearch-master-1                  1/1     Running             0          131m   10.244.6.114   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
elstest-elasticsearch-master-2                  1/1     Running             0          131m   10.244.6.115   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
flutest-fluentd-elasticsearch-7rph9             1/1     Running             0          35s    10.244.6.121   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
flutest-fluentd-elasticsearch-rbjrx             1/1     Running             0          35s    10.244.1.92    hw-apptest02-11-174.7dtest.cn    <none>           <none>
flutest-fluentd-elasticsearch-wkj24             0/1     ContainerCreating   0          6s     <none>         hw-apptest01-11-172.7dtest.cn    <none>           <none>
kibtest-kibana-7c4c7cd9df-whqtz                 1/1     Running             0          118m   10.244.1.88    hw-apptest02-11-174.7dtest.cn    <none>           <none>

6.5 测试es是否收集到数据

[root@hw-apptest01-11-172 fluentd-elasticsearch]# curl 10.103.83.183:9200/_cat/indices?v
health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   logstash-2019.06.25          bEYYLdDiSMaq-C3iMXsl0g   5   1       5577            0      3.9mb            2mb
green  open   logstash-2019.05.04          ZiWVSgV_TTKduZa3qmyQqA   5   1        124            0    510.1kb        291.7kb
green  open   logstash-2019.06.23          V4R9twEfTry-T5vazFAX4Q   5   1       7644            0      5.8mb            3mb
green  open   logstash-2019.06.11          SdS9nrATTwmZ8E3yi83_nw   5   1          2            0     29.4kb           460b
green  open   logstash-2019.06.12          aktDQ6IwTnyd78TkEeHuGQ   5   1         17            0     47.7kb           460b
green  open   logstash-2019.05.30          HYXFwTHPRsCx7V-QJ-FBNw   5   1        164            0    412.1kb        227.1kb
green  open   logstash-2019.06.06          umcCJRKFS9CnN-zD0eeL6A   5   1        522            0    802.3kb          422kb
green  open   logstash-2019.06.24          65DW7sVVS5qo17vEmxib6A   5   1      10212            0      6.8mb          3.6mb
green  open   logstash-2019.06.22          Rg6mACLBQlWjVFkcpmd83A   5   1       1872            0      1.7mb        904.1kb
yellow open   logstash-2019.06.26          NLNGeADgT4avG7khGocqfg   5   1          0            0       460b           460b

七、安装kibana

7.1 查询 kibana 的可用包

[root@hw-apptest01-11-172 elasticsearch]# helm search kibana
NAME          	CHART VERSION	APP VERSION	DESCRIPTION                                                 
elastic/kibana	7.1.1        	7.1.1      	Kibana                                                      
local/kibana  	3.2.0        	6.7.0      	Kibana is an open source data visualization plugin for El...
stable/kibana 	3.2.0        	6.7.0      	Kibana is an open source data visualization plugin for El...
[root@hw-apptest01-11-172 elasticsearch]# helm search stable/kibana -l
NAME         	CHART VERSION	APP VERSION	DESCRIPTION                                                 
stable/kibana	3.2.0        	6.7.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	3.1.1        	6.7.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	3.1.0        	6.7.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	3.0.0        	6.7.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	2.3.1        	6.7.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	2.3.0        	6.7.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	2.2.0        	6.7.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	2.1.0        	6.7.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	2.0.0        	6.6.1      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.6.0        	6.6.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.5.2        	6.6.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.5.1        	6.6.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.5.0        	6.6.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.4.1        	6.6.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.4.0        	6.6.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.3.0        	6.6.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.2.2        	6.5.4      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.2.1        	6.5.4      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.2.0        	6.5.4      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.1.2        	6.5.4      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.1.1        	6.5.3      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.1.0        	6.5.3      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.0.3        	6.5.3      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.0.2        	6.5.1      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.0.1        	6.5.1      	Kibana is an open source data visualization plugin for El...
stable/kibana	1.0.0        	6.5.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.20.0       	6.5.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.19.0       	6.5.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.18.0       	6.4.3      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.17.1       	6.4.3      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.17.0       	6.4.2      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.16.4       	6.4.2      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.16.3       	6.4.2      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.16.2       	6.4.2      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.16.1       	6.4.2      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.16.0       	6.4.2      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.15.0       	6.4.2      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.14.8       	6.4.2      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.14.7       	6.4.2      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.14.6       	6.4.1      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.14.5       	6.4.1      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.14.4       	6.4.1      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.14.3       	6.4.1      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.14.2       	6.4.1      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.14.1       	6.4.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.13.1       	6.4.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.13.0       	6.4.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.12.1       	6.3.2      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.12.0       	6.3.2      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.11.0       	6.3.2      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.10.1       	6.3.1      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.10.0       	6.3.1      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.9.0        	6.3.1      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.8.0        	6.3.1      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.7.0        	6.3.1      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.6.0        	6.0.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.5.0        	6.0.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.4.1        	6.0.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.4.0        	6.0.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.3.0        	6.0.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.2.3        	6.0.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.2.2        	6.0.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.2.1        	6.0.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.2.0        	6.0.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.1.1        	5.4.3      	Kibana is an open source data visualization plugin for El...
stable/kibana	0.1.0        	5.4.3      	Kibana is an open source data visualization plugin for El...

7.2 获取 kibana 安装包

[root@hw-apptest01-11-172 ~]# helm fetch stable/kibana --version=3.2.0

[root@hw-apptest01-11-172 ~]# tar -xf kibana-3.2.0.tgz 
tar: kibana/Chart.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/NOTES.txt: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/_helpers.tpl: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/configmap-dashboardimport.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/configmap.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/deployment.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/ingress.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/service.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/serviceaccount.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/tests/test-configmap.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/tests/test.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/volume-claim.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/.helmignore: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/OWNERS: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/README.md: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/authproxy-enabled.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/dashboard-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/extra-configmap-mounts.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/ingress-hosts-paths.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/ingress-hosts.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/initcontainers-all-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/initcontainers-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/plugin-install.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/pvc.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/security-context.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/service-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/url_dashboard-values.yaml: implausibly old time stamp 1970-01-01 08:00:00

[root@hw-apptest01-11-172 ~]# cd kibana/
[root@hw-apptest01-11-172 kibana]# ls
Chart.yaml  ci  OWNERS  README.md  templates  values.yaml

7.3 修改 values 配置文件

[root@hw-apptest01-11-172 kibana]# vim values.yaml 

# 修改 kibana 连接elasticsearch地址
[root@hw-apptest01-11-172 kibana]# grep -Rn -A5  kibana.yml values.yaml 
24:  kibana.yml:
25-    ## Default Kibana configuration from kibana-docker.
26-    server.name: kibana
27-    server.host: "0"
28-    ## For kibana < 6.6, use elasticsearch.url instead
29-    elasticsearch.hosts: http://elstest-elasticsearch-client.efk.svc:9200

# 修改 service  的类型为NodePort ,方便我们外网访问
[root@hw-apptest01-11-172 kibana]# grep -Rn -A5  ^service: values.yaml 
40:service:
41-  type: NodePort
42-  # clusterIP: None
43-  # portName: kibana-svc
44-  externalPort: 443
45-  internalPort: 5601

7.4 安装 kibana 生产release

[root@hw-apptest01-11-172 kibana]# helm install --name=kibtest --namespace=efk -f values.yaml stable/kibana --version=3.2.0
NAME:   kibtest
LAST DEPLOYED: Wed Jun 26 22:08:35 2019
NAMESPACE: efk
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                 DATA  AGE
kibtest-kibana       1     0s
kibtest-kibana-test  1     0s

==> v1/Pod(related)
NAME                             READY  STATUS             RESTARTS  AGE
kibtest-kibana-7c4c7cd9df-whqtz  0/1    ContainerCreating  0         0s

==> v1/Service
NAME            TYPE      CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
kibtest-kibana  NodePort  10.103.27.234  <none>       443:30517/TCP  0s

==> v1beta1/Deployment
NAME            READY  UP-TO-DATE  AVAILABLE  AGE
kibtest-kibana  0/1    1           0          0s


NOTES:
To verify that kibtest-kibana has started, run:

  kubectl --namespace=efk get pods -l "app=kibana"

Kibana can be accessed:

  * From outside the cluster, run these commands in the same shell:

    export NODE_PORT=$(kubectl get --namespace efk -o jsonpath="{.spec.ports[0].nodePort}" services kibtest-kibana)
    export NODE_IP=$(kubectl get nodes --namespace efk -o jsonpath="{.items[0].status.addresses[0].address}")
    echo http://$NODE_IP:$NODE_PORT

7.5 查看kibana的启动过程

[root@hw-apptest01-11-172 kibana]# kubectl get pod -n efk -o wide -w
NAME                                            READY   STATUS    RESTARTS   AGE   IP             NODE                             NOMINATED NODE   READINESS GATES
elstest-elasticsearch-client-5f68dfbddc-9h6ds   1/1     Running   0          15m   10.244.1.86    hw-apptest02-11-174.7dtest.cn    <none>           <none>
elstest-elasticsearch-client-5f68dfbddc-lqhx2   1/1     Running   0          15m   10.244.6.113   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
elstest-elasticsearch-data-0                    1/1     Running   0          15m   10.244.1.85    hw-apptest02-11-174.7dtest.cn    <none>           <none>
elstest-elasticsearch-data-1                    1/1     Running   0          12m   10.244.6.116   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
elstest-elasticsearch-master-0                  1/1     Running   0          15m   10.244.1.87    hw-apptest02-11-174.7dtest.cn    <none>           <none>
elstest-elasticsearch-master-1                  1/1     Running   0          13m   10.244.6.114   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
elstest-elasticsearch-master-2                  1/1     Running   0          12m   10.244.6.115   hw-apptest03-11-1761.7dtest.cn   <none>           <none>
flutest-fluentd-elasticsearch-4llr2             1/1     Running   3676       33d   10.244.0.21    hw-apptest01-11-172.7dtest.cn    <none>           <none>
flutest-fluentd-elasticsearch-j8sv4             1/1     Running   3637       33d   10.244.6.89    hw-apptest03-11-1761.7dtest.cn   <none>           <none>
flutest-fluentd-elasticsearch-xfqzm             1/1     Running   3027       33d   10.244.1.54    hw-apptest02-11-174.7dtest.cn    <none>           <none>
kibtest-kibana-7c4c7cd9df-whqtz                 1/1     Running   0          22s   10.244.1.88    hw-apptest02-11-174.7dtest.cn    <none>           <none>

7.6 查看 kibana service,访问测试

[root@hw-apptest01-11-172 kibana]# kubectl -n efk get svc 
NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
elstest-elasticsearch-client      ClusterIP   10.103.83.183   <none>        9200/TCP        15m
elstest-elasticsearch-discovery   ClusterIP   None            <none>        9300/TCP        15m
flutest-fluentd-elasticsearch     ClusterIP   10.96.212.237   <none>        24231/TCP       33d
kibtest-kibana                    NodePort    10.103.27.234   <none>        443:30517/TCP   41s

八、安装过程中的坑

8.1 helm 仓库地址

问题:

默认情况下的 helm stable版本是google的地址,google的地址再国内无法访问,或时而无法访问。这钟情况下helm仓库使用的第一步就卡死了。
https://kubernetes-charts.storage.googleapis.com/

解决方案:
  1. 阿里云helm仓库代理
    地址:https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts/
    已经好久没有更新过了

  2. 翻墙
    linux 使用vpn 或ss

  3. 内部仓库–git
    github官方地址: https://github.com/helm/charts.git

  4. 本地文件–git [详情加helm 进阶文档]

helm serve --address 0.0.0.0:8080 --repo-path /app/dcos/chart-repo
helm repo index /app/dcos/chart-repo
  1. 第三方代理
    代理地址: https://burdenbear.github.io/kube-charts-mirror/
    代理工具地址: https://github.com/taowujie/kube-charts-mirror

8.2 访问国外gcr docker镜像

问题:

gcr docker镜像 是google的docker仓库镜像,国内无法访问。无论是后期软件 合适kubeadm安装kubernetes 集群都是一个卡点。

解决方案:
  1. 方案一 国内加速镜像 https://github.com/anjia0532/gcr.io_mirror
 gcr.io/namespace/image_name:image_tag 
 #eq
 gcr.azk8s.cn/namespace/image_name:image_tag 
 #替换方式---> 微软云国内加速镜像
  1. 方案二 自己通过 Dockerfile 制作镜像 利用dockerhub生成
    https://blog.csdn.net/weixin_39961559/article/details/80739352
    https://blog.csdn.net/qq_27028561/article/details/79064414
    https://blog.csdn.net/nklinsirui/article/details/80581286

  2. 方案三 vpn 代理

  3. 方案四 使用第三方镜像 安全性无法保障
    docker search xx

8.3 kibana 无法启动

查看日志:

"message":"child \"elasticsearch\" fails because [\"hosts\" is not allowed]"

官方charts 文件bug,版本es 6.5 版本

url链接:
https://github.com/helm/charts/issues/14062
https://github.com/elastic/kibana/issues/32303

8.4 kibana 不兼容 es

问题:

Kibana server is not ready yet

排查思路
  1. kibana版本要和elasticsearch版本相同
  2. elasticsearch 集群状态不正常。
    使用kubectl log xxpod名查看日志

8.5 fluentd 无法收集日志到es中

启动正常就是es上面没有日志索引。
原因:
  fluentd 的镜像保存在grc docker 仓库中,国内无法访问。起初使用的是第三方镜像,是版本不兼容造成的。

九、优缺点

Kubernetes官方提供了EFK的日志收集解决方案,但是这种方案并不适合所有的业务场景,它本身就有一些局限性,例如:

所有日志都必须是out前台输出,真实业务场景中无法保证所有日志都在前台输出
只能有一个日志输出文件,而真实业务场景中往往有多个日志输出文件
Fluentd并不是常用的日志收集工具,我们更习惯用logstash,现使用filebeat替代
我们已经有自己的ELK集群且有专人维护,没有必要再在kubernetes上做一个日志收集服务
基于以上几个原因,我们决定使用自己的ELK集群。
编号方案优点缺点
1每个app的镜像中都集成日志收集组件部署方便,kubernetes的yaml文件无须特别配置,可以为每个app自定义日志收集配置强耦合,不方便应用和日志收集组件升级和维护且会导致镜像过大
2单独创建一个日志收集组件跟app的容器一起运行在同一个pod中低耦合,扩展性强,方便维护和升级需要对kubernetes的yaml文件进行单独配置,略显繁琐
3将所有的Pod的日志都挂载到宿主机上,每台主机上单独起一个日志收集Pod完全解耦,性能最高,管理起来最方便需要统一日志收集规则,目录和输出方式
Logo

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐