参考文档：

使用 Graylog 和 Prometheus 监视 Kubernetes 集群

Kubernetes documentation

Prometheus2 #22
Grafana #40
Telegraf #41

Graylog #47
Elasticsearch #49
Mongodb #59
Apache2
多节点openstack charms 部署指南0.0.1-36-graylog实际-1

Rsyslog #18
Rsyslog Forwarder Ha #20
多节点OpenStack Charms 部署指南0.0.1.dev223–26–跨model 建立关系构建rsyslog 日志服务器构建实际
多节点OpenStack Charms 部署指南0.0.1.dev223–25–rsyslog 日志服务器构建实际

下面几个章节将涉及使用graylog和prometheus2来部署k8s的日志和监控系统。

为了部署和调试方便，将日志和监控系统分为3个模型：rsyslog、graylog和prometheus2，这样万一哪个部分出了问题，其他部分不会受到大的影响。

另外，其实可以不部署rsyslog部分来转发log，而直接用graylog相关的filebeat来转发log，但还未衡量出哪个优势更大，故两个暂都保留。

部署主备rsyslog服务器

注：可以参照

#增加模型rsyslog
juju add-model rsyslog


#部署组件
juju deploy postgresql pg-a  --series focal --debug
juju deploy rsyslog primary-rsyslog  --series focal --debug
juju deploy rsyslog secondary-rsyslog  --series focal --debug
juju deploy rsyslog-forwarder-ha --series focal --debug
juju deploy filebeat --series focal --debug

#增加关系
juju add-relation rsyslog-forwarder-ha primary-rsyslog
juju add-relation rsyslog-forwarder-ha secondary-rsyslog

#建立rsyslog-forward-ha和组件转发关系
juju add-relation rsyslog-forwarder-ha pg-a
juju add-relation rsyslog-forwarder-ha primary-rsyslog

#建立filebeat和组件的转发关系，用来和以后的graylog模型通讯。
juju add-relation filebeat pg-a
juju add-relation filebeat primary-rsyslog

#提供跨模型rsyslog接口
juju offer primary-rsyslog:aggregator

查看状态：

juju status

输出类似如下：

注：此处输出和上面配置略有不同，但是可以忽略不计

App                   Version  Status   Scale  Charm                 Store       Channel  Rev  OS      Message
filebeat              6.8.16   active       2  filebeat              charmstore            33  ubuntu  Filebeat ready.
pg-a                  12.7     active       1  postgresql            charmstore           233  ubuntu  Live master (12.7)
primary-rsyslog                active       1  rsyslog               charmstore            18  ubuntu  Unit is ready
rsyslog-forwarder-ha           unknown      0  rsyslog-forwarder-ha  charmstore            20  ubuntu

Unit                Workload  Agent  Machine  Public address  Ports     Message
pg-a/0*             active    idle   0/lxd/0  10.0.3.210      5432/tcp  Live master (12.7)
  filebeat/0*       active    idle            10.0.3.210                Filebeat ready.
primary-rsyslog/0*  active    idle   0/lxd/1  10.0.3.209      514/udp   Unit is ready
  filebeat/1        active    idle            10.0.3.209                Filebeat ready.

Machine  State    DNS         Inst id              Series  AZ       Message
0        started  10.0.9.23   vm-157-1             focal   default  Deployed
0/lxd/0  started  10.0.3.210  juju-df4942-0-lxd-0  focal   default  Container started
0/lxd/1  started  10.0.3.209  juju-df4942-0-lxd-1  focal   default  Container started

Offer            Application      Charm    Rev  Connected  Endpoint    Interface  Role
primary-rsyslog  primary-rsyslog  rsyslog  18   1/1        aggregator  syslog     provider

Relation provider           Requirer                     Interface    Type         Message
pg-a:coordinator            pg-a:coordinator             coordinator  peer
pg-a:juju-info              filebeat:beats-host          juju-info    subordinate
pg-a:replication            pg-a:replication             pgpeer       peer
pg-a:syslog                 rsyslog-forwarder-ha:syslog  syslog       regular
primary-rsyslog:aggregator  rsyslog-forwarder-ha:syslog  syslog       regular
primary-rsyslog:juju-info   filebeat:beats-host          juju-info    subordinate

可以看到rsyslog-forward-ha在本模型的组件转发并未生效。生效的只有filebeat。

切换到k8s模型，并配置rsyslog-forward-ha：

juju switch k8s

#部署rsyslog-forwarder-ha
juju deploy rsyslog-forwarder-ha --series focal --debug

#增加欲转发log的应用和rsyslog-forwarder-ha的关系
juju add-relation rsyslog-forwarder-ha easyrsa
juju add-relation rsyslog-forwarder-ha etcd
juju add-relation rsyslog-forwarder-ha kubeapi-load-balancer
juju add-relation rsyslog-forwarder-ha kubernetes-master
juju add-relation rsyslog-forwarder-ha kubernetes-worker

#连接rsyslog模块的接口
 juju consume admin/rsyslog.primary-rsyslog
#建立本模块rsyslog-forwarder-ha和rsyslong模块的primary-rsyslog关系
juju add-relation rsyslog-forwarder-ha primary-rsyslog

显示状态

juju status

输出类似：

Model  Controller       Cloud/Region    Version  SLA          Timestamp
k8s    maas-controller  mymaas/default  2.8.10   unsupported  09:12:13+08:00

SAAS             Status  Store            URL
primary-rsyslog  active  maas-controller  admin/rsyslog.primary-rsyslog
prometheus2      active  maas-controller  admin/prometheus2.prometheus2

App                    Version   Status   Scale  Charm                  Store       Channel  Rev  OS      Message
containerd             go1.13.8  active       5  containerd             charmstore           130  ubuntu  Container runtime available
easyrsa                3.0.1     active       1  easyrsa                local                  0  ubuntu  Certificate Authority connected.
etcd                   3.4.5     active       3  etcd                   charmstore           594  ubuntu  Healthy with 3 known peers
flannel                0.11.0    active       5  flannel                charmstore           558  ubuntu  Flannel subnet 10.1.47.1/24
kubeapi-load-balancer  1.18.0    active       1  kubeapi-load-balancer  charmstore           798  ubuntu  Loadbalancer ready.
kubernetes-master      1.21.1    active       2  kubernetes-master      local                  0  ubuntu  Kubernetes master running.
kubernetes-worker      1.21.1    active       3  kubernetes-worker      charmstore           768  ubuntu  Kubernetes worker running.
rsyslog-forwarder-ha             unknown      9  rsyslog-forwarder-ha   charmstore            20  ubuntu
telegraf                         active       5  telegraf               charmstore            41  ubuntu  Monitoring kubernetes-master/1 (source version/commit dec0633)

Unit                       Workload  Agent  Machine  Public address  Ports           Message
easyrsa/0*                 active    idle   0        10.0.3.189                      Certificate Authority connected.
  rsyslog-forwarder-ha/0*  unknown   idle            10.0.3.189
etcd/0*                    active    idle   1        10.0.3.200      2379/tcp        Healthy with 3 known peers
  rsyslog-forwarder-ha/3   unknown   idle            10.0.3.200
etcd/1                     active    idle   2        10.0.3.201      2379/tcp        Healthy with 3 known peers
  rsyslog-forwarder-ha/2   unknown   idle            10.0.3.201
etcd/2                     active    idle   3        10.0.3.204      2379/tcp        Healthy with 3 known peers
  rsyslog-forwarder-ha/1   unknown   idle            10.0.3.204
kubeapi-load-balancer/0*   active    idle   4        10.0.3.208      443/tcp         Loadbalancer ready.
kubernetes-master/0        active    idle   5        10.0.3.202      6443/tcp        Kubernetes master running.
  containerd/4             active    idle            10.0.3.202                      Container runtime available
  flannel/4                active    idle            10.0.3.202                      Flannel subnet 10.1.22.1/24
  rsyslog-forwarder-ha/5   unknown   idle            10.0.3.202
  telegraf/1               active    idle            10.0.3.202      9103/tcp        Monitoring kubernetes-master/0 (source version/commit dec0633)
kubernetes-master/1*       active    idle   6        10.0.3.207      6443/tcp        Kubernetes master running.
  containerd/2             active    idle            10.0.3.207                      Container runtime available
  flannel/2                active    idle            10.0.3.207                      Flannel subnet 10.1.18.1/24
  rsyslog-forwarder-ha/4   unknown   idle            10.0.3.207
  telegraf/0*              active    idle            10.0.3.207      9103/tcp        Monitoring kubernetes-master/1 (source version/commit dec0633)
kubernetes-worker/0*       active    idle   7        10.0.3.203      80/tcp,443/tcp  Kubernetes worker running.
  containerd/0*            active    idle            10.0.3.203                      Container runtime available
  flannel/0*               active    idle            10.0.3.203                      Flannel subnet 10.1.47.1/24
  rsyslog-forwarder-ha/6   unknown   idle            10.0.3.203
  telegraf/3               active    idle            10.0.3.203      9103/tcp        Monitoring kubernetes-worker/0 (source version/commit dec0633)
kubernetes-worker/1        active    idle   8        10.0.3.206      80/tcp,443/tcp  Kubernetes worker running.
  containerd/3             active    idle            10.0.3.206                      Container runtime available
  flannel/3                active    idle            10.0.3.206                      Flannel subnet 10.1.4.1/24
  rsyslog-forwarder-ha/8   unknown   idle            10.0.3.206
  telegraf/4               active    idle            10.0.3.206      9103/tcp        Monitoring kubernetes-worker/1 (source version/commit dec0633)
kubernetes-worker/2        active    idle   9        10.0.3.205      80/tcp,443/tcp  Kubernetes worker running.
  containerd/1             active    idle            10.0.3.205                      Container runtime available
  flannel/1                active    idle            10.0.3.205                      Flannel subnet 10.1.69.1/24
  rsyslog-forwarder-ha/7   unknown   idle            10.0.3.205
  telegraf/2               active    idle            10.0.3.205      9103/tcp        Monitoring kubernetes-worker/2 (source version/commit dec0633)

Machine  State    DNS         Inst id       Series  AZ       Message
0        started  10.0.3.189  busy-raptor   focal   default  Deployed
1        started  10.0.3.200  crisp-swift   focal   default  Deployed
2        started  10.0.3.201  vital-tick    focal   default  Deployed
3        started  10.0.3.204  stable-dory   focal   default  Deployed
4        started  10.0.3.208  upward-ibex   focal   default  Deployed
5        started  10.0.3.202  ideal-oyster  focal   default  Deployed
6        started  10.0.3.207  safe-goat     focal   default  Deployed
7        started  10.0.3.203  glad-hen      focal   default  Deployed
8        started  10.0.3.206  cool-aphid    focal   default  Deployed
9        started  10.0.3.205  epic-moose    focal   default  Deployed

测试日志转发：

juju ssh etcd/0
logger “testing1”

切换到rsyslog模块

juju switch rsyslog

juju ssh primary-rsyslog/0 
grep testing /var/log/syslog

输出类似：

Apr 15 08:13:46 juju-db6013-0-lxd-10 ubuntu: testing1