K8S/Kubernetes

正确姿势启动metricbeat

metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - filesystem
    - memory
    - network
    - process
  enabled: true
  period: 10s
  processes: ['.*']
  cpu_ticks: false
  
output.elasticsearch:
    hosts: ["http://192.168.x.x:9200"]

setup.template.name: "metricbeat"
setup.template.fields: "fields.yml"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1

setup.kibana.host: "192.168.x.x:5601"
setup.dashboards.enabled: true

./metricbeat -e -c metricbeat.yml -d "publish"

metricbeat的Kubernetes插件

metricbeat.modules:
# Node metrics, from kubelet:
- module: kubernetes
  metricsets:
    - node
    - system
    - pod
    - container
    - volume
  period: 10s
  hosts: ["192.168.x.132:10255","192.168.x.133:10255"]

# State metrics from kube-state-metrics service:  #这部分需要kube-state-metrics支撑(Nodes/deploy个数),因此集群里需要安装kube-state-metrics组件.默认情况下这里没有的话图中上半部分没有数据
- module: kubernetes
  enabled: false  ## 这里一定要是true
  metricsets:
    - state_node
    - state_deployment
    - state_replicaset
    - state_pod
    - state_container
  period: 10s
  hosts: ["192.168.x.x:8080"]

# Kubernetes events
- module: kubernetes
  enabled: false
  metricsets:
    - event
    
output.elasticsearch:
    hosts: ["http://192.168.x.x:9200"]

既然需要kube-state-metrics了,那就研究下吧.

监控思路

• vm思路:遵从大道至简的原则
  

• 容器监控思路:
  pod的:
  top cpu
  top mem

• 别人家的监控-只能流口水了
  

kube-state-metrics提供监控k8s集群的metric

kube-state-metrics本质上是提供了k8s集群的metric
部署参考:
https://github.com/kubernetes/kube-state-metrics

克隆代码: create这几个即可

访问:

报错解决:
kubectl -n kube-system logs -f kube-state-metrics-215312065-1vzg2 -c kube-state-metrics
时候发现一些api错误

解决: 给apiserver启动添加这个参数: 它github上有说

--runtime-config=batch/v2alpha1=true

即目前完整的apiserver参数如下

 kube-apiserver     --service-cluster-ip-range=10.254.0.0/16     --etcd-servers=http://127.0.0.1:2379     --insecure-bind-address=0.0.0.0     --admission-control=ServiceAccount     --service-account-key-file=/root/ssl/ca.key --client-ca-file=/root/ssl/ca.crt    --tls-cert-file=/root/ssl/server.crt --tls-private-key-file=/root/ssl/server.key --allow-privileged=true --runtime-config=batch/v2alpha1=true --v=2

k8s插件监控集群效果图

vm监控效果图

贴上kube-metric的yaml

参考: https://github.com/kubernetes/kube-state-metrics

修改点:

• deploy的api版本

• svc为nodeport模式

• 镜像拉取

docker pull lanny/gcr.io_google_containers_kube-state-metrics_v1.1.0
dokcer pull lanny/gcr.io_google_containers_addon-resizer_1.0
docker tag lanny/gcr.io_google_containers_kube-state-metrics_v1.1.0 gcr.io/google_containers/kube-state-metrics:v1.1.0 
docker tag lanny/gcr.io_google_containers_addon-resizer_1.0 gcr.io/google_containers/addon-resizer:1.0 

$ cat kube-state-metrics-deployment.yaml 
apiVersion: apps/v1beta1 # for versions before 1.8.0 use apps/v1beta1  ## 这里默认是v1beat1,我用的是1.7.8,因此apiVersion: apps/v1beta2是不适用的.
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kube-state-metrics
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: quay.io/coreos/kube-state-metrics:v1.1.0
        ports:
        - name: http-metrics
          containerPort: 8080
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
      - name: addon-resizer
        image: gcr.io/google_containers/addon-resizer:1.0
        resources:
          limits:
            cpu: 100m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 30Mi
        env:
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        command:
          - /pod_nanny
          - --container=kube-state-metrics
          - --cpu=100m
          - --extra-cpu=1m
          - --memory=100Mi
          - --extra-memory=2Mi
          - --threshold=5
          - --deployment=kube-state-metrics

$ cat kube-state-metrics-service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    k8s-app: kube-state-metrics
  annotations:
    prometheus.io/scrape: 'true'
spec:
  type: NodePort ## 这里需要改下,外界通过nodeport方式访问metric api
  ports:
  - name: http-metrics
    port: 8080
    targetPort: http-metrics
    protocol: TCP
  selector:
    k8s-app: kube-state-metrics

$ cat kube-state-metrics-service-account.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: kube-system