K8S笔记

k8s学习k8s功能

月光光照大江

1357人浏览 · 2022-05-29 20:39:04

月光光照大江 · 2022-05-29 20:39:04 发布

k8s图谱
k8s功能
k8s架构

4. k8s架构部署

服务器准备

kubeadm 安装 kubernetes 集群要求最少2核，操作系统采用 CentOS-7.6.1810，建议采用 7.6.1810 及以上的版本
建议实验环境配置如下：4核心，8GB内存，100G硬盘空间(50G系统盘，/var/lib/docker 50G,50G用于 Ceph-RBD )
服务器 IP 地址及主机名规划如下：
master1, worker1, worker2 既是 master1 同时也是 node 节点

IP	Role	CPU	内存	数据盘
192.168.47.128	master1	2	2G	100G
192.168.47.129	worker1	2	2G	100G
192.168.47.130	worker2	2	2G	100G

2. 用到的组建版本预览

软件	版本	发布日期
kubernetes	v1.22.4	2021
docker-ce	20.10.10	2021
etcd	3.5.0-0	2021-06-16
coredns	v1.8.4	2021-05-28
calico	v3.21.0	2021-07-31
dashboard	v2.3.1	2021-06-16
ingress-nginx	v1.0.0	2021-08-24
metrics-server	v0.5.0	2021-05-28
prometheus	v2.26.0	2021-03-31
grafana	7.5.4	2021-04-14
istio	1.11.1	2021-08-25

3. 环境搭建安装好vmware软件构建虚拟机装好centos7.8系统

4. 安装之前需要对环境参数进行配置

4.1 关闭 selinux 和防火墙

关闭防火墙

systemctl stop firewalld // 关闭服务
systemctl disable firewalld //关闭开机启动服务
firewall-cmd --state//测试

关闭 selinux 重启

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config  // 修改
getenforce // 测试

4.2 主机名

# master1
hostnamectl set-hostname master1
# worker1
hostnamectl set-hostname worker1
# worker2
hostnamectl set-hostname worker2

4.2 主机IP和域名配置

查询IP和子网掩码：ip a s

查询DNS1： cat /etc/resolv.conf

查询默认网关: route -n (可能要安装route命令工具 yum -y install net-tools)

配置IP：vi /etc/sysconfig/network-scripts/ifcfg-ens33

# master
...
BOOTPROTO=none
IPADDR=192.168.47.128
NETMASK=255.255.255.0
GATEWAY=192.168.47.2
DNS1=119.29.29.29

配置域名：vi /etc/hosts

192.168.47.128 master1
192.168.47.129 worker1
192.168.47.130 worker2

4.3 配置时间同步定时任务(同步阿里云时间服务器)

yum -y install ntpdate

crontab -e 0 */5 * ** netdate ntp.aliyun.com

4.4 禁用 swap

swapoff -a
sed -i '/swap/s/^\(.*\)$/#\1/g' /etc/fstab
free -m // 测试是否分配虚拟内存

4.5 配置三台机器免密钥登陆（每台机器都有全部的公钥）

ssh-keygen -t rsa -b 2048 -P '' -f ~/.ssh/id_rsa
ssh-copy-id k8s-m1
ssh-copy-id k8s-m2
ssh-copy-id k8s-m3

4.6 配置内核参数网桥过滤功能将桥接的IPv4流量传递到iptables的链

4.7 部署ipvs 效率要比iptables高

安装ipset、ipvsadm

yum -y install ipset ipvsadm

运行脚本

#!/bin/bash
varr ='ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4'

for mod in varr
do
  modprobe $mod
done

4.8 ssh 连接优化

sed -ri 's/^#(UseDNS )yes/\1no/' /etc/ssh/sshd_config

5. 使用kubeadm来部署

安装docker-ce 看runoot手册（一般使用清华源）

安装完后查看docker启动配置

vi /usr/lib/systemd/system/docker.service

删掉ExecStart=/usr/bin/dockerd后面有-H（包括）的数据

配置daemon.json

配置docker镜像加速以及日志相关配置

Kubernetes官方推荐使用cgroup driver为systemd

mkdir /etc/docker
tee /etc/docker/daemon.json <<EOF
{
  "registry-mirrors": [
    "https://mciwm180.mirror.aliyuncs.com",
    "https://docker.mirrors.ustc.edu.cn/",
    "https://registry.docker-cn.com"
  ],
  "log-driver": "json-file",
  "log-opts": {
    "max-file": "10",
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

systemctl enable --now docker

2.安装 kubeadm、kubelet、kubectl（这里安装之前最好也配置阿里云的镜像库，）

添加k8s的阿里源（这里可以不配置，如果安装不了再配置）：

[root@master01 ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF
之后安装：

yum -y install kubeadm-1.22.4 kubectl-1.22.4 kubelet-1.22.4

配置 kubelet

指定使用 cgroup driver 为 systemd

tee /etc/sysconfig/kubelet <<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"
EOF

设置 kubelet 为开机启动

systemctl daemon-reload
systemctl enable kubelet

3. 拉取k8s的镜像，这里需要你有阿里云的账号，先登录，然后才能拉取（同时修改docker镜像加速器为阿里云）

首先查看需要的k8s镜像信息

kubeadm config images list

由于k8s默认是谷歌，国内网访问不了，所以我们用阿里

我们拉取k8s镜像时直接改为阿里的镜像地址

拉取完成之后再修改镜像的tag使其与kubeadm config images list结果名称保持一致

4. k8s初始化

再master1上初始化

kubeadm init --kubernetes-version=v1.22.4 --pod-network-cidr=172.16.0.0/16 --apiserver-advertise-address=192.168.47.128

修改拷贝文件

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

配置安装calico(配置pod-network)

网络插件有：flannel、calico、canal、kube-router、weave net等，这里使用calico网络插件

下载yaml

curl https://docs.projectcalico.org/manifests/calico.yaml -O

修改yaml文件添加并修改 eth->ens

在执行calico文件执之前最好手动下载镜像配置好阿里云加速器并使用命令行查询calico需要的镜像：

最后执行文件安装：

kubectl apply -f calico.yaml

初始化最后子节点加入- 分别再worker1、worker2执行

kubeadm join 192.168.47.128:6443 --token e2qxrq.8ijfcx3muenl7sh7 \
--discovery-token-ca-cert-hash sha256:c530e95aa852fd21fd91b2b975f83d355fe5fa40b7eb8d32cc3f684c3d081b81

5. 测试查看集群状态

kubectl get nodes

kubectl get cs

kubectl get cluster-info

kubectl get pods -n kube-system

创建一个 deployment 和 service 进行测试
kubectl create deployment nginx --image=nginx:alpine
kubectl expose deployment/nginx --name=nginx-svc --port=80 --type=NodePort
kubectl get po
Copy
输出如下

NAME READY STATUS RESTARTS AGE
nginx-565785f75c-rr8vh 0/1 Pending 0 88s
Copy
查看描述

kubectl describe po nginx-565785f75c-rr8vh
0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
Copy
因为 master 节点同时当 node 节点用，需要把 master 标签和污点去掉，默认 master 无法调度

去除 master 标签

kubectl label node k8s-m1 node-role.kubernetes.io/master-
kubectl label node k8s-m2 node-role.kubernetes.io/master-
kubectl label node k8s-m3 node-role.kubernetes.io/master-
Copy
去除污点

kubectl taint node k8s-m1 node-role.kubernetes.io/master:NoSchedule-
kubectl taint node k8s-m2 node-role.kubernetes.io/master:NoSchedule-
kubectl taint node k8s-m3 node-role.kubernetes.io/master:NoSchedule-
Copy
再次检查，已经正常调度到 master 节点了

kubectl get po
NAME READY STATUS RESTARTS AGE
nginx-565785f75c-rr8vh 0/1 ContainerCreating 0 3m55s
Copy
查看 pod 和 service
kubectl get svc -l app=nginx
Copy
输出如下

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-svc NodePort 10.110.108.4 <none> 80:31026/TCP 4m34s
Copy
访问测试
curl 10.110.108.4
Copy
输出如下

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

看到上面的，说明已经正常使用 ipvs 模式了。

5. kubectl 的运用

6. 使用yaml文件来管理k8s集群资源管理

7. namespace学习（c r d）

8. pod学习

9. Controller学习

10. Service学习

11. k8s部署完成之后需要安装核心组件 ingress/metrics-server/dashboard/部署 EFK 日志服务/prometheus+granfa 这其中ingress是

11-1 ingress裸机安装

1.部署metallb

kubectl get configmap kube-proxy -n kube-system -o yaml | \
sed -e "s/strictARP: false/strictARP: true/" | \
kub

2. 创建 namespace

这里要注意 raw.githubusercontent.com隔段时间就会被污染，所以需要查询新的ip在hosts中配置好这实在是令人骂人的问题域名解析地址：www.ipaddress.com

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/namespace.yaml

3. 部署 metallb

这里要多试几次才行，注意一下哦

wget https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/metallb.yaml

# 测试发现中科大的并没有加速效果
#sed -i 's@quay.io@quay.mirrors.ustc.edu.cn@g' metallb.yaml
kubectl apply -f metallb.yaml

4. 支持二层，BGP 等方式，这里简单的使用二层配置为 matallb 创建 cm

tee metallb-config.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.147.150-192.168.47.250
EOF

kubectl apply -f metallb-config.yaml

kubectl get po -n metallb-system

5. 部署 ingress-nginx

curl -o ingress-nginx.yaml \
  https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.0/deploy/static/provider/baremetal/deploy.yaml

sed -i 's@k8s.gcr.io/ingress-nginx/controller:v1.0.0\(.*\)@willdockerhub/ingress-nginx-controller:v1.0.0@' ingress-nginx.yaml
sed -i 's@k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0\(.*\)$@hzde0128/kube-webhook-certgen:v1.0@' \
  ingress-nginx.yaml

kubectl apply -f ingress-nginx.yaml

kubectl get po -n ingress-nginx
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create--1-k9b75     0/1     Completed   0          14s
ingress-nginx-admission-patch--1-jsrsj      0/1     Completed   0          14s
ingress-nginx-controller-79887d48bf-txxvd   0/1     Running     0          15s


// 如果出现问题可以插叙日志
kubectl describe pod ingress-nginx-controller-6b64bc6f47-294jr  --namespace=ingress-nginx

6.部署了 Metallb 的可以将 NodePort 修改为 LoadBalancer

kubectl patch svc -n ingress-nginx ingress-nginx-controller  -p '{"spec":{"type": "LoadBalancer"}}'
service/ingress-nginx-controller patched

检查安装情况

kubectl get po -n ingress-nginx
NAME                                        READY   STATUS      RESTARTS     AGE
ingress-nginx-admission-create--1-k9b75     0/1     Completed   0            72s
ingress-nginx-admission-patch--1-jsrsj      0/1     Completed   0            72s
ingress-nginx-controller-79887d48bf-txxvd   0/1     Running     1 (2s ago)   73s

kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.111.114.169   192.168.100.101   80:32236/TCP,443:30083/TCP   84s
ingress-nginx-controller-admission   ClusterIP      10.96.117.247    <none>            443/TCP                      85s

检查版本安装情况

POD_NAMESPACE=ingress-nginx
POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/component=controller -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.0.0
  Build:         041eb167c7bfccb1d1653f194924b0c5fd885e10
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.20.1

-------------------------------------------------------------------------------

后端 `ingress` 代理应用示例

创建 myapp 应用

kubectl apply -f - <<EOF
---
kind: Service
apiVersion: v1
metadata:
  name: myapp
spec:
  selector:
    app: myapp
  type: ClusterIP
  ports:
  - name: http
    port: 80
    targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: nginx:alpine
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 80
EOF
service/myapp created
deployment.apps/myapp created
Copy

创建 ingress 文件

kubectl apply -f - <<EOF
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-myapp
  annotations:
    # 指定 Ingress Controller 的类型
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: myapp.hzde.com
    http:
      paths:
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: myapp
            port:
              number: 80
EOF
ingress.networking.k8s.io/test-myapp created
Copy

添加 hosts 并尝试访问

查看 ingress 对应节点的端口

kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.101.207.65   <none>        80:30348/TCP,443:30473/TCP   9m9s
ingress-nginx-controller-admission   ClusterIP   10.96.36.183    <none>        443/TCP                      9m9s
Copy

echo '192.168.100.10 myapp.hzde.com' >> /etc/hosts
curl myapp.hzde.com:30348
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Copy

第二种安装方式，直接使用hostNetwork

curl -o ingress-nginx.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.0-beta.3/deploy/static/provider/baremetal/deploy.yaml

sed -i 's@k8s.gcr.io/ingress-nginx/controller@willdockerhub/ingress-nginx-controller@' ingress-nginx.yaml
sed -i 's@k8s.gcr.io/ingress-nginx/kube-webhook-certgen@hzde0128/kube-webhook-certgen@' ingress-nginx.yaml
sed -i 'N;315a\ \ \ \ \ \ hostNetwork: true' ingress-nginx.yaml
kubectl apply -f ingress-nginx.yaml
Copy

kubectl get po -n ingress-nginx -owide
NAME                                        READY   STATUS    RESTARTS   AGE   IP              NODE     NOMINATED NODE   READINESS GATES
nginx-ingress-controller-5dfbcfd5d9-k2fjc   1/1     Running   0          33s   192.168.100.10   k8s-m1   <none>           <none>
Copy

不修改hosts访问myapp.hzde.com

在头部指定host

curl 192.168.100.10 -H "host:myapp.hzde.com"
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

ingress HTTPS 访问

参考文档TLS/HTTPS

创建自签证书文件

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginx/O=nginx"
Generating a 2048 bit RSA private key
......................................................+++
..........................+++
writing new private key to 'tls.key'
-----
Copy

创建 secret

kubectl create secret tls tls-secret --key tls.key --cert tls.crt
secret/tls-secret created
Copy

创建 tls ingress

kubectl apply -f - <<EOF
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-myapp
  annotations:
    # 指定 Ingress Controller 的类型
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - myapp2.hzde.com
    secretName: tls-secret
  rules:
  - host: myapp2.hzde.com
    http:
      paths:
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: myapp
            port:
              number: 80
EOF
ingress.networking.k8s.io/tls-myapp created
Copy

测试

echo '192.168.100.10 myapp2.hzde.com' >> /etc/hosts

curl -sSk https://myapp2.hzde.com:30473 // 注意这个端口是443 对应的端口
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

11-2 部署 prometheus+granfa 监控服务

1. 下载 kube-prometheus（域名污染的话域名解析后再配置吧）

curl -o kube-prometheus_v0.9.0.tar.gz \
https://github.com/prometheus-operator/kube-prometheus/archive/refs/tags/v0.9.0.tar.gz

tar xf kube-prometheus_v0.9.0.tar.gz

cd kube-prometheus-0.9.0

2.修改 yaml 清单文件

修改 grafana-service

cp manifests/grafana-service.yaml{,.ori}
sed -i '/spec:/a\  type: NodePort' manifests/grafana-service.yaml
sed -i '/targetPort:/a\    nodePort: 30200' manifests/grafana-service.yaml

查看修改

diff manifests/grafana-service.yaml.ori  manifests/grafana-service.yaml
11a12
>   type: NodePort
15a17
>     nodePort: 30200

修改promethesu-service

cp manifests/prometheus-service.yaml{,.ori}
sed -i '/spec:/a\  type: NodePort' manifests/prometheus-service.yaml
sed -i '/targetPort:/a\    nodePort: 30100' manifests/prometheus-service.yaml

查看修改

diff manifests/prometheus-service.yaml.ori manifests/prometheus-service.yaml
12a13
>   type: NodePort
16a18
>     nodePort: 30100

3. 修改镜像库

sed -i '/image:/s@k8s.gcr.io/kube-state-metrics@willdockerhub@' $(grep -l image: manifests/*.yaml)

4. 部署 CRD

kubectl apply -f manifests/setup
namespace/monitoring created
customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
service/prometheus-operator created
serviceaccount/prometheus-operator created

5. 部署 kube-prometheus

kubectl apply -f manifests

输入如下：

alertmanager.monitoring.coreos.com/main created
Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionbudget.policy/alertmanager-main created
prometheusrule.monitoring.coreos.com/alertmanager-main-rules created
secret/alertmanager-main created
service/alertmanager-main created
serviceaccount/alertmanager-main created
servicemonitor.monitoring.coreos.com/alertmanager created
clusterrole.rbac.authorization.k8s.io/blackbox-exporter created
clusterrolebinding.rbac.authorization.k8s.io/blackbox-exporter created
configmap/blackbox-exporter-configuration created
deployment.apps/blackbox-exporter created
service/blackbox-exporter created
serviceaccount/blackbox-exporter created
servicemonitor.monitoring.coreos.com/blackbox-exporter created
secret/grafana-datasources created
configmap/grafana-dashboard-apiserver created
configmap/grafana-dashboard-cluster-total created
configmap/grafana-dashboard-controller-manager created
configmap/grafana-dashboard-k8s-resources-cluster created
configmap/grafana-dashboard-k8s-resources-namespace created
configmap/grafana-dashboard-k8s-resources-node created
configmap/grafana-dashboard-k8s-resources-pod created
configmap/grafana-dashboard-k8s-resources-workload created
configmap/grafana-dashboard-k8s-resources-workloads-namespace created
configmap/grafana-dashboard-kubelet created
configmap/grafana-dashboard-namespace-by-pod created
configmap/grafana-dashboard-namespace-by-workload created
configmap/grafana-dashboard-node-cluster-rsrc-use created
configmap/grafana-dashboard-node-rsrc-use created
configmap/grafana-dashboard-nodes created
configmap/grafana-dashboard-persistentvolumesusage created
configmap/grafana-dashboard-pod-total created
configmap/grafana-dashboard-prometheus-remote-write created
configmap/grafana-dashboard-prometheus created
configmap/grafana-dashboard-proxy created
configmap/grafana-dashboard-scheduler created
configmap/grafana-dashboard-statefulset created
configmap/grafana-dashboard-workload-total created
configmap/grafana-dashboards created
Warning: spec.template.spec.nodeSelector[beta.kubernetes.io/os]: deprecated since v1.14; use "kubernetes.io/os" instead
deployment.apps/grafana created
service/grafana created
serviceaccount/grafana created
servicemonitor.monitoring.coreos.com/grafana created
prometheusrule.monitoring.coreos.com/kube-prometheus-rules created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
prometheusrule.monitoring.coreos.com/kube-state-metrics-rules created
service/kube-state-metrics created
serviceaccount/kube-state-metrics created
servicemonitor.monitoring.coreos.com/kube-state-metrics created
prometheusrule.monitoring.coreos.com/kubernetes-monitoring-rules created
servicemonitor.monitoring.coreos.com/kube-apiserver created
servicemonitor.monitoring.coreos.com/coredns created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
servicemonitor.monitoring.coreos.com/kube-scheduler created
servicemonitor.monitoring.coreos.com/kubelet created
clusterrole.rbac.authorization.k8s.io/node-exporter created
clusterrolebinding.rbac.authorization.k8s.io/node-exporter created
daemonset.apps/node-exporter created
prometheusrule.monitoring.coreos.com/node-exporter-rules created
service/node-exporter created
serviceaccount/node-exporter created
servicemonitor.monitoring.coreos.com/node-exporter created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io configured
clusterrole.rbac.authorization.k8s.io/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created
clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created
clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created
configmap/adapter-config created
deployment.apps/prometheus-adapter created
rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created
service/prometheus-adapter created
serviceaccount/prometheus-adapter created
servicemonitor.monitoring.coreos.com/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/prometheus-k8s created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created
prometheusrule.monitoring.coreos.com/prometheus-operator-rules created
servicemonitor.monitoring.coreos.com/prometheus-operator created
poddisruptionbudget.policy/prometheus-k8s created
prometheus.monitoring.coreos.com/k8s created
prometheusrule.monitoring.coreos.com/prometheus-k8s-prometheus-rules created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s-config created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
service/prometheus-k8s created
serviceaccount/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus-k8s created

6.检查部署情况

这里有时会发现好多pod不健康，可以查看日志，发现好多镜像pull出问题，我们改为手动拉取就可以了

有些镜像实在下载不下来就去阿里云上找找总有办法的唉唉唉唉。。。。。。。。。。。。

kubectl get po -n monitoring

输出如下

NAME                                   READY   STATUS    RESTARTS        AGE
alertmanager-main-0                    2/2     Running   0               9m2s
alertmanager-main-1                    2/2     Running   0               9m2s
alertmanager-main-2                    2/2     Running   0               9m2s
blackbox-exporter-567dc8c7d4-ts5rr     3/3     Running   0               11m
grafana-6dd5b5f65-vxld6                1/1     Running   0               11m
kube-state-metrics-69b5f46bb9-bjjch    3/3     Running   0               11m
node-exporter-7rpts                    2/2     Running   0               11m
node-exporter-f9m77                    2/2     Running   0               11m
node-exporter-xlz2t                    2/2     Running   0               11m
prometheus-adapter-59df95d9f5-kglql    1/1     Running   0               11m
prometheus-adapter-59df95d9f5-n99fb    1/1     Running   0               11m
prometheus-k8s-0                       2/2     Running   1 (9m ago)      9m2s
prometheus-k8s-1                       2/2     Running   1 (8m59s ago)   9m2s
prometheus-operator-7775c66ccf-npmxw   2/2     Running   0               11m

kubectl get svc -n monitoring

NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
alertmanager-main       ClusterIP   10.109.167.164   <none>        9093/TCP                     12m
alertmanager-operated   ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   9m46s
blackbox-exporter       ClusterIP   10.109.154.125   <none>        9115/TCP,19115/TCP           12m
grafana                 NodePort    10.99.112.125    <none>        3000:30200/TCP               12m
kube-state-metrics      ClusterIP   None             <none>        8443/TCP,9443/TCP            11m
node-exporter           ClusterIP   None             <none>        9100/TCP                     11m
prometheus-adapter      ClusterIP   10.107.102.121   <none>        443/TCP                      11m
prometheus-k8s          NodePort    10.103.179.200   <none>        9090:30100/TCP               11m
prometheus-operated     ClusterIP   None             <none>        9090/TCP                     9m46s
prometheus-operator     ClusterIP   None             <none>        8443/TCP                     12m

如果报错的话而且无法弄时，直接删除干净，需要删除monitoring 操作如下

kubectl get namespace monitoring -o json > tmp.json

将spec里的内容删除干净，保存退出：

然后新开一个窗口运行kubectl proxy跑一个API代理在本地的8081端口

# kubectl proxy --port=8081

最后执行命令：

# curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8081/api/v1/namespaces/monitoring/finalize

然后再看一下命名空间，发现monitoring已经被删除了。

然后重启虚拟机或者看看分配内存是不是够用 reboot 结果OK

之后重新来安装 kube-prometheus

另外还有个问题就是kube-prometheus 你下载的是kube-prometheus-0.9.0

在部署 kube-prometheus的时候需要的镜像包为

k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.0

这个镜像你是找不到的除非下载git源码自己打镜像唉唉唉怎么办呢直接下v0.8.4版本然后再改一下 tag

docker pull directxman12/k8s-prometheus-adapter:v0.8.4

docker tag 9d7c9987f24e k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.0

做完再检查完美哈哈哈哈

7. 访问浏览器

打开浏览器，输入：http://192.168.47.128:30200/访问grafana admin lovetcj520

默认用户名和密码为：admin，admin 要修改密码

11-3. 部署 EFK 日志服务（用户WEB端日志查看）

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub