第二章 Kubernetes集群探索

2-1 探索K8S集群路线

• 1. kubeadm做了那些事情
• 2. 各组件的作用
  • kubelet
  • kube-apiserver
  • etct
  • kube-controller-manager
  • kube-scheduler
  • kube-proxy
• 3. kubectl使用方法

2-2 kubeadm init流程揭秘

2-2.1 引导前的检查

各环境的检查，如果swap，如不符合查件，则终止初始化

kubeadm init pre-flight check

• kubeadm版本与要安装的kubernetes版本的比对检查
• kubernetes安装的系统需求检查
  • docker服务
  • cgroups相关
• 其他检查
  • 用户: user是否为root
  • 主机: 域名是否合法，主机名不能带有下划线，主机名绑定，或者是公网可以正常解析的域名，DNS子域
  • 端口: apiserver绑定的10250, 10251, 10252端口是否被占用，ip, iptables, mount是否存在且在环境变量PATH中
  • swap: 需要禁用swap分区
  • 工具

2-2.2 生成私钥和数字证书

生成各组件通信的证书及私钥文件，默认在/etc/kubernetes/pki/目录下

• 1. 自建CA，生成ca.key, ca.crt
  • ca.crt是标准的ca509证书
  • 使用openssl查看ca证书

  $ pwd
  /etc/kubernetes/pki# 
  
  $ ls
  ls
  apiserver.crt		   apiserver.key		 ca.crt  front-proxy-ca.crt	 front-proxy-client.key
  apiserver-etcd-client.crt  apiserver-kubelet-client.crt  ca.key  front-proxy-ca.key	 sa.key
  apiserver-etcd-client.key  apiserver-kubelet-client.key  etcd	 front-proxy-client.crt  sa.pub
  
  $ openssl x509 -in ca.crt -noout -text
  Certificate:
      Data:
          Version: 3 (0x2)
          Serial Number: 0 (0x0)
      Signature Algorithm: sha256WithRSAEncryption
          Issuer: CN = kubernetes
          Validity
              Not Before: May 18 04:38:46 2019 GMT
              Not After : May 15 04:38:46 2029 GMT
          Subject: CN = kubernetes
          Subject Public Key Info:
              Public Key Algorithm: rsaEncryption
                  Public-Key: (2048 bit)
                  Modulus:
                      00:cc:7f:f9:b3:47:92:0f:f3:42:73:65:f0:ac:53:
                      46:08:ce:1d:8e:a1:3f:16:9b:fe:e5:fb:83:e0:1e:
                      ... ...
                      d5:9f:d8:35:24:4d:ca:ef:81:c7:de:44:e9:37:f4:
                      6e:ac:e0:16:b5:bf:72:20:60:71:b3:fa:fa:8b:34:
                      bc:b5
                  Exponent: 65537 (0x10001)
          X509v3 extensions:
              X509v3 Key Usage: critical
                  Digital Signature, Key Encipherment, Certificate Sign
              X509v3 Basic Constraints: critical
                  CA:TRUE
      Signature Algorithm: sha256WithRSAEncryption
          09:56:89:45:52:47:ff:36:ea:34:c9:ca:bd:d8:8d:c2:6f:4e:
          52:b0:cf:51:1a:3e:a7:b7:bd:0a:f8:a8:f0:cb:2e:fc:f7:53:
          ... ...
          62:f5:48:37:1d:68:ed:a6:17:8f:5d:3c:79:35:9e:37:b0:fa:
          e5:af:f2:29
  

  • 验证apiserver-etcd-client.crt证书是不是由kubernetes的根证书(ca.crt)签发的

  $ openssl verify -CAfile ca.crt ./apiserver-etcd-client.crt
  O = system:masters, CN = kube-apiserver-etcd-client
  error 7 at 0 depth lookup: certificate signature failure
  error ./apiserver-etcd-client.crt: verification failed    # 这里可以看到验证失败了
  140224101396544:error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67:
  140224101396544:error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:573:
  140224101396544:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:../crypto/asn1/a_verify.c:171:
      
  # 这里说明：该证书是由etcd的ca.crt证书签发的
  $ openssl verify -CAfile etcd/ca.crt apiserver-etcd-client.crt
  apiserver-etcd-client.crt: OK
  

• 2. apiserver的私钥与公钥证书
• 3. apiserver访问kubelet使用的客户端私钥与证书
• 4. serviceaccount的私钥sa.key, sa.pub
• 5. etcd相关的私钥和数字证书

2-2.3 生成控制平台组件kubeconfig文件

生成配置文件，用于各组件间通信使用

• kubeconfig配置文件是一系列配置文件的统称
• 组件kubeconfig文件
  • ~/.kube/config 在初始化过程成功后，打印输出提示的配置
  • /etc/kubernetes/*.conf
  • KUBECONFIG环境变量
• kubeconfig配置: 包含cluster, user, context信息
  • 查看配置文件信息

  $ kubectl config view
  apiVersion: v1
  clusters:
  - cluster:
      certificate-authority-data: REDACTED
      server: https://172.16.81.161:6443
  name: kubernetes
  contexts:
  - context:
      cluster: kubernetes
      user: kubernetes-admin
  name: kubernetes-admin@kubernetes
  current-context: kubernetes-admin@kubernetes
  kind: Config
  preferences: {}
  users:
  - name: kubernetes-admin
  user:
      client-certificate-data: REDACTED
      client-key-data: REDACTED
  

• 允许kubectl快速切换context，管理多集群

2-2.4 生成控制平面组件minifest文件

生成的manifest文件被master节点读取，启动控制平面组件，并维护控制平面组件的状态

组件 manifest 文件

• /etc/kubernetes/manifests

etcd.yaml  kube-apiserver.yaml	kube-controller-manager.yaml  kube-scheduler.yaml

• 控制平面组件以Static Pod形式运行
  • 所谓 Static Pod 就是以节点上的kubelet来管理运行的
  • 而 kubelet 服务则是由 master 节点上的 apiserver 来管理的
• kubelet 读取 manifests 目录并管理各控制平面组件 Pod 启停

2-2.5 下载镜像，等待控制平面启动

默认从google的官方镜像站下载，国内需要事件下载镜像并打好标签

• 从k8s.gcr.io下载组件镜像
• kubeadm会一起等待探测并等待 localhost:6443/healthz 服务返回成功
• 过一段时间后，控制平面主要组件都启动OK
• 安装 dns 和 kube-prosy 插件
  • 以 DaemonSet 方式部署 kube-proxy

  $ kubectl get daemonset -n kube-system
  NAME         DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
  kube-proxy   3         3         3         3            3           <none>          1d
  weave-net    3         3         3         3            3           <none>          1d
  

  • 部署 kube-dns(也可以用 core-dns 替代)
  • dns 插件处于 Pending 状态，直到 cluster 网络就绪

2-2.6 保存MasterConfiguration配置信息

保存MasterConfiguration配置信息

2-2.7 设定master标志

将当前节点设定为master节点，默认工作负载不会被调度到master节点

2-2.8 进行基于TLS的安全引导的相关配置

2-2.9 安装DNS及kube-rposy组件

DNS插件安装成功后会牌Pending状态，需要网络插件安装完成后才会切换成正常状态(Running)

2-3 kubeadm join揭秘

• join: 将节点加入集群
  • join前检查
  • discovery-token-ca-cert-hash: 用于Node验证Master身份
    • 如何根据数字证书公钥算出对应的哈希值

    $ openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -pubkey | openssl rsa -pubin -outform DER 2> /dev/null | sha256sum | cut -d ' ' -f 1
    59012afebb4d66d6204f27598d634d53a94d2f296b5c49df670c6e47ed255799
    
    # 官方提供的方式
    $ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
    59012afebb4d66d6204f27598d634d53a94d2f296b5c49df670c6e47ed255799
    

  • token: 用于master验证node身份

2-4 kubernetes核心组件详解

2-4.1 kubelet: 节点pod管家

运行于集群的所有节点上

• 每个节点上的kubelet由操作系统init(如：systemd)进程启动
• kubelet服务的配置文件及启动参数配置文件

/lib/systemd/system/kubelet.service 
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf    # 主要配置文件

• 配置文件修改后，需要使用下面的命令生效

$ systemctl daemon-reload
$ systemctl restart kubelet

2-4.2 kube-apiserver: 集群管理入口

kube-apiserver:

• 由kubectl启动的static pod
• apiserver的pod spec: /etc/kubernetes/manifests/kube-apiserver.yaml
• kube监听/etc/kubernetes/manifests/下文件的变化，自动重启发生变化的apiserver pod

2-4.3 etcd: 配置中心

• 由kubelet启动的static popd
• apiserver 与 etcd 之间采用基于 tls 的安全通信
• etcd 挂载 master 节点本地路径 /var/lib/etcd 用于运行时数据存储
• 如果要做etcd备份，需要关注下此路径

$ tree /var/lib/etcd
/var/lib/etcd
└── member
    ├── snap
    │   ├── 0000000000000007-000000000000c355.snap
    │   ├── 0000000000000007-000000000000ea66.snap
    │   ├── 0000000000000007-0000000000011177.snap
    │   ├── 0000000000000008-0000000000013888.snap
    │   ├── 0000000000000008-0000000000015f99.snap
    │   └── db
    └── wal
        ├── 0000000000000000-0000000000000000.wal
        ├── 0000000000000001-000000000000f5dc.wal
        └── 0.tmp

3 directories, 9 files

2-4.4 kube-controller-manager: 管理控制中心

• 负责集群内Node, Pod副本, 服务的Endpoint, 命名空间, ServiceAccount, 资源配额等管理
• 由 kubelet 启动的 static pod

2-4.5 kube-scheduler: 调度器

Scheduler: 单纯地调度Pod

• 按照特定的策略的调度算法，将等待调度Pod绑定到集群中某个适合的Node，并写入绑定信息
• 由 kubelet 启动的 static pod
• 配置文件: /etc/kubernetes/manifests/kube-scheduler.yaml

2-5.6 kube-proxy: 服务抽象实现

kube-proxy运行于kubernetes集群中每个节点上

• kube-proxy由 daemonset 控制器在各个节点上启动唯一实例
• 配置文件: /var/lib/kube-proxy/config.conf（在Pod内，容器内）

$ kubectl -n kube-system get pods -o wide | grep 'kube-proxy'
kube-proxy-2bj4c                        1/1       Running            2          2d        172.16.81.162    dbk8s-node-01
kube-proxy-bf7lf                        1/1       NodeLost           1          2d        172.16.81.163    dbk8s-node-02
kube-proxy-p7n5s                        1/1       Running            7          2d        172.16.81.161    dbk8s-master

# 这里以 master 节点的kube-proxy为例，exec可以执行命令
$ kubectl -n kube-system exec kube-proxy-2bj4c -- cat /var/lib/kube-proxy/config.conf
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
  qps: 5
clusterCIDR: 192.168.1.0/24
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  minSyncPeriod: 0s
  scheduler: ""
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: ""
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
resourceContainer: /kube-proxy

2-5 kubectl详解

kubectl 是目前管理 k8s 集群的最强利器

2-5.1 集群访问配置

usage: kubectl config

• 查看配置

kubectl config view 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: https://172.16.81.161:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

• 创建一个集群入口

kubectl config set-cluster k8s1 --server=https://1.2.3.4
Cluster "k8s1" set.

• 查看集群

kubectl config get-clusters 
NAME
kubernetes
k8s1

• 删除集群

$ kubectl config delete-cluster k8s1 
deleted cluster k8s1 from /root/.kube/config

• 创建context入口

$ kubectl config set-context admin1@k8s1 --user=admin1
Context "admin1@k8s1" created.

• 查看context

$ kubectl config get-contexts 
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
          admin1@k8s1                                admin1             
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin

• 选择默认context

$ kubectl config use-context admin1@k8s1 
Switched to context "admin1@k8s1".

• 获取当前的context

$ kubectl config get-contexts 
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         admin1@k8s1                                admin1             
          kubernetes-admin@kubernetes   kubernetes   kubernetes-admin   
$ kubectl config current-context 
admin1@k8s1

• 删除context

$ kubectl config delete-context admin1@k8s1

• 创建凭证

$ kubectl config set-credentials testing-01 --username=testing-01 --password=abcdef@
User "testing-01" set.

# 这里可以看到已经有用户生成了
$ kubectl config view  | tail -4
- name: testing-01
  user:
    password: abcdef@
    username: testing-01

2-5.2 集群控制

• kubectl [ create|apply|delete|label|edit|expose|scale ]

# 创建nginx-deployment配置文件
$ cat nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:alpine
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nginx-config
              mountPath: /etc/nginx/conf.d
            - name: web-root
              mountPath: /usr/share/nginx/html
      volumes:
        - name: nginx-config
          configMap:
            name: nginx-config
        - name: web-root
          hostPath:
            path: /var/www/html

• 根据配置文件来创建

$ kubectl create -f nginx-deployment.yaml
deployment.apps "nginx-deployment-example" created

• 查看刚刚创建的deployent

$ kubectl get pods --show-labels  | grep deployment
nginx-deployment-example-7945d4c5f-cg85r   0/1       ContainerCreating   0          14m       app=nginx,pod-template-hash=350180719

• 给pod打标签

$ kubectl label pod/nginx-deployment-example-7945d4c5f-cg85r status=health
pod "nginx-deployment-example-7945d4c5f-cg85r" labeled

$ kubectl get pods --show-labels  | grep deployment
nginx-deployment-example-7945d4c5f-cg85r   0/1       ContainerCreating   0          16m       app=nginx,pod-template-hash=350180719,status=health

• 通过kubect edit命令修改deployment的配置

root@dbk8s-master:~# kubectl get pods
NAME                                       READY     STATUS              RESTARTS   AGE
nginx-deployment-example-7945d4c5f-cg85r   0/1       ContainerCreating   0          20m

root@dbk8s-master:~# kubectl get deployment
NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment-example   1         1         1            0           20m
root@dbk8s-master:~# kubectl edit deployment nginx-deployment-example
##################
# 这里修改的部分如下，replicas: 3, 即创建的副本数
spec:
  progressDeadlineSeconds: 600
  replicas: 3
##################
deployment.extensions "nginx-deployment-example" edited

# 查看已存在的deployment
root@dbk8s-master:~# kubectl get deployment
NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment-example   3         3         3            0           21m

# 这里可以看到已经有3个副本了
root@dbk8s-master:~# kubectl get pods
NAME                                       READY     STATUS              RESTARTS   AGE
nginx-deployment-example-7945d4c5f-cg85r   0/1       ContainerCreating   0          21m
nginx-deployment-example-7945d4c5f-j9pr7   0/1       ContainerCreating   0          10s
nginx-deployment-example-7945d4c5f-s7rc8   0/1       ContainerCreating   0          10s

• 使用kubectl scale扩容

# 扩容至10个副本
root@dbk8s-master:~# kubectl scale --replicas=10 deployment nginx-deployment-example
deployment.extensions "nginx-deployment-example" scaled

# 看到有10个副本正在被创建
root@dbk8s-master:~# kubectl get pods
NAME                                       READY     STATUS              RESTARTS   AGE
nginx-deployment-example-7945d4c5f-777rp   0/1       ContainerCreating   0          3s
nginx-deployment-example-7945d4c5f-cg85r   0/1       ContainerCreating   0          25m
nginx-deployment-example-7945d4c5f-dwwt6   0/1       ContainerCreating   0          3s
nginx-deployment-example-7945d4c5f-gcc8m   0/1       ContainerCreating   0          3s
nginx-deployment-example-7945d4c5f-j9pr7   0/1       ContainerCreating   0          4m
nginx-deployment-example-7945d4c5f-jnfmh   0/1       ContainerCreating   0          3s
nginx-deployment-example-7945d4c5f-rqq6h   0/1       ContainerCreating   0          3s
nginx-deployment-example-7945d4c5f-s7rc8   0/1       ContainerCreating   0          4m
nginx-deployment-example-7945d4c5f-ssxgt   0/1       ContainerCreating   0          3s
nginx-deployment-example-7945d4c5f-xn87h   0/1       ContainerCreating   0          3s

# 这里可以看到简图信息
root@dbk8s-master:~# kubectl get deployment
NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment-example   10        10        10           0           26m

• 使用kubectl apply恢复原始配置

root@dbk8s-master:~# kubectl apply -f nginx-deployment.yaml
deployment.apps "nginx-deployment-example" configured
root@dbk8s-master:~# kubectl get deployment
NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment-example   1         1         1            0           28m

• 将deployment删除

root@dbk8s-master:~# kubectl delete -f nginx-deployment.yaml
deployment.apps "nginx-deployment-example" deleted

# 正在终止pod
root@dbk8s-master:~# kubectl get pods --show-labels
NAME                                       READY     STATUS        RESTARTS   AGE       LABELS
nginx-deployment-example-7945d4c5f-cg85r   0/1       Terminating   0          29m       app=nginx,pod-template-hash=350180719,status=health

# 这里已经看不到了
root@dbk8s-master:~# kubectl get pods --show-labels
No resources found.

2-5.3 集群查看和问题调试

• kubectl [ get|describe|logs|exec|attach ]

# 查看pod信息
$ kubectl get pods --all-namespaces -o wide

# 查看pod详细信息
$ kubectl -n kube-system describe pods weave-net-4mwv6

# 查看pod日志
$ kubectl -n kube-system logs -f weave-net-4mwv6

# 在pod中执行命令
$ kubectl -n kube-system exec monitoring-influxdb-cc95575b9-7d9xj -- uptime
 13:27:47 up 50 min,  0 users,  load average: 0.04, 0.10, 0.08

# 通过kubectl attach挂载到某个容器上, 如果该容器中有日志产生，则会实时打印出来
$ kubectl -n kube-system attach monitoring-influxdb-cc95575b9-7d9xj
Defaulting container name to influxdb.
Use 'kubectl describe pod/monitoring-influxdb-cc95575b9-7d9xj -n kube-system' to see all of the containers in this pod.
If you don't see a command prompt, try pressing enter.