第二章 Kubernetes集群探索

2-1 探索K8S集群路线

    1. kubeadm做了那些事情
    1. 各组件的作用
    • kubelet
    • kube-apiserver
    • etct
    • kube-controller-manager
    • kube-scheduler
    • kube-proxy
    1. kubectl使用方法

2-2 kubeadm init流程揭秘

2-2.1 引导前的检查

各环境的检查,如果swap,如不符合查件,则终止初始化

kubeadm init pre-flight check

  • kubeadm版本与要安装的kubernetes版本的比对检查
  • kubernetes安装的系统需求检查
    • docker服务
    • cgroups相关
  • 其他检查
    • 用户: user是否为root
    • 主机: 域名是否合法,主机名不能带有下划线,主机名绑定,或者是公网可以正常解析的域名,DNS子域
    • 端口: apiserver绑定的10250, 10251, 10252端口是否被占用,ip, iptables, mount是否存在且在环境变量PATH中
    • swap: 需要禁用swap分区
    • 工具

2-2.2 生成私钥和数字证书

生成各组件通信的证书及私钥文件,默认在/etc/kubernetes/pki/目录下

    1. 自建CA,生成ca.key, ca.crt
    • ca.crt是标准的ca509证书
    • 使用openssl查看ca证书
    $ pwd
    /etc/kubernetes/pki# 
    
    $ ls
    ls
    apiserver.crt		   apiserver.key		 ca.crt  front-proxy-ca.crt	 front-proxy-client.key
    apiserver-etcd-client.crt  apiserver-kubelet-client.crt  ca.key  front-proxy-ca.key	 sa.key
    apiserver-etcd-client.key  apiserver-kubelet-client.key  etcd	 front-proxy-client.crt  sa.pub
    
    $ openssl x509 -in ca.crt -noout -text
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 0 (0x0)
        Signature Algorithm: sha256WithRSAEncryption
            Issuer: CN = kubernetes
            Validity
                Not Before: May 18 04:38:46 2019 GMT
                Not After : May 15 04:38:46 2029 GMT
            Subject: CN = kubernetes
            Subject Public Key Info:
                Public Key Algorithm: rsaEncryption
                    Public-Key: (2048 bit)
                    Modulus:
                        00:cc:7f:f9:b3:47:92:0f:f3:42:73:65:f0:ac:53:
                        46:08:ce:1d:8e:a1:3f:16:9b:fe:e5:fb:83:e0:1e:
                        ... ...
                        d5:9f:d8:35:24:4d:ca:ef:81:c7:de:44:e9:37:f4:
                        6e:ac:e0:16:b5:bf:72:20:60:71:b3:fa:fa:8b:34:
                        bc:b5
                    Exponent: 65537 (0x10001)
            X509v3 extensions:
                X509v3 Key Usage: critical
                    Digital Signature, Key Encipherment, Certificate Sign
                X509v3 Basic Constraints: critical
                    CA:TRUE
        Signature Algorithm: sha256WithRSAEncryption
            09:56:89:45:52:47:ff:36:ea:34:c9:ca:bd:d8:8d:c2:6f:4e:
            52:b0:cf:51:1a:3e:a7:b7:bd:0a:f8:a8:f0:cb:2e:fc:f7:53:
            ... ...
            62:f5:48:37:1d:68:ed:a6:17:8f:5d:3c:79:35:9e:37:b0:fa:
            e5:af:f2:29
    
    • 验证apiserver-etcd-client.crt证书是不是由kubernetes的根证书(ca.crt)签发的
    $ openssl verify -CAfile ca.crt ./apiserver-etcd-client.crt
    O = system:masters, CN = kube-apiserver-etcd-client
    error 7 at 0 depth lookup: certificate signature failure
    error ./apiserver-etcd-client.crt: verification failed    # 这里可以看到验证失败了
    140224101396544:error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67:
    140224101396544:error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:573:
    140224101396544:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:../crypto/asn1/a_verify.c:171:
        
    # 这里说明:该证书是由etcd的ca.crt证书签发的
    $ openssl verify -CAfile etcd/ca.crt apiserver-etcd-client.crt
    apiserver-etcd-client.crt: OK
    
    1. apiserver的私钥与公钥证书
    1. apiserver访问kubelet使用的客户端私钥与证书
    1. serviceaccount的私钥sa.key, sa.pub
    1. etcd相关的私钥和数字证书

2-2.3 生成控制平台组件kubeconfig文件

生成配置文件,用于各组件间通信使用

  • kubeconfig配置文件是一系列配置文件的统称
  • 组件kubeconfig文件
    • ~/.kube/config 在初始化过程成功后,打印输出提示的配置
    • /etc/kubernetes/*.conf
    • KUBECONFIG环境变量
  • kubeconfig配置: 包含cluster, user, context信息
    • 查看配置文件信息
    $ kubectl config view
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: REDACTED
        server: https://172.16.81.161:6443
    name: kubernetes
    contexts:
    - context:
        cluster: kubernetes
        user: kubernetes-admin
    name: kubernetes-admin@kubernetes
    current-context: kubernetes-admin@kubernetes
    kind: Config
    preferences: {}
    users:
    - name: kubernetes-admin
    user:
        client-certificate-data: REDACTED
        client-key-data: REDACTED
    
  • 允许kubectl快速切换context,管理多集群

2-2.4 生成控制平面组件minifest文件

生成的manifest文件被master节点读取,启动控制平面组件,并维护控制平面组件的状态

组件 manifest 文件

  • /etc/kubernetes/manifests
etcd.yaml  kube-apiserver.yaml	kube-controller-manager.yaml  kube-scheduler.yaml
  • 控制平面组件以Static Pod形式运行
    • 所谓 Static Pod 就是以节点上的kubelet来管理运行的
    • 而 kubelet 服务则是由 master 节点上的 apiserver 来管理的
  • kubelet 读取 manifests 目录并管理各控制平面组件 Pod 启停

2-2.5 下载镜像,等待控制平面启动

默认从google的官方镜像站下载,国内需要事件下载镜像并打好标签

  • 从k8s.gcr.io下载组件镜像
  • kubeadm会一起等待探测并等待 localhost:6443/healthz 服务返回成功
  • 过一段时间后,控制平面主要组件都启动OK
  • 安装 dns 和 kube-prosy 插件
    • 以 DaemonSet 方式部署 kube-proxy
    $ kubectl get daemonset -n kube-system
    NAME         DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    kube-proxy   3         3         3         3            3           <none>          1d
    weave-net    3         3         3         3            3           <none>          1d
    
    • 部署 kube-dns(也可以用 core-dns 替代)
    • dns 插件处于 Pending 状态,直到 cluster 网络就绪

2-2.6 保存MasterConfiguration配置信息

保存MasterConfiguration配置信息

2-2.7 设定master标志

将当前节点设定为master节点,默认工作负载不会被调度到master节点

2-2.8 进行基于TLS的安全引导的相关配置

2-2.9 安装DNS及kube-rposy组件

DNS插件安装成功后会牌Pending状态,需要网络插件安装完成后才会切换成正常状态(Running)

2-3 kubeadm join揭秘

  • join: 将节点加入集群
    • join前检查
    • discovery-token-ca-cert-hash: 用于Node验证Master身份
      • 如何根据数字证书公钥算出对应的哈希值
      $ openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -pubkey | openssl rsa -pubin -outform DER 2> /dev/null | sha256sum | cut -d ' ' -f 1
      59012afebb4d66d6204f27598d634d53a94d2f296b5c49df670c6e47ed255799
      
      # 官方提供的方式
      $ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
      59012afebb4d66d6204f27598d634d53a94d2f296b5c49df670c6e47ed255799
      
    • token: 用于master验证node身份

2-4 kubernetes核心组件详解

2-4.1 kubelet: 节点pod管家

运行于集群的所有节点上

  • 每个节点上的kubelet由操作系统init(如:systemd)进程启动
  • kubelet服务的配置文件及启动参数配置文件
/lib/systemd/system/kubelet.service 
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf    # 主要配置文件
  • 配置文件修改后,需要使用下面的命令生效
$ systemctl daemon-reload
$ systemctl restart kubelet

2-4.2 kube-apiserver: 集群管理入口

kube-apiserver:

  • 由kubectl启动的static pod
  • apiserver的pod spec: /etc/kubernetes/manifests/kube-apiserver.yaml
  • kube监听/etc/kubernetes/manifests/下文件的变化,自动重启发生变化的apiserver pod

2-4.3 etcd: 配置中心

  • 由kubelet启动的static popd
  • apiserver 与 etcd 之间采用基于 tls 的安全通信
  • etcd 挂载 master 节点本地路径 /var/lib/etcd 用于运行时数据存储
  • 如果要做etcd备份,需要关注下此路径
$ tree /var/lib/etcd
/var/lib/etcd
└── member
    ├── snap
    │   ├── 0000000000000007-000000000000c355.snap
    │   ├── 0000000000000007-000000000000ea66.snap
    │   ├── 0000000000000007-0000000000011177.snap
    │   ├── 0000000000000008-0000000000013888.snap
    │   ├── 0000000000000008-0000000000015f99.snap
    │   └── db
    └── wal
        ├── 0000000000000000-0000000000000000.wal
        ├── 0000000000000001-000000000000f5dc.wal
        └── 0.tmp

3 directories, 9 files

2-4.4 kube-controller-manager: 管理控制中心

  • 负责集群内Node, Pod副本, 服务的Endpoint, 命名空间, ServiceAccount, 资源配额等管理
  • 由 kubelet 启动的 static pod

2-4.5 kube-scheduler: 调度器

Scheduler: 单纯地调度Pod

  • 按照特定的策略的调度算法,将等待调度Pod绑定到集群中某个适合的Node,并写入绑定信息
  • 由 kubelet 启动的 static pod
  • 配置文件: /etc/kubernetes/manifests/kube-scheduler.yaml

2-5.6 kube-proxy: 服务抽象实现

kube-proxy运行于kubernetes集群中每个节点上

  • kube-proxy由 daemonset 控制器在各个节点上启动唯一实例
  • 配置文件: /var/lib/kube-proxy/config.conf(在Pod内,容器内)
$ kubectl -n kube-system get pods -o wide | grep 'kube-proxy'
kube-proxy-2bj4c                        1/1       Running            2          2d        172.16.81.162    dbk8s-node-01
kube-proxy-bf7lf                        1/1       NodeLost           1          2d        172.16.81.163    dbk8s-node-02
kube-proxy-p7n5s                        1/1       Running            7          2d        172.16.81.161    dbk8s-master

# 这里以 master 节点的kube-proxy为例,exec可以执行命令
$ kubectl -n kube-system exec kube-proxy-2bj4c -- cat /var/lib/kube-proxy/config.conf
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
  qps: 5
clusterCIDR: 192.168.1.0/24
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  minSyncPeriod: 0s
  scheduler: ""
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: ""
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
resourceContainer: /kube-proxy

2-5 kubectl详解

kubectl 是目前管理 k8s 集群的最强利器

2-5.1 集群访问配置

usage: kubectl config

  • 查看配置
kubectl config view 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: https://172.16.81.161:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

  • 创建一个集群入口
kubectl config set-cluster k8s1 --server=https://1.2.3.4
Cluster "k8s1" set.
  • 查看集群
kubectl config get-clusters 
NAME
kubernetes
k8s1

  • 删除集群
$ kubectl config delete-cluster k8s1 
deleted cluster k8s1 from /root/.kube/config
  • 创建context入口
$ kubectl config set-context admin1@k8s1 --user=admin1
Context "admin1@k8s1" created.
  • 查看context
$ kubectl config get-contexts 
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
          admin1@k8s1                                admin1             
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin
  • 选择默认context
$ kubectl config use-context admin1@k8s1 
Switched to context "admin1@k8s1".
  • 获取当前的context
$ kubectl config get-contexts 
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         admin1@k8s1                                admin1             
          kubernetes-admin@kubernetes   kubernetes   kubernetes-admin   
$ kubectl config current-context 
admin1@k8s1
  • 删除context
$ kubectl config delete-context admin1@k8s1
  • 创建凭证
$ kubectl config set-credentials testing-01 --username=testing-01 --password=abcdef@
User "testing-01" set.

# 这里可以看到已经有用户生成了
$ kubectl config view  | tail -4
- name: testing-01
  user:
    password: abcdef@
    username: testing-01

2-5.2 集群控制

  • kubectl [ create|apply|delete|label|edit|expose|scale ]
# 创建nginx-deployment配置文件
$ cat nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:alpine
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nginx-config
              mountPath: /etc/nginx/conf.d
            - name: web-root
              mountPath: /usr/share/nginx/html
      volumes:
        - name: nginx-config
          configMap:
            name: nginx-config
        - name: web-root
          hostPath:
            path: /var/www/html
  • 根据配置文件来创建
$ kubectl create -f nginx-deployment.yaml
deployment.apps "nginx-deployment-example" created
  • 查看刚刚创建的deployent
$ kubectl get pods --show-labels  | grep deployment
nginx-deployment-example-7945d4c5f-cg85r   0/1       ContainerCreating   0          14m       app=nginx,pod-template-hash=350180719
  • 给pod打标签
$ kubectl label pod/nginx-deployment-example-7945d4c5f-cg85r status=health
pod "nginx-deployment-example-7945d4c5f-cg85r" labeled

$ kubectl get pods --show-labels  | grep deployment
nginx-deployment-example-7945d4c5f-cg85r   0/1       ContainerCreating   0          16m       app=nginx,pod-template-hash=350180719,status=health
  • 通过kubect edit命令修改deployment的配置
root@dbk8s-master:~# kubectl get pods
NAME                                       READY     STATUS              RESTARTS   AGE
nginx-deployment-example-7945d4c5f-cg85r   0/1       ContainerCreating   0          20m

root@dbk8s-master:~# kubectl get deployment
NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment-example   1         1         1            0           20m
root@dbk8s-master:~# kubectl edit deployment nginx-deployment-example
##################
# 这里修改的部分如下,replicas: 3, 即创建的副本数
spec:
  progressDeadlineSeconds: 600
  replicas: 3
##################
deployment.extensions "nginx-deployment-example" edited

# 查看已存在的deployment
root@dbk8s-master:~# kubectl get deployment
NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment-example   3         3         3            0           21m

# 这里可以看到已经有3个副本了
root@dbk8s-master:~# kubectl get pods
NAME                                       READY     STATUS              RESTARTS   AGE
nginx-deployment-example-7945d4c5f-cg85r   0/1       ContainerCreating   0          21m
nginx-deployment-example-7945d4c5f-j9pr7   0/1       ContainerCreating   0          10s
nginx-deployment-example-7945d4c5f-s7rc8   0/1       ContainerCreating   0          10s
  • 使用kubectl scale扩容
# 扩容至10个副本
root@dbk8s-master:~# kubectl scale --replicas=10 deployment nginx-deployment-example
deployment.extensions "nginx-deployment-example" scaled

# 看到有10个副本正在被创建
root@dbk8s-master:~# kubectl get pods
NAME                                       READY     STATUS              RESTARTS   AGE
nginx-deployment-example-7945d4c5f-777rp   0/1       ContainerCreating   0          3s
nginx-deployment-example-7945d4c5f-cg85r   0/1       ContainerCreating   0          25m
nginx-deployment-example-7945d4c5f-dwwt6   0/1       ContainerCreating   0          3s
nginx-deployment-example-7945d4c5f-gcc8m   0/1       ContainerCreating   0          3s
nginx-deployment-example-7945d4c5f-j9pr7   0/1       ContainerCreating   0          4m
nginx-deployment-example-7945d4c5f-jnfmh   0/1       ContainerCreating   0          3s
nginx-deployment-example-7945d4c5f-rqq6h   0/1       ContainerCreating   0          3s
nginx-deployment-example-7945d4c5f-s7rc8   0/1       ContainerCreating   0          4m
nginx-deployment-example-7945d4c5f-ssxgt   0/1       ContainerCreating   0          3s
nginx-deployment-example-7945d4c5f-xn87h   0/1       ContainerCreating   0          3s

# 这里可以看到简图信息
root@dbk8s-master:~# kubectl get deployment
NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment-example   10        10        10           0           26m
  • 使用kubectl apply恢复原始配置
root@dbk8s-master:~# kubectl apply -f nginx-deployment.yaml
deployment.apps "nginx-deployment-example" configured
root@dbk8s-master:~# kubectl get deployment
NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment-example   1         1         1            0           28m
  • 将deployment删除
root@dbk8s-master:~# kubectl delete -f nginx-deployment.yaml
deployment.apps "nginx-deployment-example" deleted

# 正在终止pod
root@dbk8s-master:~# kubectl get pods --show-labels
NAME                                       READY     STATUS        RESTARTS   AGE       LABELS
nginx-deployment-example-7945d4c5f-cg85r   0/1       Terminating   0          29m       app=nginx,pod-template-hash=350180719,status=health

# 这里已经看不到了
root@dbk8s-master:~# kubectl get pods --show-labels
No resources found.

2-5.3 集群查看和问题调试

  • kubectl [ get|describe|logs|exec|attach ]
# 查看pod信息
$ kubectl get pods --all-namespaces -o wide

# 查看pod详细信息
$ kubectl -n kube-system describe pods weave-net-4mwv6

# 查看pod日志
$ kubectl -n kube-system logs -f weave-net-4mwv6

# 在pod中执行命令
$ kubectl -n kube-system exec monitoring-influxdb-cc95575b9-7d9xj -- uptime
 13:27:47 up 50 min,  0 users,  load average: 0.04, 0.10, 0.08

# 通过kubectl attach挂载到某个容器上, 如果该容器中有日志产生,则会实时打印出来
$ kubectl -n kube-system attach monitoring-influxdb-cc95575b9-7d9xj
Defaulting container name to influxdb.
Use 'kubectl describe pod/monitoring-influxdb-cc95575b9-7d9xj -n kube-system' to see all of the containers in this pod.
If you don't see a command prompt, try pressing enter.

Logo

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐