cfssl 工具

cfssl-certinfo 验证证书信息
  1. 用法:

    cfssl-certinfo -cert xxx.pem
    cfssl-certinfo -domain www.baidu.com
    

    将证书还原成 json 结构

    md5sum命令:
    验证一个文件的md5值

    md5sum 文件
    
  2. 从 .kubeconfig 配置文中反解获取证书

    echo “LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR3RENDQXFpZ0F3SUJBZ0lVYkh1czBSZkE2dHA1TjRYZnYvRWhkSlA4dytvd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1hqRUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjJKbGFXcHBibWN4RURBT0JnTlZCQWNUQjJKbAphV3BwYm1jeEN6QUpCZ05WQkFvVEFtOWtNUXd3Q2dZRFZRUUxFd052Y0hNeEVEQU9CZ05WQkFNVEIwWmxibWRaCmRXNHdIaGNOTWpFd09ESXpNRGt5TnpBd1doY05OREV3T0RFNE1Ea3lOekF3V2pCZk1Rc3dDUVlEVlFRR0V3SkQKVGpFUU1BNEdBMVVFQ0JNSFltVnBhbWx1WnpFUU1BNEdBMVVFQnhNSFltVnBhbWx1WnpFTE1Ba0dBMVVFQ2hNQwpiMlF4RERBS0JnTlZCQXNUQTI5d2N6RVJNQThHQTFVRUF4TUlhemh6TFc1dlpHVXdnZ0VpTUEwR0NTcUdTSWIzCkRRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRRGNnS2M1VzNER3hjUklkeGR0OXh3aEdrWDdOVXREM21pamNqTmMKTnVYb21KYTlpMGlySU9ObVNyNm1GV05NZU14MXdmeW9aY2J4OVdSRXZHcTYwRW0xZmo3SWpyRDByVDZRa1Ivbwp2QkRpWFJ5dWdjMGZPYUVMcG01OWc3SDJXUkpFVjhOV1RXSnBpUTdNcUloYVJqOW1QODEzejlHZnJiZ0hzUnBlCkQ4RGRXem1ZNklXbjlRaHYxcnZ3U1ZDVVJsZ0tUS1pvd0VCclEwZk9BZ0U3Nm5ibDZXZktIRjB0SUpFcTY1ZU0KZngvbExJRExuRGpIWE5SMWorenBvWGZjNWlBNU9jQ1A0bGhUUlBMZW9CUUF1WWpjc3Z4SGY4UVMrSzRvTTVwNwpRWGRVc0RPUk16dW96TktCVFRKUElEZlpXOFZaNTA0eGo4L2UxOXNrS1pYRUFRYzFBZ01CQUFHamRUQnpNQTRHCkExVWREd0VCL3dRRUF3SUZvREFUQmdOVkhTVUVEREFLQmdnckJnRUZCUWNEQWpBTUJnTlZIUk1CQWY4RUFqQUEKTUIwR0ExVWREZ1FXQkJSTjJDQnM0RWl5cjJOZDAzaFBGVEwwbFdPRXNUQWZCZ05WSFNNRUdEQVdnQlRnRnhmLwo4c0ozcmYxcm0xdUptN21nZGZFVmh6QU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFBajl5elI4c1JaODQ5VHVQCmV4VDdoNzFkRGxJc0N5aFQzL0pwRHJ3R0Z6L0s1ZFFkbXcyRVh1RWVacXpPYzlRSzZQMEJ1UG9JRTZTcCtvZXMKbnlvTWFUcE96Q0NjT1k1WlRZRzNtQUdiVlB6SlpMWFVIMnNiRDJpYkI4RzYzUlpVV080TG9jS2RqRm9NMjlyNQo4UGdlMEJGU2s4ZXFzam1GMFNITHNLeTFKZTRwU1ZDLzlzVGlpRHllTitwQk02NStSeHJxSWRpSSs2NVNma29vClpBSE5NYnJLNGo1Rlo3ZCtESk0xREZTcjA1V3lQY1lSVTB5bkg2UGd2My9zRHRHWSt0THRVckRiZnlySFZiZTEKczRzc3Bpd0lGQnc4Tk40bitNQmxkd1VxTFB3MGI5anJUbVlRcGQ3K0JIWkMvOFFVcGk2b0NLT1RJQWZkOUxqUApVa3NuV2c9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==”|base64 -d > 123.pem

    执行命令:

    cfssl-certinfo -cert 123.pem
    
kubeconfig 文件
  1. 这是一个k8s用户的配置文件
  2. 它里面含有证书信息
  3. 证书过期或更换,需要同步替换该文件

kubectl

管理 K8S 核心资源的三种基本方法:
1. 陈述式管理方法 - 主要依赖命令行CLI工具进行管理
  1. 查看名称空间
    kubectl get namespaces
    kubectl get ns
    
  2. 查询 default 名称空间里所有的资源
    kubectl get all -n default
    
  3. 创建/删除名称空间
    kubectl create ns app
    kubectl create namespace app
    kubectl delete ns app
    kubectl delete namespace app
    
  4. 创建 deployment 资源
    kubectl create deployment nginx-dp --image=harbor.od.com/public/nginx:v1.7.9 -n kube-public # 在kube-public名称空间下创建一个pod 控制器,而控制器的类型是 deployment,根据镜像,将container 运行在pod中
    kubectl get pods -n kube-public # 查看 kube-public名称空间下的 pods
    kubectl get pods -n kube-public -o wide # 详细查看
    
  5. 查看 deployment 资源
    kubectl get deployment -n kube-public  #如果不指定名称空间,使用 default 名称空间 
    
  6. 详细查看 deployment
    kubectl describe deployment nginx-dp -n kube-public
    
  7. 进入 pod
    kubectl exec -it nginx-dp-5dfc689474-tflm4 /bin/bash -n kube-public # nginx-dp-5dfc689474-tflm4 pod的名字
    
  8. 删除 deployment
    kubectl delete deploy nginx-dp -n kube-public
    
  9. 创建 service
    kubectl expose deployment nginx-dp --port=80 -n kube-public
    kubectl scale deployment nginx-dp --replicas=2 -n kube-public # 扩容deployment 
    
  10. 查看 service
    kubectl describe service nginx-dp -n kube-public
    

kubectl 是官方的CLI命令行工具,用于与 apiserver 进行通信,将用户在命令行输入的命令,组织并转化为apiserver能识别的信息,进而实现管理K8S各种资源的一种有效途径。
K8S集群管理资源的唯一入口是通过相应的方法调用apiserver的接口。
kubectl 对于增加资源,删除资源,查看资源都很方便,但是对于资源修改,比较麻烦。

2. 声明式管理方法 - 主要依赖统一资源配置清单(manifest)进行管理
  1. 查看资源配置清单
    kubectl get svc nginx-dp -o yaml -n kube-public
    
  2. 解释资源配置清单
    kubectl explain service
    
  3. 应用资源配置清单
    kubectl create/apply -f xxxx.yaml
    
  4. 在线修改资源配置清单
    kubectl edit svc service_name
    
  5. 离线修改资源配置清单,并引用
    vi xxx.yaml #修改资源配置清单
    kubectl apply -f xxx.yaml # 修改后并应用资源配置清单
    
  6. 声明式删除一个资源
    kubectl delete -f xxxx.yaml
    
3. GUI管理方法 - 主要依赖图形化操作界面进行管理

Kubernetes 网络模型

Kubernetes 设计了网络模型,但却将它的实现交给了网络插件,CNI网络插件最主要的功能就是实现POD资源能够跨宿主机进行通信。常见的CNI网络插件有:Flannel、Calico、Canal、Contiv、OpenContrail、NSX-T、Kube-router

部署K8S的CNI网络插件 - Flannel
集群规划
主机名角色ip
hdss7-21.host.comflannel10.4.7.21
hdss7-22.host.comflannel10.4.7.22
下载 flannel、解压、做软连接
  1. 解压,做软链接
    mkdir -p /opt/flannel-v0.11.0
    tar xf flannel-v0.11.0-linux-amd64.tar.gz -C /opt/flannel-v0.11.0/
    ln -s /opt/flannel-v0.11.0/ /opt/flannel
    
  2. flannel要连接etcd,需要将hdss7-200上的证书拷贝过来
    mkdir -p /opt/flannel-v0.11.0/cert
    cd /opt/flannel-v0.11.0/cert
    scp hdss7-200:/opt/certs/ca.pem .
    scp hdss7-200:/opt/certs/client.pem .
    scp hdss7-200:/opt/certs/client-key.pem .
    
创建配置 /opt/flannel-v0.11.0/subnet.env
FLANNEL_NETWORK=172.7.0.0/16		# pod的网段	
FLANNEL_SUBNET=172.7.21.1/24		# 本机运行pod的网段	
FLANNEL_MTU=1500
FLANNEL_IPMASQ=false
创建启动脚本 /opt/flannel-v0.11.0/flanneld.sh
  1. 创建启动脚本
    #!/bin/sh
    ./flanneld \
      --public-ip=10.4.7.21 \
      --etcd-endpoints=https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
      --etcd-keyfile=./cert/client-key.pem \
      --etcd-certfile=./cert/client.pem \
      --etcd-cafile=./cert/ca.pem \
      --iface=ens33 \
      --subnet-file=./subnet.env \
      --healthz-port=2401
    
  2. 增加执行权限,并创建日志目录
    chmod +x  /opt/flannel-v0.11.0/flanneld.sh
    mkdir -p /data/logs/flanneld
    
操作etcd,增加host-gw
cd /opt/etcd/
./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}'
./etcdctl member list # 查看etcd节点
./etcdctl get /coreos.com/network/config # 查看网络
创建 supervisor 配置
  1. 创建 /etc/supervisord.d/flannel.ini
    [program:flanneld-7-21]
    command=/opt/flannel/flanneld.sh                             ; the program (relative uses PATH, can take args)
    numprocs=1                                                   ; number of processes copies to start (def 1)
    directory=/opt/flannel                                       ; directory to cwd to before exec (def no cwd)
    autostart=true                                               ; start at supervisord start (default: true)
    autorestart=true                                             ; retstart at unexpected quit (default: true)
    startsecs=30                                                 ; number of secs prog must stay running (def. 1)
    startretries=3                                               ; max # of serial start failures (default 3)
    exitcodes=0,2                                                ; 'expected' exit codes for process (default 0,2)
    stopsignal=QUIT                                              ; signal used to kill process (default TERM)
    stopwaitsecs=10                                              ; max num secs to wait b4 SIGKILL (default 10)
    user=root                                                    ; setuid to this UNIX account to run the program
    redirect_stderr=true                                         ; redirect proc stderr to stdout (default false)
    stdout_logfile=/data/logs/flanneld/flanneld.stdout.log       ; stderr log path, NONE for none; default AUTO
    stdout_logfile_maxbytes=64MB                                 ; max # logfile bytes b4 rotation (default 50MB)
    stdout_logfile_backups=4                                     ; # of stdout logfile backups (default 10)
    stdout_capture_maxbytes=1MB                                  ; number of bytes in 'capturemode' (default 0)
    stdout_events_enabled=false                                  ; emit events on stdout writes (default false)
    
  2. 启动flannel 插件
    supervisorctl update 
    supervisorctl start flannel-7-21
    
Flannel 的 host-gw 模型

在这里插入图片描述

Flannel 的 VxLan 模型

在这里插入图片描述

iptables,解决集群内部的 SNAT 转换问题
  1. 安装iptables

    yum install iptables-services -y
    
  2. 启动iptables并设置开机自启动

    systemctl start iptables
    systemctl enable iptables 
    
  3. 优化 iptables 规则

    iptables-save |grep -i postrouting # 查看iptables规则
    iptables -t nat -D POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE # 删除一条iptables规则 -D表示删除
    # 插入一条规则 -I 表示插入 -s 表示源地址 ! 表示否定:插入一条,源地址网络为172.7.21.0/24网络地址,不是去往172.7.0.0/16
    # 这个网络,也不是从docker0出网,那么才做网络地址转换
    iptables -t nat -I POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE 
    iptables-save > /etc/sysconfig/iptables #将规则保存
    

    10.4.7.21主机上,来源是172.7.21.0/24段的docker的ip,目标ip不是172.7.0.0/16段,网络发包不从docker0网桥设备出站的,才进行SNAT转换。

    iptables-save |grep -i reject # 查看iptables的拒绝规则
    iptables -t filter -D INPUT -j REJECT --reject-with icmp-host-prohibited # 删除 INPUT 链拒绝规则
    iptables -t filter -D FORWARD -j REJECT --reject-with icmp-host-prohibited # 删除 FORWARD 链拒绝规则
    iptables-save > /etc/sysconfig/iptables
    
服务发现
  1. 简单来说,服务发现就是服务(应用)之间相互定位的过程。
  2. 在K8S集群里,POD的IP是不断变换的,如何”以不变应万变“?
    1. 抽象出Service资源,通过标签选择器,关联一组POD。
    2. 抽象出集群网络,通过相对固定的”集群IP“,使服务接入点固定。
部署K8S的服务发现插件 - CoreDNS
  1. 部署coredns (在运维主机上hdss7-200上执行)
    docker pull coredns/coredns:1.6.1 # 从docker仓库中拉取镜像
    docker tag coredns/coredns:1.6.1 harbor.od.com/public/coredns:v1.6.1 # 给镜像打tag
    docker push harbor.od.com/public/coredns:v1.6.1 # 将镜像推送到私有仓库
    
  2. 运维主机上配置nginx(hdss7-200上执行)
    mkdir /data/k8s-yaml
    vi /etc/nginx/conf.d/k8s-yaml.od.com.conf
    server {
        listen       80;
        server_name  k8s-yaml.od.com;
    
        location / {
            autoindex on;
            default_type text/plain;
            root /data/k8s-yaml;
        }
    }
    # 保存退出后,执行下面命令
    nginx -t
    nginx -s reload
    
  3. 配置内网DNS解析 (hdss7-11上执行)
    vi /var/named/od.com.zone
    
    $ORIGIN od.com.
    $TTL 600        ; 10 minutes
    @       IN SOA  dns.od.com. dnsadmin.od.com. (
                                    2021082205 ; serial  # 注意这里的序列化增加1,每次编辑改文件,序列化增加1
                                    10800      ; refresh (3 hours)
                                    900        ; retry (15 minutes)
                                    604800     ; expire (1 week)
                                    86400      ; minimum (1 day)
                                    )
                           NS    dns.od.com.
    $TTL 60 ; 1 minute
    dns                 A     10.4.7.11
    harbor              A     10.4.7.200
    k8s-yaml            A     10.4.7.200  # 添加一条 DNS解析规则
    
    
  4. 在运维主机上 /data/k8s-yaml目录下,编辑文件rbac.yaml、ConfigMap.yaml、Deployment.yaml、Service.yaml资源文件
    1. rbac.yaml
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: coredns
        namespace: kube-system
        labels:
            kubernetes.io/cluster-service: "true"
            addonmanager.kubernetes.io/mode: Reconcile
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        labels:
          kubernetes.io/bootstrapping: rbac-defaults
          addonmanager.kubernetes.io/mode: Reconcile
        name: system:coredns
      rules:
      - apiGroups:
        - ""
        resources:
        - endpoints
        - services
        - pods
        - namespaces
        verbs:
        - list
        - watch
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        annotations:
          rbac.authorization.kubernetes.io/autoupdate: "true"
        labels:
          kubernetes.io/bootstrapping: rbac-defaults
          addonmanager.kubernetes.io/mode: EnsureExists
        name: system:coredns
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: system:coredns
      subjects:
      - kind: ServiceAccount
        name: coredns
        namespace: kube-system
      
    2. cm.yaml
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: coredns
        namespace: kube-system
      data:
        Corefile: |
          .:53 {
              errors
              log
              health
              ready
              kubernetes cluster.local 10.254.0.0/16		# service网段
              forward . 10.4.7.11						# 物理机安装dns服务的地址		
              cache 30
              loop
              reload
              loadbalance
             }
      
    3. dp.yaml
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: coredns
        namespace: kube-system
        labels:
          k8s-app: coredns
          kubernetes.io/name: "CoreDNS"
      spec:
        replicas: 1
        selector:
          matchLabels:
            k8s-app: coredns
        template:
          metadata:
            labels:
              k8s-app: coredns
          spec:
            priorityClassName: system-cluster-critical
            serviceAccountName: coredns
            containers:
            - name: coredns
              image: harbor.od.com/public/coredns:v1.6.1
              args:
              - -conf
              - /etc/coredns/Corefile
              volumeMounts:
              - name: config-volume
                mountPath: /etc/coredns
              ports:
              - containerPort: 53
                name: dns
                protocol: UDP
              - containerPort: 53
                name: dns-tcp
                protocol: TCP
              - containerPort: 9153
                name: metrics
                protocol: TCP
              livenessProbe:
                httpGet:
                  path: /health
                  port: 8080
                  scheme: HTTP
                initialDelaySeconds: 60
                timeoutSeconds: 5
                successThreshold: 1
                failureThreshold: 5
            dnsPolicy: Default
            volumes:
              - name: config-volume
                configMap:
                  name: coredns
                  items:
                  - key: Corefile
                    path: Corefile
      
    4. svc.yaml
      apiVersion: v1
      kind: Service
      metadata:
        name: coredns
        namespace: kube-system
        labels:
          k8s-app: coredns
          kubernetes.io/cluster-service: "true"
          kubernetes.io/name: "CoreDNS"
      spec:
        selector:
          k8s-app: coredns
        clusterIP: 192.168.0.2		# dns服务的ip
        ports:
        - name: dns
          port: 53
          protocol: UDP
        - name: dns-tcp
          port: 53
        - name: metrics
          port: 9153
          protocol: TCP
      
  5. 在hdss7-21上执行,创建
    kubectl apply -f http://k8s-yaml.od.com/coredns/rbac.yaml
    kubectl apply -f http://k8s-yaml.od.com/coredns/cm.yaml
    kubectl apply -f http://k8s-yaml.od.com/coredns/dp.yaml
    kubectl apply -f http://k8s-yaml.od.com/coredns/svc.yaml
    
K8S服务暴露的两种方式

K8S的DNS实现了服务在集群“内”被自动发现,那么如何使得服务在K8S集群“外”被使用和访问呢?

  1. 使用 NodePort 型的 Service,该方式无法使用 Kube-proxy 的 ipvs 模型,只能使用 iptables 模型。
  2. 使用 Ingress 资源。Ingress 只能调度并暴露7层应用,特指http和https协议。
  • Ingress 是 K8S API的标准资源类型之一,也是一种核心资源。它其实就是一组基于域名和URL路径,把用户的请求转发至指定Service 资源的规则。
  • Ingress 可以将集群外部的请求流量,转发至集群内部,从而实现“服务暴露”。
  • Ingress 控制器是能够为 Ingress 资源监听某套接字,然后根据 Ingress 规则匹配机制路由调度流量的一个组件。
部署traefik (ingress 控制器)
  1. 在运维主机hdss7-200 上,准备traefik镜像
    docker pull traefik:v1.7.2-alpine
    docker tag traefik:v1.7.2-alpine harbor.od.com/public/traefik:v1.7.2
    docker push harbor.od.com/public/traefik:v1.7.2
    
  2. 创建资源配置清单
    cd /data/k8s-yaml/
    mkdir traefik
    cd traefik/
    vim rbac.yaml
    
    rbac.yaml
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: traefik-ingress-controller
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
      name: traefik-ingress-controller
    rules:
      - apiGroups:
          - ""
        resources:
          - services
          - endpoints
          - secrets
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - extensions
        resources:
          - ingresses
        verbs:
          - get
          - list
          - watch
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: traefik-ingress-controller
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: traefik-ingress-controller
    subjects:
    - kind: ServiceAccount
      name: traefik-ingress-controller
      namespace: kube-system
    
  3. ds.yaml文档
    apiVersion: extensions/v1beta1
    kind: DaemonSet
    metadata:
      name: traefik-ingress
      namespace: kube-system
      labels:
        k8s-app: traefik-ingress
    spec:
      template:
        metadata:
          labels:
            k8s-app: traefik-ingress
            name: traefik-ingress
        spec:
          serviceAccountName: traefik-ingress-controller
          terminationGracePeriodSeconds: 60
          containers:
          - image: harbor.od.com/public/traefik:v1.7.2
            name: traefik-ingress
            ports:
            - name: controller
              containerPort: 80
              hostPort: 81
            - name: admin-web
              containerPort: 8080
            securityContext:
              capabilities:
                drop:
                - ALL
                add:
                - NET_BIND_SERVICE
            args:
            - --api
            - --kubernetes
            - --logLevel=INFO
            - --insecureskipverify=true
            - --kubernetes.endpoint=https://10.4.7.10:7443		# keepalive VIP的地址
            - --accesslog
            - --accesslog.filepath=/var/log/traefik_access.log
            - --traefiklog
            - --traefiklog.filepath=/var/log/traefik.log
            - --metrics.prometheus
    
  4. svc.yaml
    kind: Service
    apiVersion: v1
    metadata:
      name: traefik-ingress-service
      namespace: kube-system
    spec:
      selector:
        k8s-app: traefik-ingress
      ports:
        - protocol: TCP
          port: 80
          name: controller
        - protocol: TCP
          port: 8080
          name: admin-web
    
  5. ingress.yaml
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: traefik-web-ui
      namespace: kube-system
      annotations:
        kubernetes.io/ingress.class: traefik
    spec:
      rules:
      - host: traefik.od.com
        http:
          paths:
    	  - path: /
            backend:
              serviceName: traefik-ingress-service
              servicePort: 8080
    
  6. 应用资源配置清单
    kubectl apply -f http://k8s-yaml.od.com/traefik/rbac.yaml
    kubectl apply -f http://k8s-yaml.od.com/traefik/ds.yaml
    kubectl apply -f http://k8s-yaml.od.com/traefik/svc.yaml
    kubectl apply -f http://k8s-yaml.od.com/traefik/ingress.yaml
    
  7. 在HDSS7-11 和 HDSS7-12两台主机上(配置了keepalive的主机上)的nginx配置反向代理
    vi /etc/nginx/conf.d/od.com.conf
    
    upstream default_backend_traefik {
        server 10.4.7.21:81    max_fails=3 fail_timeout=10s;	# 此ip为node ip+81端口,每个node节点都需要加上
        server 10.4.7.22:81    max_fails=3 fail_timeout=10s;
    }
    server {
        server_name *.od.com;					# 泛域名匹配,凡是od.con的域名内的http服务。都给到ingress里面
      
        location / {
            proxy_pass http://default_backend_traefik;
            proxy_set_header Host       $http_host;
            proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
        }
    }
    
Logo

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐