k8s集群管理（一）
cfssl 工具cfssl-certinfo 验证证书信息用法：cfssl-certinfo -cert xxx.pemcfssl-certinfo -domain www.baidu.com将证书还原成 json 结构md5sum命令：验证一个文件的md5值md5sum 文件从 .kubeconfig 配置文中反解获取证书echo “LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0
·
cfssl 工具
cfssl-certinfo 验证证书信息
-
用法：
cfssl-certinfo -cert xxx.pem cfssl-certinfo -domain www.baidu.com将证书还原成 json 结构
md5sum命令：
验证一个文件的md5值md5sum 文件 -
从 .kubeconfig 配置文中反解获取证书
echo “LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR3RENDQXFpZ0F3SUJBZ0lVYkh1czBSZkE2dHA1TjRYZnYvRWhkSlA4dytvd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1hqRUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjJKbGFXcHBibWN4RURBT0JnTlZCQWNUQjJKbAphV3BwYm1jeEN6QUpCZ05WQkFvVEFtOWtNUXd3Q2dZRFZRUUxFd052Y0hNeEVEQU9CZ05WQkFNVEIwWmxibWRaCmRXNHdIaGNOTWpFd09ESXpNRGt5TnpBd1doY05OREV3T0RFNE1Ea3lOekF3V2pCZk1Rc3dDUVlEVlFRR0V3SkQKVGpFUU1BNEdBMVVFQ0JNSFltVnBhbWx1WnpFUU1BNEdBMVVFQnhNSFltVnBhbWx1WnpFTE1Ba0dBMVVFQ2hNQwpiMlF4RERBS0JnTlZCQXNUQTI5d2N6RVJNQThHQTFVRUF4TUlhemh6TFc1dlpHVXdnZ0VpTUEwR0NTcUdTSWIzCkRRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRRGNnS2M1VzNER3hjUklkeGR0OXh3aEdrWDdOVXREM21pamNqTmMKTnVYb21KYTlpMGlySU9ObVNyNm1GV05NZU14MXdmeW9aY2J4OVdSRXZHcTYwRW0xZmo3SWpyRDByVDZRa1Ivbwp2QkRpWFJ5dWdjMGZPYUVMcG01OWc3SDJXUkpFVjhOV1RXSnBpUTdNcUloYVJqOW1QODEzejlHZnJiZ0hzUnBlCkQ4RGRXem1ZNklXbjlRaHYxcnZ3U1ZDVVJsZ0tUS1pvd0VCclEwZk9BZ0U3Nm5ibDZXZktIRjB0SUpFcTY1ZU0KZngvbExJRExuRGpIWE5SMWorenBvWGZjNWlBNU9jQ1A0bGhUUlBMZW9CUUF1WWpjc3Z4SGY4UVMrSzRvTTVwNwpRWGRVc0RPUk16dW96TktCVFRKUElEZlpXOFZaNTA0eGo4L2UxOXNrS1pYRUFRYzFBZ01CQUFHamRUQnpNQTRHCkExVWREd0VCL3dRRUF3SUZvREFUQmdOVkhTVUVEREFLQmdnckJnRUZCUWNEQWpBTUJnTlZIUk1CQWY4RUFqQUEKTUIwR0ExVWREZ1FXQkJSTjJDQnM0RWl5cjJOZDAzaFBGVEwwbFdPRXNUQWZCZ05WSFNNRUdEQVdnQlRnRnhmLwo4c0ozcmYxcm0xdUptN21nZGZFVmh6QU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFBajl5elI4c1JaODQ5VHVQCmV4VDdoNzFkRGxJc0N5aFQzL0pwRHJ3R0Z6L0s1ZFFkbXcyRVh1RWVacXpPYzlRSzZQMEJ1UG9JRTZTcCtvZXMKbnlvTWFUcE96Q0NjT1k1WlRZRzNtQUdiVlB6SlpMWFVIMnNiRDJpYkI4RzYzUlpVV080TG9jS2RqRm9NMjlyNQo4UGdlMEJGU2s4ZXFzam1GMFNITHNLeTFKZTRwU1ZDLzlzVGlpRHllTitwQk02NStSeHJxSWRpSSs2NVNma29vClpBSE5NYnJLNGo1Rlo3ZCtESk0xREZTcjA1V3lQY1lSVTB5bkg2UGd2My9zRHRHWSt0THRVckRiZnlySFZiZTEKczRzc3Bpd0lGQnc4Tk40bitNQmxkd1VxTFB3MGI5anJUbVlRcGQ3K0JIWkMvOFFVcGk2b0NLT1RJQWZkOUxqUApVa3NuV2c9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==”|base64 -d > 123.pem
执行命令：
cfssl-certinfo -cert 123.pem
kubeconfig 文件
- 这是一个k8s用户的配置文件
- 它里面含有证书信息
- 证书过期或更换，需要同步替换该文件
kubectl
管理 K8S 核心资源的三种基本方法：
1. 陈述式管理方法 - 主要依赖命令行CLI工具进行管理
- 查看名称空间
kubectl get namespaces kubectl get ns - 查询 default 名称空间里所有的资源
kubectl get all -n default - 创建/删除名称空间
kubectl create ns app kubectl create namespace app kubectl delete ns app kubectl delete namespace app - 创建 deployment 资源
kubectl create deployment nginx-dp --image=harbor.od.com/public/nginx:v1.7.9 -n kube-public # 在kube-public名称空间下创建一个pod 控制器，而控制器的类型是 deployment，根据镜像，将container 运行在pod中 kubectl get pods -n kube-public # 查看 kube-public名称空间下的 pods kubectl get pods -n kube-public -o wide # 详细查看 - 查看 deployment 资源
kubectl get deployment -n kube-public #如果不指定名称空间，使用 default 名称空间 - 详细查看 deployment
kubectl describe deployment nginx-dp -n kube-public - 进入 pod
kubectl exec -it nginx-dp-5dfc689474-tflm4 /bin/bash -n kube-public # nginx-dp-5dfc689474-tflm4 pod的名字 - 删除 deployment
kubectl delete deploy nginx-dp -n kube-public - 创建 service
kubectl expose deployment nginx-dp --port=80 -n kube-public kubectl scale deployment nginx-dp --replicas=2 -n kube-public # 扩容deployment - 查看 service
kubectl describe service nginx-dp -n kube-public
kubectl 是官方的CLI命令行工具，用于与 apiserver 进行通信，将用户在命令行输入的命令，组织并转化为apiserver能识别的信息，进而实现管理K8S各种资源的一种有效途径。
K8S集群管理资源的唯一入口是通过相应的方法调用apiserver的接口。
kubectl 对于增加资源，删除资源，查看资源都很方便，但是对于资源修改，比较麻烦。
2. 声明式管理方法 - 主要依赖统一资源配置清单（manifest）进行管理
- 查看资源配置清单
kubectl get svc nginx-dp -o yaml -n kube-public - 解释资源配置清单
kubectl explain service - 应用资源配置清单
kubectl create/apply -f xxxx.yaml - 在线修改资源配置清单
kubectl edit svc service_name - 离线修改资源配置清单，并引用
vi xxx.yaml #修改资源配置清单 kubectl apply -f xxx.yaml # 修改后并应用资源配置清单 - 声明式删除一个资源
kubectl delete -f xxxx.yaml
3. GUI管理方法 - 主要依赖图形化操作界面进行管理
Kubernetes 网络模型
Kubernetes 设计了网络模型，但却将它的实现交给了网络插件，CNI网络插件最主要的功能就是实现POD资源能够跨宿主机进行通信。常见的CNI网络插件有：Flannel、Calico、Canal、Contiv、OpenContrail、NSX-T、Kube-router
部署K8S的CNI网络插件 - Flannel
集群规划
| 主机名 | 角色 | ip |
|---|---|---|
| hdss7-21.host.com | flannel | 10.4.7.21 |
| hdss7-22.host.com | flannel | 10.4.7.22 |
下载 flannel、解压、做软连接
- 解压，做软链接
mkdir -p /opt/flannel-v0.11.0 tar xf flannel-v0.11.0-linux-amd64.tar.gz -C /opt/flannel-v0.11.0/ ln -s /opt/flannel-v0.11.0/ /opt/flannel - flannel要连接etcd，需要将hdss7-200上的证书拷贝过来
mkdir -p /opt/flannel-v0.11.0/cert cd /opt/flannel-v0.11.0/cert scp hdss7-200:/opt/certs/ca.pem . scp hdss7-200:/opt/certs/client.pem . scp hdss7-200:/opt/certs/client-key.pem .
创建配置 /opt/flannel-v0.11.0/subnet.env
FLANNEL_NETWORK=172.7.0.0/16 # pod的网段
FLANNEL_SUBNET=172.7.21.1/24 # 本机运行pod的网段
FLANNEL_MTU=1500
FLANNEL_IPMASQ=false
创建启动脚本 /opt/flannel-v0.11.0/flanneld.sh
- 创建启动脚本
#!/bin/sh ./flanneld \ --public-ip=10.4.7.21 \ --etcd-endpoints=https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \ --etcd-keyfile=./cert/client-key.pem \ --etcd-certfile=./cert/client.pem \ --etcd-cafile=./cert/ca.pem \ --iface=ens33 \ --subnet-file=./subnet.env \ --healthz-port=2401 - 增加执行权限，并创建日志目录
chmod +x /opt/flannel-v0.11.0/flanneld.sh mkdir -p /data/logs/flanneld
操作etcd，增加host-gw
cd /opt/etcd/
./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}'
./etcdctl member list # 查看etcd节点
./etcdctl get /coreos.com/network/config # 查看网络
创建 supervisor 配置
- 创建 /etc/supervisord.d/flannel.ini
[program:flanneld-7-21] command=/opt/flannel/flanneld.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/flannel ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/flanneld/flanneld.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false) - 启动flannel 插件
supervisorctl update supervisorctl start flannel-7-21
Flannel 的 host-gw 模型
Flannel 的 VxLan 模型
iptables，解决集群内部的 SNAT 转换问题
-
安装iptables
yum install iptables-services -y -
启动iptables并设置开机自启动
systemctl start iptables systemctl enable iptables -
优化 iptables 规则
iptables-save |grep -i postrouting # 查看iptables规则 iptables -t nat -D POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE # 删除一条iptables规则 -D表示删除 # 插入一条规则 -I 表示插入 -s 表示源地址 ! 表示否定：插入一条，源地址网络为172.7.21.0/24网络地址，不是去往172.7.0.0/16 # 这个网络，也不是从docker0出网，那么才做网络地址转换 iptables -t nat -I POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE iptables-save > /etc/sysconfig/iptables #将规则保存10.4.7.21主机上，来源是172.7.21.0/24段的docker的ip，目标ip不是172.7.0.0/16段，网络发包不从docker0网桥设备出站的，才进行SNAT转换。
iptables-save |grep -i reject # 查看iptables的拒绝规则 iptables -t filter -D INPUT -j REJECT --reject-with icmp-host-prohibited # 删除 INPUT 链拒绝规则 iptables -t filter -D FORWARD -j REJECT --reject-with icmp-host-prohibited # 删除 FORWARD 链拒绝规则 iptables-save > /etc/sysconfig/iptables
服务发现
- 简单来说，服务发现就是服务（应用）之间相互定位的过程。
- 在K8S集群里，POD的IP是不断变换的，如何”以不变应万变“？
- 抽象出Service资源，通过标签选择器，关联一组POD。
- 抽象出集群网络，通过相对固定的”集群IP“，使服务接入点固定。
部署K8S的服务发现插件 - CoreDNS
- 部署coredns (在运维主机上hdss7-200上执行）
docker pull coredns/coredns:1.6.1 # 从docker仓库中拉取镜像 docker tag coredns/coredns:1.6.1 harbor.od.com/public/coredns:v1.6.1 # 给镜像打tag docker push harbor.od.com/public/coredns:v1.6.1 # 将镜像推送到私有仓库 - 运维主机上配置nginx（hdss7-200上执行）
mkdir /data/k8s-yaml vi /etc/nginx/conf.d/k8s-yaml.od.com.conf server { listen 80; server_name k8s-yaml.od.com; location / { autoindex on; default_type text/plain; root /data/k8s-yaml; } } # 保存退出后，执行下面命令 nginx -t nginx -s reload - 配置内网DNS解析 （hdss7-11上执行）
vi /var/named/od.com.zone $ORIGIN od.com. $TTL 600 ; 10 minutes @ IN SOA dns.od.com. dnsadmin.od.com. ( 2021082205 ; serial # 注意这里的序列化增加1，每次编辑改文件，序列化增加1 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.od.com. $TTL 60 ; 1 minute dns A 10.4.7.11 harbor A 10.4.7.200 k8s-yaml A 10.4.7.200 # 添加一条 DNS解析规则 - 在运维主机上 /data/k8s-yaml目录下，编辑文件rbac.yaml、ConfigMap.yaml、Deployment.yaml、Service.yaml资源文件
- rbac.yaml
apiVersion: v1 kind: ServiceAccount metadata: name: coredns namespace: kube-system labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: kubernetes.io/bootstrapping: rbac-defaults addonmanager.kubernetes.io/mode: Reconcile name: system:coredns rules: - apiGroups: - "" resources: - endpoints - services - pods - namespaces verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults addonmanager.kubernetes.io/mode: EnsureExists name: system:coredns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:coredns subjects: - kind: ServiceAccount name: coredns namespace: kube-system - cm.yaml
apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { errors log health ready kubernetes cluster.local 10.254.0.0/16 # service网段 forward . 10.4.7.11 # 物理机安装dns服务的地址 cache 30 loop reload loadbalance } - dp.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: coredns namespace: kube-system labels: k8s-app: coredns kubernetes.io/name: "CoreDNS" spec: replicas: 1 selector: matchLabels: k8s-app: coredns template: metadata: labels: k8s-app: coredns spec: priorityClassName: system-cluster-critical serviceAccountName: coredns containers: - name: coredns image: harbor.od.com/public/coredns:v1.6.1 args: - -conf - /etc/coredns/Corefile volumeMounts: - name: config-volume mountPath: /etc/coredns ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP - containerPort: 9153 name: metrics protocol: TCP livenessProbe: httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 dnsPolicy: Default volumes: - name: config-volume configMap: name: coredns items: - key: Corefile path: Corefile - svc.yaml
apiVersion: v1 kind: Service metadata: name: coredns namespace: kube-system labels: k8s-app: coredns kubernetes.io/cluster-service: "true" kubernetes.io/name: "CoreDNS" spec: selector: k8s-app: coredns clusterIP: 192.168.0.2 # dns服务的ip ports: - name: dns port: 53 protocol: UDP - name: dns-tcp port: 53 - name: metrics port: 9153 protocol: TCP
- rbac.yaml
- 在hdss7-21上执行，创建
kubectl apply -f http://k8s-yaml.od.com/coredns/rbac.yaml kubectl apply -f http://k8s-yaml.od.com/coredns/cm.yaml kubectl apply -f http://k8s-yaml.od.com/coredns/dp.yaml kubectl apply -f http://k8s-yaml.od.com/coredns/svc.yaml
K8S服务暴露的两种方式
K8S的DNS实现了服务在集群“内”被自动发现，那么如何使得服务在K8S集群“外”被使用和访问呢？
- 使用 NodePort 型的 Service，该方式无法使用 Kube-proxy 的 ipvs 模型，只能使用 iptables 模型。
- 使用 Ingress 资源。Ingress 只能调度并暴露7层应用，特指http和https协议。
- Ingress 是 K8S API的标准资源类型之一，也是一种核心资源。它其实就是一组基于域名和URL路径，把用户的请求转发至指定Service 资源的规则。
- Ingress 可以将集群外部的请求流量，转发至集群内部，从而实现“服务暴露”。
- Ingress 控制器是能够为 Ingress 资源监听某套接字，然后根据 Ingress 规则匹配机制路由调度流量的一个组件。
部署traefik （ingress 控制器）
- 在运维主机hdss7-200 上，准备traefik镜像
docker pull traefik:v1.7.2-alpine docker tag traefik:v1.7.2-alpine harbor.od.com/public/traefik:v1.7.2 docker push harbor.od.com/public/traefik:v1.7.2 - 创建资源配置清单
rbac.yamlcd /data/k8s-yaml/ mkdir traefik cd traefik/ vim rbac.yamlapiVersion: v1 kind: ServiceAccount metadata: name: traefik-ingress-controller namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: traefik-ingress-controller rules: - apiGroups: - "" resources: - services - endpoints - secrets verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses verbs: - get - list - watch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: traefik-ingress-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: traefik-ingress-controller subjects: - kind: ServiceAccount name: traefik-ingress-controller namespace: kube-system - ds.yaml文档
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: traefik-ingress namespace: kube-system labels: k8s-app: traefik-ingress spec: template: metadata: labels: k8s-app: traefik-ingress name: traefik-ingress spec: serviceAccountName: traefik-ingress-controller terminationGracePeriodSeconds: 60 containers: - image: harbor.od.com/public/traefik:v1.7.2 name: traefik-ingress ports: - name: controller containerPort: 80 hostPort: 81 - name: admin-web containerPort: 8080 securityContext: capabilities: drop: - ALL add: - NET_BIND_SERVICE args: - --api - --kubernetes - --logLevel=INFO - --insecureskipverify=true - --kubernetes.endpoint=https://10.4.7.10:7443 # keepalive VIP的地址 - --accesslog - --accesslog.filepath=/var/log/traefik_access.log - --traefiklog - --traefiklog.filepath=/var/log/traefik.log - --metrics.prometheus - svc.yaml
kind: Service apiVersion: v1 metadata: name: traefik-ingress-service namespace: kube-system spec: selector: k8s-app: traefik-ingress ports: - protocol: TCP port: 80 name: controller - protocol: TCP port: 8080 name: admin-web - ingress.yaml
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: traefik-web-ui namespace: kube-system annotations: kubernetes.io/ingress.class: traefik spec: rules: - host: traefik.od.com http: paths: - path: / backend: serviceName: traefik-ingress-service servicePort: 8080 - 应用资源配置清单
kubectl apply -f http://k8s-yaml.od.com/traefik/rbac.yaml kubectl apply -f http://k8s-yaml.od.com/traefik/ds.yaml kubectl apply -f http://k8s-yaml.od.com/traefik/svc.yaml kubectl apply -f http://k8s-yaml.od.com/traefik/ingress.yaml - 在HDSS7-11 和 HDSS7-12两台主机上（配置了keepalive的主机上）的nginx配置反向代理
vi /etc/nginx/conf.d/od.com.conf upstream default_backend_traefik { server 10.4.7.21:81 max_fails=3 fail_timeout=10s; # 此ip为node ip+81端口，每个node节点都需要加上 server 10.4.7.22:81 max_fails=3 fail_timeout=10s; } server { server_name *.od.com; # 泛域名匹配，凡是od.con的域名内的http服务。都给到ingress里面 location / { proxy_pass http://default_backend_traefik; proxy_set_header Host $http_host; proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for; } }
K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容
更多推荐
1
0- 0
已为社区贡献8条内容
所有评论(0)