k8s三部曲(2)——kubernetes(TSL)非主节点安装dashboard

环境准备安装规划节点IP角色安装的组件192.168.0.111Masteretcd、kube-apiserver、kube-controller-manager、kube-scheduler、cfssl、kubectl192.168.0.112Node1docker 、kubelet、kube-proxy、flanneld 、cfssl、kubectl...

小兔爷

804人浏览 · 2019-11-27 16:52:31

小兔爷 · 2019-11-27 16:52:31 发布

环境准备

安装注意(防止误导)

以下案例是k8s的apiserver只能通过证书访问，如果k8s能通过service
account访问则只需要运行相应的dashboard.yml文件即可

安装规划

节点IP	角色	安装的组件
192.168.0.111	Master	etcd、kube-apiserver、kube-controller-manager、kube-scheduler、cfssl、kubectl
192.168.0.112	Node1	docker 、kubelet、kube-proxy、flanneld 、cfssl、kubectl
192.168.0.113	Node2	docker 、kubelet、kube-proxy flanneld、cfssl 、kubectl

准备证书(各个工作节点)

#dashboard证书放在这
$ mkdir -p /etc/kubernetes/ca/dashboard

#没有dashboard-csr.json则创建相应文件并填写下方内容
$ cat dashboard-csr.json 
{
  "CN": "system:dashboard",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "XS",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
$ cp dashboard-csr.json  /etc/kubernetes/ca/dashboard/
$ cd /etc/kubernetes/ca/dashboard/

#使用根证书(ca.pem)签发calico证书
$ cfssl gencert \
        -ca=/etc/kubernetes/ca/ca.pem \
        -ca-key=/etc/kubernetes/ca/ca-key.pem \
        -config=/etc/kubernetes/ca/ca-config.json \
        -profile=kubernetes kube-dashboard.json | cfssljson -bare dashboard
#我们最终要的是dashboard-key.pem和dashboard.pem
$ ls
dashboard.csr  dashboard-key.pem  dashboard.pem  kube-dashboard.json

准备kube-dashboard.kubeconfig配置(各个工作节点)

#--server kube-apiserver地址
$ kubectl config set-cluster kubernetes \
        --certificate-authority=/etc/kubernetes/ca/ca.pem \
        --embed-certs=true \
        --server=https://192.168.0.111:6443 \
        --kubeconfig=kube-dashboard.kubeconfig

$ kubectl config set-credentials kube-proxy \
        --client-certificate=/etc/kubernetes/ca/dashboard/dashboard.pem \
        --client-key=/etc/kubernetes/ca/dashboard/dashboard-key.pem \
        --embed-certs=true \
        --kubeconfig=kube-dashboard.kubeconfig
    
$ kubectl config set-context default \
        --cluster=kubernetes \
        --user=kube-dashboard \
        --kubeconfig=kube-dashboard.kubeconfig
        
$ kubectl config use-context default --kubeconfig=kube-dashboard.kubeconfig

$ mv kube-dashboard.kubeconfig /etc/kubernetes/kube-dashboard.kubeconfig

准备TSL证书

$ mkdir /certs
$ openssl req -nodes -newkey rsa:2048 -keyout certs/dashboard.key -out certs/dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=kubernetes-dashboard"
Generating a 2048 bit RSA private key
................+++
..............................................+++
writing new private key to 'certs/dashboard.key'
-----
No value provided for Subject Attribute C, skipped
No value provided for Subject Attribute ST, skipped
No value provided for Subject Attribute L, skipped
No value provided for Subject Attribute O, skipped
No value provided for Subject Attribute OU, skipped
[root@elasticsearch01 /]# ls /certs
dashboard.csr  dashboard.key

$ openssl x509 -req -sha256 -days 365 -in certs/dashboard.csr -signkey certs/dashboard.key -out certs/dashboard.crt
Signature ok
subject=/CN=kubernetes-dashboard
Getting Private key
$ ls certs/
dashboard.crt  dashboard.csr  dashboard.key

[root@elasticsearch01 /]# kubectl create secret generic kubernetes-dashboard-certs --from-file=certs -n kubernetes-dashboard
secret/kubernetes-dashboard-certs created

安装准备

官网安装Web UI

(Dashboard)的入口为 https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/
虽然官网上一键部署非常华丽，但是往往轮到我们自己部署的时候就没有这么简单啦，所以我们先下载下载进行部分修改

将部署文件下载到本地

$ wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

修改kubernetes-dashboard的deployment(总共三处地方)
```
$ vim recommended.yaml
```

第一处(关键)

配置tls是为了能够访问相关的https服务
配置kubeconfig是为了能够访问主节点上的kube-apiserver地址

args:
	  #- --auto-generate-certificateskubernetes-dashboard-7479fc4647-qzhpj
	  - --namespace=kubernetes-dashboard
	  - --tls-key-file=dashboard.key    # 配置密钥文件
	  - --tls-cert-file=dashboard.crt
	  # Uncomment the following line to manually specify Kubernetes API server Host
	  # If not specified, Dashboard will attempt to auto discover the API server and connect
	  # to it. Uncomment only if the default does not work.
	  # - --apiserver-host=https://192.168.0.111:6443
	  - --kubeconfig=/etc/kubernetes/kube-dashboard.kubeconfig

第二处(关键)

将所需文件挂载到容器中，不然将会提示找不到相应文件

 volumeMounts:
	            - name: kubernetes-dashboard-certs
	              mountPath: /certs
	              # Create on-disk volume to store exec logs
	            - mountPath: /tmp
	              name: tmp-volume
	              ###################修改开始#######################
	            - mountPath: /etc/kubernetes/kube-dashboard.kubeconfig
	              name: config
	              ###################修改结束#######################
	         
	      volumes:
	        - name: kubernetes-dashboard-certs
	          secret:
	            secretName: kubernetes-dashboard-certs
	        - name: tmp-volume
	          emptyDir: {}
	          ###################修改开始#######################
	        - hostPath:
	            path: /etc/kubernetes/kube-dashboard.kubeconfig
	          name: config
	           ##################修改结束#######################
	      serviceAccountName: kubernetes-dashboard
	      nodeSelector:
	        "beta.kubernetes.io/os": linux

第三处(关键)

将dashboard类型改为NodePort方便访问

kind: Service
	apiVersion: v1
	metadata:
	  labels:
	    k8s-app: kubernetes-dashboard
	  name: kubernetes-dashboard
	  namespace: kubernetes-dashboard
	spec:
	  ports:
	    - port: 443
	      targetPort: 8443
	  type: NodePort
	  selector:
	    k8s-app: kubernetes-dashboard

配置权限角色

kubectl create clusterrolebinding system:anonymous --clusterrole=cluster-admin --user=system:anonymous

开始安装

加载部署文件
```
$ kubectl apply -f recommended.yaml
```

查看服务端口

$ kubectl get svc -n kubernetes-dashboard
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   192.168.0.44    <none>        8000/TCP        30h
kubernetes-dashboard        NodePort    192.168.0.167   <none>        443:25773/TCP   30h

游览器安装证书

#生成crt文件
grep 'client-certificate-data' /etc/kubernetes/admin.conf | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.crt
#生成key文件
grep 'client-key-data' /etc/kubernetes/admin.conf | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.key
#生成p12证书文件(证书的生成和导入需要一个密码)
openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12 -name "kubernetes-client"
#将该证书文件导入到chrome浏览器中(自行百度)

访问dashboard

获取token认证

#部署账号
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
EOF
#角色绑定
$ cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF

#获取token
$ kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

#实例如下，复制token值即可
$ kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')
Name:         admin-user-token-rtpnj
Namespace:    kubernetes-dashboard
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: admin-user
              kubernetes.io/service-account.uid: 5ea34466-b4ae-4764-8888-ce21193dd913

Type:  kubernetes.io/service-account-token

Data
====
namespace:  20 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IkZIQ3RSNGR4aFJXMV9sRzFMRnhrdEdNbUFlYlRya1F6alg5Nmg2S0x0NWsifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC
9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXJ0cG5qIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWF
jY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI1ZWEzNDQ2Ni1iNGFlLTQ3NjQtODg4OC1jZTIxMTkzZGQ5MTMiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291
bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.enf2-uLp_Kg6wUYrWQRNyh11TeZLQ1xuxj_Ykll5Gvix9zxZ7e4oppPVzkaq4AukEXvWbraR8LCKmapRu10wu8l2Nt8n5qxAUM6ECgBT8sDki1IFDpZMw9XE8F4nJqjLYGRRMiQ-4two
XcrjRmvq7mDLmzA-sEv0Iq7dO-tLeEh6iB-IWLyh5VlmNgIecPVFyjzgg3arJj414SF31PGSUT2D68jYu1zgjjLHL-tl54r0lWuG91pPnvwaiOBD8ec0aej0ULnXA376Ap1ZtAHsNd3iwtQvBYQjGUtwJH8hzF_DkKW_TTdX91Q1-TVJcgv8opXr5y0TE
DA-i9kgEQ4owQ
ca.crt:     1346 bytes

输入token

深渊巨坑

无法找到kubeconfig
```
 no file or directory
```
解决: 进行目录挂载

连接不到api-server

[root@elasticsearch01 yaml]# kubectl logs kubernetes-dashboard-7649fbd576-r4wn2 --namespace=kube-system
2018/12/29 05:52:10 Starting overwatch
2018/12/29 05:52:10 Using apiserver-host location: https://10.2.8.44:6443
2018/12/29 05:52:10 Skipping in-cluster config
2018/12/29 05:52:10 Using random key for csrf signing
2018/12/29 05:52:10 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service account's configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://192.168.0.111:6443/version: x509: failed to load system roots and no roots provided
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ

解决：使用kubeconfig进行安全验证

dashboard http服务没问题(9090端口)，https服务有问题(8443端口)

查看相关日志可以发现连接8443端口拒绝连接

解决：配置tls的key和证书

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub