1、环境准备:
虚拟机或服务器的master节点CPU需2核以上,可通过下述命令查看:
cat /proc/cpuinfo| grep "physical id"| sort| uniq| wc -l

(1)k8s各节点SSH设置免密登录
所有节点用root用户操作,全部设置免密登陆,不做细分。

(2)时间同步。
yum install -y ntpdate
ntpdate -u ntp.api.bz

(3)所有节点必须关闭防火墙及swap。
 systemctl disable firewalld.service 
 systemctl stop firewalld.service 
 systemctl status firewalld.service 
 sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
 setenforce 0
 sed -i 's/.∗swap.∗.∗swap.∗/# \1/g' /etc/fstab
 swapoff -a
如果修改/etc/fstab里的swap相关信息,需要重启。

2、节点规划:


主机名    IP&Role
10.10.1.200    master1    etcd、Master、Node、keepalived
10.10.1.199  master2    etcd、Master、Node、keepalived
10.10.1.198  master3    etcd、Master、Node、keepalived
10.10.1.201  node1
10.10.1.202  node2
10.10.1.203  node3
10.10.1.210    cluster.kube.com

所有节点主机名和IP加入/etc/hosts解析
cat /etc/hosts
10.10.1.200  master1
10.10.1.201  node1
10.10.1.202  node2
10.10.1.203  node3
10.10.1.198 master3
10.10.1.199 master2
10.10.1.210    cluster.kube.com

3、镜像清单:

k8s.gcr.io/kube-proxy                v1.13.0             8fa56d18961f        9 days ago          80.2MB
k8s.gcr.io/kube-scheduler            v1.13.0             9508b7d8008d        9 days ago          79.6MB
k8s.gcr.io/kube-controller-manager   v1.13.0             d82530ead066        9 days ago          146MB
k8s.gcr.io/kube-apiserver            v1.13.0             f1ff9b7e3d6e        9 days ago          181MB
quay.io/calico/node                  v3.3.2              4e9be81e3a59        9 days ago          75.3MB
quay.io/calico/cni                   v3.3.2              490d921fa49c        9 days ago          75.4MB
k8s.gcr.io/coredns                   1.2.6               f59dcacceff4        5 weeks ago         40MB
k8s.gcr.io/etcd                      3.2.24              3cab8e1b9802        2 months ago        220MB
quay.io/coreos/flannel               v0.10.0-s390x       463654e4ed2d        10 months ago       47MB
quay.io/coreos/flannel               v0.10.0-ppc64l      e2f67d69dd84        10 months ago       53.5MB
quay.io/coreos/flannel               v0.10.0-arm         c663d02f7966        10 months ago       39.9MB
quay.io/coreos/flannel               v0.10.0-amd64       f0fad859c909        10 months ago       44.6MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        11 months ago       742kB
--------------------- ----------------------------------------

4.部署keepalived

此处的keeplived的主要作用是为haproxy提供vip(10.10.1.210),在三个haproxy实例之间提供主备,降低当其中一个haproxy失效的时对服务的影响。
(1)系统配置
cat >> /etc/sysctl.conf << EOF
net.ipv4.ip_forward = 1
EOF
sysctl -p
(2)安装keepalived
yum install -y keepalived
(3)配置keepalived:
【注意:VIP地址是否正确,且各个节点的priority不同,master1节点为MASTER,其余节点为BACKUP,killall -0 意思是根据进程名称检测进程是否存活】
 

--------------master1:
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface ens32
    virtual_router_id 51
    priority 250
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 35f18af7190d51c9f7f78f37300a0cbd
    }
    virtual_ipaddress {
        10.10.1.210
    }
    track_script {
        check_haproxy
    }
}
EOF
--------------master2:
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens32
    virtual_router_id 51
    priority 249
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 35f18af7190d51c9f7f78f37300a0cbd
    }
    virtual_ipaddress {
        10.10.1.210
    }
    track_script {
        check_haproxy
    }
}
EOF

--------------master3:
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens32
    virtual_router_id 51
    priority 248
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 35f18af7190d51c9f7f78f37300a0cbd
    }
    virtual_ipaddress {
        10.10.1.210
    }
    track_script {
        check_haproxy
    }
}
EOF

(4)启动并检测服务
systemctl enable keepalived.service 
systemctl start keepalived.service
systemctl status keepalived.service 
ip address show ens32


5.部署haproxy【所有master】

此处的haproxy为apiserver提供反向代理,haproxy将所有请求轮询转发到每个master节点上。相对于仅仅使用keepalived主备模式仅单个master节点承载流量,这种方式更加合理、健壮。

(1)系统配置
cat >> /etc/sysctl.conf << EOF
net.ipv4.ip_nonlocal_bind = 1
EOF
sysctl -p

(2)安装haproxy
yum install -y haproxy
(3)配置haproxy【三个master节点一样】
 

cat > /etc/haproxy/haproxy.cfg << EOF
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     40000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kubernetes-apiserver
    mode                 tcp
    bind                 *:16443
    option               tcplog
    default_backend      kubernetes-apiserver

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
    mode        tcp
    balance     roundrobin
    server  master1 10.10.1.200:6443 check
    server  master2 10.10.1.199:6443 check
    server  master3 10.10.1.198:6443 check

#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
    bind                 *:1080
    stats auth           admin:awesomePassword
    stats refresh        5s
    stats realm          HAProxy\ Statistics
    stats uri            /admin?stats
EOF

(4)启动并检测服务
 systemctl enable haproxy.service 
 systemctl start haproxy.service 
 systemctl status haproxy.service 
 ss -lnt | grep -E "16443|1080"
LISTEN     0      128          *:1080                     *:*                  
LISTEN     0      128          *:16443                    *:*
--------------------- 
6.安装kubeadm、kubectl、kubelet、docker【为了方便,所有节点都直接复制粘贴进行相同操作】

(1)系统配置
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

(2)安装docker
### 设置docker-ce的yum源 ###
yum-config-manager  --add-repo  https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum -y repolist
yum -y install docker-ce-18.06.1.ce-3.el7 --disableexcludes=docker-ce

# 编辑systemctl的Docker启动文件
sed -i "13i ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT" /usr/lib/systemd/system/docker.service
cat /usr/lib/systemd/system/docker.service |grep ExecStart

(3)安装kubernetes
### 设置kubernetes的yum源 ###
cat <<EOF>> /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum -y repolist
yum -y install  kubelet-1.13.0  kubeadm-1.13.0  kubectl-1.13.0  --disableexcludes=kubernetes

(4)# 启动docker&kubelet,设置开机启动
systemctl daemon-reload
systemctl restart docker.service 
systemctl enable docker.service 
systemctl status docker.service 
systemctl enable kubelet.service 

(5)#提前拉取镜像更改标签【所有节点】
docker pull mirrorgooglecontainers/kube-apiserver:v1.13.0
docker pull mirrorgooglecontainers/kube-controller-manager:v1.13.0
docker pull mirrorgooglecontainers/kube-scheduler:v1.13.0
docker pull mirrorgooglecontainers/kube-proxy:v1.13.0
docker pull mirrorgooglecontainers/pause:3.1
docker pull mirrorgooglecontainers/etcd:3.2.24
docker pull coredns/coredns:1.2.6

docker tag mirrorgooglecontainers/kube-apiserver:v1.13.0          k8s.gcr.io/kube-apiserver:v1.13.0
docker tag mirrorgooglecontainers/kube-controller-manager:v1.13.0 k8s.gcr.io/kube-controller-manager:v1.13.0
docker tag mirrorgooglecontainers/kube-scheduler:v1.13.0          k8s.gcr.io/kube-scheduler:v1.13.0
docker tag mirrorgooglecontainers/kube-proxy:v1.13.0              k8s.gcr.io/kube-proxy:v1.13.0
docker tag mirrorgooglecontainers/pause:3.1                       k8s.gcr.io/pause:3.1
docker tag mirrorgooglecontainers/etcd:3.2.24                     k8s.gcr.io/etcd:3.2.24
docker tag coredns/coredns:1.2.6                                  k8s.gcr.io/coredns:1.2.6

docker rmi mirrorgooglecontainers/kube-apiserver:v1.13.0           
docker rmi mirrorgooglecontainers/kube-controller-manager:v1.13.0  
docker rmi mirrorgooglecontainers/kube-scheduler:v1.13.0           
docker rmi mirrorgooglecontainers/kube-proxy:v1.13.0               
docker rmi mirrorgooglecontainers/pause:3.1                        
docker rmi mirrorgooglecontainers/etcd:3.2.24                      
docker rmi coredns/coredns:1.2.6                                 
  
docker pull xiyangxixia/k8s-flannel:v0.10.0-s390x
docker tag xiyangxixia/k8s-flannel:v0.10.0-s390x quay.io/coreos/flannel:v0.10.0-s390x
docker rmi  xiyangxixia/k8s-flannel:v0.10.0-s390x

docker pull xiyangxixia/k8s-flannel:v0.10.0-ppc64le
docker tag xiyangxixia/k8s-flannel:v0.10.0-ppc64le quay.io/coreos/flannel:v0.10.0-ppc64l
docker rmi xiyangxixia/k8s-flannel:v0.10.0-ppc64le

docker pull xiyangxixia/k8s-flannel:v0.10.0-arm
docker tag xiyangxixia/k8s-flannel:v0.10.0-arm quay.io/coreos/flannel:v0.10.0-arm
docker rmi  xiyangxixia/k8s-flannel:v0.10.0-arm

docker pull xiyangxixia/k8s-flannel:v0.10.0-amd64
docker tag xiyangxixia/k8s-flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.10.0-amd64
docker rmi  xiyangxixia/k8s-flannel:v0.10.0-amd64
--------------------- 
7.部署master1

(1)编辑kubeadm配置文件
cd ~
cat > kubeadm-config.yaml << EOF
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.13.0
apiServer:
  certSANs:
    - "cluster.kube.com"
controlPlaneEndpoint: "cluster.kube.com:16443"
networking:
  podSubnet: "10.244.0.0/16"
EOF
注意:【podSubnet如果使用flannel方案,则推荐设置为10.244.0.0/16】
------------------------------------------------------------------------------------------------------
(2)初始化第一个master节点
kubeadm init --config kubeadm-config.yaml 
…………
记录加入集群的token等:
kubeadm join cluster.kube.com:16443 --token h0q766.ng8jo85gpbdffqks --discovery-token-ca-cert-hash sha256:8d493104d82j59b3c777d4bc74822ecbe21ac618ea876acafb5876ebf4c45e80
此时kuberctl get nodes查看节点状态不可能是Ready,coredns会有问题比如taint什么的因为还没有设置网络插件先不用管。

第一个master上作为root用户执行下列命令:(如果不是root用户就需要执行kubeadm初始化之后提示的三个命令)
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> /etc/profile
source /etc/profile 
echo $KUBECONFIG  
------------------------------------------------------------------------------------------------------
(3)安装网络插件

设置系统参数【所有节点】:
sysctl net.bridge.bridge-nf-call-iptables=1
创建网络:
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml

查看状态:
kubectl  get nodes
kubectl get pods --all-namespaces
-------------------------------------------------------
(4)复制相关文件到其他两个master节点上
ssh root@master2 mkdir -p /etc/kubernetes/pki/etcd
scp /etc/kubernetes/admin.conf root@master2:/etc/kubernetes 
scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@master2:/etc/kubernetes/pki
scp /etc/kubernetes/pki/etcd/ca.* root@master2:/etc/kubernetes/pki/etcd

ssh root@master3 mkdir -p /etc/kubernetes/pki/etcd
scp /etc/kubernetes/admin.conf root@master3:/etc/kubernetes
scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@master3:/etc/kubernetes/pki
scp /etc/kubernetes/pki/etcd/ca.* root@master3:/etc/kubernetes/pki/etcd
--------------------- 
8.部署其他的master

执行加入语句,后加--experimental-control-plane参数即可。
kubeadm join cluster.kube.com:16443 --token h0q766.ngajo85gpbdffqks --discovery-token-ca-cert-hash sha256:8d493104d87059b3c777d4bc74822ecbe21ac618ea876acafb5876ebf4c45e80  --experimental-control-plane
完成后执行:
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> /etc/profile
source /etc/profile 
echo $KUBECONFIG
===================================================================
2.6、查看kubernetes集群各个状态【任意master执行即可】
(1)kubectl get nodes -o wide
NAME       STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
master1   Ready    master   11m     v1.13.0   10.10.1.200   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0
master2   Ready    master   2m16s   v1.13.0   10.10.1.199   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0
master3   Ready    master   45s     v1.13.0   10.10.1.198   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0

(2)kubectl get pods --all-namespaces -o wide

(3)kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   

(4)查看etcd集群状态:
进入容器:
kubectl exec -ti -n kube-system etcd-master1 sh
设定环境变量:
export ETCDCTL_API=3
执行带证书的语句:
etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key member list
得到类似如下的结果:
4e0333cc4a713ecd, started, master3, https://10.10.1.198:2380, https://10.10.1.198:2379
865b63300bc38ab7, started, master2, https://10.10.1.199:2380, https://10.10.1.199:2379
d5c79fd433825701, started, master1, https://10.10.1.200:2380, https://10.10.1.200:2379

===============================================================================
三、Node节点的配置【所有Worker】
确认已经完成了前面提到的“所有节点”的设置,包括hosts设置、SSH设置、安装了docker$k8s、提前拉取了镜像
3.1、参数设置
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe ip_vs83【这个可能会提示没有,没关系】

3.2、执行加入语句:
kubeadm join cluster.kube.com:16443 --token h0q766.ngajo85gpbdffqks --discovery-token-ca-cert-hash sha256:8d493104d87059b3c777d4bc74822ecbe21ac618ea876acafb5876ebf4c45e80
忘记或token超期的处理方式:
Master上执行以下命令获取新的token值,:
kubeadm token create
在master节点上执行以下命令链来获取新的hash值:
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null |   openssl dgst -sha256 -hex | sed 's/^.* //'
然后其他节点可以再通过命令加入:
kubeadm join 192.168.1.201:6443 --token `新的token值` --discovery-token-ca-cert-hash `新的hash值`

3.3、master1查看状态
kubectl get nodes -o wide
kubectl get pods --all-namespaces -o wide
--------------------- 

四、在k8s基础上配置dashboard
4.1、所有节点提前下载镜像并修改标签:【用1.8.3的】可以上https://hub.docker.com/r/library/查具体信息;
docker pull k8scn/kubernetes-dashboard-amd64:v1.8.3
docker tag k8scn/kubernetes-dashboard-amd64:v1.8.3  k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3
docker rmi k8scn/kubernetes-dashboard-amd64:v1.8.3
docker images|grep dashboard

4.2、在master1上,应用dashboard的部署文件
(1)、下载和修改文件
wget  https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml  
修改上述文件,修改里面的镜像版本策略和版本,修改nodeport,默认的是v1.10.0的;
---------镜像策略部分--------------------------------
containers:
- name: kubernetes-dashboard
  imagePullPolicy: IfNotPresent
  image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3
---------------------------------------
targetPort部分如下:改为nodePort类型并指定30080端口(实际看情况可以指定其他端口)
# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30080
  selector:
    k8s-app: kubernetes-dashboard
--------------------------------------------------
然后执行:
kubectl create -f kubernetes-dashboard.yaml

4.3、查看创建情况:
kubectl get pods -n kube-system
创建不对的话可以使用kubectl  delete  -f  kubernetes-dashboard.yaml删掉一切;
创建正常之后继续:

4.4、创建serviceaccount 用于登陆dashboard【名为dashboard-admin】
kubectl create serviceaccount dashboard-admin -n kube-system

4.5、创建clusterrolebinding
kubectl create clusterrolebinding cluster-dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
kubectl  get svc --all-namespaces可以查看服务的IP及端口对应如下:
kube-system   kubernetes-dashboard   NodePort    10.97.145.51   <none>        443:30080/TCP   

4.6、查找刚刚生成的secret的token
先找secret:格式为 kubectl  get  secret -n kube-system|grep dashboard-admin
找到刚创建的secret名为:dashboard-admin-token-xxxx
实际查得:dashboard-admin-token-2kw7n
再找token,格式为:kubectl describe secret dashboard-admin-token-xxxx -n kube-system
实际操作
kubectl describe secret dashboard-admin-token-2kw7n   -n kube-system
复制token:
etoken:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY2654nQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4td3ZmaGoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiN2YwNDY0ZGUtZmYzZS0xMWU4LWI0YTYtMDAwYzI5N2ZkNzk0Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.h3GdIgtH3KAr58GVsxqSyj-i9lnreRPGiResvof5viWP_P0X5l7q_dUclZf9UWhcPV7gaomOLZfttzsYNBfvb0KJgl6PMmnzbjaYHGbcpyFLWF0_51XxvjSnFpLdkH8x_bcI3ZewMHoYRu5-X1gjfNofHOXm4TaX9pMtXqSih-TKhwlUoVAN2lQ2pBcSdlpFHCg2gf85jb3Bm6LGXT5oBWkupXPmFKGX2EX3YgX4J8hPo-gmE-yHKIcRvVy_cfpAjJlmKUAnbXQSJCban0R7GIpYhNPrwJoMuOqCSLGgTPvG1kfvdeANHbA6wVm70SBc50STmrJJFGaML1urQzDtIA

7、登陆dashboard:使用火狐浏览器【chrome的添加例外比较麻烦】
https://10.10.1.200:30080【访问其他的集群里的IP也是一样的流程和结果,如访问某个Node的IP或者VIP都可以】
会有提示不安全,高级--添加例外,对10.10.1.200:30080添加例外;
然后会有两个选项分别是kubeconfig和token,
选择token
输入上面复制来的secret的token。
登录,即可查看各项内容。

五、配置监控
参考:http://blog.51cto.com/kaliarch/2160569

5.1、master/node节点环境部署
master1安装git,并下载相关yaml文件
git clone https://github.com/redhatxl/k8s-prometheus-grafana.git

5.2、在node节点下载监控所需镜像
docker pull prom/node-exporter
docker pull prom/prometheus:v2.0.0
docker pull grafana/grafana:4.2.0

5.3、采用daemonset方式部署node-exporter组件
cd  k8s-prometheus-grafana
kubectl create -f  node-exporter.yaml 

5.4、部署prometheus组件
rbac文件
kubectl create -f  prometheus/rbac-setup.yaml
以configmap的形式管理prometheus组件的配置文件
kubectl create -f  prometheus/configmap.yaml 
Prometheus deployment 文件
kubectl create -f  prometheus/prometheus.deploy.yml 
Prometheus service文件
kubectl create -f  prometheus/prometheus.svc.yml 

5.5、部署grafana组件
grafana deployment配置文件
kubectl create -f   grafana/grafana-deploy.yaml
grafana service配置文件
kubectl create -f   grafana/grafana-svc.yaml
grafana ingress配置文件
kubectl create -f   grafana/grafana-ing.yaml

5.6、查看node-exporter
http://10.10.1.210:31672/metrics
prometheus对应的nodeport端口为30003,通过访问http://10.10.1.210:30003/target(这个要等一会儿) 可以看到prometheus已经成功连接上了k8s的apiserver,可以查看到一系列值。
kubectl get svc --all-namespaces查看grafana暴露的nodePort是32092;
通过端口进行granfa访问,http://10.10.1.210:32092默认用户名密码均为admin
登录后点击选择添加数据源,按如下填写:
数据源名称Prometheus,type选择prometheus;
HTTP settings:
Url:http://prometheus:9090
Access:proxy
其他不填,最下方选择添加,然后选择Save&Test,会有成功提示。
然后点击上方的保存dashboard按钮或者Ctrl+s保存。
导入Dashboard,点击import,会出现相关界面,直接输入模板编号315在线导入,然后选择数据源名称为prometheus,点击import即可。
或者下载好对应的json模板文件本地导入,面板模板下载地址https:///dashboards/315。
即可查看展示效果。

六、集群功能测试:
(1)首先验证kube-apiserver, kube-controller-manager, kube-scheduler, pod network 是否正常:
master上部署一个 Nginx Deployment,包含两个Pod
参考:https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
kubectl create deployment nginx --image=nginx:alpine
等待一会儿之后创建成功;
kubectl scale deployment nginx --replicas=2

验证Nginx Pod是否正确运行,并且会分配10.244.开头的集群IP
kubectl get pods -l app=nginx -o wide
 
输出如下:
NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE      NOMINATED NODE
nginx-65d5c4f7cc-7pzgp   1/1     Running   0          88s   10.244.1.2   ubuntu2   <none>
nginx-65d5c4f7cc-l2h26   1/1     Running   0          82s   10.244.1.3   ubuntu2   <none>
如果哪个不正常,可以使用
kubectl  describe  pod  nginx-xxxxxx来查看相信信息;

(2)再验证一下kube-proxy是否正常:Master上操作
# 以 NodePort 方式对外提供服务 https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/
kubectl expose deployment nginx --port=80 --type=NodePort
# 查看集群外可访问的Port
kubectl get services nginx
# 输出如下:
NAME      TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
nginx     NodePort   10.98.44.131   <none>        80:32137/TCP   10s
 
# 可以通过集群内的任意 NodeIP:Port 在集群外部访问这个服务,或者通过浏览器访问也可以;
curl http://10.10.1.201:32137
curl http://10.10.1.202:32137
curl http://10.10.1.203:32137

(3)验证一下dns, pod network是否正常:
查看内网IP:
NAME                   READY     STATUS    RESTARTS   AGE       IP           NODE
nginx-f9f67b99-tbdhf   1/1       Running   0          4m        10.244.3.2   node3
nginx-f9f67b99-w4zrp   1/1       Running   0          3m        10.244.1.2   node1

(4) 运行Busybox并进入交互模式
kubectl run -it curl --image=radial/busyboxplus:curl
 
# 输入`nslookup nginx`查看是否可以正确解析出集群内的IP,已验证DNS是否正常
nslookup nginx
 # 输出如下:
[ root@curl-87b54756-bt8bs:/ ]$ nslookup nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nginx
Address 1: 10.100.213.93 nginx.default.svc.cluster.local
[ root@curl-87b54756-bt8bs:/ ]$ 
 
# 通过服务名进行访问,验证kube-proxy是否正常
[ root@curl-5cc7b478b6-tlf46:/ ]$ curl http://nginx/
 # 输出如下:
# <!DOCTYPE html> ---省略,但主要内容应该跟前几步curl出来的nginx信息一样;
 # 分别访问一下2个Pod的内网IP,验证跨Node的网络通信是否正常(内网IP是从上面查来的)
[ root@curl-5cc7b478b6-tlf46:/ ]$ curl http://10.244.3.2/
[ root@curl-5cc7b478b6-tlf46:/ ]$ curl http://10.244.1.2/
--------------------- 

 

转载于:https://my.oschina.net/wwh/blog/3050652

Logo

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐