K8S/Kubernetes

一、准备实验环境

1.准备两台centos7虚拟机，用来安装k8s集群，下面是两台虚拟机的配置情况

k8s-master(192.168.1.237)配置：

操作系统：centos7.4、centos7.5、centos7.6以及更高版本都可以配置：4核cpu，8G内存，两块60G硬盘网络：桥接网络

k8s-node1(192.168.1.238)配置：

操作系统：centos7.6配置：4核cpu，4G内存，两块60G硬盘网络：桥接网络

k8s-node2(192.168.1.244)配置：

操作系统：euleros SP5配置：4核cpu，4G内存，两块60G硬盘网络：桥接网络

2.安装基础软件包，各个节点操作

yum -y install wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel  python-devel epel-release lrzsz  openssh-server socat  ipvsadm conntrack

3.关闭firewalld防火墙，各个节点操作，centos7系统默认使用的是firewalld防火墙，停止firewalld防火墙，并禁用这个服务

systemctl  stop firewalld  && systemctl  disable  firewalld

4.安装iptables，各个节点操作，如果你用firewalld不是很习惯，可以安装iptables，这个步骤可以不做，根据大家实际需求

4.1 安装iptables

yum install iptables-services -y

4.2 禁用iptables

service iptables stop   && systemctl disable iptables

5.时间同步，各个节点操作

5.1 时间同步

ntpdate cn.pool.ntp.org

5.2 编辑计划任务，每小时做一次同步

crontab -e

* */1 * * * /usr/sbin/ntpdate   cn.pool.ntp.org

6. 关闭selinux，各个节点操作

关闭selinux，设置永久关闭，这样重启机器selinux也处于关闭状态

修改/etc/sysconfig/selinux文件

sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinuxsed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

上面文件修改之后，需要重启虚拟机，可以强制重启：

reboot -r

7.关闭交换分区，各个节点操作

swapoff -a# 永久禁用，打开/etc/fstab注释掉swap那一行。sed -i 's/.*swap.*/#&/' /etc/fstab

8.优化内核参数

cat > /etc/sysctl.d/kubernetes.conf << EOF

 net.bridge.bridge-nf-call-iptables=1

 net.bridge.bridge-nf-call-ip6tables=1

 net.ipv4.ip_forward=1

 net.ipv4.tcp_tw_recycle=0

 vm.swappiness=0 # 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它

 vm.overcommit_memory=1 # 不检查物理内存是否够用

 vm.panic_on_oom=0 # 开启 OOM

 fs.inotify.max_user_instances=8192

 fs.inotify.max_user_watches=1048576

 fs.file-max=52706963

 fs.nr_open=52706963

 net.ipv6.conf.all.disable_ipv6=1

 net.netfilter.nf_conntrack_max=2310720

EOF

sysctl -p /etc/sysctl.d/kubernetes.conf

9.设置主机名&修改hosts文件

每台服务器上设置主机名以及修改hosts文件

# 设置主机名

vi /etc/hostname# 每台服务器上设置自己的主机名,修改hosts文件

vi /etc/hosts    # 设定节点ip对应关系

192.168.1.237 k8s-master

192.168.1.238 k8s-node1

192.168.1.244 k8s-node2

10.master免密登录其它节点

此处以k8s-master作为master，k8s-node1和k8snode2作为node，只在master上进行操作

创建密钥

[root@K8S00 ~]# ssh-keygen -t rsa  #一直回车

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Created directory '/root/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:c9mUKafubFrLnpZGEnQflcSYu7KPc64Mz/75WPwLvJY root@K8S00

The key's randomart image is:

+---[RSA 2048]----+

|             *o. |

|        . . +oo  |

|       . ...=o   |

|        .  Bo    |

|        S.+ ..   |

|        .+o o.   |

|        .oo+ o+  |

|         XB+.Eo. |

|        .B#BBo..o|

+----[SHA256]-----+

分发密钥

设置 K8S01 的 root 账户可以无密码登录所有节点：

ssh-copy-id -i .ssh/id_rsa.pub root@k8s-node1

ssh-copy-id -i .ssh/id_rsa.pub root@k8s-node1

#上面需要输入密码，输入k8s-node物理机密码即可

11.命令补全

安装bash-completion

 yum -y install bash-completion

加载bash-completion

source /etc/profile.d/bash_completion.sh

二、安装kubernetes1.17.3

1.修改yum源，各个节点操作

(1)备份原来的yum源

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

(2)下载阿里的yum源

centos7:

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

Euleros SP5:

cat /etc/yum.repos.d/EulerOS.repo

[base]

name=EulerOS-2.0SP5 base

baseurl=http://repo.huaweicloud.com/euler/2.5/os/x86_64/

enabled=1

gpgcheck=1

gpgkey=http://repo.huaweicloud.com/euler/2.5/os/RPM-GPG-KEY-EulerOS

(3)生成新的yum缓存

yum makecache fast

(4)配置安装k8s需要的yum源

cat /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64

enabled=1

gpgcheck=0

(5)清理yum缓存

yum clean all

(6)生成新的yum缓存

yum makecache fast

(7)更新软件包

yum -y update

(8)安装软件包

yum -y install yum-utils device-mapper-persistent-data lvm2

(9)添加新的软件源

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

※euleros系统可直接安装docker软件

2.安装docker19.03，各个节点操作

2.1 查看支持的docker版

yum list docker-ce --showduplicates |sort -r

2.2 下载19.03版本

yum install -y docker-ce-19*  && systemctl enable docker && systemctl start docker

#查看docker状态，如果状态是active(running)，说明docker是正常运行状态systemctl status docker  

2.3 修改docker配置文件，配置镜像加速器

[root@k8s-master ~]# cat > /etc/docker/daemon.json <<EOF

{

"exec-opts": ["native.cgroupdriver=systemd"],

"log-driver": "json-file",

"log-opts": {

"max-size": "100m"

},

"storage-driver": "overlay2",

"storage-opts": [

"overlay2.override_kernel_check=true"

]

}

EOF

2.4 重启docker使配置生效

systemctl restart docker

3.安装kubernetes1.17.3

3.1 在k8s-master和k8s-node上安装kubeadm和kubelet

yum install kubeadm-1.17.3 kubelet-1.17.3

systemctl enable kubelet

3.2上传镜像到k8s_master和k8s_node节点之后，手动解压镜像，镜像在百度网盘，我的是从官方下载的镜像，大家可以放心使用

链接：https://pan.baidu.com/s/1eiU69FKkgz0wBl5Lf1ClHw
提取码：vrgv

docker load -i  kube-apiserver.tar.gz

docker load -i   kube-scheduler.tar.gz

docker load -i   kube-controller-manager.tar.gz

docker load -i  pause.tar.gz

docker load -i  cordns.tar.gz

docker load -i  etcd.tar.gz

docker load -i  kube-proxy.tar.gz

docker load -i cni.tar.gz

docker load -i calico-node.tar.gz
docker load -i  kubernetes-dashboard_1_10.tar.gz

docker load -i  metrics-server-amd64_0_3_1.tar.gz

docker load -i  addon.tar.gz

说明：

pause版本是3.1etcd版本是3.4.3          

cordns版本是1.6.5 

cni版本是3.5.3 

calico版本是3.5.3         

apiserver、scheduler、controller-manager、kube-proxy版本是1.17.3 

kubernetes dashboard版本1.10.1 

metrics-server版本0.3.1         

addon-resizer版本是1.8.4

3.3 在k8s_master节点初始化k8s集群
kubeadm init --kubernetes-version=v1.17.3 --pod-network-cidr=10.244.0.0/16  --apiserver-advertise-address=192.168.1.237

显示如下，说明初始化成功了

To start using your cluster, you need to run the following as a regular user:
  mkdir -p $HOME/.kube  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config  sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:  https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.124.16:6443 --token i9m8e5.z12dnlekjbmuebsk \    --discovery-token-ca-cert-hash sha256:2dc931b4508137fbe1bcb93dc84b7332e7e874ec5862a9e8b8fff9f7c2b57621

注：

kubeadm join ... 这条命令需要记住，下面我们把k8s的node节点加入到集群需要在node节点输入这条命令，每次执行这个结果都是不一样的，

大家记住自己执行的结果，在下面会用到

3.4 在k8s-master节点执行如下，这样才能有权限操作k8s资源

mkdir -p $HOME/.kube

cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

chown $(id -u):$(id -g) $HOME/.kube/config

kubectl get nodes 

显示如下

NAME         STATUS     ROLES    AGE     VERSIONk8s-master   NotReady   master   2m13s   v1.17.3

3.5 把k8s-node节点加入到k8s集群，在node节点操作

kubeadm join 192.168.1.237:6443 --token i9m8e5.z12dnlekjbmuebsk \    --discovery-token-ca-cert-hash sha256:2dc931b4508137fbe1bcb93dc84b7332e7e874ec5862a9e8b8fff9f7c2b57621

注：上面的这个加入到k8s节点的一串命令就是在3.3初始化的时候生成的

3.6 在k8s-master节点查看集群节点状态
kubectl get nodes  显示看到Status状态都是NotReady，这是因为没有安装网络插件，如calico或者flannel
3.7 在k8s-master节点安装calico网络插件在master节点执行如下：
kubectl apply -f calico.yaml

# Calico Version v3.5.3
# https://docs.projectcalico.org/v3.5/releases#v3.5.3
# This manifest includes the following component versions:
#   calico/node:v3.5.3
#   calico/cni:v3.5.3

# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Typha is disabled.
  typha_service_name: "none"
  # Configure the Calico backend to use.
  calico_backend: "bird"

  # Configure the MTU to use
  veth_mtu: "1440"

  # The CNI network configuration to install on each node.  The special
  # values in this config will be automatically populated.
  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.0",
      "plugins": [
        {
          "type": "calico",
          "log_level": "info",
          "datastore_type": "kubernetes",
          "nodename": "__KUBERNETES_NODE_NAME__",
          "mtu": __CNI_MTU__,
          "ipam": {
            "type": "host-local",
            "subnet": "usePodCidr"
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        }
      ]
    }

---

# This manifest installs the calico/node container, as well
# as the Calico CNI plugins and network config on
# each master and worker node in a Kubernetes cluster.
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: calico-node
  namespace: kube-system
  labels:
    k8s-app: calico-node
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        k8s-app: calico-node
      annotations:
        # This, along with the CriticalAddonsOnly toleration below,
        # marks the pod as a critical add-on, ensuring it gets
        # priority scheduling and that its resources are reserved
        # if it ever gets evicted.
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      nodeSelector:
        beta.kubernetes.io/os: linux
      hostNetwork: true
      tolerations:
        # Make sure calico-node gets scheduled on all nodes.
        - effect: NoSchedule
          operator: Exists
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoExecute
          operator: Exists
      serviceAccountName: calico-node
      # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
      # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
      terminationGracePeriodSeconds: 0
      initContainers:
        # This container installs the Calico CNI binaries
        # and CNI network config file on each node.
        - name: install-cni
          image: quay.io/calico/cni:v3.5.3
          command: ["/install-cni.sh"]
          env:
            # Name of the CNI config file to create.
            - name: CNI_CONF_NAME
              value: "10-calico.conflist"
            # The CNI network config to install on each node.
            - name: CNI_NETWORK_CONFIG
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: cni_network_config
            # Set the hostname based on the k8s node name.
            - name: KUBERNETES_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # CNI MTU Config variable
            - name: CNI_MTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Prevents the container from sleeping forever.
            - name: SLEEP
              value: "false"
          volumeMounts:
            - mountPath: /host/opt/cni/bin
              name: cni-bin-dir
            - mountPath: /host/etc/cni/net.d
              name: cni-net-dir
      containers:
        # Runs calico/node container on each Kubernetes node.  This
        # container programs network policy and routes on each
        # host.
        - name: calico-node
          image: quay.io/calico/node:v3.5.3
          env:
            # Use Kubernetes API as the backing datastore.
            - name: DATASTORE_TYPE
              value: "kubernetes"
            # Wait for the datastore.
            - name: WAIT_FOR_DATASTORE
              value: "true"
            # Set based on the k8s node name.
            - name: NODENAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # Choose the backend to use.
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
            # Cluster type to identify the deployment type
            - name: CLUSTER_TYPE
              value: "k8s,bgp"
            # Auto-detect the BGP IP address.
            - name: IP
              value: "autodetect"
            - name: IP_AUTODETECTION_METHOD
              value: "can-reach=192.168.124.56"
            # Enable IPIP
            - name: CALICO_IPV4POOL_IPIP
              value: "Always"
            # Set MTU for tunnel device used if ipip is enabled
            - name: FELIX_IPINIPMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            - name: CALICO_IPV4POOL_CIDR
              value: "10.244.0.0/16"
            # Disable file logging so `kubectl logs` works.
            - name: CALICO_DISABLE_FILE_LOGGING
              value: "true"
            # Set Felix endpoint to host default action to ACCEPT.
            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
              value: "ACCEPT"
            # Disable IPv6 on Kubernetes.
            - name: FELIX_IPV6SUPPORT
              value: "false"
            # Set Felix logging to "info"
            - name: FELIX_LOGSEVERITYSCREEN
              value: "info"
            - name: FELIX_HEALTHENABLED
              value: "true"
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 250m
          livenessProbe:
            httpGet:
              path: /liveness
              port: 9099
              host: localhost
            periodSeconds: 10
            initialDelaySeconds: 10
            failureThreshold: 6
          readinessProbe:
            exec:
              command:
              - /bin/calico-node
              - -bird-ready
              - -felix-ready
            periodSeconds: 10
          volumeMounts:
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /run/xtables.lock
              name: xtables-lock
              readOnly: false
            - mountPath: /var/run/calico
              name: var-run-calico
              readOnly: false
            - mountPath: /var/lib/calico
              name: var-lib-calico
              readOnly: false
      volumes:
        # Used by calico/node.
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: var-run-calico
          hostPath:
            path: /var/run/calico
        - name: var-lib-calico
          hostPath:
            path: /var/lib/calico
        - name: xtables-lock
          hostPath:
            path: /run/xtables.lock
            type: FileOrCreate
        # Used to install CNI.
        - name: cni-bin-dir
          hostPath:
            path: /opt/cni/bin
        - name: cni-net-dir
          hostPath:
            path: /etc/cni/net.d
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-node
  namespace: kube-system

---
# Create all the CustomResourceDefinitions needed for
# Calico policy and networking mode.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
   name: felixconfigurations.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: FelixConfiguration
    plural: felixconfigurations
    singular: felixconfiguration
---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: bgppeers.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: BGPPeer
    plural: bgppeers
    singular: bgppeer

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: bgpconfigurations.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: BGPConfiguration
    plural: bgpconfigurations
    singular: bgpconfiguration

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ippools.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: IPPool
    plural: ippools
    singular: ippool

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: hostendpoints.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: HostEndpoint
    plural: hostendpoints
    singular: hostendpoint

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: clusterinformations.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: ClusterInformation
    plural: clusterinformations
    singular: clusterinformation

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: globalnetworkpolicies.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: GlobalNetworkPolicy
    plural: globalnetworkpolicies
    singular: globalnetworkpolicy

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: globalnetworksets.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: GlobalNetworkSet
    plural: globalnetworksets
    singular: globalnetworkset

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: networkpolicies.crd.projectcalico.org
spec:
  scope: Namespaced
  group: crd.projectcalico.org
  version: v1
  names:
    kind: NetworkPolicy
    plural: networkpolicies
    singular: networkpolicy
---

# Include a clusterrole for the calico-node DaemonSet,
# and bind it to the calico-node serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: calico-node
rules:
  # The CNI plugin needs to get pods, nodes, and namespaces.
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - endpoints
      - services
    verbs:
      # Used to discover service IPs for advertisement.
      - watch
      - list
      # Used to discover Typhas.
      - get
  - apiGroups: [""]
    resources:
      - nodes/status
    verbs:
      # Needed for clearing NodeNetworkUnavailable flag.
      - patch
      # Calico stores some configuration information in node annotations.
      - update
  # Watch for changes to Kubernetes NetworkPolicies.
  - apiGroups: ["networking.k8s.io"]
    resources:
      - networkpolicies
    verbs:
      - watch
      - list
  # Used by Calico for policy information.
  - apiGroups: [""]
    resources:
      - pods
      - namespaces
      - serviceaccounts
    verbs:
      - list
      - watch
  # The CNI plugin patches pods/status.
  - apiGroups: [""]
    resources:
      - pods/status
    verbs:
      - patch
  # Calico monitors various CRDs for config.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - globalfelixconfigs
      - felixconfigurations
      - bgppeers
      - globalbgpconfigs
      - bgpconfigurations
      - ippools
      - globalnetworkpolicies
      - globalnetworksets
      - networkpolicies
      - clusterinformations
      - hostendpoints
    verbs:
      - get
      - list
      - watch
  # Calico must create and update some CRDs on startup.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
      - felixconfigurations
      - clusterinformations
    verbs:
      - create
      - update
  # Calico stores some configuration information on the node.
  - apiGroups: [""]
    resources:
      - nodes
    verbs:
      - get
      - list
      - watch
  # These permissions are only requried for upgrade from v2.6, and can
  # be removed after upgrade or on fresh installations.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - bgpconfigurations
      - bgppeers
    verbs:
      - create
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: calico-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-node
subjects:
- kind: ServiceAccount
  name: calico-node
  namespace: kube-system
---

View Code

在k8s-master节点查看calico是否处于running状态

kubectl  get pods -n kube-system

显示如下，说明calico部署正常，如果calico没有部署成功，cordns会一直显示在ContainerCreating

calico-node-rkklw                1/1     Running   0          3m4scalico-node-rnzfq            1/1     Running   0          3m4scoredns-6955765f44-jzm4k     1/1     Running   0          25mcoredns-6955765f44-mmbr7    1/1   Running  0  25m

在k8s-master节点查看STATUS状态
kubectl get nodes 

显示如下，STATUS状态是ready，表示集群处于正常状态

NAME       STATUS  ROLES   AGE      VERSION

k8s-master  Ready  master   9m52s   v1.17.3

k8s-node1   Ready   <none>   6m10s   v1.17.3

k8s-node2   Ready   <none>   6m10s   v1.17.3