前置知识：docker入门，如何部署Django uwsgi nginx应用
参考文档：kubernetes setup doc

环境准备

练习环境采用VirtualBox运行ubuntu18.04，有现成的机器此环节可略过。

1、VirtualBox设置

1、新建虚拟电脑
2、设置》系统》cpu 2，内存2G（k8s最低要求）
3、下载Ubuntu Server 18.04 LTS，设置》存储》没有盘片》分配光驱，
选择刚刚下载的ubuntu镜像文件
4、启动，初次注意勾选安装sshd
5、配置网络：在VertualBox中，将虚拟机网络设置为桥接模式；参考宿主机网络信息，
修改虚拟机文件cat /etc/netplan/50-cloud-init.yaml，然后 netplan apply

network:
    ethernets:
        enp0s3:
            addresses: [192.168.0.191/24]            # [ip/子网掩码]
            gateway4: 192.168.0.1                    # 网关
            dhcp4: no                               # 是否动态获取ip
            nameservers:
                    addresses: [114.114.114.114]    # dns
    version: 2

备注：子网掩码设置请搜索CIDR（无类别域间路由，Classless Inter-Domain Routing），点击查看可用dns
6、安装docker apt-get install -y docker.io

2、安装k8s（阿里云源）

• 添加软件包密钥(apt-key)
  sudo curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg --output /etc/apt/trusted.gpg.d/apt-key.gpg
• 添加源

# /etc/apt/sources.list.d/kubernetes.list
deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main

• 安装
  sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl

3、创建集群

1、确认机器开放端口， 如果使用aws或者阿里云机器务必配置开启相应端口的安全组

master节点开放端口

规则	方向	端口范围	作用	使用者
TCP	Inbound	6443*	Kubernetes API server	All
TCP	Inbound	2379-2380	etcd server client API	kube-apiserver, etcd
TCP	Inbound	10250	Kubelet API	Self, Control plane
TCP	Inbound	10251	kube-scheduler	Self
TCP	Inbound	10252	kube-controller-manager	Self

node 节点开放端口

规则	方向	端口范围	作用	使用者
TCP	Inbound	10250	Kubelet API	Self, Control plane
TCP	Inbound	30000-32767	NodePort Services**	All

2、所有节点启用 docker service，关闭 swap
systemctl enable docker.service
swapoff -a
3、在master启动kubeadm
sudo kubeadm init --pod-network-cidr=182.168.0.0/16 --image-repository registry.aliyuncs.com/google_containers
输出：

Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.191:6443 --token lgm81w.xjktiv2w32jozvgt --discovery-token-ca-cert-hash sha256:a3a535428d2302acfc4cdff1dc3a6db29bd01bbbbccae824f902842523035edd

4、 设置 pod network calico
下载网络插件配置文件 calico.yaml
curl https://docs.projectcalico.org/v3.6/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml > calico.yaml
注意修改参数 CALICO_IPV4POOL_CIDR，与 kubeadm init 的参数 pod-network-cidr保持一致！

# The default IPv4 pool to create on startup if none exists. Pod IPs will be
# chosen from this range. Changing this value after installation will have
# no effect. This should fall within `--cluster-cidr`.
- name: CALICO_IPV4POOL_CIDR
  value: "182.168.0.0/16"

执行 sudo kubectl apply -f calico.yaml

5、加入节点，在节点机器执行命令

yetongxue@vm2:~$ kubeadm join 192.168.0.191:6443 --token lgm81w.xjktiv2w32jozvgt --discovery-token-ca-cert-hash sha256:a3a535428d2302acfc4cdff1dc3a6db29bd01bbbbccae824f902842523035edd
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

在 master 检查 vm2 加入状态

yetongxue@vm1:~$ sudo kubectl get nodes
NAME   STATUS   ROLES    AGE   VERSION
vm1    Ready    master   21m   v1.15.0
vm2    Ready    <none>   19m   v1.15.0

如果出现问题，执行kubectl describe nodes <node_name>查看原因
k8s环境安装完成 ✌️

部署应用集群

前置知识

• pod 一个或一组容器组成，共同提供一种服务；定义文件pod.yaml中包含使用的镜像、端口、数据卷挂载、启动命令，为pod打标签label这些，类似编排多个容器的 docker-compose.yaml 的作用
• service 多个相同服务的pod组成一个服务，通过service.yaml中spec.selecter=[pod label]选择pod，负载均衡的感觉
• volume 数据卷，有很多类型：emptyDir适合pod内部各个容器共享内容，hostPath则会将目录挂载到宿主机，还有网络数据卷、分布式文件系统这些；还有一种专门当做配置文件来用的数据挂载方式——configMap，可以用一份文件来生成configMap，用来替换某些软件原来的配置文件；也可以设置很多的key value，在各种yaml文件中引用，以设置容器所需要的环境变量参数

参考资料

• kubernetes yaml格式的pod定义文件完整内容
• Creating the pod using the YAML file
• config map
• 通过configmap更新k8s里的mysql配置文件
• kubernetes 特殊存储卷ConfigMap

下面就将项目 django_demo 部署到 k8s 集群，参考 docker-compose.yaml、mysqld.cnf内容，分别创建如下文件（k8s命名一般是小驼峰/中划线）

1、通过文件 mysqld.cnf 创建 configmap

yetongxue@vm1:~$ sudo kubectl create configmap mysqld-config --from-file=mysqld.cnf
configmap/mysqld-config created
yetongxue@vm1:~$ sudo kubectl describe configmaps  mysqld-config
[sudo] password for master: 
Name:         mysqld-config
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
mysqld.cnf:
----
[mysqld]
pid-file    = /var/run/mysqld/mysqld.pid
socket        = /var/run/mysqld/mysqld.sock
datadir       = /var/lib/mysql
#log-error  = /var/log/mysql/error.log
# By default we only accept connections from localhost
bind-address  = 0.0.0.0
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

# utf8mb4
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
init_connect='SET NAMES utf8mb4'
skip-character-set-client-handshake = true

Events:  <none>

2、将 configmap mysql-config 挂载到 pod，替换 mysql 镜像原有的myqld.cnf；将mysql数据目录（见mysqd.cnf 中 datadir 字段）/var/lib/mysql 挂载到宿主机/var/lib/mysql-datadir

# mysql-pod.yaml
apiVersion: v1
kind: Pod
metadata:
    name: mysql
    labels:
        app: database
        version: v1
        release: dev
spec:
    restartPolicy: Always
    containers:
        - name: mysql
          image: mysql:5.7
          imagePullPolicy: IfNotPresent
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: MYSQL_ROOT_PASSWORD
              value: root
            - name: MYSQL_USER
              value: yetongxue
            - name: MYSQL_PASSWORD
              value: qwerasdf
            - name: MYSQL_DATABASE
              value: django_demo
          volumeMounts:
            - name: config-volume-mysql
              mountPath: /etc/mysql/mysql.conf.d/
            - name: mysql-datadir
              mountPath: /var/lib/mysql
    volumes:
        - name: config-volume-mysql
          configMap:
            name: mysqld-config
        - name: mysql-datadir
          hostPath:
            path: /var/lib/mysql-datadir

启动并验证 mysql pod 是否正常

yetongxue@vm1:~$ sudo kubectl apply -f mysql-pod.yaml 
pod/mysql created
yetongxue@vm1:~$ sudo kubectl get pod mysql
NAME    READY   STATUS    RESTARTS   AGE
mysql   1/1     Running   0          9d

yetongxue@vm1:~$ sudo kubectl exec -it mysql bash
root@mysql:/# mysql -u root -p
Enter password: 
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| django_demo        |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

检查数据挂载是否成功

yetongxue@vm2:~$ ls /var/lib/mysql-datadir/
auto.cnf    ca.pem           client-key.pem  ib_buffer_pool  ib_logfile0  ibtmp1  performance_schema  public_key.pem   server-key.pem
ca-key.pem  client-cert.pem  django_demo     ibdata1         ib_logfile1  mysql   private_key.pem     server-cert.pem  sys

这里有一个优化的地方，因为 mysql-pod.yaml 中使用了许多的环境变量，这些变量在下面的 django-demo 部署中依然会使用到，所以我们将这些变量剥离出来，通过 configMap 引用。
创建 mysql 环境变量 configMap，参考 Configure all key-value pairs in a ConfigMap as container environment variables

# env-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: env-config
data:
  MYSQL_ROOT_PASSWORD: root
  MYSQL_USER: yetongxue
  MYSQL_PASSWORD: qwerasdf
  MYSQL_DATABASE: django_demo

使用前先创建 configMap

yetongxue@vm1:~$ sudo kubectl apply -f env-config.yaml 
configmap/env-config created

下面将 mysql-pod.yaml 改造一下，环境变量通过 configMap 的方式引用，如下：

# mysql-pod.yaml
apiVersion: v1
kind: Pod
metadata:
    name: mysql
    labels:
        app: database
        version: v1
        release: dev
spec:
    restartPolicy: Always
    containers:
        - name: mysql
          image: mysql:5.7
          imagePullPolicy: IfNotPresent
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: env-config
                  key: MYSQL_ROOT_PASSWORD
            - name: MYSQL_USER
              valueFrom:
                configMapKeyRef:
                  name: env-config
                  key: MYSQL_USER
            - name: MYSQL_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: env-config
                  key: MYSQL_PASSWORD 
            - name: MYSQL_DATABASE
              valueFrom:
                configMapKeyRef:
                  name: env-config
                  key: MYSQL_DATABASE 
          volumeMounts:
            - name: config-volume-mysql
              mountPath: /etc/mysql/mysql.conf.d/
            - name: mysql-datadir
              mountPath: /var/lib/mysql
    volumes:
        - name: config-volume-mysql
          configMap:
            name: mysqld-config
        - name: mysql-datadir
          hostPath: 
            path: /var/lib/mysql-datadir 

这里还有一个优化的地方 😂，一般我们都不通过 pod.yaml 直接创建pod，而是通过 Deployment Controller 来自动创建 pod，Deployment 定义文件 deployment.yaml 中参数replicas设置期望运行的 pod 数量，Deployment 会自动增加或者删除 pod 来维持；另外参数template的内容便是刚刚的 pod.yaml 内容，所以 RC 便知道如何去创建 pod；弹性伸缩、滚动升级也是通过它来实现的。

# mysql-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
    name: mysql-deployment
spec:
    selector:
        matchLabels:
            app: database
    replicas: 1
    template:
        metadata:
            name: mysql
            labels:
                app: database
        spec:
            restartPolicy: Always
            containers:
            - name: mysql
              image: mysql:5.7
              imagePullPolicy: IfNotPresent
              env:
                - name: TZ
                  value: Asia/Shanghai
                - name: MYSQL_ROOT_PASSWORD
                  valueFrom:
                    configMapKeyRef:
                      name: env-config
                      key: MYSQL_ROOT_PASSWORD
                - name: MYSQL_USER
                  valueFrom:
                    configMapKeyRef:
                      name: env-config
                      key: MYSQL_USER
                - name: MYSQL_PASSWORD
                  valueFrom:
                    configMapKeyRef:
                      name: env-config
                      key: MYSQL_PASSWORD 
                - name: MYSQL_DATABASE
                  valueFrom:
                    configMapKeyRef:
                      name: env-config
                      key: MYSQL_DATABASE 
              volumeMounts:
                - name: config-volume-mysql
                  mountPath: /etc/mysql/mysql.conf.d/
                - name: mysql-datadir
                  mountPath: /var/lib/mysql
            volumes:
              - name: config-volume-mysql
                configMap:
                    name: mysqld-config
              - name: mysql-datadir
                hostPath: 
                    path: /var/lib/mysql-datadir

删除刚刚创建的 pod: sudo kubectl delete pod mysql
通过 RC 创建 pod: sudo kubectl apply -f mysql-deployment.yaml

这里有两个地方需要注意：

• spec.selector.matchLabels务必和 spec.template.matadata.labels保持一致，否则RC通过template创建出pod后，通过matchLabels匹配不到相应的pod就会不停的创建。
• 对于mysql这种有数据状态的应用，不能简单的设置replicas为大于1的数值来扩容，这又是个优化的地方，有兴趣的同学可参考 Run a Replicated Stateful Application

3、创建 msyql service

Kubernetes 中 service 有三种类型:

• ClusterIP:提供一个集群内部的虚拟IP以供Pod访问。
• NodePort:在每个Node上打开一个端口以供外部访问。
• LoadBalancer:通过外部的负载均衡器来访问。

显然，这里的mysql用ClusterIP就好；待会下面创建django-demo service就用LoadBalancer。

# mysql-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: mysql-service        
spec:
  selector:
    app: database    #通过label选择加入service的pod
  ports:                    
  - port: 3306
    targetPort: 3306
    protocol: TCP

启动mysql service

yetongxue@vm1:~$ sudo kubectl apply -f mysql-service.yaml 
service/mysql-service created
yetongxue@vm1:~$ sudo kubectl get service
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
kubernetes      ClusterIP   10.96.0.1        <none>        443/TCP    10h
mysql-service   ClusterIP   10.108.192.143   <none>        3306/TCP   46s
yetongxue@vm1:~$ sudo kubectl get endpoints mysql-service 
NAME            ENDPOINTS              AGE
mysql-service   182.168.185.194:3306   24m

ENDPOINTS表明已经将刚刚创建的mysql pod纳入了mysql service.

4、创建 django-demo deployment

先把通过 Dockerfile 构建的镜像上传到阿里云镜像仓库 registry.cn-hangzhou.aliyuncs.com/yetongxue/django_demo:dev。
由于RC的好处是明显的，所以现在我们就直接通过RC来创建pod
注意将MYSQL_HOST修改成刚刚创建的mysql-service的ClusterIP，或者直接就用service name: mysql-service

# django-demo-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
    name: django-demo-deployment
spec:
    selector:
        matchLabels:
            app: web
    replicas: 2
    template:
        metadata:
            name: django-demo
            labels:
                app: web
        spec:
            restartPolicy: Always
            containers:
              - name: demo
                image: registry.cn-hangzhou.aliyuncs.com/yetongxue/django_demo:dev
                imagePullPolicy: IfNotPresent
                env:
                  - name: TZ
                    value: Asia/Shanghai
                  - name: MYSQL_HOST
                    value: mysql-service
                  - name: MYSQL_USER
                    valueFrom:
                      configMapKeyRef:
                        name: env-config
                        key: MYSQL_USER
                  - name: MYSQL_PASSWORD
                    valueFrom:
                      configMapKeyRef:
                        name: env-config
                        key: MYSQL_PASSWORD 
                  - name: MYSQL_DATABASE
                    valueFrom:
                      configMapKeyRef:
                        name: env-config
                        key: MYSQL_DATABASE
                ports:
                  - containerPort: 80

启动 django-demo deployment，并验证连接数据库是否正常。

yetongxue@vm1:~$ sudo kubectl apply -f django-demo-deployment.yaml 
deployment.apps/django-demo-deployment created
yetongxue@vm1:~$ sudo kubectl get pods
NAME                                      READY   STATUS              RESTARTS   AGE
django-demo-deployment-8477cd6dbb-4dz9b   0/1     ContainerCreating   0          57s
django-demo-deployment-8477cd6dbb-7jssg   0/1     ContainerCreating   0          57s
mysql-deployment-588c96799f-7md5n         1/1     Running             0          24m
yetongxue@vm1:~$ sudo kubectl exec -it mysql-deployment-588c96799f-7md5n bash
root@mysql-deployment-588c96799f-7md5n:/# mysql -u root -p
Enter password: 
mysql> use django_demo
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+--------------------------------+
| Tables_in_django_demo          |
+--------------------------------+
| auth_group                     |
| auth_group_permissions         |
| auth_permission                |
| authtoken_token                |
| django_admin_log               |
| django_content_type            |
| django_migrations              |
| django_session                 |
| test_app_book                  |
| test_app_user                  |
| test_app_user_groups           |
| test_app_user_user_permissions |
+--------------------------------+
12 rows in set (0.01 sec)

5、创建django-demo service

# django-demo-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: django-demo-service
spec:
  selector:
    app: web
  externalIPs: ["192.168.0.191"]      # 指定一个外部IP，这里就用master节点IP
  ports:
  - name: http
    port: 8000
    targetPort: 80
    protocol: TCP
  type: LoadBalancer        # 类型负载均衡

启动并验证是否成功

yetongxue@vm1:~$ sudo kubectl apply -f django-demo-service.yaml 
service/django-demo-service created

访问地址：http://192.168.0.191:8000/admin
账号：admin
密码：qwerasdf
截图纪念如下：

问题收集

在参照文档进行练习时，遇到的一些问题记录如下

1、在master执行kubeadm reset之后，如果你之前执行过命令sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config，则需要再次执行，否则权限无法验证，因为$HOME/.kube/config里面存的还是上次kubeadm init生成的文件；node被移除或者master执行了kubeadm reset， 在执行kubeadm join之前还需要执行 kubeadm reset，否则在进行 kubeadm join时产生如下错误

[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] Initial timeout of 40s passed.
packet_write_wait: Connection to 192.168.0.192 port 22: Broken pipe

2、在使用VirtualBox时，安装完ubuntu我reboot了（以后一定要通过VirtualBox操作），结果当前虚拟机在VirtualBox的状态就变成了Guru Meditation，无法关闭启动删除，只可查看日志，又没看懂日志。不知道为啥，最后直接kill相应进程解决

常用命令

• 查看 pod: kubectl describe pod --namespace=<name_space> <pod_name>
• 进入pod: kubectl exec -it <pod_name> bash
• 查看pod日志：kubectl logs <pod_name>