自定义StorageClass

因为mysql和nacos都是有状态服务，所以我们需要指定存储的方式。

这里，我们使用的是nfs。你需要提前安装好nfs服务器并且在客户端完成挂载。参考博文：nfs服务器搭建

对每个有状态服务，手动部署pv和pvc是麻烦的，所以我们可以用StorageClass来自动的为我们生成pv和pvc。

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  namespace: kube-system

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: kube-system 
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-provisioner-01
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-provisioner-01
  template:
    metadata:
      labels:
        app: nfs-provisioner-01
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: jmgao1983/nfs-client-provisioner:latest
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: nfs-provisioner-01  # 此处供应者名字供storageclass调用
            - name: NFS_SERVER
              value: 192.168.31.200   # 填入NFS的地址(换成自己服务器的地址)
            - name: NFS_PATH
              value: /public   # 填入NFS挂载的目录(换成自己的目录)
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.31.200   # 填入NFS的地址(换成自己服务器的地址)
            path: /public   # 填入NFS挂载的目录(换成自己的目录)

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-ocean # (名字自己取)
provisioner: nfs-provisioner-01
# Supported policies: Delete、 Retain ， default is Delete
reclaimPolicy: Retain

我们在系统的命名空间下创建了一个账号，并且创建了一个角色，该角色能够访问StorageClass。然后我们完成账号和角色的绑定。

我们在kube-system这个命名空间下部署了一个nfs客户端供应者。因为在创建StorageClass的时候我们必要指明provisioner，它表明供应pv的插件是nfs。

最后我们创建StorageClass资源。回收策略默认是Delete，我们选择手工回收。

apply一下后我们查看一下部署的资源：

[root@MiWiFi-R4CM-srv k8s]# kubectl get sc -n kube-system 
NAME                 PROVISIONER                AGE
nfs-ocean            nfs-provisioner-01         4d1h

[root@MiWiFi-R4CM-srv k8s]# kubectl get ClusterRole 
NAME                                                                   AGE
admin                                                                  4d23h
cluster-admin                                                          4d23h
edit                                                                   4d23h
kubernetes-dashboard                                                   4d23h
nfs-client-provisioner-runner (这是我们创建的)                                          4d1h
...

[root@MiWiFi-R4CM-srv k8s]# kubectl get ClusterRoleBinding 
NAME                                                   AGE
cluster-admin                                          4d23h
kubeadm:kubelet-bootstrap                              4d23h
kubeadm:node-autoapprove-bootstrap                     4d23h
kubeadm:node-autoapprove-certificate-rotation          4d23h
kubeadm:node-proxier                                   4d23h
kubernetes-dashboard                                   4d23h
minikube-rbac                                          4d23h
run-nfs-client-provisioner (这是我们创建的)              4d1h
 ...

[root@MiWiFi-R4CM-srv k8s]# kubectl get deployment -n kube-system
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
coredns              2/2     2            2           4d23h
nfs-provisioner-01   1/1     1            1           4d1h

[root@MiWiFi-R4CM-srv k8s]# kubectl get sc
NAME                 PROVISIONER                AGE
nfs-ocean            nfs-provisioner-01         4d1h
standard (default)   k8s.io/minikube-hostpath   4d23h

可以看到StorageClass是不区分命名空间的。

部署单节点mysql

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: StatefulSet
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  serviceName: "mysql-svc"
  template:
    metadata:
      labels:
        app: mysql # has to match .spec.selector.matchLabels
    spec:
      containers:
      - image: mysql:5.7
        name: mysql
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          value: 123456 # mysql初始化密码
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-volumeclaim
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: mysql-volumeclaim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "nfs-ocean" # 名字为我们上面创建的StorageClass的名字
      resources:
        requests:
          storage: 5Gi

你只需要改一下mysql的密码，存储类的名称。至于serviceName，我们接着创建：

apiVersion: v1
kind: Service
metadata:
  name: mysql-svc
spec:
  type: NodePort
  selector:
    app: mysql
  ports:
   - port: 3306
     targetPort: 3306

因为我是给自己项目开发用的，所以我就用NodePort对外暴露了。同时，我们最好再暴露一个无头服务，供给集群内其他服务用(比如之后要部署的nacos)：

apiVersion: v1
kind: Service
metadata:
  name: mysql-headless
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None

把这三个yaml都apply一下，我们会发现pvc和pv自动创建了：

[root@MiWiFi-R4CM-srv mysql]# kubectl get pvc
NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql-volumeclaim-mysql-0   Bound    pvc-e83a928f-2b0b-4a52-a5d7-21622631bc70   5Gi        RWO            nfs-ocean      4d

[root@MiWiFi-R4CM-srv mysql]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                               STORAGECLASS   REASON   AGE
pvc-e83a928f-2b0b-4a52-a5d7-21622631bc70   5Gi        RWO            Retain           Bound    default/mysql-volumeclaim-mysql-0   nfs-ocean               4d

其实pvc的取名是很有规律的：mysql-volumeclaim-mysql-0，其中mysql-volumeclaim是挂载模板的名字，而mysql-0就是我mysql单节点pod的名字。

在nfs服务端也出现了新的目录：

[root@MiWiFi-R4CM-srv public]# pwd
/public
[root@MiWiFi-R4CM-srv public]# ll
total 8
drwxrwxrwx. 7 polkitd root 4096 Apr 22 12:22 default-mysql-volumeclaim-mysql-0-pvc-e83a928f-2b0b-4a52-a5d7-21622631bc70

当然，nfs客户端也会有同样的目录。

部署单节点nacos

直接上部署的yaml：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nacos
spec:
  serviceName: nacos-svc
  replicas: 1
  template:
    metadata:
      labels:
        app: nacos
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
    spec:
      containers:
        - name: nacos
          imagePullPolicy: IfNotPresent 
          image: nacos/nacos-server:1.4.0
          resources:
            requests:
              memory: "2Gi"
              cpu: "500m"
          ports:
            - containerPort: 8848
              name: client-port
          env:
            - name: NACOS_REPLICAS
              value: "1"
            - name: SPRING_DATASOURCE_PLATFORM
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: spring.platform
            - name: MYSQL_SERVICE_DB_NAME
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.db.name
            - name: MYSQL_SERVICE_PORT
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.port
            - name: MYSQL_SERVICE_USER
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.user
            - name: MYSQL_SERVICE_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.password
            - name: MYSQL_SERVICE_HOST
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.db.host
            - name: NACOS_SERVER_PORT
              value: "8848"
            - name: PREFER_HOST_MODE
              value: "hostname"
            - name: MODE
              value: "standalone"
            - name: NACOS_SERVERS
              value: "nacos-headless.default.svc.cluster.local:8848"
            - name: nacos.naming.data.warmup
              value: "false"
          volumeMounts:
            - name: nacos-datadir
              mountPath: /home/nacos/data
            - name: nacos-logdir
              mountPath: /home/nacos/logs
  volumeClaimTemplates:
    - metadata:
        name: nacos-datadir
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "nfs-ocean"
        resources:
          requests:
            storage: 2Gi
    - metadata:
        name: nacos-logdir
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "nfs-ocean"
        resources:
          requests:
            storage: 2Gi
  selector:
    matchLabels:
      app: nacos

因为是单节点，所以replicas为1，模式是standalone。

我们等下还要部署一个名为nacos-cm的ConfigMap来告诉nacos我们的mysql在哪里。这里的- name: SPRING_DATASOURCE_PLATFORM必须要配置，也就是要指定为mysql，否则nacos还是会用默认的数据库derby去存储配置信息和nacos的角色用户信息。

至于serviceName，我们也是等下创建。从NACOS_SERVERS的名称nacos-headless.default.svc.cluster.local:8848我们可以看到，等下需要一个名为nacos-headless的无头服务。

另外一个环境变量nacos.naming.data.warmup=false也是必须的，这可以保证单节点服务发现不出错。

挂载的目录，我选择了data和logs挂载，至于那个application.properties，因为它主要的功能就是指定mysql的地址，既然我们已经在环境变量里说了，所以就不挂载了。如果你有个性化定制，就可以也搞一个ConfigMap把配置文件挂载出来。

与集群发现有关的，我们一律不写。

(友情提示：写yaml的时候千万不要用tab键！！！)

下面是nacos-cm：

apiVersion: v1
kind: ConfigMap
metadata:
  name: nacos-cm
data:
  spring.platform: "mysql"
  mysql.db.name: "nacos_config"
  mysql.db.host: "mysql-headless.default.svc.cluster.local"
  mysql.port: "3306"
  mysql.user: "root"
  mysql.password: "123456"

你自己建一个库，我这里名称为nacos_config，然后在里面建nacos需要的表：nacos-mysql

mysql的地址就是mysql暴露出来的无头服务的名字+命名空间+svc+cluster+local，它也是有规律的。

对内暴露：

apiVersion: v1
kind: Service
metadata:
  name: nacos-headless
  labels:
    app: nacos
  annotations:
    service.alpha.kubernetes.io/publishNotReadyAddresses: "true"
spec:
  ports:
    - port: 8848
      name: server
      targetPort: 8848
  clusterIP: None
  selector:
    app: nacos

关于是否使用service.alpha.kubernetes.io/publishNotReadyAddresses: "true"这个注解，参照这个 issue。

对外暴露的话，可以使用无头服务+Ingress，我在这里因为是给自己本地用，就用NodePort来暴露了：

apiVersion: v1
kind: Service
metadata:
  name: nacos-svc
spec:
  type: NodePort
  selector:
    app: nacos
  ports:
   - port: 8848
     targetPort: 8848

以上yaml全部apply就可以用ip:port/nacos访问了。