我们主要通过ceph-csi进行kK8s集群和ceph rbd的连接。

获取Ceph-CSI

git clone https://github.com/ceph/ceph-csi.git
cd ceph-csi/deploy/rbd/kubernetes

修改文件

• 修改csi-rbdplugin-provisioner.yaml和csi-rbdplugin.yaml文件

将csi-rbdplugin-provisioner.yaml和csi-rbdplugin.yaml文件中ceph-csi-encryption-kms-config配置注释掉

#- name: ceph-csi-encryption-kms-config
# mountPath: /etc/ceph-csi-encryption-kms-config/
#- name: ceph-csi-encryption-kms-config
# configMap:
# name: ceph-csi-encryption-kms-config

配置csi-config-map.yaml文件中ceph集群的信息

---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "ec15eb3e-eb66-4431-acda-428e91658560",    // 通过ceph集群的ID
        "monitors": [
          "192.168.2.90:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config

config-map.yaml

csi-rbdplugin-provisioner.yaml和csi-rbdplugin.yaml里面用到了ConfigMap ceph-config该文件在example文件夹中，也可以自己创建

apiVersion: v1
kind: ConfigMap
data:
  ceph.conf: |
    [global]
    auth_cluster_required = cephx
    auth_service_required = cephx
    auth_client_required = cephx

  # keyring is a required key and its value should be empty
  keyring: |
metadata:
  name: ceph-config

修改下载镜像地址

csi-rbdplugin.yaml和csi-rbdplugin-provisioner.yaml中的镜像地址是google的需要修改，我这里部署的时候修改成了阿里云和对应的地址

  registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.5.1
  quay.io/cephcsi/cephcsi:canary
  gcr.lank8s.cn/k8s-staging-sig-storage/csi-provisioner:canary
  registry.aliyuncs.com/google_containers/csi-snapshotter:v6.0.1
  registry.aliyuncs.com/google_containers/csi-attacher:v3.4.0
  registry.aliyuncs.com/google_containers/csi-resizer:v1.4.0

部署csi

kubectl apply -f kubernetes/
kubectl get pods
csi-rbdplugin-ms5lk                          3/3     Running   0          7h12m
csi-rbdplugin-provisioner-597875dbb4-l49ht   7/7     Running   0          7h12m
csi-rbdplugin-rz8px                          3/3     Running   0          7h12m
csi-rbdplugin-shjb8                          3/3     Running   0          7h12m
nfs-client-provisioner-8cbf68bfd-ptzz2       1/1     Running   2          3d10h

Ceph服务器创建存储池

#创建pool
ceph osd pool create rbddata 8 8
rbd pool init rbddata
# 获取admin密钥
ceph auth get-key client.admin

创建SC

创建ceph密钥

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: default
stringData:
  userID: kubernetes
  userKey: AQCOp6ZiP42IBBAA4J+Eeg7/oGflMMkDN15XYw==
  encryptionPassphrase: test_passphrase

创建storageclass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-rbd-sc
  namespace: default
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: ec15eb3e-eb66-4431-acda-428e91658560
  # monitors: 192.168.2.90:6789
  pool: rbddata
  # imageFormat: "2"
  imageFeatures: "layering"
  csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
  csi.storage.k8s.io/provisioner-secret-namespace: default
  csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: default
  csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
  csi.storage.k8s.io/node-stage-secret-namespace: default
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
   - discard

部署

kubectl apply -f ceph-secret.yaml
kubectl apply -f storageclass.yaml

测试

创建pvc

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc
kubectl apply -f pvc.yaml
kubectl get pvc rbd-pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rbd-pvc   Bound    pvc-11b931b0-7cb5-40e1-815b-c15659310593   1Gi      RWO            csi-rbd-sc        0

POD

---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx
      volumeMounts:
        - name: mypvc
          mountPath: /var/lib/www/html
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false
kubectl apply -f pod.yaml
kubectl get pods csi-rbd-demo-pod
NAME               READY   STATUS    RESTARTS   AGE
csi-rbd-demo-pod   1/1     Running   0          0

kubectl exec -ti csi-rbd-demo-pod -- bash
df -h
Filesystem               Size  Used Avail Use% Mounted on
overlay                  199G  7.4G  192G   4% /
tmpfs                     64M     0   64M   0% /dev
tmpfs                    7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root  199G  7.4G  192G   4% /etc/hosts
shm                       64M     0   64M   0% /dev/shm
/dev/rbd0                976M  2.6M  958M   1% /var/lib/www/html
tmpfs                    7.8G   12K  7.8G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                    7.8G     0  7.8G   0% /proc/acpi
tmpfs                    7.8G     0  7.8G   0% /proc/scsi
tmpfs                    7.8G     0  7.8G   0% /sys/firmware