kubernetes集群中ceph集群使用

块设备创建和使用

Ceph 能够为 pod 提供裸的块设备卷，定义在Ceph数据冗余级别的一个池中

1.storageclass 配置

• storageclass.yaml

该配置文件为生产场景提供了3个副本，至少需要3个节点
数据会间歇性的在3个不同的k8s节点上复制，单节点故障不会造成数据的丢失或不可用

• storageclass-ec.yaml

以纠错码来替代镜像达到数据持久性的功能，需要至少3个节点
Ceph 的纠错码比镜像的效率更高，所以能够提供高可用性而无需3份镜像的代价
但会造成节点较高的编码解码计算，也就是会提高节点的cpu资源消耗

• storageclass-test.yaml:

测试场景，只需要单个节点，只有一份镜像，有数据丢失的风险
使用 csi/rbd 目录下的配置文间，csi是推荐的驱动，flex驱动不推荐使用

[root@master ~]# cd /tmp/rook/cluster/examples/kubernetes/ceph/csi/rbd/
[root@master rbd]# sed -i 's/failureDomain: host/failureDomain: osd/g' storageclass.yaml
[root@master rbd]# vim storageclass.yaml
[root@master rbd]# kubectl apply -f storageclass.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block created
[root@master rbd]# kubectl get sc -n rook-ceph

2.查看配置结果

[root@master rbd]# kubectl -n rook-ceph get pod -l "app=rook-ceph-tools"
NAME   READY   STATUS    RESTARTS   AGE
rook-ceph-tools-7476c966b7-5f5kg   1/1  Running 0    46h
[root@master rbd]# NAME=$(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}')
[root@master rbd]# kubectl -n rook-ceph exec -it ${NAME} ceph status
[root@master rbd]# kubectl -n rook-ceph exec -it ${NAME} ceph osd status
[root@master rbd]# kubectl -n rook-ceph exec -it ${NAME} ceph osd df

3.kubernetes Dashboard 查看结果

4.创建 Wordpress 进行测试

[root@master rbd]# cd /tmp/rook/cluster/examples/kubernetes
[root@master kubernetes]# sed -i 's|mysql:5.6|registry.cn-hangzhou.aliyuncs.com/vinc-auto/mysql:5.6|g' mysql.yaml
[root@master kubernetes]# sed -i 's|wordpress:4.6.1-apache|registry.cn-hangzhou.aliyuncs.com/vinc-auto/wordpress:4.6.1-apache|g' wordpress.yaml
[root@master kubernetes]# sed -i 's/LoadBalancer/NodePort/g' wordpress.yaml
[root@master kubernetes]# kubectl apply -f mysql.yaml
service/wordpress-mysql created
persistentvolumeclaim/mysql-pv-claim created
deployment.apps/wordpress-mysql created
[root@master kubernetes]# kubectl apply -f wordpress.yaml
service/wordpress created
persistentvolumeclaim/wp-pv-claim created
deployment.apps/wordpress created

1.查看创建的资源

[root@master kubernetes]# kubectl get pvc -o wide
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE   VOLUMEMODE
mysql-pv-claim   Bound    pvc-8a58a8cc-2620-47ba-8182-58e7b0ef998e   20Gi       RWO            rook-ceph-block   12m   Filesystem
wp-pv-claim      Bound    pvc-50626a36-64f3-4fb8-a83f-e4dff26be1f4   20Gi       RWO            rook-ceph-block   12m   Filesystem
[root@master kubernetes]#  kubectl get deploy -o wide
NAME              READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                               SELECTOR
wordpress         1/1     1            1           2m24s   wordpress    registry.cn-hangzhou.aliyuncs.com/vinc-auto/wordpress:4.6.1-apache   app=wordpress,tier=frontend
wordpress-mysql   1/1     1            1           2m43s   mysql        registry.cn-hangzhou.aliyuncs.com/vinc-auto/mysql:5.6                app=wordpress,tier=mysql
[root@master kubernetes]#  kubectl get pod -o wide
NAME                              READY   STATUS    RESTARTS   AGE     IP            NODE     NOMINATED NODE   READINESS GATES
nginx-xingdian                    1/1     Running   1          2d9h    10.244.2.3    node-2   <none>           <none>
nginx-xingdian-2                  1/1     Running   0          2d8h    10.244.1.2    node-1   <none>           <none>
wordpress-5587564997-lgp2w        1/1     Running   0          113s    10.244.2.26   node-2   <none>           <none>
wordpress-mysql-55456f44b-kgnj6   1/1     Running   0          2m13s   10.244.2.25   node-2   <none>           <none>

2.浏览器访问 wordpress 进行部署

3.查看Ceph集群中的相关数据

[root@master ~]# kubectl -n rook-ceph get pod -l "app=rook-ceph-tools"
NAME        READY   STATUS    RESTARTS   AGE
rook-ceph-tools-7476c966b7-5f5kg   1/1     Running   0    47h
[root@master ~]# NAME=$(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}')
[root@master ~]# kubectl -n rook-ceph exec -it ${NAME} sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
sh-4.2# ceph osd pool stats
pool replicapool id 1
  client io 3.7 KiB/s rd, 195 KiB/s wr, 2 op/s rd, 2 op/s wr
sh-4.2# rbd ls -p replicapool
csi-vol-0377015d-4624-11eb-80eb-12916ae40d83
csi-vol-864d78df-4628-11eb-80eb-12916ae40d83
csi-vol-904c6c28-4628-11eb-80eb-12916ae40d83
sh-4.2# rbd info replicapool/'csi-vol-0377015d-4624-11eb-80eb-12916ae40d83'
rbd image 'csi-vol-0377015d-4624-11eb-80eb-12916ae40d83':
	size 20 GiB in 5120 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 8a347eb76d55
	block_name_prefix: rbd_data.8a347eb76d55
	format: 2
	features: layering
	op_features: 
	flags: 
	create_timestamp: Tue Dec 22 11:20:30 2020
	access_timestamp: Tue Dec 22 11:20:30 2020
	modify_timestamp: Tue Dec 22 11:20:30 2020

4.删除测试环境

[root@k8s-master kubernetes]# cd /tmp/rook/cluster/examples/kubernetes
[root@k8s-master kubernetes]# kubectl delete -f wordpress.yaml
[root@k8s-master kubernetes]# kubectl delete -f mysql.yaml
[root@k8s-master kubernetes]# kubectl delete -n rook-ceph cephblockpools.ceph.rook.io replicapool
[root@k8s-master kubernetes]# kubectl delete storageclass rook-ceph-block