前言

环境:centos 7.9 k8s 1.22.17

为什么要是使用rook部署ceph?从官网https://docs.ceph.com/en/latest/install/#recommended-methods可以得知ceph官方推荐在k8s集群中使用rook部署和管理ceph集群。所以本篇讲解如何在k8s中使用rook部署ceph集群。

什么是rook

rook官网:https://rook.io/
官网介绍,rook是开源的,Kubernetes的云原生存储。
rook是Kubernetes的存储操作员,Rook将分布式存储系统转变为自管理、自扩展、自修复的存储服务。它自动化了存储管理员的任务:部署、引导、配
置、发放、扩展、升级、迁移、灾难恢复、监控和资源管理。
Rook使用Kubernetes平台的强大功能,通过Kubernetes Operator为Ceph提供服务。
rook是Ceph存储提供商,Rook编排Ceph存储解决方案,使用专门的Kubernetes Operator实现自动化管理。Rook确保Ceph在Kubernetes上运行良好,
并简化部署和管理。

Rook是一个开源的云原生存储编排器,为Ceph存储提供平台、框架和支持,以便与云原生环境进行本地集成。
rook既可以k8s集群里创建一个ceph集群,rook还可以配置链接外部的ceph集群供k8s使用,本篇使用rook在k8s集群内部创建ceph集群。

先决条件

1、安装依赖
#k8s所有节点安装lvm,因为OSD pods运行的节点需要
sudo yum install -y lvm2
2、确认内核是否支持rbd
#确认内核是否支持rbd
lsmod | grep rbd		#有输出说明支持
rbd                    83640  0 
libceph               306625  1 rbd
#如果没有输出,则临时载入rbd模块,重启失效
modprobe  rbd			#内核加载rbd
#写入配置文件,开机自动加载,
[root@master ~]# vim  /etc/modules-load.d/rbd.conf		#文件名任意,以.conf结尾即可
rbd										#内容写rbd即可
[root@master ~]# 
3、内核要求
如果从CephFS创建RWX卷,推荐的最低内核版本是4.17。如果内核版本低于4.17,则不会强制执行所请求的PVC大小。存储配额只在较新的内核上强制执行。
4、k8s版本要求
Kubernetes v1.21 or higher is supported by Rook.
5、服务器架构要求
Architectures supported are amd64 / x86_64 and arm64.
6、存储要求,满足一种即可
原始设备(没有分区或格式化的文件系统)
原始分区(没有格式化的文件系统)
LVM逻辑卷(没有格式化的文件系统)
块模式下存储类中可用的持久卷
使用命令lsblk -f查看,如果FSTYPE字段不为空,则在相应的设备上没有文件系统,则rook可以使用该设备。

开始部署rook

#官方文档https://rook.io/docs/rook/v1.11/Getting-Started/quickstart/#tldr
#在k8smaster节点上克隆rook的源码
mkdir  ~/rook-ceph
cd ~/rook-ceph
yum install git -y
git clone --single-branch --branch v1.11.7 https://github.com/rook/rook.git
cd rook/deploy					#deploy目录下有两种安装方式
[root@master deploy]# ll							
drwxr-xr-x 5 root root   63 Jun 13 17:44 charts			#helm安装的ceph
drwxr-xr-x 4 root root 4096 Jun 13 17:44 examples		#yaml文件手动安装的rook-ceph
drwxr-xr-x 3 root root   22 Jun 13 17:44 olm

#本次采用yaml文件手动安装的rook-ceph
[root@master deploy]# cd examples

[root@master examples]# ls 	crds.yaml -f common.yaml -f operator.yaml	cluster.yaml images.txt 
crds.yaml  common.yaml  operator.yaml  cluster.yaml
[root@master examples]# 
#从上面的名字可以得知,crd.yaml就是自定义的crd资源清单,common.yaml定义了各种各样的rbac相关的role、rolebinding和serviceaccount
#operator.yaml文件里面用于创建一个configmap和deployment,该deployment就是Rook Ceph Operator
#cluster.yaml创建了kind: CephCluster的自定义资源,这个资源其实会安装Ceph daemon pods (mon, mgr, osd, mds, rgw)
#images.txt 镜像文件,可以知道需要哪些镜像,可以提前下载好镜像

#开始安装rook-ceph,安装顺序crds.yaml、common.yaml、operator.yaml、cluster.yaml
[root@master examples]# kubectl create -f crds.yaml -f common.yaml
[root@master examples]# vim operator.yaml			#可以先看下operator.yaml,该文件里面用于创建一个configmap和deployment
 ROOK_CSI_KUBELET_DIR_PATH: "/var/lib/kubelet"	#这个变量配置的是kubelet的路径,如果你的kubelet安装在其它目录请更改该参数
 ROOK_ENABLE_DISCOVERY_DAEMON: "true"		#是否启动discovery守护进程监视集群内节点上的裸存储设备,默认是false,这里设为true
 #如果你只打算基于StorageClassDeviceSets和pvc创建你的osd,这个守护进程不需要运行

operator.yaml文件configmap里面有一些注释掉的默认的镜像可能下载不了,找了个别人的镜像仓库,使用下面的方法解决:
docker pull registry.aliyuncs.com/it00021hot/csi-attacher:v4.1.0
docker tag registry.aliyuncs.com/it00021hot/csi-attacher:v4.1.0 registry.k8s.io/sig-storage/csi-attacher:v4.1.0
docker rmi registry.aliyuncs.com/it00021hot/csi-attacher:v4.1.0

docker pull registry.aliyuncs.com/it00021hot/csi-node-driver-registrar:v2.7.0
docker tag registry.aliyuncs.com/it00021hot/csi-node-driver-registrar:v2.7.0  registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.7.0
docker rmi registry.aliyuncs.com/it00021hot/csi-node-driver-registrar:v2.7.0

docker pull registry.aliyuncs.com/it00021hot/csi-resizer:v1.7.0
docker tag registry.aliyuncs.com/it00021hot/csi-resizer:v1.7.0 registry.k8s.io/sig-storage/csi-resizer:v1.7.0
docker rmi registry.aliyuncs.com/it00021hot/csi-resizer:v1.7.0 

docker pull  registry.aliyuncs.com/it00021hot/csi-provisioner:v3.4.0
docker tag registry.aliyuncs.com/it00021hot/csi-provisioner:v3.4.0 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0
docker rmi registry.aliyuncs.com/it00021hot/csi-provisioner:v3.4.0

docker pull  registry.aliyuncs.com/it00021hot/csi-snapshotter:v6.2.1
docker tag registry.aliyuncs.com/it00021hot/csi-snapshotter:v6.2.1  registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1
docker rmi registry.aliyuncs.com/it00021hot/csi-snapshotter:v6.2.1 

[root@master examples]# kubectl create  -f operator.yaml	#安装operator.yaml

cluster.yaml文件讲解

[root@master examples]# grep -Ev '^$|#' cluster.yaml 
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph # namespace:cluster
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v17.2.6
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook		# 配置文件将被持久化的主机路径,必须指定.
  skipUpgradeChecks: false
  continueUpgradeAfterChecksEvenIfNotHealthy: false
  waitTimeoutForHealthyOSDInMinutes: 10
  mon:
    count: 3							#mon的个数,默认3个,对于生产环境,建议至少配置3个mon,应该指定奇数个mon
    allowMultiplePerNode: false			#是否允许多个mon在一个主机上,默认false不允许
  mgr:
    count: 2							#mgr的个数,默认2个,当需要mgr高可用性时,应当设置为2,实现一主一备
    allowMultiplePerNode: false			#是否允许多个mon在一个主机上,默认false不允许
    modules:
      - name: pg_autoscaler
        enabled: true
  dashboard:
    enabled: true						#ceph的dashboard面板,默认是开启的
    ssl: true
  monitoring:				
    enabled: false
    metricsDisabled: false
  network:
    connections:
      encryption:
        enabled: false
      compression:
        enabled: false
      requireMsgr2: false
  crashCollector:
    disable: false
  logCollector:
    enabled: true
    periodicity: daily # one of: hourly, daily, weekly, monthly
    maxLogSize: 500M # SUFFIX may be 'M' or 'G'. Must be at least 1M.
  cleanupPolicy:
    confirmation: ""
    sanitizeDisks:
      method: quick
      dataSource: zero
      iteration: 1
    allowUninstallWithVolumes: false
  annotations:
  labels:
  resources:
  removeOSDsIfOutAndSafeToRemove: false
  priorityClassNames:
    mon: system-node-critical
    osd: system-node-critical
    mgr: system-cluster-critical
  storage: 					#集群级存储配置和选则,即选择哪些磁盘作为OSD存储
    useAllNodes: true		#默认使用所有节点,如果有节点规划,则设为false
    useAllDevices: true		#默认使用所有没有文件系统的分区,如果有节点磁盘规划,则设为false
    config:
    onlyApplyOSDPlacement: false
  disruptionManagement:
    managePodBudgets: true
    osdMaintenanceTimeout: 30
    pgHealthCheckTimeout: 0

  healthCheck:
    daemonHealth:
      mon:
        disabled: false
        interval: 45s
      osd:
        disabled: false
        interval: 60s
      status:
        disabled: false
        interval: 60s
    livenessProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false
    startupProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false

[root@master examples]# 



[root@master examples]# vim cluster.yaml					#查看cluster.yaml,关注下下面这个参数

#重要提示:如果您重新安装集群,请确保在每个主机上删除此目录,否则在新集群上启动mons将失败。
#在Minikube中,'/data'目录被配置为在重启时持久化。在Minikube环境中使用“/data/rook”
dataDirHostPath: /var/lib/rook		#保持默认,不用改


[root@master examples]# kubectl create -f cluster.yaml		#安装cluster.yaml

查看cephcluster

[root@ceph-1 examples]# kubectl -n rook-ceph get cephcluster
NAME        DATADIRHOSTPATH   MONCOUNT   AGE    PHASE   MESSAGE                        HEALTH      EXTERNAL   FSID
rook-ceph   /var/lib/rook     3          2d1h   Ready   Cluster created successfully   HEALTH_OK              25f12c96-29b0-4487-8e2d-ec24525e33f9
[root@ceph-1 examples]# 

查看pod

[root@master examples]# kubectl  get all -n rook-ceph
NAME                                                   READY   STATUS      RESTARTS   AGE
pod/csi-cephfsplugin-c4dt7                             2/2     Running     0          15m
pod/csi-cephfsplugin-ddk4x                             2/2     Running     0          15m
pod/csi-cephfsplugin-nv2m8                             2/2     Running     0          3m42s
pod/csi-cephfsplugin-provisioner-58948fc785-jbmxs      5/5     Running     0          4m20s
pod/csi-cephfsplugin-provisioner-58948fc785-sr9r6      5/5     Running     0          15m
pod/csi-rbdplugin-ht49j                                2/2     Running     0          15m
pod/csi-rbdplugin-k5lwp                                2/2     Running     0          15m
pod/csi-rbdplugin-provisioner-84bfcb8bfc-8xb4f         5/5     Running     0          15m
pod/csi-rbdplugin-provisioner-84bfcb8bfc-m8kgp         5/5     Running     0          15m
pod/csi-rbdplugin-x9p4g                                2/2     Running     0          3m42s
pod/rook-ceph-crashcollector-ceph-1-6f67c84f4b-p2rxt   1/1     Running     0          15m
pod/rook-ceph-crashcollector-ceph-2-76d94b6769-x2q95   1/1     Running     0          13m
pod/rook-ceph-crashcollector-ceph-3-5ffd7fc4d-4tkfc    1/1     Running     0          13m
pod/rook-ceph-mgr-a-745c8fb9c-8lxkm                    3/3     Running     0          15m
pod/rook-ceph-mgr-b-77bdd5584b-wqb6w                   3/3     Running     0          15m
pod/rook-ceph-mon-a-5bdf5ccd9-ptzhl                    2/2     Running     0          17m
pod/rook-ceph-mon-b-577d5bfcd7-22vtt                   2/2     Running     0          15m
pod/rook-ceph-mon-c-84bccb66f5-k9lzj                   2/2     Running     0          15m
pod/rook-ceph-operator-58d8b7b5df-8blw4                1/1     Running     0          166m
pod/rook-ceph-osd-0-844ffbd8d7-9cclh                   2/2     Running     0          13m
pod/rook-ceph-osd-1-6ccd9cc645-jvqjh                   2/2     Running     0          13m
pod/rook-ceph-osd-2-54866f6f46-c7vpx                   2/2     Running     0          13m
pod/rook-ceph-osd-prepare-ceph-1-w5cnd                 0/1     Completed   0          13m
pod/rook-ceph-osd-prepare-ceph-2-rxml8                 0/1     Completed   0          13m
pod/rook-ceph-osd-prepare-ceph-3-dkq98                 0/1     Completed   0          12m

NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/rook-ceph-mgr             ClusterIP   10.233.42.104   <none>        9283/TCP            14m
service/rook-ceph-mgr-dashboard   ClusterIP   10.233.36.230   <none>        8443/TCP            14m
service/rook-ceph-mon-a           ClusterIP   10.233.3.156    <none>        6789/TCP,3300/TCP   17m
service/rook-ceph-mon-b           ClusterIP   10.233.62.240   <none>        6789/TCP,3300/TCP   15m
service/rook-ceph-mon-c           ClusterIP   10.233.34.110   <none>        6789/TCP,3300/TCP   15m

NAME                              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/csi-cephfsplugin   3         3         3       3            3           <none>          15m
daemonset.apps/csi-rbdplugin      3         3         3       3            3           <none>          15m

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/csi-cephfsplugin-provisioner      2/2     2            2           15m
deployment.apps/csi-rbdplugin-provisioner         2/2     2            2           15m
deployment.apps/rook-ceph-crashcollector-ceph-1   1/1     1            1           15m
deployment.apps/rook-ceph-crashcollector-ceph-2   1/1     1            1           15m
deployment.apps/rook-ceph-crashcollector-ceph-3   1/1     1            1           15m
deployment.apps/rook-ceph-mgr-a                   1/1     1            1           15m
deployment.apps/rook-ceph-mgr-b                   1/1     1            1           15m
deployment.apps/rook-ceph-mon-a                   1/1     1            1           17m
deployment.apps/rook-ceph-mon-b                   1/1     1            1           15m
deployment.apps/rook-ceph-mon-c                   1/1     1            1           15m
deployment.apps/rook-ceph-operator                1/1     1            1           3h54m
deployment.apps/rook-ceph-osd-0                   1/1     1            1           13m
deployment.apps/rook-ceph-osd-1                   1/1     1            1           13m
deployment.apps/rook-ceph-osd-2                   1/1     1            1           13m

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/csi-cephfsplugin-provisioner-58948fc785      2         2         2       15m
replicaset.apps/csi-rbdplugin-provisioner-84bfcb8bfc         2         2         2       15m
replicaset.apps/rook-ceph-crashcollector-ceph-1-6f67c84f4b   1         1         1       15m
replicaset.apps/rook-ceph-crashcollector-ceph-2-76d94b6769   1         1         1       13m
replicaset.apps/rook-ceph-crashcollector-ceph-2-fd87584cf    0         0         0       15m
replicaset.apps/rook-ceph-crashcollector-ceph-3-5ffd7fc4d    1         1         1       13m
replicaset.apps/rook-ceph-crashcollector-ceph-3-6955f6c55d   0         0         0       15m
replicaset.apps/rook-ceph-mgr-a-745c8fb9c                    1         1         1       15m
replicaset.apps/rook-ceph-mgr-b-77bdd5584b                   1         1         1       15m
replicaset.apps/rook-ceph-mon-a-5bdf5ccd9                    1         1         1       17m
replicaset.apps/rook-ceph-mon-b-577d5bfcd7                   1         1         1       15m
replicaset.apps/rook-ceph-mon-c-84bccb66f5                   1         1         1       15m
replicaset.apps/rook-ceph-operator-58d8b7b5df                1         1         1       3h54m
replicaset.apps/rook-ceph-osd-0-844ffbd8d7                   1         1         1       13m
replicaset.apps/rook-ceph-osd-1-6ccd9cc645                   1         1         1       13m
replicaset.apps/rook-ceph-osd-2-54866f6f46                   1         1         1       13m

NAME                                     COMPLETIONS   DURATION   AGE
job.batch/rook-ceph-osd-prepare-ceph-1   1/1           2s         13m
job.batch/rook-ceph-osd-prepare-ceph-2   1/1           4s         13m
job.batch/rook-ceph-osd-prepare-ceph-3   1/1           7s         12m
[root@master examples]# 

安装ceph-toolbox客户端工具

以上我们已经搭建好了ceph集群,那么如何验证ceph集群是否健康的呢?官方提供了ceph-toolbox这个容器客户端工具来使用ceph相关命令操作ceph集群,ceph-toolbox也是一个容器,按照官网步骤创建ceph-toolbox即可:

ceph-toolbox官网:https://rook.io/docs/rook/v1.11/Troubleshooting/ceph-toolbox/
官网说toolbox有两种运行模式:
交互式:启动一个toolbox pod,您可以在其中连接并从shell执行Ceph命令;
一次性作业:使用Ceph命令运行脚本,并从作业日志中收集结果。
我们这里使用交互式,即启动一个toolbox pod,然后在其中使用shell执行Ceph命令。
toolbox的yaml文件就在源码包里面,直接安装即可。

[root@master examples]# ll toolbox.yaml toolbox-job.yaml 
-rw-r--r-- 1 root root 1868 Jul 11 17:31 toolbox-job.yaml	#一次性作业
-rw-r--r-- 1 root root 4220 Jul 11 17:31 toolbox.yaml		#交互式
[root@master examples]# 
[root@master examples]# kubectl  apply  -f toolbox.yaml 
deployment.apps/rook-ceph-tools created
[root@master examples]# 
[root@master examples]# kubectl  get pod -n rook-ceph 
NAME                                               READY   STATUS      RESTARTS   AGE
rook-ceph-tools-657868c8cf-sz5b7                   1/1     Running     0          25s
[root@master examples]# kubectl  -n rook-ceph exec -it rook-ceph-tools-657868c8cf-sz5b7 -- bash
bash-4.4$ ceph status			#查看ceph集群状态
  cluster:
    id:     25f12c96-29b0-4487-8e2d-ec24525e33f9
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum a,b,c (age 54m)
    mgr: a(active, since 51m), standbys: b
    osd: 3 osds: 3 up (since 51m), 3 in (since 52m)
 
  data:
    pools:   1 pools, 1 pgs
    objects: 2 objects, 449 KiB
    usage:   62 MiB used, 147 GiB / 147 GiB avail
    pgs:     1 active+clean
 
bash-4.4$ ceph fsid						#查看ceph集群cluster ID
25f12c96-29b0-4487-8e2d-ec24525e33f9
bash-4.4$ 
bash-4.4$ cat /etc/ceph/ceph.conf   	#可以看到,容器里面其实保存了ceph.conf文件
[global]
mon_host = 10.233.34.110:6789,10.233.3.156:6789,10.233.62.240:6789		#3个mon的svc IP

[client.admin]
keyring = /etc/ceph/keyring
bash-4.4$ cat /etc/ceph/keyring   		#可以看到,容器里面其实保存了keyring文件
[client.admin]
key = AQBuZa1kTXIsLBAATbTu19OPcVATJX4rgnJFCQ==
bash-4.4$ ceph auth  get-key client.admin	#这条命令也可以看到keyring
AQBuZa1kTXIsLBAATbTu19OPcVATJX4rgnJFCQ==
bash-4.4$

启用ceph-dashboard

官网:https://rook.io/docs/rook/v1.11/Storage-Configuration/Monitoring/ceph-dashboard/#enable-the-ceph-dashboard

#在安装cluster.yaml是我们启用了ceph的dashboard面板
[root@ceph-1 examples]# grep -B9  -i dashboard cluster.yaml 
# enable the ceph dashboard for viewing cluster status
  dashboard:
    enabled: true
    # serve the dashboard under a subpath (useful when you are accessing the dashboard via a reverse proxy)
    # urlPrefix: /ceph-dashboard
    # serve the dashboard at the given port.
    # port: 8443
    # serve the dashboard using SSL
[root@ceph-1 examples]# kubectl -n rook-ceph get service	#查看svc
NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
rook-ceph-mgr             ClusterIP   10.233.42.104   <none>        9283/TCP            43h
rook-ceph-mgr-dashboard   ClusterIP   10.233.36.230   <none>        8443/TCP            43h
..........................
说明:
rook-ceph-mgr这个service是报告Prometheus metrics的;
rook-ceph-mgr-dashboard这个service就是ceph的dashboard面板
[root@ceph-1 examples]# 
Rook为dashboard创建了一个默认用户,名称叫做admin,该用户密码保存在rook-ceph-dashboard-password这个secret里面
可以这样解密:kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{.data.password}" | base64 --decode && echo 

访问dashboard

由于rook-ceph-mgr-dashboard svc默认是ClusterIP类型的,所以外部浏览器无法访问,尝试编辑rook-ceph-mgr-dashboard 这个svc为NodePort类型,即使编辑时保存成功,但是发现它会自动变成ClusterIP类型,即自动还原了。所以这种方法行不通。
根据官网介绍,可以有多种方式实现,NodePort是最简单的一种,官网提供了对应的svc,就在源码里面:

[root@ceph-1 examples]# ll dashboard-external-https.yaml
-rw-r--r-- 1 root root 432 Jul 11 17:31 dashboard-external-https.yaml
[root@ceph-1 examples]# kubectl  apply  -f dashboard-external-https.yaml
[root@ceph-1 examples]# kubectl  get svc -n rook-ceph 
NAME                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
rook-ceph-mgr                            ClusterIP   10.233.42.104   <none>        9283/TCP            44h
rook-ceph-mgr-dashboard                  ClusterIP   10.233.36.230   <none>        8443/TCP            44h
rook-ceph-mgr-dashboard-external-https   NodePort    10.233.63.61    <none>        8443:17468/TCP      8s
[root@ceph-1 examples]# 
浏览器访问:
https://192.168.244.6:17468/	
注意是https访问,不是http,账号admin,密码保存在rook-ceph-dashboard-password这个secret里面

删除rook-ceph集群

官网清除集群说明:https://rook.io/docs/rook/v1.11/Getting-Started/ceph-teardown/
#如果yaml文件还存在,则可以根据yaml删除资源
kubectl delete -f cluster.yaml
kubectl delete -f common.yaml
kubectl delete -f operator.yaml
rm -rf /var/lib/rook		#删除每台宿主机上的ceph默认配置目录
Logo

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐