一、环境信息
主机名IPk8s角色rook角色磁盘20G
rook01192.168.86.36master,slavemon,mgr,osd,csi/dev/sdb-sdd
rook02192.168.86.37master,slavemon,mgr,osd,csi/dev/sdb-sdd
rook03192.168.86.38master,slavemon,mgr,osd,csi/dev/sdb-sdd
rook04192.168.86.39slaveosd,csi/dev/sdb-sdd
二、部署和卸载

1、获取安装源码(使用写博客时最新的版本)

git clone --single-branch --branch v1.7.2 https://github.com/rook/rook.git

2、调整mon调度策略参数,⽣产环境有⼀些专⻔的节点⽤于mon、mgr,存储节点节点使⽤单独的节点承担,利⽤调度机制实现
rook/cluster/examples/kubernetes/ceph/cluster.yaml

#定义mon节点分布和mgr节点分布
  placement:
    mon:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: ceph-mon
              operator: In
              values:
              - enable
    mgr:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: ceph-mgr
              operator: In
              values:
              - enable

设置磁盘的参数,调整为false,⽅便后⾯定制

  storage: # cluster level storage configuration and selection
    useAllNodes: false
    useAllDevices: false
# nodes below will be used as storage resources.  Each node's 'name' field should match their 'kubernetes.io/hostname' label.
    nodes:
    - name: "rook01"
      devices: # specific devices to use for storage can be specified for each node
      - name: "sdb"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdc"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdd"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
    - name: "rook02"
      devices: # specific devices to use for storage can be specified for each node
      - name: "sdb"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdc"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdd"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
    - name: "rook03"
      devices: # specific devices to use for storage can be specified for each node
      - name: "sdb"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdc"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdd"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
    - name: "rook04"
      devices: # specific devices to use for storage can be specified for each node
      - name: "sdb"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdc"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdd"
        config:
          storeType: bluestore
          journalSizeMB: "4096"

设置资源限制,默认组件没有设置资源分配,当出现资源争抢的时候可能会出现驱逐,为了保证Ceph的核⼼组件能分配到特定的资源,需要设置合理的资源分配;
mon,内存推荐128G
mds
osd,每T磁盘建议需要有4G以上内存

  resources:
    mon:
      limits:
        cpu: "500m"
        memory: "1Gi"
      requests:
        cpu: "500m"
        memory: "1Gi"
    mgr:
      limits:
        cpu: "500m"
        memory: "1Gi"
      requests:
        cpu: "500m"
        memory: "1Gi"
    osd:
      limits:
        cpu: "1000m"
        memory: "2Gi"
      requests:
        cpu: "1000m"
        memory: "2Gi"

3、给节点打label并部署rook

kubectl label node rook01 ceph-mon=enable
kubectl label node rook01 ceph-mgr=enable
kubectl label node rook02 ceph-mon=enable
kubectl label node rook02 ceph-mgr=enable
kubectl label node rook03 ceph-mon=enable
kubectl label node rook03 ceph-mgr=enable

cd rook/cluster/examples/kubernetes/ceph/
kubectl apply -f common.yaml
kubectl apply -f crds.yaml
kubectl apply -f operator.yaml
kubectl apply -f cluster.yaml
#部署客户端,提供ceph集群命令操作入口
kubectl apply -f toolbox.yaml

扩容osd或者主机直接在cluster.yaml中添加新节点信息即可,加入节点后osd无法被创建需要修改operator.yaml中的rook-discover扫描时间,默认1小时

            # The duration between discovering devices in the rook-discover daemonset.
            - name: ROOK_DISCOVER_DEVICES_INTERVAL
              value: "60s"
Logo

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐