我的博客原文链接:https://blog.gcc.ac.cn/post/2023/%E8%BF%B7%E4%BD%A0ceph%E9%9B%86%E7%BE%A4%E6%90%AD%E5%BB%BA/

环境

机器列表:

IP角色说明
10.0.0.15osdARMv7,512M内存,32G存储,百兆网口
10.0.0.16clientARM64,2G内存,测试集群性能用,千兆网口
10.0.0.17osdAMD64,2G内存,32G存储,百兆网口
10.0.0.18mon,mgr,mds,osdAMD64,2G内存,32G存储,百兆网口
10.0.0.19osdARM64,1G内存,32G存储,百兆网口

这集群有x86,有ARM,ARM里面还分ARMv7和ARM64,你问我是有意的还是不小心的?我是有意用三种架构的CPU的。

注意:老版本(14.2.21)不同架构不能混合部署,否则osd通信有问题。后续尝试17.2.5版本发现已修复,不过新版本又出现了个RAW USED不准的bug(可能是因为ARMv7是32位,只支持4G空间)以及osd无法删除的bug。

系统采用ubuntu或debian。参考文档:https://docs.ceph.com/en/quincy/install/manual-deployment

官方推荐内存最少4G,我最低用512M内存的板子,纯属娱乐。

部署步骤

【可选】增加软件源/etc/apt/sources.list.d/ceph.list

deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-quincy/ bullseye main

【可选】添加公钥:

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E84AC2C0460F3994

在所有ceph节点和客户端安装软件包:

apt-get update
apt-get install -y ceph ceph-mds ceph-volume

在所有ceph节点和客户端创建配置文件/etc/ceph/ceph.conf

[global]
fsid = 4a0137cc-1081-41e4-bacd-5ecddc63ae2d
mon initial members = intel-compute-stick
mon host = 10.0.0.17
public network = 10.0.0.0/24
cluster network = 10.0.0.0/24
osd journal size = 64
osd pool default size = 2
osd pool default min size = 1
osd pool default pg num = 64
osd pool default pgp num = 64
osd crush chooseleaf type = 1

在mon节点执行初始化命令:

ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
chown ceph:ceph /tmp/ceph.mon.keyring
monmaptool --create --add intel-compute-stick 10.0.0.17 --fsid 4a0137cc-1081-41e4-bacd-5ecddc63ae2d /tmp/monmap
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-intel-compute-stick
sudo -u ceph ceph-mon --mkfs -i intel-compute-stick --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
systemctl start ceph-mon@intel-compute-stick

部署MGR:

name=intel-compute-stick
mkdir -p /var/lib/ceph/mgr/ceph-$name
ceph auth get-or-create mgr.$name mon 'allow profile mgr' osd 'allow *' mds 'allow *' > /var/lib/ceph/mgr/ceph-$name/keyring
ceph-mgr -i $name

【旧版本需要做此操作】在ARMv7架构的osd节点修改文件nano /usr/lib/python3/dist-packages/ceph_argparse.py,将timeout = kwargs.pop('timeout', 0)改成timeout = 24 * 60 * 60

/var/lib/ceph/bootstrap-osd/ceph.keyring同步到所有osd节点。

部署OSD:

truncate --size 16G osd.img
losetup /dev/loop5 /root/osd.img 
ceph-volume raw prepare --data /dev/loop5 --bluestore
systemctl start ceph-osd@{osd-num} # 注意这个num要和前面输出一致

部署mds:

mkdir -p /var/lib/ceph/mds/ceph-intel-compute-stick
ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-intel-compute-stick/keyring --gen-key -n mds.intel-compute-stick
ceph auth add mds.intel-compute-stick osd "allow rwx" mds "allow *" mon "allow profile mds" -i /var/lib/ceph/mds/ceph-intel-compute-stick/keyring

修改/etc/ceph/ceph.conf,增加如下内容

[mds.intel-compute-stick]
host = intel-compute-stick

执行命令:

ceph-mds --cluster ceph -i intel-compute-stick -m 10.0.0.17:6789
systemctl start ceph.target

创建cephfs:

ceph osd pool create perf_test 64
ceph osd pool create cephfs_data 64
ceph osd pool create cephfs_metadata 16
ceph fs new cephfs cephfs_metadata cephfs_data
ceph fs authorize cephfs client.cephfs_user / rw
# 将上面的输出复制到客户端`/etc/ceph/ceph.client.cephfs_user.keyring`文件中

挂载cephfs:

mkdir /mnt/cephfs
mount -t ceph cephfs_user@.cephfs=/ /mnt/cephfs

其它命令

显示 bluestore 缓存的当前值:ceph daemon /var/run/ceph/ceph-osd.1.asok config show | grep bluestore_cache_size

设置OSD缓存用量:ceph daemon /var/run/ceph/ceph-osd.1.asok config set bluestore_cache_size_ssd 134217128

ceph-volume raw list

重启后重新挂载osd:ceph-volume raw activate --device /dev/loop5 --no-systemd

手动删除osd:ceph osd out {osd-num}

ceph osd safe-to-destroy osd.2

ceph osd destroy {id} --yes-i-really-mean-it

性能测试

节点数随机写(IOPS)随机读(IOPS)顺序写(MiB/s)顺序读(MiB/s)rados写(MB/s)rados读(MB/s)rados随机(MB/s)
2160352958.0121.810.673321.559722.2359
31310605814.027.912.95228.079729.3502
41392408715.532.315.339833.215734.9272

fio随机读写
fio顺序读写
rados性能测试

两节点

两节点状态截图

cephfs:

  • dd写入:16.3 MB/s (dd if=/dev/zero of=test.img bs=1M count=1024)
  • dd读取:17.9 MB/s (echo 3 > /proc/sys/vm/drop_caches; dd if=test.img of=/dev/null bs=1M)
  • fio随机写:1603
  • fio随机读:5295
  • fio顺序写:8201KiB/s
  • fio顺序读:21.8MiB/s

rados bench -p perf_test 30 write --no-cleanup

hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_orangepi3-lts_2101
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        16         0         0         0           -           0
    2      16        16         0         0         0           -           0
    3      16        16         0         0         0           -           0
    4      16        19         3   2.99955         3     3.64087     3.38759
    5      16        22         6   4.79925        12     4.34316     3.83332
    6      16        25         9   5.99904        12     5.38901     4.34105
    7      16        28        12   6.85602        12     6.50295     4.87251
    8      16        31        15   7.49875        12     4.71442      4.9794
    9      16        36        20   8.88738        20     4.61761     5.24509
   10      16        38        22   8.79849         8     9.75918     5.40151
   11      16        41        25   9.08934        12     5.56684     5.44177
   12      16        46        30   9.99826        20     3.34787     5.21296
   13      16        49        33   10.1521        12     4.10847     5.17286
   14      16        51        35   9.99824         8     3.49611     5.16674
   15      16        55        39   10.3982        16     6.06545       5.098
   16      16        58        42   10.4981        12     6.23518     5.07308
   17      16        59        43   10.1158         4     6.45517     5.10522
   18      16        62        46   10.2204        12     6.48019     5.08282
   19      16        63        47   9.89297         4     6.57521     5.11457
2023-04-18T15:12:33.881436+0800 min lat: 1.95623 max lat: 9.75918 avg lat: 5.12979
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16        68        52   10.3981        20     6.95187     5.12979
   21      16        68        52   9.90296         0           -     5.12979
   22      16        70        54    9.8164         4     7.85105     5.23762
   23      16        72        56   9.73736         8      8.1019     5.33872
   24      16        76        60   9.99818        16     8.86371     5.40133
   25      16        77        61   9.75822         4     3.14892      5.3644
   26      16        80        64   9.84436        12     9.63972     5.45864
   27      16        82        66     9.776         8     3.18262     5.48564
   28      16        84        68   9.71252         8     3.89847     5.52669
   29      16        87        71   9.79132        12     9.72659     5.63168
   30      16        91        75   9.99818        16     9.93409     5.63449
   31      13        91        78   10.0627        12     4.09479     5.65952
   32      11        91        80   9.99817         8      4.0216     5.68887
   33       6        91        85   10.3012        20     3.65814      5.7294
   34       1        91        90   10.5863        20     4.55552     5.71818
Total time run:         34.1039
Total writes made:      91
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     10.6733
Stddev Bandwidth:       6.19765
Max bandwidth (MB/sec): 20
Min bandwidth (MB/sec): 0
Average IOPS:           2
Stddev IOPS:            1.58086
Max IOPS:               5
Min IOPS:               0
Average Latency(s):     5.70886
Stddev Latency(s):      2.2914
Max latency(s):         11.0174
Min latency(s):         1.95623

rados bench -p perf_test 30 seq

hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        18         2   7.99563         8    0.749535    0.640048
    2      16        24         8   15.9934        24     1.82768      1.2209
    3      16        30        14   18.6602        24     2.90052     1.76318
    4      16        36        20   19.9939        24     3.22248     2.10121
    5      16        41        25   19.9941        20     2.38014     2.20456
    6      16        46        30   19.9943        20     3.69409     2.33743
    7      16        52        36   20.5655        24     3.58809      2.4047
    8      16        58        42   20.9942        24     3.94671     2.48664
    9      15        63        48   21.3151        24     2.99373     2.55527
   10      16        69        53   21.1832        20     3.06825     2.56988
   11      16        75        59   21.4387        24     3.56289     2.56456
   12      16        80        64   21.3185        20     1.57452     2.55362
   13      16        86        70   21.5243        24     1.79171     2.56933
   14      16        91        75   21.4152        20      4.4904      2.6246
   15      10        91        81   21.5871        24      4.2782     2.67077
   16       4        91        87   21.7375        24     2.84972     2.68923
Total time run:       16.8834
Total reads made:     91
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   21.5597
Average IOPS:         5
Stddev IOPS:          1.03078
Max IOPS:             6
Min IOPS:             2
Average Latency(s):   2.69827
Max latency(s):       4.4904
Min latency(s):       0.530561

rados bench -p perf_test 30 rand

hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        20         4   15.9921        16    0.756867    0.562019
    2      16        26        10   19.9929        24     1.82705     1.09684
    3      16        32        16   21.3269        24     2.89898     1.63216
    4      16        38        22   21.9939        24      2.8592     1.96528
    5      16        42        26   20.7945        16     2.89643     2.08897
    6      16        48        32   21.3279        24     3.25362     2.24424
    7      16        54        38   21.7089        24     2.53504     2.33103
    8      16        60        44   21.9947        24     2.49214     2.39352
    9      16        66        50   22.2169        24      2.8506     2.46248
   10      16        70        54   21.5948        16     2.89337     2.49112
   11      16        76        60   21.8128        24     2.13852     2.49727
   12      16        82        66   21.9946        24     2.53781     2.54012
   13      16        88        72   22.1485        24     2.53625     2.57632
   14      16        94        78   22.2804        24     2.89682     2.59727
   15      15        98        83   22.1192        20     2.80375     2.61577
   16      16       104        88   21.9865        20     2.54292     2.62566
   17      16       110        94   22.1046        24     2.84867     2.64381
   18      16       116       100   22.2097        24     3.20428     2.65252
   19      16       122       106   22.3036        24      2.5454      2.6641
2023-04-18T15:13:26.426279+0800 min lat: 0.367019 max lat: 3.92008 avg lat: 2.6753
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       127       111   22.1883        20     2.84974      2.6753
   21      16       132       116   22.0838        20     2.90241     2.68289
   22      16       138       122   22.1707        24     3.20846     2.68523
   23      16       144       128     22.25        24     3.25698     2.69284
   24      16       150       134   22.3228        24     3.26158     2.69223
   25      16       155       139   22.2297        20     1.06362     2.67481
   26      16       160       144   22.1438        20      3.9225     2.67843
   27      16       166       150   22.2124        24     4.27967     2.68765
   28      16       172       156    22.276        24     4.33003     2.69881
   29      16       178       162   22.3353        24      3.9836     2.71073
   30      16       183       167   22.2573        20     2.13414     2.71708
   31      11       183       172   22.1844        20     3.57702     2.72915
   32       5       183       178    22.241        24     3.63257     2.73706
Total time run:       32.9197
Total reads made:     183
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   22.2359
Average IOPS:         5
Stddev IOPS:          0.669015
Max IOPS:             6
Min IOPS:             4
Average Latency(s):   2.74483
Max latency(s):       4.63637
Min latency(s):       0.367019

三节点

三节点状态截图

cephfs:

  • dd写入:21.3 MB/s (dd if=/dev/zero of=test.img bs=1M count=1024)
  • dd读取:19.5 MB/s (echo 3 > /proc/sys/vm/drop_caches; dd if=test.img of=/dev/null bs=1M)
  • fio随机写:1310
  • fio随机读:6058
  • fio顺序写:14.0MiB/s
  • fio顺序读:27.9MiB/s

rados bench -p perf_test 30 write --no-cleanup

hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_orangepi3-lts_2188
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        16         0         0         0           -           0
    2      16        17         1   1.99967         2     1.99725     1.99725
    3      16        19         3   3.99915         8     2.92674     2.50932
    4      16        21         5   4.99899         8     3.68936     2.91102
    5      16        25         9   7.19858        16     4.44208     3.50803
    6      16        29        13   8.66497        16     3.32056     3.77577
    7      16        31        15   8.56977         8     6.59867     3.95511
    8      16        33        17   8.49837         8     2.95588     3.89244
    9      16        39        23   10.2203        24     8.91377      4.5287
   10      16        42        26    10.398        12     4.00285     4.61726
   11      16        46        30    10.907        16     1.88708     4.32788
   12      16        48        32   10.6647         8     3.22648     4.22892
   13      16        52        36   11.0748        16     1.51764     4.17041
   14      16        53        37   10.5695         4     2.53073      4.1261
   15      16        55        39   10.3981         8     8.91467     4.39938
   16      16        60        44   10.9979        20     7.31564      4.7521
   17      16        61        45   10.5862         4     2.00639     4.69108
   18      16        66        50    11.109        20     2.34879     4.76501
   19      16        69        53   11.1558        12     2.87372     4.81177
2023-04-18T15:18:28.656929+0800 min lat: 1.51764 max lat: 10.6957 avg lat: 4.81906
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16        74        58   11.5978        20     2.18304     4.81906
   21      16        75        59   11.2359         4     7.06397     4.85711
   22      16        80        64   11.6341        20     3.49498     4.85191
   23      16        80        64   11.1283         0           -     4.85191
   24      16        82        66   10.9979         4      6.3591     4.83667
   25      16        84        68   10.8779         8     6.34421     4.91693
   26      16        89        73   11.2286        20     8.52854     5.06126
   27      16        92        76   11.2571        12     5.63676     5.07197
   28      16        97        81   11.5692        20     6.39309     5.04017
   29      16       100        84    11.584        12     7.29099     5.01084
   30      16       105        89   11.8644        20     2.50054     4.96138
   31      10       105        95   12.2558        24     9.50269     4.94482
   32       4       105       101   12.6226        24     4.52251     4.83104
Total time run:         32.4273
Total writes made:      105
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     12.952
Stddev Bandwidth:       7.39196
Max bandwidth (MB/sec): 24
Min bandwidth (MB/sec): 0
Average IOPS:           3
Stddev IOPS:            1.87271
Max IOPS:               6
Min IOPS:               0
Average Latency(s):     4.774
Stddev Latency(s):      2.42686
Max latency(s):         10.6957
Min latency(s):         1.51764

rados bench -p perf_test 30 seq

hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        19         3   11.9944        12    0.935954    0.768977
    2      16        26        10   19.9933        28     1.96017     1.31348
    3      16        33        17   22.6602        28     2.77006     1.68533
    4      16        42        26   25.9932        36     2.27413     1.75351
    5      16        51        35   27.9931        36     2.27554     1.77164
    6      16        59        43   28.6598        32     2.90927     1.80862
    7      16        65        49   27.9935        24     1.89015     1.83058
    8      16        72        56   27.9936        28    0.468285     1.88307
    9      16        81        65   28.8824        36    0.785162     1.90263
   10      15        88        73   29.1822        32      2.5208     1.94424
   11      16        95        79   28.7108        24      2.0044     1.96069
   12      16       103        87   28.9843        32     3.24909     1.95645
   13       9       105        96   29.5231        36     2.53005     1.97092
   14       3       105       102   29.1284        24     1.87098     2.00023
Total time run:       14.9574
Total reads made:     105
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   28.0797
Average IOPS:         7
Stddev IOPS:          1.68379
Max IOPS:             9
Min IOPS:             3
Average Latency(s):   2.02704
Max latency(s):       3.42015
Min latency(s):       0.417385

rados bench -p perf_test 30 rand

hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        22         6    23.986        24    0.824307    0.603606
    2      16        31        15   29.9884        36     1.90078     1.13916
    3      16        40        24   31.9898        36     2.15175     1.41768
    4      16        47        31   30.9908        28     1.42468      1.5251
    5      16        55        39    31.191        32     2.52356      1.5776
    6      16        63        47   31.3248        32     2.89946     1.61864
    7      16        71        55   31.4203        32    0.368204     1.66398
    8      16        78        62   30.9915        28     2.37269     1.70559
    9      16        87        71   31.5467        36     2.84709     1.77106
   10      16        95        79   31.5911        32     1.07912     1.79016
   11      16       104        88   31.9911        36     1.44303     1.80956
   12      16       112        96   31.9913        32     1.42344      1.8181
   13      16       119       103   31.6838        28     2.69186     1.80644
   14      16       127       111   31.7059        32     3.20146     1.82418
   15      16       133       117   31.1918        24    0.423103     1.84334
   16      16       141       125   31.2418        32    0.992599     1.85535
   17      16       148       132   31.0508        28     0.97594     1.84928
   18      16       155       139   30.8809        28     1.49825      1.8682
   19      16       161       145   30.5185        24     4.27347     1.90106
2023-04-18T15:19:17.492678+0800 min lat: 0.366624 max lat: 4.73499 avg lat: 1.92024
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      15       167       152   30.3887        28     4.73499     1.92024
   21      16       175       159   30.2746        28    0.367838     1.90697
   22      16       184       168   30.5345        36    0.745395     1.90833
   23      16       193       177   30.7718        36     1.09359     1.92217
   24      16       199       183   30.4895        24     4.54686     1.95203
   25      16       207       191   30.5496        32     4.18695     1.97411
   26      16       215       199   30.6052        32     1.11538     1.96496
   27      16       223       207   30.6566        32     1.41712     1.96474
   28      16       230       214   30.5616        28     2.08073     1.98234
   29      16       236       220   30.3352        24      2.4372     2.00951
   30      16       241       225   29.9906        20     3.20498       2.026
   31       9       241       232   29.9262        28     2.49127     2.02668
   32       3       241       238   29.7409        24      2.5638     2.05061
Total time run:       32.8447
Total reads made:     241
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   29.3502
Average IOPS:         7
Stddev IOPS:          1.10534
Max IOPS:             9
Min IOPS:             5
Average Latency(s):   2.06437
Max latency(s):       4.91188
Min latency(s):       0.366624

四节点

四节点状态截图

cephfs:

  • dd写入:22.0 MB/s (dd if=/dev/zero of=test.img bs=1M count=1024)
  • dd读取:19.7 MB/s (echo 3 > /proc/sys/vm/drop_caches; dd if=test.img of=/dev/null bs=1M)
  • fio随机写:1392
  • fio随机读:4087
  • fio顺序写:15.5MiB/s
  • fio顺序读:32.3MiB/s

rados bench -p perf_test 30 write --no-cleanup

hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_orangepi3-lts_35619
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        16         0         0         0           -           0
    2      16        17         1   1.99975         2     1.78202     1.78202
    3      16        20         4   5.33256        12     2.84877     2.43044
    4      16        21         5   4.99923         4     3.74263     2.69287
    5      16        23         7    5.5991         8     4.76583     2.85451
    6      16        32        16   10.6649        36     5.86804     3.70616
    7      16        34        18    10.284         8     1.92668      3.4596
    8      16        39        23    11.498        20     7.93078      3.6863
    9      16        42        26   11.5535        12     8.75803     3.93689
   10      16        47        31   12.3978        20     9.69823     3.95612
   11      16        50        34   12.3614        12     3.04304      3.9946
   12      16        54        38   12.6644        16     3.45564     3.99375
   13      16        60        44    13.536        24     3.44875     3.95588
   14      16        63        47   13.4261        12     1.81607     3.85018
   15      16        67        51   13.5975        16     2.73712     3.76863
   16      16        71        55   13.7475        16     4.15855     3.72367
   17      16        76        60    14.115        20     10.2432     3.75291
   18      16        79        63   13.9973        12     2.17743     3.81827
   19      16        83        67   14.1024        16     8.20002     3.90448
2023-04-30T16:14:03.140857+0800 min lat: 1.0234 max lat: 10.2441 avg lat: 3.90471
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16        84        68   13.5973         4     3.91999     3.90471
   21      16        87        71   13.5211        12     3.44596     3.99304
   22      16        93        77   13.9972        24     3.23659     3.97209
   23      16        96        80   13.9102        12     1.23555     4.00768
   24      16       102        86   14.3305        24     1.35805     4.04373
   25      16       103        87   13.9172         4     6.09656     4.06732
   26      16       107        91   13.9972        16      3.5407     4.08549
   27      16       109        93    13.775         8     6.03469     4.08524
   28      16       118       102   14.5686        36     2.17569     4.06772
   29      16       121       105   14.4799        12     4.49289     4.11014
   30      16       125       109   14.5305        16     2.87309     4.08652
   31       9       125       116   14.9648        28     2.25205      4.0526
   32       6       125       119   14.8721        12     1.79392     4.00744
Total time run:         32.5949
Total writes made:      125
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     15.3398
Stddev Bandwidth:       8.72312
Max bandwidth (MB/sec): 36
Min bandwidth (MB/sec): 0
Average IOPS:           3
Stddev IOPS:            2.20611
Max IOPS:               9
Min IOPS:               0
Average Latency(s):     4.0305
Stddev Latency(s):      2.20517
Max latency(s):         10.2441
Min latency(s):         1.0234

rados bench -p perf_test 30 seq

hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        20         4   15.9905        16    0.853527    0.650566
    2      16        32        16    31.987        48     1.92335      1.1849
    3      15        42        27   35.9872        44    0.728561     1.28407
    4      16        54        38   37.9873        44     1.45691     1.36427
    5      16        63        47   37.5885        36    0.867527     1.32744
    6      16        71        55   36.6558        32    0.388871     1.33514
    7      16        81        65   37.1323        40    0.402616     1.36986
    8      16        90        74     36.99        36      1.0629     1.41874
    9      16        99        83   36.8793        36     1.75055     1.48116
   10      16       106        90   35.9908        28    0.407852     1.52166
   11      16       117       101   36.7179        44     1.74839     1.55673
   12      13       125       112    37.324        44     1.12977     1.56548
   13       6       125       119   36.6064        28     2.87119      1.5828
   14       3       125       122   34.8488        12     3.12215     1.61803
   15       1       125       124   33.0589         8      3.1234     1.64145
Total time run:       15.0531
Total reads made:     125
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   33.2157
Average IOPS:         8
Stddev IOPS:          3.12745
Max IOPS:             12
Min IOPS:             2
Average Latency(s):   1.65394
Max latency(s):       4.13081
Min latency(s):       0.388871

rados bench -p perf_test 30 rand

hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        22         6   23.9897        24    0.745057    0.560803
    2      16        34        18   35.9886        48    0.372282    0.938734
    3      16        45        29   38.6558        44     2.19349     1.17867
    4      16        54        38   37.9898        36     1.37898     1.21186
    5      16        63        47   37.5905        36    0.368202     1.26999
    6      16        73        57   37.9907        40    0.369987     1.34372
    7      16        84        68   38.8479        44    0.379556     1.35436
    8      15        92        77   38.4844        36    0.366431     1.35522
    9      16        98        82   36.4306        20    0.369742     1.38116
   10      16       108        92   36.7866        40     1.07875     1.40909
   11      16       119       103   37.4414        44    0.411923     1.42015
   12      16       128       112   37.3207        36    0.721847     1.46517
   13      15       136       121   37.2149        36     3.88744     1.51544
   14      16       145       129   36.8421        32     1.48207     1.53257
   15      16       153       137    36.519        32    0.374107     1.56824
   16      16       162       146   36.4861        36    0.369296     1.56676
   17      16       174       158   37.1627        48     0.73181     1.55813
   18      16       179       163   36.2091        20    0.402144     1.57222
   19      16       190       174   36.6186        44     0.37141     1.58182
2023-04-30T16:16:14.252493+0800 min lat: 0.366431 max lat: 5.00059 avg lat: 1.59628
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       201       185   36.9871        44    0.707558     1.59628
   21      16       212       196   37.3204        44     0.72207     1.59639
   22      16       222       206   37.4418        40     4.26157     1.60271
   23      16       231       215   37.3786        36    0.863178     1.59685
   24      16       240       224   37.3208        36    0.703603     1.59669
   25      16       248       232   37.1078        32     3.21437       1.601
   26      16       260       244   37.5262        48    0.545021     1.58514
   27      16       269       253   37.4695        36     4.48831     1.59588
   28      16       277       261   37.2739        32    0.369147     1.59645
   29      16       287       271   37.3677        40    0.732066     1.60558
   30      16       295       279   37.1885        32    0.421207     1.61968
   31       8       295       287    37.021        32     4.43585     1.63341
   32       5       295       290   36.2391        12     4.24594     1.66179
   33       3       295       292   35.3834         8     3.91917     1.67774
Total time run:       33.7845
Total reads made:     295
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   34.9272
Average IOPS:         8
Stddev IOPS:          2.41248
Max IOPS:             12
Min IOPS:             2
Average Latency(s):   1.70153
Max latency(s):       5.00059
Min latency(s):       0.366431

卸载命令

apt remove ceph-* --purge -y
rm osd.img
rm -rf /var/lib/ceph/
rm -rf /usr/share/ceph
reboot

B站视频

我在家用了一块512M内存的ARMv7板子、一块1G内存的ARM64板子和一块2G内存的AMD64的HTPC和一根2G内存的Intel电脑棒搭建了一个Ceph集群。其中HTPC作为mon,mgr,mds,osd,而其它三个设备作为osd。他们都是百兆网络。我的目的是为了学习Ceph集群的搭建,此外也对Ceph集群的性能比较感兴趣。因此我用了一块ARM64的板子(有千兆网络)作为客户端,挂载cephfs进行性能测试。除了cephfs,我还用rados进行了性能测试。我测试了两节点、三节点和四节点的情况下的性能并进行了对比。我的Ceph集群是双副本的。总的来说顺序读写随着节点越多,性能越好,但随机读写因节点数量太少,暂时看不出趋势。

视频文字稿

低配设备搭建Ceph集群,性能如何?(分布式存储)

上次我用家里的电子垃圾搭了一个k8s集群,这次我来搭一个Ceph集群。

Ceph是一个开源的分布式存储系统。我这次用了一块ARMv7的板子,只有512M内存,还有一块1G内存ARM64板子(树莓派3B),另外用了一根Intel的电脑棒和一台HTPC,都是2G内存的。官方文档里面建议内存最少4G,不过我搭这个集群只是图一乐。

我本来还想用256M内存的那块板子的,但是内存太小,经过优化还是跑不起来,所以只能放弃。

我是根据官方文档来手动部署的,我一开始用的比较旧的版本,然后遇到一堆坑,后面换了新版就正常了,虽然新版也是各种bug,但是不影响我测试。

我使用双副本配置,所有的存储节点都是百兆内网。我另外使用一个千兆内网的ARM64板子作为客户端,对集群进行性能测试。我首先挂载了cephfs。也就是基于Ceph的分布式文件系统。挂载之后,在上面读写,这些读写请求会分发到所有的存储节点上,所以理论上存储节点越多,读写速度就越快。

我分别对两节点、三节点和四节点的情况进行测试。首先挂载cephfs并用fio进行性能测试,结果如图,随机读写性能比较差,而且和节点数量关系不大,原因可能是存储节点的闪存比较垃圾。不过连续读写都随着节点数量增加而增加,在两节点和三节点的时候,连续读取基本能跑满存储节点带宽。另外我还用rados进行了性能测试,结果如图,都是随节点数量增加而增加,不过不是线性增长。

如果大家喜欢这期视频,欢迎点赞投币收藏转发,有什么想说也可以在评论区留言。

Logo

欢迎加入我们的广州开发者社区,与优秀的开发者共同成长!

更多推荐