drbd9
利用k8s制作独立的数据库,每个数据库对应一个pvc,这样数据就不会丢失,每一个数据库都是独立svc;但是怎么应用呢?本来想着把前端加一个mycat,或atlas作一个中间件,预想是能把数据库分开,一些写库1,一些写库2,但是没有那么简单;数据分开写就是分片,分片是对一个表进行分片;这样就比较麻烦,需要把所有的表都先预设好然后,每一个库再新建这个表,再启动服务才能使用;...
安装drbd9
- 注意,需要更新内核,/boot的大小要200M以上,反正我更新的时候提示空间不足
- 内核更新后,有遇到drm:drm_atomic_helper wait for dependencies / flip_done timed out,原因应该是内核跟CPU的型号有关系,网上也有类似情况
- 可恢复系统或降内核处理
- 解决办法, 新建一个500M的盘,分区格式化,把/boot的内容全部复制到新的分区里(需要挂载到一个目录),
- 然后umount /boot, 把新的分区挂载到/boot上,然后才更新成功
[root@drab ~]# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
# yum install drbd90 kmod-drbd90 -y #需要EPEL yum源;
- 设置裸设备,需要分区,整个磁盘 或 LVM磁盘,磁盘陈列,LVM逻辑盘等块储存设备
两台机器都增加一个10G的分区,可以分区,不能格式化,能使用的空间是没有格式化的空间
[root@primary ~]# fdisk -l |grep -A 10 "/dev/sdb" #两台机器都新增一个磁盘;
磁盘 /dev/sdb:10.7 GB, 10737418240 字节,20971520 个扇区
Units = 扇区 of 1 * 512 = 512 bytes
扇区大小(逻辑/物理):512 字节 / 512 字节
I/O 大小(最小/最佳):512 字节 / 512 字节
磁盘标签类型:dos
磁盘标识符:0xd795a1c3
设备 Boot Start End Blocks Id System
/dev/sdb1 2048 20971519 10484736 83 Linux
- 制作配置文件 - 两台机器配置文件是一样的;
路径:/etc/drbd.conf; /etc/drbd.d/global_common.conf
[root@primary drbd.d]# cat ../drbd.conf
# You can find an example in /usr/share/doc/drbd.../drbd.conf.example
include "drbd.d/global_common.conf";
include "drbd.d/*.res";
[root@primary drbd.d]# cat global_common.conf
global {
usage-count yes;
# minor-count dialog-refresh disable-ip-verification
}
common {
protocol C;
handlers {
# These are EXAMPLE handlers only.
# They may have severe implications,
# like hard resetting the node under certain circumstances.
# Be careful when chosing your poison.
# pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
# pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
# local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup {
wfc-timeout 240;
degr-wfc-timeout 240;
outdated-wfc-timeout 240;
}
disk {
on-io-error detach;
}
net {
cram-hmac-alg md5;
shared-secret "testdrbd";
}
syncer {
# rate 30M;
}
}
[root@primary drbd.d]# cat r0.res
resource r0 {
on primary {
device /dev/drbd1;
disk /dev/sdb1;
address 192.168.3.51:7789;
meta-disk internal;
}
on secondary {
device /dev/drbd1;
disk /dev/sdb1;
address 192.168.3.52:7789;
meta-disk internal;
}
}
- 启动方法,两台机器都要进行的操作
1 create
2 启动服务
[root@tanyvlinux drbd.d]# drbdadm create-md r0
md_offset 10736365568
al_offset 10736332800
bm_offset 10736005120
Found some data
==> This might destroy existing data! <==
Do you want to proceed?
[need to type 'yes' to confirm] yes
initializing activity log
initializing bitmap (320 KB) to all zero
Writing meta data...
New drbd meta data block successfully created.
success
[root@tanyvlinux drbd.d]# systemctl start drbd
- 其他信息
- 第二台启动服务时,会尝试连接,如果没有获取到信息,第一台机器在上可以drbdadm up r0试试; 启动里如果是使用ssh登陆,有一些信息在ssh客户端是看不到的,可能会显示为等待状态,那些信息是系统直接返回的,虚拟机上才能看到; 连接上后,按回车,就正常了;
- 这里还没有正常工作,需要设置一个primary
drbdadm primary --force r0 #设置好后,secondary会开始同步数据,后续再说明查看状态的命令;
-
primary会产生一个新的分区/dev/drbd1
这个分区就是我们需要使用的分区,把drbd1格式化后使用,不要对/dev/sdb进行操作,那样操作都会引起问题,实验过程多做快照,错了可以再来,要不然效率就会很低;
格式化挂载保存数据后,等待secondary同步好数据后up to date; 可以尝试错误恢复; -
错误恢复1;
-
正常关闭primary, secondary 提升为primary, 就可以挂载/dev/drbd1使用;如果操作没问题,就可以正常看到之前primary更新的数据;
-
大概操作
primary:
umount 挂载点 #有挂载点是不能将主变成从机;但是即使用umount -l 也会出现提示,始终有进程挂载着/dev/drbd1;不能变成从机;这个情况是比较麻烦;经过很多尝试,终于找到不算太好的办法,就是drbdadm disconnect r0; systemctl stop drbd; 然后再重启吧;重启后变成secondary;
drbdadm secondary r0 #如果/dev/drbd1在格式化之前,尝试主从切换,应该就是主降为从,从升为主就可以;
secondary:
drbdadm primary --force r0
-
错误恢复2
如果主机挂机后,再启动回来,secondary实际上是不需要动的;前提是像上面一样正常关闭服务,然后再等机器启动,再把服务启动,然后再设定主机为primary;(如果做了systemctl enable drbd, 再把挂载信息写入/etc/fstab这样启动后,是可以重新连接上) -
错误恢复3
在虚拟机上,直接把primary重启,试了两次都进入了emergency mode,没有研究怎么恢复;
所以如果在使用drbd的情况下,不正常关机,机器多半是用不了,需要用快照恢复;
这个情况只能把secondary启动起来,跟第一个例子一样,升为primary, 挂载,即可使用; -
转移主次
由于数据是完全一致的,在网络没有问题的情况下,是可以任意切换主次;(前提两边都是up to date的状态;连接好以后,数据同步以后(感觉主机的格式化信息也会同步到从机所以要等一段时间),如果网络没有断,其实一直都是保持up to date的状态,因为是C协议,数据同时是写进两台机器;同步媒介是网络,所以会比在磁盘里写慢很多,后面有一些简单的数据;)
跟错误恢复1是一样的;
原来主的启动,会不会产生脑裂呢?是不会的,每次关闭后,再把服务启动,原来的主也会变成secondary;(为保安全,切换的时候,在关闭服务之前,把主变成从,再把从提升)
- 从机同步数据相关
同步时关闭主,同步会重新开始
如果从机原来在同步数据,把主正常关闭,再打开,主的角色是secondary,但不影响那个在同步数据的从机同步数据,机器一上线其马上就开始同步数据,应该是drbd还是知道哪边的数据更新一些;
drbd状态查询
[root@primary ~]# drbdadm status r0
r0 role:Primary #本机状态;
disk:UpToDate #最新状态;
secondary role:Secondary #主机名为secondary的角色为secondary;
replication:SyncSource peer-disk:Inconsistent done:2.34 #同步的比例2.34%;
[root@secondary ~]# drbdadm status r0
r0 role:Secondary
disk:Inconsistent
primary role:Primary
replication:SyncTarget peer-disk:UpToDate done:6.38
[root@primary drbdtop-0.2.2]# drbdsetup status r0 #同步完成后的状态;
r0 role:Primary
disk:UpToDate
secondary role:Secondary
peer-disk:UpToDate
- drbdmon 可以看到同步进展
- 主机
- 从机
完全同步后的状态
其他查看状态的命令;
[root@primary drbdtop-0.2.2]# drbdsetup status r0 --verbose --statistics
r0 node-id:0 role:Primary suspended:no
write-ordering:flush
volume:0 minor:1 disk:UpToDate quorum:yes
size:10484380 read:5652221 written:10590 al-writes:8 bm-writes:0 upper-pending:0 lower-pending:0 al-suspended:no blocked:no
secondary node-id:1 connection:Connected role:Secondary congested:no ap-in-flight:0 rs-in-flight:288
volume:0 replication:SyncSource peer-disk:Inconsistent done:43.44 resync-suspended:no #同步了43.44;
received:0 sent:4554192 out-of-sync:5930332 pending:2 unacked:0 dbdt1:1.89 eta:3057
[root@primary drbdtop-0.2.2]# drbdsetup events2 --now r0 #一个时刻的状态;
exists resource name:r0 role:Primary suspended:no
exists connection name:r0 peer-node-id:1 conn-name:secondary connection:Connected role:Secondary
exists device name:r0 volume:0 minor:1 disk:UpToDate client:no quorum:yes
exists peer-device name:r0 peer-node-id:1 conn-name:secondary volume:0 replication:SyncSource peer-disk:Inconsistent peer-client:no resync-suspended:no done:49.33
exists -
[root@primary drbdtop-0.2.2]# drbdsetup events2 r0 #动态更新;更新完成,也是静态的;
exists resource name:r0 role:Primary suspended:no
exists connection name:r0 peer-node-id:1 conn-name:secondary connection:Connected role:Secondary
exists device name:r0 volume:0 minor:1 disk:UpToDate client:no quorum:yes
exists peer-device name:r0 peer-node-id:1 conn-name:secondary volume:0 replication:SyncSource peer-disk:Inconsistent peer-client:no resync-suspended:no done:54.52
exists -
change peer-device name:r0 peer-node-id:1 conn-name:secondary volume:0 done:54.53
change peer-device name:r0 peer-node-id:1 conn-name:secondary volume:0 done:54.59
change peer-device name:r0 peer-node-id:1 conn-name:secondary volume:0 done:54.65
^C
[root@primary drbdtop-0.2.2]# drbdsetup events2 --statistics --now r0 #增加信息显示;
exists resource name:r0 role:Primary suspended:no write-ordering:flush
exists connection name:r0 peer-node-id:1 conn-name:secondary connection:Connected role:Secondary congested:no ap-in-flight:0 rs-in-flight:2256
exists device name:r0 volume:0 minor:1 disk:UpToDate client:no quorum:yes size:10484380 read:6925717 written:10590 al-writes:8 bm-writes:0 upper-pending:0 lower-pending:0 al-suspended:no blocked:no
exists peer-device name:r0 peer-node-id:1 conn-name:secondary volume:0 replication:SyncSource peer-disk:Inconsistent peer-client:no resync-suspended:no done:55.57 received:0 sent:5827688 out-of-sync:4657820 pending:20 unacked:0 dbdt1:2.28 eta:1991
exists -
[root@primary drbdtop-0.2.2]# drbdadm cstate r0 #查看是否连接正常
Connected
[root@primary drbdtop-0.2.2]# drbdadm role r0
Primary
[root@primary drbdtop-0.2.2]# drbdadm dstate r0 # 磁盘状态;
UpToDate/Inconsistent #一台机器还在同步信息;
[root@secondary ~]# drbdadm dstate r0 #同步完成的状态;
UpToDate/UpToDate
drbdadm up r0 #enable resource;
drbdadm down r0 #disable resource;
drbdadm primary r0 #Promoting and demoting resources;
drbdadm secondary r0
速度测试
- 测试 - 网络远离路由器
网络很差时,rsync比drbd快,这是不是跟数据较验这方面有关系?
[root@primary src]# rsync -avP mariadb-10.4.7-linux-systemd-x86_64.tar.gz /mnt/ #/mnt挂载了/dev/drbd1;
sending incremental file list
mariadb-10.4.7-linux-systemd-x86_64.tar.gz
1,068,409,785 100% 2.76MB/s 0:06:09 (xfr#1, to-chk=0/1)
sent 1,068,670,744 bytes received 35 bytes 2,884,401.56 bytes/sec
total size is 1,068,409,785 speedup is 1.00
[root@primary src]# rsync -avP mariadb-10.4.7-linux-systemd-x86_64.tar.gz 192.168.3.52:/usr/local/src/ #同步同一个文件到同一台机器;
The authenticity of host '192.168.3.52 (192.168.3.52)' can't be established.
ECDSA key fingerprint is SHA256:iXyDbBSYHuTlZHpO4n1Of+aAKOBgU0hrsQmWBzm8avc.
ECDSA key fingerprint is MD5:58:80:5c:0f:27:27:14:b5:23:a7:fe:b8:a1:e9:95:6d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.3.52' (ECDSA) to the list of known hosts.
root@192.168.3.52's password:
sending incremental file list
mariadb-10.4.7-linux-systemd-x86_64.tar.gz
1,068,409,785 100% 4.37MB/s 0:03:53 (xfr#1, to-chk=0/1)
sent 1,068,670,744 bytes received 35 bytes 4,353,037.80 bytes/sec
total size is 1,068,409,785 speedup is 1.00
[root@primary src]# rsync -avP mariadb-10.4.7-linux-systemd-x86_64.tar.gz /root/ #同步到本地,非drbd;
sending incremental file list
mariadb-10.4.7-linux-systemd-x86_64.tar.gz
1,068,409,785 100% 98.07MB/s 0:00:10 (xfr#1, to-chk=0/1)
sent 1,068,670,744 bytes received 35 bytes 101,778,169.43 bytes/sec
total size is 1,068,409,785 speedup is 1.00
- 测试 (网络离路由器比较近)
网络比较差的时候,速度差不多,rsync比 drdb快一点;
[root@primary src]# rsync -avP mariadb-10.4.7-linux-systemd-x86_64.tar.gz /mnt/
sending incremental file list
mariadb-10.4.7-linux-systemd-x86_64.tar.gz
1,068,409,785 100% 4.41MB/s 0:03:51 (xfr#1, to-chk=0/1)
sent 1,068,670,744 bytes received 35 bytes 4,616,288.46 bytes/sec
total size is 1,068,409,785 speedup is 1.00
[root@primary src]# rsync -avP mariadb-10.4.7-linux-systemd-x86_64.tar.gz 192.168.3.52:/usr/local/src/
root@192.168.3.52's password:
sending incremental file list
mariadb-10.4.7-linux-systemd-x86_64.tar.gz
1,068,409,785 100% 4.65MB/s 0:03:38 (xfr#1, to-chk=0/1)
sent 1,068,670,744 bytes received 35 bytes 4,697,453.97 bytes/sec
total size is 1,068,409,785 speedup is 1.00
[root@primary src]# ls /root/mariadb-10.4.7-linux-systemd-x86_64.tar.gz
/root/mariadb-10.4.7-linux-systemd-x86_64.tar.gz
[root@primary src]# rm -rf !$
rm -rf /root/mariadb-10.4.7-linux-systemd-x86_64.tar.gz
[root@primary src]# rsync -avP mariadb-10.4.7-linux-systemd-x86_64.tar.gz /root/
sending incremental file list
mariadb-10.4.7-linux-systemd-x86_64.tar.gz
1,068,409,785 100% 136.85MB/s 0:00:07 (xfr#1, to-chk=0/1)
sent 1,068,670,744 bytes received 35 bytes 142,489,437.20 bytes/sec
total size is 1,068,409,785 speedup is 1.00
- 测试 (一台机器由2Gwifi转成网线,另一台应该是5G的wifi)
网速提高了以后,drbd速度比rsync快;
[root@primary src]# rsync -avP mariadb-10.4.7-linux-systemd-x86_64.tar.gz /mnt/
sending incremental file list
mariadb-10.4.7-linux-systemd-x86_64.tar.gz
1,068,409,785 100% 12.42MB/s 0:01:22 (xfr#1, to-chk=0/1)
sent 1,068,670,744 bytes received 35 bytes 12,953,585.20 bytes/sec
total size is 1,068,409,785 speedup is 1.00
[root@primary src]# rsync -avP mariadb-10.4.7-linux-systemd-x86_64.tar.gz 192.168.3.52:/usr/local/src/
root@192.168.3.52's password:
sending incremental file list
mariadb-10.4.7-linux-systemd-x86_64.tar.gz
1,068,409,785 100% 11.08MB/s 0:01:31 (xfr#1, to-chk=0/1)
sent 1,068,670,744 bytes received 35 bytes 11,074,308.59 bytes/sec
total size is 1,068,409,785 speedup is 1.00
更多推荐
所有评论(0)