linux多路径Device-Mapper+Multipath

DM-Multipath概述DM-Multipath 能够使服务器与存储控制器间multiple I/O路径变成一个单一的设备。I/O路径是由线缆、交换机、控制器组成的物理SAN。DM-Multipath能够创建一个由I/O路径聚集组成的新设备。在不配置DM-Multipath的情况下，盘阵的一个LUN从控制器主机端口映射到服务器，在操作系统里被识别成一个独立的设备，这样就会造成同一个LUN通过盘

末点

3771人浏览 · 2022-06-10 13:52:02

末点 · 2022-06-10 13:52:02 发布

一 DM-Multipath概述

DM-Multipath 能够使服务器与存储控制器间multiple I/O路径变成一个单一的设备。I/O路径是由线缆、交换机、控制器组成的物理SAN。DM-Multipath能够创建一个由I/O路径聚集组成的新设备。

在不配置DM-Multipath的情况下，盘阵的一个LUN从控制器主机端口映射到服务器，在操作系统里被识别成一个独立的设备，这样就会造成同一个LUN通过盘阵不同的主机端口映射到服务器被识别成不同的设备。作为一种解决方案，DM-Multipath通过在物理设备上创建一个单独的多路径设备，提供了一种在逻辑上管理I/O路径的机制，这样盘阵的LUN从控制器主机端口映射到服务器，在操作系统里被识别成一个多路径设备。

每个多路径设备拥有一个唯一标识的World Wide Identifier（全球识别号，WWID），默认情况下，多路径设备的名称被设置成它的WWID。通过修改multipath.conf文件中的user_friendly_names选项参数，可以设置多路径设备的别名为mpathn。例如如下的配置环境：一个拥有两个HBA的服务器通过一个未配置zone的FC交换机连接到有两个主机端口的盘阵控制器（盘阵仅有一个LUN）上，在操作系统里能够看到四个设备：/dev/sda, /dev/sdb, dev/sdc, 和/dev/sdd。通过配置multipath.conf文件，DM-Multipath就会创建一个拥有WWID的多路径设备，多路径设备受控于DM-Multipath，我们可以在三个不同目录查看多路径设备：/dev/目录；/dev/mapper/mpathn,/dev/mpath/mpathn；/dev/dm-n.。使用/dev/mapper目录中的设备名对多路径设备进行管理，如创建逻辑卷，创建文件系统等。/dev/mpath将所有的多路径设备放到该目录下，只是便于查看，不要使用该目录下的设备进行创建逻辑卷，创建文件系统等操作。/dev/dm-n只是用于系统内部的使用，不要使用这里面的设备。

二 DM-Multipath功能

DM-Multipath能够提供：

冗余

在active/passive模式中，DM-Multipath提供失效转移功能。active/passive环境中，在任何时间内只有一半的I/O路径被使用。如果一个I/O路径内的元素例如线缆、交换机、控制器失效，DM-Multipath切换到轮换的路径。

提高性能

DM-Multipath能够被配置在active/ active模式下，在这种模式下，I/O路径处于round-robin方式。DM-Multipath能够动态的平衡I/O负荷。

三 DM-Multipath应用环境示例

Active/ Passive Multipath Configuration with One RAID Device

该配置里服务器有两个HBA，两个SAN交换机和两个控制器。从服务器到一个RAID存在两个I/O路径。如下图：

该配置中，第一条I/O路径通过hba1,SAN1和controller1 。第二条I/O路径通过hba2,SAN2和controller2。这个配置中存在多种潜在的失效因素：

HBA failure

FC cable failure

SAN switch failure

Array controller port failure

如果出现以上任何一个失效点，DM-Multipath会切换到轮换的路径。

② Active/Active Multipath Configuration with One RAID Device

该配置里服务器有两个HBA，一个SAN交换机和两个控制器，如下图：

该配置中，从服务器到一个存储存在四条I/O路径：

hba1 to controller1

hba1 to controller2

hba2 to controller1

hba2 to controller2

配置DM-Multipath后，DM-Multipath能够监测每条I/O路径的负载，动态的平衡I/O负荷。

四Setting Up DM-Multipath

在系统准备DM-Multipath之前，请确保系统包含device-mapper-multipath包，操作系统安装光盘内包含该rpm包。

①编辑/etc/multipath.conf文件，将以下内容的注释移除，使multipathing功能开启。

blacklist {

devnode "*"

}

修改完应如下

# blacklist {

# devnode "*"

# }

②保存配置

③modprobe dm-multipath

service multipathd start

multipath –v2 // multipath –v2命令输出多路径设备，如果没有任何输出，确保SAN正确连接。

④chkconfig multipathd on //确保multipath daemon 开机自动启动

⑤本地的硬盘设备不建议使用DM-Multipath，需要将本地硬盘从多路径设备中忽略。有两种方式，方式一：

vi /etc/multipath.conf

将blacklist {

devnode "*"

}

修改成

blacklist {

devnode "^sda" //扫描除本地硬盘的多路径设备

}

方式二：

/etc/multipath.conf文件中添加

blacklist{

wwid SIBM-ESXSST336732LC____F3ET0EP0Q000072428BX1(该值为本地硬盘的WWID)

}

然后再运行

multipath –F

multipath –v2

[root@rh4cluster1 ~]# multipath –F //移除多路径设备

[root@rh4cluster1 ~]# multipath -v2

create: 3600a0b80001327d80000006d43621677

[size=12 GB][features="0"][hwhandler="0"]

\_ round-robin 0

\_ 2:0:0:0 sdb 8:16

\_ 3:0:0:0 sdf 8:80

create: 3600a0b80001327510000009a436215ec

[size=12 GB][features="0"][hwhandler="0"]

\_ round-robin 0

\_ 2:0:0:1 sdc 8:32

\_ 3:0:0:1 sdg 8:96

create: 3600a0b80001327d800000070436216b3

[size=12 GB][features="0"][hwhandler="0"]

\_ round-robin 0

\_ 2:0:0:2 sdd 8:48

\_ 3:0:0:2 sdh 8:112

create: 3600a0b80001327510000009b4362163e

[size=12 GB][features="0"][hwhandler="0"]

\_ round-robin 0

\_ 2:0:0:3 sde 8:64

\_ 3:0:0:3 sdi 8:128

⑥大多数的磁盘阵列型号支持DM-Multipath，如果要添加的设备不是/etc/multipath.conf文件里默认支持的设备，可以参照默认配置添加设备情况，例如添加一个如下的设备：

devices {

device {

vendor "HP"

product "OPEN-V."

getuid_callout "/sbin/scsi_id -g -u -p0x80 -s /block/%n"

}

附：北大某学院项目实际案例：

项目实施过程中由于要实现光纤链路的多路径，由于DS6310FE没有提供LINUX下的多路径冗余软件，所以采用了LINUX系统下的device mapper multipath解决方案。

SAN组织结构如下：

DS6310FE：双控制器，每个控制器2个4G FC主机端口，全部连接到FC交换机。4组RAID,其中一个LUN同时映射给三个主机，其它三个LUN分别映射给各自的主机。

Brocade 200E FC Switch

Server：每个配置一个Qlogic QLE2460 HBA卡，连接交换机

这样LUN到主机的光纤链路存在多个。

以下为配置步骤

①# rpm -qa | grep device-mapper

device-mapper-1.02.13-1.el5

device-mapper-multipath-0.4.7-8.el5

如果显示如上说明系统已经安装了device-mapper-multipath，如果未安装，在安装光盘可以找到该rpm包进行安装

# rpm -ivh device-mapper-multipath-0.4.7-8.el5.i386.rpm

Preparing... ##################### [100%]

1:device-mapper-multipath##################### [100%]

②配置/etc/multipath.conf文件

进行备份，cp /etc/multipath.conf /etc/multipath.conf.bak

vi /etc/multipath.conf

将devnode_blacklist {

devnode "*"

}

修改成

devnode_blacklist {

devnode "^sda" //扫描除本地硬盘的多路径设备

}

③执行# multipath -v3

load path identifiers cache

# all paths in cache :

222770001551df74d 6:0:0:0 sdb 8:16 1 [active] Dawning /Diskarray DS6310/0330

222f5000155f4aab4 6:0:0:1 sdc 8:32 1 [active] Dawning /Diskarray DS6310/0330

222770001551df74d 6:0:1:0 sdd 8:48 1 [active] Dawning /Diskarray DS6310/0330

222f5000155f4aab4 6:0:1:1 sde 8:64 1 [active] Dawning /Diskarray DS6310/0330

222770001551df74d 6:0:2:0 sdf 8:80 1 [active] Dawning /Diskarray DS6310/0330

222f5000155f4aab4 6:0:2:1 sdg 8:96 1 [active] Dawning /Diskarray DS6310/0330

222770001551df74d 6:0:3:0 sdh 8:112 1 [active] Dawning /Diskarray DS6310/0330

222f5000155f4aab4 6:0:3:1 sdi 8:128 1 [active] Dawning /Diskarray DS6310/0330

sda blacklisted

===== path info sdb (mask 0x1f) =====

bus = 1

dev_t = 8:16

size = 5847653376

vendor = Dawning

product = Diskarray DS6310

rev = 0330

h:b:t:l = 6:0:0:0

tgt_node_name =

serial = 495345200000000000000000972AA3D84BF39EF4

path checker = readsector0 (internal default)

state = 2

getprio = (null) (internal default)

prio = 1

uid = 222770001551df74d (cache)

===== path info sdc (mask 0x1f) =====

bus = 1

dev_t = 8:32

size = 1949217792

vendor = Dawning

product = Diskarray DS6310

rev = 0330

h:b:t:l = 6:0:0:1

tgt_node_name =

serial = 4953452000000000000000006B1ED98B16A2FB26

path checker = readsector0 (internal default)

state = 2

getprio = (null) (internal default)

prio = 1

uid = 222f5000155f4aab4 (cache)

===== path info sdd (mask 0x1f) =====

bus = 1

dev_t = 8:48

size = 5847653376

vendor = Dawning

product = Diskarray DS6310

rev = 0330

h:b:t:l = 6:0:1:0

tgt_node_name =

serial = 495345200000000000000000972AA3D84BF39EF4

path checker = readsector0 (internal default)

state = 2

getprio = (null) (internal default)

prio = 1

uid = 222770001551df74d (cache)

===== path info sde (mask 0x1f) =====

bus = 1

dev_t = 8:64

size = 1949217792

vendor = Dawning

product = Diskarray DS6310

rev = 0330

h:b:t:l = 6:0:1:1

tgt_node_name =

serial = 4953452000000000000000006B1ED98B16A2FB26

path checker = readsector0 (internal default)

state = 2

getprio = (null) (internal default)

prio = 1

uid = 222f5000155f4aab4 (cache)

===== path info sdf (mask 0x1f) =====

bus = 1

dev_t = 8:80

size = 5847653376

vendor = Dawning

product = Diskarray DS6310

rev = 0330

h:b:t:l = 6:0:2:0

tgt_node_name =

serial = 495345200000000000000000972AA3D84BF39EF4

path checker = readsector0 (internal default)

state = 2

getprio = (null) (internal default)

prio = 1

uid = 222770001551df74d (cache)

===== path info sdg (mask 0x1f) =====

bus = 1

dev_t = 8:96

size = 1949217792

vendor = Dawning

product = Diskarray DS6310

rev = 0330

h:b:t:l = 6:0:2:1

tgt_node_name =

serial = 4953452000000000000000006B1ED98B16A2FB26

path checker = readsector0 (internal default)

state = 2

getprio = (null) (internal default)

prio = 1

uid = 222f5000155f4aab4 (cache)

===== path info sdh (mask 0x1f) =====

bus = 1

dev_t = 8:112

size = 5847653376

vendor = Dawning

product = Diskarray DS6310

rev = 0330

h:b:t:l = 6:0:3:0

tgt_node_name =

serial = 495345200000000000000000972AA3D84BF39EF4

path checker = readsector0 (internal default)

state = 2

getprio = (null) (internal default)

prio = 1

uid = 222770001551df74d (cache)

===== path info sdi (mask 0x1f) =====

bus = 1

dev_t = 8:128

size = 1949217792

vendor = Dawning

product = Diskarray DS6310

rev = 0330

h:b:t:l = 6:0:3:1

tgt_node_name =

serial = 4953452000000000000000006B1ED98B16A2FB26

path checker = readsector0 (internal default)

state = 2

getprio = (null) (internal default)

prio = 1

uid = 222f5000155f4aab4 (cache)

# all paths :

222770001551df74d 6:0:0:0 sdb 8:16 1 [active][ready] Dawning /Diskarray DS631

222f5000155f4aab4 6:0:0:1 sdc 8:32 1 [active][ready] Dawning /Diskarray DS631

222770001551df74d 6:0:1:0 sdd 8:48 1 [active][ready] Dawning /Diskarray DS631

222f5000155f4aab4 6:0:1:1 sde 8:64 1 [active][ready] Dawning /Diskarray DS631

222770001551df74d 6:0:2:0 sdf 8:80 1 [active][ready] Dawning /Diskarray DS631

222f5000155f4aab4 6:0:2:1 sdg 8:96 1 [active][ready] Dawning /Diskarray DS631

222770001551df74d 6:0:3:0 sdh 8:112 1 [active][ready] Dawning /Diskarray DS63

222f5000155f4aab4 6:0:3:1 sdi 8:128 1 [active][ready] Dawning /Diskarray DS63

params = 0 0 4 1 round-robin 0 1 1 8:32 1000 round-robin 0 1 1 8:64 1000 round-robin 0 1 1 8:96 1000 round-robin 0 1 1 8:128 1000

status = 1 0 0 4 1 A 0 1 0 8:32 A 0 E 0 1 0 8:64 A 0 E 0 1 0 8:96 A 0 E 0 1 0 8:128 A 0

params = 0 0 4 1 round-robin 0 1 1 8:16 1000 round-robin 0 1 1 8:48 1000 round-robin 0 1 1 8:80 1000 round-robin 0 1 1 8:112 1000

status = 1 0 0 4 1 A 0 1 0 8:16 A 0 E 0 1 0 8:48 A 0 E 0 1 0 8:80 A 0 E 0 1 0 8:112 A 0

Found matching wwid [222770001551df74d] in bindings file.

Setting alias to mpath0

pgpolicy = failover (internal default)

selector = round-robin 0 (internal default)

features = 0 (internal default)

hwhandler = 0 (internal default)

rr_weight = 1 (internal default)

rr_min_io = 1000 (config file default)

no_path_retry = NONE (internal default)

pg_timeout = NONE (internal default)

0 5847653376 multipath 0 0 4 1 round-robin 0 1 1 8:16 1000 round-robin 0 1 1 8:48 1000 round-robin 0 1 1 8:80 1000 round-robin 0 1 1 8:112 1000

set ACT_NOTHING: map unchanged

Found matching wwid [222f5000155f4aab4] in bindings file.

Setting alias to mpath1

pgpolicy = failover (internal default)

selector = round-robin 0 (internal default)

features = 0 (internal default)

hwhandler = 0 (internal default)

rr_weight = 1 (internal default)

rr_min_io = 1000 (config file default)

no_path_retry = NONE (internal default)

pg_timeout = NONE (internal default)

0 1949217792 multipath 0 0 4 1 round-robin 0 1 1 8:32 1000 round-robin 0 1 1 8:64 1000 round-robin 0 1 1 8:96 1000 round-robin 0 1 1 8:128 1000

set ACT_NOTHING: map unchanged

显示出所有的多路径设备，从sdb到sdi共8个分区，以上的信息尤其是红色标明的是需要添加到/etc/multipath.conf的重要信息。

④执行# multipath -ll

mpath1 (222f5000155f4aab4)

[size=929 GB][features="0"][hwhandler="0"]

\_ round-robin 0 [prio=1][active]

\_ 6:0:0:1 sdc 8:32 [active][ready]

\_ round-robin 0 [prio=1][enabled]

\_ 6:0:1:1 sde 8:64 [active][ready]

\_ round-robin 0 [prio=1][enabled]

\_ 6:0:2:1 sdg 8:96 [active][ready]

\_ round-robin 0 [prio=1][enabled]

\_ 6:0:3:1 sdi 8:128 [active][ready]

mpath0 (222770001551df74d)

[size=2788 GB][features="0"][hwhandler="0"]

\_ round-robin 0 [prio=1][active]

\_ 6:0:0:0 sdb 8:16 [active][ready]

\_ round-robin 0 [prio=1][enabled]

\_ 6:0:1:0 sdd 8:48 [active][ready]

\_ round-robin 0 [prio=1][enabled]

\_ 6:0:2:0 sdf 8:80 [active][ready]

\_ round-robin 0 [prio=1][enabled]

\_ 6:0:3:0 sdh 8:112 [active][ready]

显示未作多路径冗余时的状态。

⑤加载dm_multipath模块 modprobe dm_multipath

打开守护进程/etc/init.d/multipathd start chkconfig multipathd on

⑥配置/etc/multipath.conf文件

# This is an example configuration file for device mapper multipath.

# For a complete list of the default configuration values, see

# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.conf.defaults

# For a list of configuration options with descriptions, see

# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.conf.annotated

# Blacklist all devices by default. Remove this to enable multipathing

# on the default devices.

devnode_blacklist {

devnode "^sda"

}

## Use user friendly names, instead of using WWIDs as names.

defaults {

user_friendly_names yes

}

## By default, devices with vendor = "IBM" and product = "S/390.*" are

## blacklisted. To enable mulitpathing on these devies, uncomment the

## following lines.

#devices {

# device {

# vendor "IBM"

# product "S/390 DASD ECKD"

# path_grouping_policy multibus

# getuid_callout "/sbin/dasdview -j -f /dev/%n"

# path_checker directio

# }

## This is a template multipath-tools configuration file

## Uncomment the lines relevent to your environment

#defaults {

# udev_dir /dev

# polling_interval 10

# selector "round-robin 0"

# path_grouping_policy multibus

# getuid_callout "/sbin/scsi_id -g -u -s /block/%n"

# prio_callout /bin/true

# path_checker readsector0

# rr_min_io 100

# rr_weight priorities

# failback immediate

# no_path_retry fail

# user_friendly_name yes

## The wwid line in the following blacklist section is shown as an example

## of how to blacklist devices by wwid. The 3 devnode lines are the

## compiled in default blacklist. If you want to blacklist entire types

## of devices, such as all scsi devices, you should use a devnode line.

## However, if you want to blacklist specific devices, you should use

## a wwid line. Since there is no guarantee that a specific device will

## not change names on reboot (from /dev/sda to /dev/sdb for example)

## devnode lines are not recommended for blacklisting specific devices.

#devnode_blacklist {

# wwid 26353900f02796769

# devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"

# devnode "^hd[a-z]"

multipaths {

# multipath {

# wwid 3600508b4000156d700012000000b0000

# alias yellow

# path_grouping_policy multibus

# path_checker readsector0

# path_selector "round-robin 0"

# failback manual

# rr_weight priorities

# no_path_retry 5

# }

# multipath {

# wwid 1DEC_____321816758474

# alias red

# }

multipath {

wwid 222770001551df74d

alias data

path_grouping_policy multibus

path_checker ture

path_selector "round-robin 0"

}

multipath {

wwid 222f5000155f4aab4

alias scratch

path_grouping_policy multibus

path_checker ture

path_selector "round-robin 0"

}

devices {

# device {

# vendor "COMPAQ "

# product "HSV110 (C)COMPAQ"

# path_grouping_policy multibus

# getuid_callout "/sbin/scsi_id -g -u -s /block/%n"

# path_checker readsector0

# path_selector "round-robin 0"

# hardware_handler "0"

# failback 15

# rr_weight priorities

# no_path_retry queue

# }

# device {

# vendor "COMPAQ "

# product "MSA1000 "

# path_grouping_policy multibus

# }

device {

vendor "Dawning "

product "Diskarray DS6310"

#path_grouping_policy multibus

path_grouping_policy group_by_serial

getuid_callout "/sbin/scsi_id -g -u -s /block/%n"

path_checker readsector0

path_selector "round-robin 0"

hardware_handler "0"

failback immediate

rr_weight uniform

no_path_retry 12

}

}红色为改动部分

⑦执行multipath -v3