The SCSI passthrough over multipath is very commonly used in virtualization environments. The SCSI IO commands are forwarded from the virtual machine to the KVM hypervisor side storage via SG_IO. The KVM hypervisor side storage is always provied in a multipath configuration in order to facilitate high availability. Therefore, the developers may require a solution that allows them to quickly and efficiently setup an environment to diagnose and observe the behavior of virtio-scsi + QEMU + SG_IO + multipath + SCSI, e.g., how the path error is propagated to the virtual machine.
In this article, we demonstrate how to setup a software-only KVM environment on a regular Oracle Cloud Infrastructure (OCI) virtual machine, without any special SCSI/storage hardware requirements.
Note: The OCI Instance must be an Oracle Linux 7 Intel architecture instance.
Software Installation
Once you’ve created your OL7 based OCI Intel instance, SSH into the instance to perform software installation. Only targetcli
and QEMU
are required for installation, since the device-mapper-multipath
is already available.
[root@vm opc]# yum install targetcli [root@vm opc]# yum install qemu
Create Two Paths
We use the targetcli
to setup two loopback devices referring to the same image file: /home/opc/storage.raw
:
[root@vm opc]# dd if=/dev/zero of=storage.raw bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 0.616011 s, 1.7 GB/s [root@vm opc]# targetcli /backstores/fileio create name=storage file_or_dev=/home/opc/storage.raw Created fileio storage with size 1073741824
The 1st path is via naa.5001405f01234ba3
, and the 2nd path is via naa.500230567123438b
.
[root@vm opc]# targetcli /loopback create naa.5001405f01234ba3 Created target naa.5001405f01234ba3. [root@vm opc]# targetcli /loopback/naa.5001405f01234ba3/luns create /backstores/fileio/storage Created LUN 0. [root@vm opc]# targetcli /loopback create naa.500230567123438b Created target naa.500230567123438b. [root@vm opc]# targetcli /loopback/naa.500230567123438b/luns create /backstores/fileio/storage Created LUN 0.
There will be two loopback SCSI devices (and disks) available.
[root@vm opc]# lsscsi [2:0:0:1] disk ORACLE BlockVolume 1.0 /dev/sda [3:0:1:0] disk LIO-ORG storage 4.0 /dev/sdb [4:0:1:0] disk LIO-ORG storage 4.0 /dev/sdc
Both loopback SCSI devices are based on the same backstore file /home/opc/storage.raw
.
[root@vm opc]# targetcli ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 0] | o- fileio ................................................................................................. [Storage Objects: 1] | | o- storage ............................................................. [/home/opc/storage.raw (1.0GiB) write-back activated] | | o- alua ................................................................................................... [ALUA Groups: 1] | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 0] o- loopback ......................................................................................................... [Targets: 2] | o- naa.5001405f01234ba3 ................................................................................. [naa.50014055b3b6d883] | | o- luns ............................................................................................................ [LUNs: 1] | | o- lun0 ........................................................ [fileio/storage (/home/opc/storage.raw) (default_tg_pt_gp)] | o- naa.500230567123438b ................................................................................. [naa.5001405f51ac04e4] | o- luns ............................................................................................................ [LUNs: 1] | o- lun0 ........................................................ [fileio/storage (/home/opc/storage.raw) (default_tg_pt_gp)] o- vhost ............................................................................................................ [Targets: 0]
Setup multipath
Create or modify the configuration file /etc/multipath.conf
, to rule out sda
which is already used as boot image of the virtual machine:
[root@vm opc]# cat /etc/multipath.conf defaults { user_friendly_names yes no_path_retry 10 } blacklist { devnode "sda" }
To ensure this configuration is not persistent across reboot, ensure to only start the systemd service, do not enable it. Once the multipathd service is started, only sdb
will be active:
[root@vm opc]# systemctl start multipathd [root@vm opc]# multipath -ll mpathc (360014052d347b7c8b8041e9bc817234e) dm-0 LIO-ORG ,storage size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 3:0:1:0 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 4:0:1:0 sdc 8:32 active ready running
Virtual Machine Creation
Create a virtual machine with scsi-block
, using /dev/mapper/mpathc
. The file ol7.qcow2
is an OL7-based boot image:
$ qemu-system-x86_64 -smp 4 -m 8G -enable-kvm -cpu host -vnc :7 \ -net nic -net user,hostfwd=tcp::5027-:22 -hda ol7.qcow2 -serial stdio \ -device virtio-scsi-pci,id=virtio-scsi01,bus=pci.0,addr=0x4 \ -drive file=/dev/mapper/mpathc,format=raw,if=none,id=drvpath,cache=none,aio=native \ -device scsi-block,bus=virtio-scsi01.0,channel=0,scsi-id=0,lun=0,drive=drvpath
Failover
Fail the sdb
device on purpose to mimick a failover. Then you will notice how sdc
becomes active. The developers may diagnose and observe at either virtual machine side or the KVM hypervisor side during the failover.
[root@vm opc]# echo 1 > /sys/block/sdb/device/delete [root@vm opc]# multipath -ll mpathc (360014052d347b7c8b8041e9bc817234e) dm-0 LIO-ORG ,storage size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active `- 4:0:1:0 sdc 8:32 active ready running
The following is an extract from multipathd
syslog:
May 20 06:17:50 vm kernel: sd 3:0:1:0: [sdb] Synchronizing SCSI cache May 20 06:17:50 vm multipathd: sdb: remove path (uevent) May 20 06:17:50 vm multipathd: mpathc: load table [0 2097152 multipath 1 queue_if_no_path 0 1 1 service-time 0 1 1 8:32 1] May 20 06:17:50 vm multipathd: sdb [8:16]: path removed from map mpathc May 20 06:17:50 vm kernel: scsi 3:0:1:0: alua: Detached
To recover, please scan the loopback device again. Now both sdb
and sdc
are available again, although sdc
remains active:
[root@vm opc]# echo "- - -" > /sys/class/scsi_host/host3/scan [root@vm opc]# multipath -ll mpathc (360014052d347b7c8b8041e9bc817234e) dm-0 LIO-ORG ,storage size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 4:0:1:0 sdc 8:32 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 3:0:1:0 sdb 8:16 active ready running
Summary
The virtualization environment is quite complex involving a very deep software stack, which includes software components from the virtual machine side filesystem to the KVM hypervisor side SCSI storage. In this article, we demonstrated how to quickly and efficiently setup a full-software environment for QEMU scsi-block + multipath failover. This helps developers diagnose or observe the behavior of SG_IO during a multipath failover. Remember the only prerequisite for following the instructions provided here is that the OCI instance must be based on Oracle Linux 7 and Intel architecture.