Software-Only Setup to Diagnose QEMU SCSI Passthrough and Multipath Failover

May 31, 2022 | 5 minute read
Text Size 100%:

The SCSI passthrough over multipath is very commonly used in virtualization environments. The SCSI IO commands are forwarded from the virtual machine to the KVM hypervisor side storage via SG_IO. The KVM hypervisor side storage is always provied in a multipath configuration in order to facilitate high availability. Therefore, the developers may require a solution that allows them to quickly and efficiently setup an environment to diagnose and observe the behavior of virtio-scsi + QEMU + SG_IO + multipath + SCSI, e.g., how the path error is propagated to the virtual machine.

In this article, we demonstrate how to setup a software-only KVM environment on a regular Oracle Cloud Infrastructure (OCI) virtual machine, without any special SCSI/storage hardware requirements.

Note: The OCI Instance must be an Oracle Linux 7 Intel architecture instance.

Software Installation

Once you’ve created your OL7 based OCI Intel instance, SSH into the instance to perform software installation. Only targetcli and QEMU are required for installation, since the device-mapper-multipath is already available.

[root@vm opc]# yum install targetcli
[root@vm opc]# yum install qemu

Create Two Paths

We use the targetcli to setup two loopback devices referring to the same image file: /home/opc/storage.raw:

[root@vm opc]# dd if=/dev/zero of=storage.raw bs=1M count=1024
1024+0 records in
1024+0 records out 
1073741824 bytes (1.1 GB) copied, 0.616011 s, 1.7 GB/s

[root@vm opc]# targetcli /backstores/fileio create name=storage file_or_dev=/home/opc/storage.raw
Created fileio storage with size 1073741824

The 1st path is via naa.5001405f01234ba3, and the 2nd path is via naa.500230567123438b.

[root@vm opc]# targetcli /loopback create naa.5001405f01234ba3
Created target naa.5001405f01234ba3.

[root@vm opc]# targetcli /loopback/naa.5001405f01234ba3/luns create /backstores/fileio/storage
Created LUN 0.

[root@vm opc]# targetcli /loopback create naa.500230567123438b
Created target naa.500230567123438b.

[root@vm opc]# targetcli /loopback/naa.500230567123438b/luns create /backstores/fileio/storage
Created LUN 0.

There will be two loopback SCSI devices (and disks) available.

[root@vm opc]# lsscsi
[2:0:0:1]    disk    ORACLE   BlockVolume      1.0   /dev/sda 
[3:0:1:0]    disk    LIO-ORG  storage          4.0   /dev/sdb
[4:0:1:0]    disk    LIO-ORG  storage          4.0   /dev/sdc

Both loopback SCSI devices are based on the same backstore file /home/opc/storage.raw.

[root@vm opc]# targetcli ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 1]
  | | o- storage ............................................................. [/home/opc/storage.raw (1.0GiB) write-back activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 0]
  o- loopback ......................................................................................................... [Targets: 2]
  | o- naa.5001405f01234ba3 ................................................................................. [naa.50014055b3b6d883]
  | | o- luns ............................................................................................................ [LUNs: 1]
  | |   o- lun0 ........................................................ [fileio/storage (/home/opc/storage.raw) (default_tg_pt_gp)]
  | o- naa.500230567123438b ................................................................................. [naa.5001405f51ac04e4]
  |   o- luns ............................................................................................................ [LUNs: 1]
  |     o- lun0 ........................................................ [fileio/storage (/home/opc/storage.raw) (default_tg_pt_gp)]
  o- vhost ............................................................................................................ [Targets: 0]

Setup multipath

Create or modify the configuration file /etc/multipath.conf, to rule out sda which is already used as boot image of the virtual machine:

[root@vm opc]# cat /etc/multipath.conf
defaults {
    user_friendly_names yes
    no_path_retry 10
}
blacklist {
    devnode "sda"
}

To ensure this configuration is not persistent across reboot, ensure to only start the systemd service, do not enable it. Once the multipathd service is started, only sdb will be active:

[root@vm opc]# systemctl start multipathd

[root@vm opc]# multipath -ll
mpathc (360014052d347b7c8b8041e9bc817234e) dm-0 LIO-ORG ,storage
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 3:0:1:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 4:0:1:0 sdc 8:32 active ready running

Virtual Machine Creation

Create a virtual machine with scsi-block, using /dev/mapper/mpathc. The file ol7.qcow2 is an OL7-based boot image:

$ qemu-system-x86_64 -smp 4 -m 8G -enable-kvm -cpu host -vnc :7 \
-net nic -net user,hostfwd=tcp::5027-:22 -hda ol7.qcow2 -serial stdio \
-device virtio-scsi-pci,id=virtio-scsi01,bus=pci.0,addr=0x4 \
-drive file=/dev/mapper/mpathc,format=raw,if=none,id=drvpath,cache=none,aio=native \
-device scsi-block,bus=virtio-scsi01.0,channel=0,scsi-id=0,lun=0,drive=drvpath

Failover

Fail the sdb device on purpose to mimick a failover. Then you will notice how sdc becomes active. The developers may diagnose and observe at either virtual machine side or the KVM hypervisor side during the failover.

[root@vm opc]# echo 1 > /sys/block/sdb/device/delete

[root@vm opc]# multipath -ll
mpathc (360014052d347b7c8b8041e9bc817234e) dm-0 LIO-ORG ,storage
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 4:0:1:0 sdc 8:32 active ready running

The following is an extract from multipathd syslog:

May 20 06:17:50 vm kernel: sd 3:0:1:0: [sdb] Synchronizing SCSI cache
May 20 06:17:50 vm multipathd: sdb: remove path (uevent)
May 20 06:17:50 vm multipathd: mpathc: load table [0 2097152 multipath 1 queue_if_no_path 0 1 1 service-time 0 1 1 8:32 1]
May 20 06:17:50 vm multipathd: sdb [8:16]: path removed from map mpathc
May 20 06:17:50 vm kernel: scsi 3:0:1:0: alua: Detached

To recover, please scan the loopback device again. Now both sdb and sdc are available again, although sdc remains active:

[root@vm opc]# echo "- - -" > /sys/class/scsi_host/host3/scan

[root@vm opc]# multipath -ll
mpathc (360014052d347b7c8b8041e9bc817234e) dm-0 LIO-ORG ,storage
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 4:0:1:0 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 3:0:1:0 sdb 8:16 active ready running

Summary

The virtualization environment is quite complex involving a very deep software stack, which includes software components from the virtual machine side filesystem to the KVM hypervisor side SCSI storage. In this article, we demonstrated how to quickly and efficiently setup a full-software environment for QEMU scsi-block + multipath failover. This helps developers diagnose or observe the behavior of SG_IO during a multipath failover. Remember the only prerequisite for following the instructions provided here is that the OCI instance must be based on Oracle Linux 7 and Intel architecture.

Dongli Zhang


Previous Post

Simplify managing Oracle Linux 8 Application Streams with OS Management Service

Julie Wong | 6 min read

Next Post


Verrazzano 1.3 – Our users have spoken, and we listened!

David Cabelus | 3 min read