While traditional QEMU vhost-scsi offloads I/O processing from QEMU to kernel space vhost threads, the vhost-user-scsi can leverage SPDK (Storage Performance Development Kit) as the backend to delegate I/O processing to external userspace processes. In this blog article, we demonstrate how to build SPDK and launch a QEMU instance with vhost-user-scsi on Oracle Linux 8, running on a VM in Oracle Cloud Infrastructure (OCI), providing a hands-on experience of vhost-user-scsi and SPDK. We are not going to cover the underlying mechanisms, instead concentrating on how to build and utilize the feature. This article aims to facilitate Linux administrators or developers, to study, debug and develop QEMU vhost-user-scsi and SPDK.

Step 1. Create an OCI VM

For the purposes of this article we are using an Oracle Linux 8 instance in OCI with the VM.Standard.E5.Flex shape (4-OCPU and 48GB of memory).

Step 2. Configure dnf and install packages

Once your instance is created, enable the yum repositories ol8_kvm_appstream, ol8_codeready_builder and ol8_developer_EPEL to install the prerequisite packages.

$ sudo dnf config-manager --enable ol8_kvm_appstream
$ sudo dnf module remove virt -y --all
$ sudo dnf module reset virt -y
$ sudo dnf module enable virt:kvm_utils3 -y

$ sudo dnf config-manager --enable ol8_codeready_builder
$ sudo dnf config-manager --enable ol8_developer_EPEL

Next install the required packages, including qemu-kvm provided by Oracle Linux.

$ sudo dnf install -y git fuse3-devel meson python3-pyelftools \
                      numactl-devel uuid-devel libuuid-devel \
                      ncurses-devel CUnit-devel libaio-devel \
                      qemu-kvm

Step 3. Configure gcc

In order to use gcc-11 we need to enable the gcc-toolset-11 software collection:

$ scl enable gcc-toolset-11 bash

Double check that the gcc version is 11.5.0.

$ gcc -v
... ...
gcc version 11.5.0 20240719 (Red Hat 11.5.0-2.0.1) (GCC)

To ensure this software collection is enabled permanently across logins, add the above scl command to /etc/profile.

Step 4. Download and build SPDK source code

We use SPDK v25.09 in this article.

$ cd /home/opc
$ git clone https://github.com/spdk/spdk
$ cd spdk
$ git checkout -b v25.09_build v25.09

$ git submodule update --init
$ ./configure
$ make -j8

Step 5. Create vhost target

Start SPDK vhost daemon. Pre-allocate 10GB of hugepages. We did not configure CPU affinity mask to bind any SPDK threads, as this article is intended for study purposes only.

$ cd /home/opc/spdk
$ sudo HUGEMEM=10240 scripts/setup.sh
$ sudo ./build/bin/vhost -S /tmp

Start a new SSH session to create the vhost target. We utilize a raw image file as the storage.

$ dd if=/dev/zero of=/home/opc/storage.raw bs=1M count=1024 oflag=direct
$ cd /home/opc/spdk
$ sudo ./scripts/rpc.py vhost_create_scsi_controller vhost.0
$ sudo ./scripts/rpc.py bdev_aio_create /home/opc/storage.raw bdev.0 512
$ sudo ./scripts/rpc.py vhost_scsi_controller_add_target vhost.0 0 bdev.0
$ sudo ./scripts/rpc.py bdev_get_bdevs

Step 6. Create QEMU instance

Create a QEMU instance with vhost-user-scsi. The file ol8.qcow2 is an OL8-based boot image.

$ sudo /usr/libexec/qemu-kvm \
-machine q35 -accel kvm \
-cpu host -smp 4 -m 8G \
-hda /home/opc/ol8.qcow2 \
-net nic -net user,hostfwd=tcp::5028-:22 \
-vnc :8 -serial stdio \
-object memory-backend-file,id=mem0,size=8G,mem-path=/dev/hugepages,share=on \
-numa node,memdev=mem0 \
-chardev socket,id=char0,path=/tmp/vhost.0 \
-device vhost-user-scsi-pci,id=vscsi0,chardev=char0,num_queues=4

The vhost-scsi-user is present in guest VM as /dev/sda.

[root@vm ~]# lsscsi
[0:0:0:0]    disk    ATA      QEMU HARDDISK    2.5+  /dev/sdb 
[2:0:0:0]    cd/dvd  QEMU     QEMU DVD-ROM     2.5+  /dev/sr0 
[3:0:0:0]    disk    INTEL    AIO disk         0001  /dev/sda

There are four virtio-scsi queues in guest VM.

[root@vm ~]# cat /proc/interrupts  | grep virtio
 25:          0          0          0          0   PCI-MSI 49152-edge      virtio0-config
 26:          1          0          0          0   PCI-MSI 49153-edge      virtio0-control
 27:          0          1          0          0   PCI-MSI 49154-edge      virtio0-event
 28:          8          0          0          0   PCI-MSI 49155-edge      virtio0-request
 29:          0         70          0          0   PCI-MSI 49156-edge      virtio0-request
 30:          0          0         52          0   PCI-MSI 49157-edge      virtio0-request
 31:          0          0          0        265   PCI-MSI 49158-edge      virtio0-request

In addition, many SPDK debug logs are generated through the SSH session of “./build/bin/vhost -S /tmp“.

Summary

In this article we demonstrate how to build and setup the QEMU vhost-user-scsi and SPDK environment on Oracle Linux 8. We utilize a raw image file as the storage. We did not configure CPU affinity mask to bind any SPDK threads, as this article is intended for study purposes only. Developers may use OCI and Oracle Linux 8 for vhost-user-scsi/SPDK related development/research projects.