In this blog, the second in a series of three, Oracle Linux kernel developer Elena Ufimtseva demonstrates how to configure and build our disaggregated QEMU.
Configure and build multi-process QEMU
To build the system that supports multi-process device emulation in QEMU, the build system was modified to add new objects. To get the latest development tree with multi-process support clone it from the git repository and branch multi-process-qemu-v0.1:
git clone -b multi-process-qemu-v0.1 https://github.com/oracle/qemu.gitRun configure with –enable-mpqemu to enable multi-process qemu and run make:
./configure --disable-xen --disable-tcg --disable-tcg-interpreter --target-list=x86_64-softmmu --enable-guest-agent --enable-mpqemu
make all
make installNotes on Xen:
If no support of Xen on the system is needed, --disable-xen should be used. On OL7 --disable-xen should be used.
There are few executable files, some of which are the remote programs. Depending on the options used while configuring Qemu, one may need to add the location of those remote programs to the PATH environment variable. In current version the program name is “qemu-scsi-dev”. configure script can be used with option –install= to specify the installation directory.
Running multi-process QEMU
To run qemu device emulation in a separate process, there are following options that are different from the original qemu: rdevice; rdrive; These options are similar to the ones in original qemu and can be used in the same way. For example, to run disk attached to LSI SCSI controller in remote process, the following command line can be used:
/usr/local/bin/qemu-system-x86_64 -name vm -m 6G -drive file=/root/ol7.qcow2,format=raw -enable-kvm -machine q35,accel=kvm -rdevice lsi53c895a,rid=0,id=scsi0,command=qemu-scsi-dev -rdevice scsi-hd,rid=0,drive=drive0,bus=scsi0.0,scsi-id=0 -rdrive id=drive0,rid=0,file=/root/cirros-0.4.0-x86_64-disk.img,format=qcow2 -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=6G,share=on -numa node,memdev=mem -display none -vnc :0 -monitor stdio -device e1000,netdev=net0 -netdev user,id=net0,hostfwd=tcp::5555-:22Required options are:
-
remote device options:
-rdevice lsi53c895a,rid=0,id=scsi0,command=qemu-scsi-dev -rdevice scsi-hd,rid=0,drive=drive0,bus=scsi0.0,scsi-id=0 -rdrive id=drive0,rid=0,file=/root/cirros-0.4.0-x86_64-disk.img,format=qcow2 -
memory object to support file descriptor based memory synchronization between remote process and qemu:
-object memory-backend-file,id=mem,mem-path=/dev/shm/,size=6G,share=on -numa node,memdev=mem
The result of running multi-process qemu with one remote process:
There are two processes listed here, one is the main qemu and the second is the qemu-scsi-dev remote process.
Debugging and troubleshooting
There are additional options to provide more diagnostics for debugging.
To enable logging for multi process qemu, -D option can be specified with mask “rdebug”: -D /tmp/qemu.log -d rdebug
To enable Qemu debugging with gdb, it can be configured with --enable-debug-info to include debug symbols.
Since multi-process qemu has additional processes that are spawned during the execution, to use gdb to debug child processes the following settings can be used to launch gdb:
set detach-on-fork off
set follow-exec-mode new
set follow-fork-mode child
set print inferior-events onThis will allow debug of the child process automatically.
Below is the example of such a debug session:
[root@localhost ~]# gdb /usr/local/bin/qemu-system-x86_64
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/local/bin/qemu-system-x86_64...done.
(gdb) r -enable-kvm -machine q35 -smp 4 -m 8000M -vnc :0 -net nic -net user,hostfwd=tcp::5022-:22 -drive file=/root/ol7.qcow2,format=raw -rdevice lsi53c895a,rid=0,id=scsi0 -rdevice scsi-hd,rid=0,drive=drive0,bus=scsi0.0,scsi-id=0 -rdrive id=drive0,rid=0,file=/root/cirros-0.4.0-x86_64-disk.img -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=8000M,share=on -numa node,memdev=mem
Starting program: /usr/local/bin/qemu-system-x86_64 -enable-kvm -machine q35 -smp 4 -m 8000M -vnc :0 -net nic -net user,hostfwd=tcp::5022-:22 -drive file=/root/ol7.qcow2,format=raw -rdevice lsi53c895a,rid=0,id=scsi0 -rdevice scsi-hd,rid=0,drive=drive0,bus=scsi0.0,scsi-id=0 -rdrive id=drive0,rid=0,file=/root/cirros-0.4.0-x86_64-disk.img -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=8000M,share=on -numa node,memdev=mem
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7fffef5fe700 (LWP 14001)]
[New Thread 0x7ffdfac1a700 (LWP 14003)]
[New Thread 0x7ffdfa419700 (LWP 14005)]
[New Thread 0x7ffdf9c18700 (LWP 14006)]
[New Thread 0x7ffdf9417700 (LWP 14007)]
[New Thread 0x7ffdebfff700 (LWP 14009)]
[New inferior 14010]
[New process 14010]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Thread 0x7ffff7fc5c00 (LWP 14010) is executing new program: /usr/local/bin/qemu-scsi-dev
[New inferior 14010]
(gdb) info inferior
Num Description Executable
3 process 14010 /usr/local/bin/qemu-scsi-dev
2 <null> /usr/local/bin/qemu-system-x86_64
1 process 13997 /usr/local/bin/qemu-system-x86_64 To see part 2 in this blog series, go to: https://blogs.oracle.com/linux/post/towards-a-more-secure-qemu-hypervisor-part-3-of-3