Disk dynamic reconfiguration in Oracle VM Server for SPARC
By jsavit on Nov 21, 2013
Adding a virtual disk to a running domain
The entire sequence of commands in the control domain defines, adds and removes a disk while the guest domain runs:
# mkfile -n 20g /ldomsnfs/ldom0/disk1.img # 1. create a disk image file # ldm add-vdsdev /ldomsnfs/ldom0/disk1.img vol01@primary-vds0 # 2. define vdisk # ldm add-vdisk vdisk01 vol01@primary-vds0 ldom0 # 3. add disk to the domain # ldm rm-vdisk vdisk01 ldom0 # 4. take it away from the domain. # ldm rm-vdsdev vol01@primary-vds0 # 5. undefine the virtual disk # rm disk1.img # 6. save a little space.
That's all there is to it. The new disk is available for the domain's use after step 3 until I take it away in step 4.
Viewing reconfiguration from within the guest
Let's take a look from the guest domain's perspective. In the guest, you can see one disk before adding more (before command 3, above) via the format command:
# format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c3d1
/virtual-devices@100/channel-devices@200/disk@1 Specify disk (enter its number): ^C
There's one disk until ldm add-vdisk is issued in the control domain. That results in a dynamic reconfiguration event that can be seen, if you are curious, by entering dmesg within the guest:
# dmesg|tail ...snip... Nov 20 12:03:23 ldom0 vdc: [ID 625787 kern.info] vdisk@0 is online using ldc@16,0 Nov 20 12:03:23 ldom0 cnex: [ID 799930 kern.info] channel-device: vdc0 Nov 20 12:03:23 ldom0 genunix: [ID 936769 kern.info] vdc0 is /virtual-devices@100/channel-devices@200/disk@0 Nov 20 12:03:23 ldom0 genunix: [ID 408114 kern.info] /virtual-devices@100/channel-devices@200/disk@0 (vdc0) online
You can see the added disk using format and then use it. In this case I created a temporary ZFS pool.
# format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c3d0At this point I can just go ahead and use the added disk space. I could have done other things like add it to the existing ZFS pool to make it a mirror, but this illustrates the point.
/virtual-devices@100/channel-devices@200/disk@0 1. c3d1 /virtual-devices@100/channel-devices@200/disk@1 Specify disk (enter its number): ^C # zpool create temp c3d0 # zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT rpool 19.9G 5.27G 14.6G 26% 1.00x ONLINE - temp 19.9G 112K 19.9G 0% 1.00x ONLINE -
What happens if I try to remove an in-use disk
It could be very damaging to remove a virtual device while it is in use, so the default behavior is that Solaris tells logical domains manager that the device is in use and cannot be removed. That's a very important advantage of Oracle VM Server for SPARC: the logical domains framework and Solaris work cooperatively, in this and many other aspects.
In this case, we're prevented from yanking a disk while it is in use. If I try to remove the disk while it's in use, I get an error message - exactly what you want:
# ldm rm-vdisk vdisk01 ldom0 Dynamic reconfiguration of the virtual device on domain ldom0 failed with error code (-122). The OS on domain ldom0 did not report a reason for the failure. Check the logs on that OS instance for any further information. Failed to remove vdisk instanceThe reason is "because it's in use!" :-) An administrator would log into the guest to see what file systems are mounted. This behavior can be overridden using the "-f" option if you are certain you know what you're doing.
Removing the disk
I issued zpool destroy temp in the guest and repeated the ldm rm-vdsdev and it worked. Using zpool export temp would work just as well, and if I choose I can add that virtual disk to a different domain and it could use zpool import temp to access data created by ldom0. With other file systems, a regular umount would have the same effect, making it possible to remove the disk without -f.
The format command now shows only one disk again, and dmesg shows kernel messages when disk went offline:
Nov 20 12:42:10 ldom0 vdc: [ID 990228 kern.info] vdisk@0 is offline Nov 20 12:42:10 ldom0 genunix: [ID 408114 kern.info] /virtual-devices@100/channel-devices@200/disk@0 (vdc0) offline
Solaris and the logical domain manager are engineered to work together in a coordinated fashion to provide operational flexibility. One of the values this provides is that administrators can safely add and remove virtual devices while domains run. This can be used for operational tasks like adding or removing disk capacity or IOPS as needed. The same capabilities are also available for virtual network devices.