Layered Virtualization

It's time for another guest blogger.

Solving a Corner Case

One of my former colleagues, Joe Yanushpolsky (josephy100 -AT- was recently involved in the movement of a latency-sensitive Linux application to Solaris as part of platform consolidation. The code was old and it required access to kernel routines not available under BrandZ. Using VirtualBox as a virtual x86 system, the task was easier than expected.


VirtualBox enables you to run multiple x86-based operating system "guests" on an x86 computer - desktop or server. Unlike other virtualization tools, like VMware ESX, VirtualBox allows you to keep your favorite operating system as the 'base' operating system. This is called a Type 2 hypervisor. For existing systems - especially desktops and laptops - this means you can keep your current setup and applications and maintain their current performance. Only the guests will have reduced performance - more on that later.

Here is Joe's report of his tests.

The goals included allowing many people to independently run this application while sharing a server. It would be important to isolate each user from other users. But the resource controls included with VirtualBox were not sufficiently granular for the overall purpose. Solaris Containers (zones) have a richer set of resource controls. Would it be possible to combine Containers and VirtualBox?

The answer was 'yes' - I tried two slightly different methods. Each method starts by installing VirtualBox in the global zone to set up a device entry and some of the software. Details are provided later. After that is complete, the two methods differ.

  1. Create a Container and install VirtualBox in it. This is the Master WinXP VirtualBox (MWVB) Container. If any configuration steps specific to a WinXP environment are needed, they can be done now. When a Windows XP environment is needed, clone the MWVB Container and install WinXP in the clone. Management of the Container can be delegated to the user of the WinXP environment if you want.
  2. Create a Container and install VirtualBox in it. This is the Master CentOS VirtualBox (MCVB) Container. Install CentOS in the Container. When a CentOS environment is needed, clone the MCVB - including the copy of CentOS that's already in the Container - to create a new Container. Management of the Container can be delegated to the user of the CentOS environment if you want.
In each case, resource controls can be applied to the Container to ensure that everyone gets a fair share of the system's resources like CPU, RAM, virtual memory, etc.

When the process is complete, you have a guest OS, shown here via X Windows.

CentOS picture

Not only did the code run well but it did so in a sparse root non-global zone

Well that was easy! How about Windows?
Windows XP picture

Now, this is interesting. As long as the client VM is supported by VirtualBox, it can be installed and run in a Solaris/OpenSolaris Container. I immediately thought of several useful applications of this combination of virtualization technologies:

  • migrate existing applications that are deemed "unmovable" to latest eco-friendly x64 (64-bit x86) platforms
  • reduce network latency of distributed applications by collapsing the network onto a large memory system with zones, regardless of which OS the application components were originally written in
  • on-demand provisioning, as a service, an entire development environment for Linux or Windows developers. When using ZFS, this could be accomplished in seconds - is this a "poor man's" cloud or what?!
  • eliminate ISV support issues that are currently associated with BrandZ's lack of support for recent Linux kernels or Solaris 8 or 9 kernel
  • what else can you create?
Best of all, Solaris, OpenSolaris and VirtualBox can be downloaded and used free of charge. Simple to build, easy to deploy, low overhead, free - I love it!


The advantage of having access to application code through Containers more than compensated for a 5% overhead (on a laptop) due to having a second kernel. The overall environment seems to be disk-sensitive (SSDs to the rescue!). Given that typical server load in a large IT shop is 15-20%, a number of such "foreign" zones could be added without impacting overall server performance.

Future Investigations

It would be interesting to evaluate scalability of the overall environment by testing different resource controls in Solaris Containers and in VirtualBox. I'd need a machine bigger than the laptop for that :-).

Installation Details

Here are the highlights of "How to install." For more details, follow instructions in the VirtualBox User manual.

  • Install VirtualBox on a Solaris x64 machine in the global zone so that the vboxdrv driver is available in the Solaris kernel.
  • Create a target zone with access to the vboxdrv device ("add device; set match=/dev/vboxdrv; end").
  • In the zone, clean up the artifacts of the previous VirtualBox installation in the global zone. All you need to do is to uninstall the SUNWvbox package and remove references to /opt/VirtualBox directory.
  • Install VirtualBox package in the zone.
  • Copy the OS distro into a file system in the global zone (e.g. /export/distros/centos.iso, and configure a loopback mount into the zone ("add fs; set dir=/mnt/images; set special=/export/distros; set type=lofs; end").
  • Start VirtualBox in the zone and install the client OS distro.
What advantages does this model have over other virtualization solutions?
  • The Solaris kernel is the software layer closest to the hardware. With Solaris, you benefit from the industry-leading scalability of Solaris and all of its innovations, like:
    • ZFS for data protection - currently, neither Windows nor Linux distros have ZFS. You can greatly improve storage robustness of your Windows or Linux system by running it as a VirtualBox guest.
    • SMF/FMA, which allows the whole system to tolerate hardware problems
    • DTrace, which allows you to analyze system performance issues while the apps are running. Although you can use DTrace in the 'base' Solaris OS environment to determine which guest is causing the performance issue, and whether the problem is network I/O, disk I/O, or something else, DTrace will not be able to "see" into VirtualBox guests to help figure out which particular application is the culprit - unless the guest is running Solaris, in which case you run DTrace in the guest!
  • Cost: You can download and use Solaris and OpenSolaris without cost. You can download and use VirtualBox without cost. Some Linux distros are also free. What costs less than 'free?'
What can you do with this concept? Here are some more ideas:
  • Run almost any Linux apps on a Solaris system by running that Linux distro in VirtualBox - or a combination of different Linux distros.
  • Run multiple Windows apps - even on different versions of Windows - on Solaris.
Additional notes are available from the principal investigator, Joseph Yanushpolsky: josephy100 -AT- .

Nice, would never have thought of using Containers in conjunction with VirtualBox in this manner.

Thanks for the insight (and thanks to you colleague for his work).

Posted by Mike on May 05, 2009 at 03:00 AM EDT #

Wow, I was actually wondering about doing something like this! I'm new to Solaris but immediately saw the vast capabilities Solaris has. This is definitely something to look into.

Posted by Patrick Cantu on May 18, 2009 at 06:35 AM EDT #

If you want to play dirty then create x zones and install virtualbox in them. Then install Windows XP on all virtualboxes.

Now you have the most rubust ICA or RDP environment for your office. All XP installations have their own IP-adres and you can manage their behavior (like firewalling,shared disks etc) from solaris.

or share it with your family: Your family can use their old computers with the performance of a new one....

Never have seen windows XP install so fast..(nor boot)

Posted by Maddie on May 18, 2009 at 10:52 AM EDT #

Just wanted to give a update that I have sucessfully gotten this to work in Solaris 10 u7. Once initial tests are complete I'll be rolling this out in production for my users! (All 60+ of them)

Posted by Patrick Cantu on June 05, 2009 at 07:01 AM EDT #

Initially I tried VirtualBox 2.2.4 on an SXCE build 115 and OpenSolaris 0906 using UFS and ZFS and each time I received an NS_ERROR_FAILURE for Machine when trying to start a guest in VirtualBox inside a non-global zone. Howerver, running a guest VM on 2.2.4 in the global zone worked.

Thanks to Joseph Yanushpolsky for information I received from him via email, I was successful using VirtualBox 2.2.0 and running a FreeBSD 7.2 guest in VirtualBox in a non-global zone using an OpenSolaris host.

Note that VirtualBox 2.2.2 has the same NS_ERROR_FAILURE when trying to run a guest VM inside a non-global zone.

Also, if you use OpenSolaris you will need to create an /etc/fonts directory and download the fonts.conf file or VirtualBox will not start. SXCE doesn't have this problem.

Additionally, when I tried to use the pkgcmds (e.g., pkgadd) in the non-global zone on OpenSolaris 0906 it erred out because it couldn't find the default install file. To solve this I copied the /var/sadm/install/admin/default file from the global zone to the non-global zone and also created the /var/sadm/security directory which is included in the default file.

Posted by Gary on June 11, 2009 at 01:54 PM EDT #

This is great stuff!
Will this work on a Zone clusters and will it handle fail over of the running virtualbox instances?

Zone clusters is described here:

Posted by Jaco Vosloo on July 31, 2009 at 04:05 AM EDT #

Post a Comment:
Comments are closed for this entry.

Jeff Victor writes this blog to help you understand Oracle's Solaris and virtualization technologies.

The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.


« August 2016