Thursday Sep 17, 2009

Answering a customer's LDOMs security questions

Recently a customer in the Federal Government asked some fairly straightforward security questions about Logical Domains.  In doing my research, I found it wasn't that straight forward to get the answers from the standard Logical Domains (LDOMs) documentation.  Luckily, our engineering and marketing team stepped up to provide clear, concise answers so that this customer (who prefers to remain anonymous) can move forward and implement their virtualization strategy on Sun's T2 class of processors.

Logical Domains (LDoms) provide built-in and no-cost virtualization capabilities for Sun Chip Multithreading (CMT) Servers. Unlike proprietary virtualization technologies, LDoms can save you up to $10,000 per server. It allows you to create virtual machines that take advantage of the massive thread scale offered by these platforms. Create up to 128 virtual servers on one system... for free!  Customers have used Logical Domains to reduce their costs and consolidate their server farms for significant returns in operations and energy savings. For example, using LDOMs and Solaris containers, the United States Air Force was able to reduce rack space to achieve a 13:1 consolidation ratio, decreased server deployment time by more than 90% and cut datacenter power consumption by more than 25%. Download the software for Solaris 10 or OpenSolaris today.

Logical Domains allow the primary Solaris domain (sometimes known as the control domain) to create virtual disks and assign CPU thread, network, memory and I/O resources to other virtual Solaris machines to run on a single system.  The control domain uses the Logical Domains Manager (LDM) to control, monitor and manage the running domains.  Live migration of domains is supported.

LDoms 1.2 adds a number of new features, including:

  • Improved Network performance with the introduction of support for jumbo frames
  • Reduced power footprint with CPU power management, powering off cores that aren't in use automatically
  • Easier adoption with support for physical-to-virtual migration tool
  • Quick start with support for configuration assistant tools
  • Faster agility with enhancements to Domain Mobility
  • Increased control and response to guest availability with Domain dependencies
  • In-built protection from corruption with Auto-recovery of configurations
And now on to the Q and A:


Q: Can the Control domain access/utilize the CPU threads of a guest without shutting down the guest?

Answer: A Control domain cannot access the CPU threads assigned to a guest domain unless the threads are removed from the guest, and then added to the control domain, such as with CPU Dynamic Reconfiguration, or by rebooting both the guest and control domain after a Static Reconfiguration. LDoms fundamentally partitions CPU resources and there is no sharing of CPU thread resources. Enforcement of this partitioning and separation is done at the Hypervisor level, so it cannot be circumvented by the Control domain.

Virtualization solutions for x86 and IBM Power systems typically time-slice access to threads across multiple guests. This is because IBM and Intel CPU's have very few threads per socket. With SPARC CMT, we have up to 128 threads per socket, and we take advantage of the hardware by using a much safer and simpler partitioning approach in the SPARC Hypervisor and LDoms.

Q: Can a guest domain access the CPU threads of another guest?

Answer: No. LDoms partitions threads and does not share them across logical domain boundaries. See detailed explanation above.

Q: Can a guest domain access the CPU threads of the control domain?

Answer: No. See answers above.


Q: Can the Control domain alter the active memory space of a running guest?

Answer: There are two types of memory “alteration” in a system, first is modifying the contents of existing memory in a guest, and second, is the reconfiguration of memory size within a guest. For LDoms, guests have no knowledge of one another, nor are there any interfaces to allow one guest to gain access to or modify the memory of another guest. Memory separation and partitioning is enforced by the SPARC Hypervisor.

As of LDoms 1.2, Any request to change the memory configuration (i.e. How much memory a guest has allocated to it), through the LDM command line interface on the Control Domain would queue a “Delayed Reconfiguration” operation, which would take effect upon the next reboot of the guest. Beginning in LDoms 2.0, we will support the dynamic reconfiguration of a guest domain's memory configuration.

There are some memory transfer or shared memory access between domains done in order to implement virtual device and domain services. These transfers and sharing are strictly controlled by each domain and by the SPARC hypervisor: a domain will define, with the hypervisor, the memory data it is going to transfer or share with another domain

Q: Can a guest domain access the memory of another guest?

Answer: No. Guests have no knowledge of one another, nor are there any interfaces to allow one guest to gain access to or modify the memory of another guest. Memory separation and partitioning is enforced by the SPARC Hypervisor.

Q: Can a guest domain access the memory of the control domain?

Answer: No. There are no interfaces which allow for a guest to modify the configuration of or gain access to any part of the control domain's memory.

Virtual Network

Q: Can the control domain alter the network traffic of guest domains? The concern is about a compromised Control Domain becoming a man-in-the-middle. How can this condition be identified/reported?

Answer: Yes. The network switching of the packets is done in a software driver(vsw), its harder to alter the network traffic to Guest domains, but a compromised control(or service) domain \*can\* alter the traffic. Our Security model assumes that the domain(s) that host services such as vsw, are trusted, so they need to be secured as per the local security guidelines. Compromising or accessing the network traffic of guest domains from the control domain requires root access on the control domain.

Q: Can a guest domain access the network traffic for another guest? The assumption is yes, since an IP network is being shared. A scenario of interest - or pre condition - is if the physical NIC is disconnected, other than via the physical IP network. The key concern is a guest domain accessing the IP traffic of another guest domain via the virtual switch.

Answer: No. The traffic between the virtual switch(vsw) and the virtual network device(vnet) uses Logical Domain Channels(LDCs) that are a point-to-point type of connection. As a result, the traffic between the virtual switch and a guest domain is not visible to other guest domains. Note, switching is based on mac-addresses and LDoms doesn't allow the change of mac-address of a vnet device in a guest domain, so guest domains cannot spoof by changing their mac-addresses.

Q: Can a guest domain access the network traffic of the control domain?

Answer: No. Guest domains will only see the traffic that fits the following:

  • Unicast traffic that matches the virtual network device's mac-address in the guest domain.

  • Broadcast traffic.

  • Multicast traffic for which the guest domain registered to receive.

No other packets will be seen by the Guest domains.

Virtual Disks

Q: Can a guest domain access virtual disk devices that it has not been allocated, e.g., other guests, Control Domains?

Answer: No. A guest domain can only access virtual disk devices that have been explicitly assigned to it. It will not see, nor can the guest access any other disk.

Virtual Console

Q: Can a guest domain access the virtual console of another guest domain?

Q: Can a guest domain access the console for the control domain?

Answers: A guest domain cannot access the console interface for a different guest domain, nor can a guest domain access the console for the control domain. The only console access is via a privileged user on the control domain itself. There are no interfaces available in any other scenario for access a guest console, including over the general network interface.

Special Interest

Once the LDoms are running in our environment, there is very little need to log into the Control Domain (CD) and this is preferred behavior.

Q: Can a Control Domain be shut down and the LDOMS continue to run? If not, are there other options for maximally restricting access to, e.g., "locking" a CD once the LDoms are configured? An acceptable instance of "locking"is restricting access to the CD from Virtual Console only. Ideally, access via SSH would also be highly restricted. Limited access for maintenance and configuration are also acceptable.

In summary, the primary objective of these features is to secure the CD from a malicious user gaining access and changing LDom configuration without detection.

Answer: one of the architectural principles of LDoms has been that a guest domain can operate independently of the control domain. For example, If a control domain were to fail and reboot, the guests will continue to operate. Extending this logic, yes, you can currently shutdown the control domain and the guest environment will continue to operate. However, this holds only if the guests are using virtual I/O (assuming that I/O is being served from an I/O service domain that's not the control domain) or have been granted direct ownership of one or more PCI-E busses. But with the advent of upcoming projects like direct I/O (the ability to assign individual PCI-E slots to a guest) and SR-IOV (the ability to assign individual PCI-E virtual functions to a guest), it will not be possible to shut down the control domain without impacting guest domains that have been allocated individual PCI-E slots or functions.

In addition, other caveats, or things to consider are:

  • Without a control domain, there is no console access to the guests unless the console service is hosted elsewhere.

  • With no control domain, there's no LDoms Manager, which precludes any monitoring or reconfiguration of the guests. It also precludes capabilities such as domain mobility (i.e. migration) and power management.

  • All IO used by the guest must continue to be available – i.e. If the control domain is also operating as an IO service domain, those IO devices being served by the control domain will cease to be available for the duration that the control domain is down.

  • FMA (the Solaris Fault Management Architecture) will be unavailable

  • Certain Sun as well as third party management tools require access to the control domain, if the control domain goes down, those tools will have degraded capability

In terms of "locking" or severely limiting access to the control domain, that is certainly possible, but would be subject to its own set of constraints:

  • Without control domain access, there is no console access to the guests unless the console service is hosted elsewhere.

  • There's no way to interact with the LDoms Manager directly, which limits the ability to monitor, manage, or reconfigure the guests. The current lack of a suitable standalone LDoms management capability exacerbates this issue.

  • The inability to login to the control domain makes it extremely difficult to discover or manage any I/O (e.g. disks & network interfaces) bound to that domain.

  • Certain Sun as well as third party management tools require access to the control domain, if the control domain is locked down, those tools will have degraded capability

The control domain is usually configured as a service domain. In that case,the control domain needs to be up and running in order to provide service for virtual devices used by guest domains. If the control domain is down then access to virtual devices is suspended until the control domain comes back up.

On appropriate platforms, I/O domains can be created and used as service domains instead of using the control domain as a service domain. That way, guest domains will not depend on the control domain to access their virtual devices.


Jim Laurent is an Oracle Sales consultant based in Reston, Virginia. He supports US DoD customers as part of the North American Public Sector hardware organization. With over 17 years experience at Sun and Oracle, he specializes in Solaris and server technologies. Prior to Oracle, Jim worked 11 years for Gould Computer Systems (later known as Encore).


« April 2014