Friday Mar 18, 2016

Root domains and I/O on SPARC M7

Please see the excellent blog entry on root domains and how they've changed (for the better) on SPARC M7 servers at the blog article Complex Root Domains. The article refers to SR-IOV but doesn't discuss it, in order to focus on root domains, but SR-IOV also remains available on M7 systems for physical I/O with high resource granularity.

Wednesday Nov 11, 2015

Virtual HBA in Oracle VM Server for SPARC

Oracle VM Server for SPARC 3.3 added an important new feature, virtual HBA (vHBA), which adds flexibility and relieves prior limitations of virtual I/O without sacrificing performance. This blog entry describes this new feature and shows how to use it.
[Read More]

Thursday Oct 08, 2015

Oracle VM Server for SPARC Best practices: naming virtual network devices

This blog shows a simple usability best practice to make it easier to identify network resources using 'ldm list-netdev'.[Read More]

Thursday Apr 09, 2015

Oracle VM Server for SPARC 3.2 - Enhanced Virtual Disk Multipathing

Last month, Oracle released Oracle VM Server for SPARC release 3.2 which includes numerous enhancements. One of these is improvement for virtual disk multipathing, which provides redundant paths to virtual disk so that disk access continues even if a path or service domain fails.

Multipath groups are arranged in an active/standby pair of connections to the same physical media. In case of a path or service domain failure, I/O activity continues on a surviving path. This is also helpful for rolling upgrades: a service domain can be rebooted for an upgrade, and virtual disk I/O continues without interruption. That's important for continuous availability while upgrading system software.

A previous limitation was that you could not determine by commands which path was active, and you couldn't force activity onto a selected path. That meant that all the I/O for multiple virtual disks went (typically) to the primary path instead of being load balanced across service domains and HBAs. You could deduce which service domains were actively doing disk I/O by using commands like iostat, but there was no visibility, and no way to spread the load. Oracle VM Server for SPARC addresses this by adding command output that shows which path is active, and let you switch the active path to one of the available paths. Now, the command 'ldm list-bindings' shows which path is active, and the command 'ldm set-vdisk' lets you set which path is active. For further details and syntax, please see the documentation at Configuring Virtual Disk Multipathing

Monday Mar 23, 2015

Oracle VM Server for SPARC 3.2 - Live Migration

Oracle has just released Oracle VM Server for SPARC release 3.2. This update has been integrated into Oracle Solaris 11.2 beginning with SRU 8.4. Please refer to Oracle Solaris 11.2 Support Repository Updates (SRU) Index [ID 1672221.1]. 

This new release introduces the following features:

  • Improved multipath virtual disk I/O (mpgroup): view, set active I/O path
  • Improved domain observability to show dependencies between service and guest domains
  • Improved network observability, quality of service and security management, and PVLAN
  • I/O Resiliency (IOR) for physical I/O
  • Dynamic Bus (dynamically assign PCI bus to domains)
  • Live migration improvements
  • Guest additions (VM API to interact with host environment)
  • Guest access to SPARC performance counters

Live migration performance and security enhancements

This blog entry details 3.2 improvements to live migration. Oracle VM Server for SPARC has supported live migration since release 2.1, and has been enhanced over time to provide features like cross-CPU live migration to permit migrating domains across different SPARC CPU server types. Oracle VM Server for SPARC 3.2 improves live migration performance and security.

Live migration performance

The time to migrate a domain is reduced in Oracle VM Server for SPARC 3.2 by the following improvements:

  • Parallel page copying and memory mapped I/O: data compression and transmission were alredy multi-threaded, but copying from hypervisor memory was single-threaded and buffers were copied twice. This change adds worker threads for parallelism and reduces the number of times data is copied, including for network I/O
  • LZJB used to compress: Memory is compressed before it is encrypted and transmitted over the network. This change uses the fast, lightweight LZJB (Lempel Zev Jeff Bonwick) algorithm to quickly compress and decompress memory pages. Zero-fill pages are skipped, and pages that are only slightly reduced in size are sent unchanged. That reduces overall processing time.
These and other changes reduce overall migration time, reduce domain suspension time (the time at the end of migration when the domain is paused to retransmit the last remaining pages). and reduces CPU utilization. In my own testing I've seen speedups from 50% to 500% faster migration depending on the guest domain activity and memory size. Others may experience different times, depending on network and CPU speeds and domain configuration.

This improvement is available on all SPARC servers supporting Oracle VM Server for SPARC, including the older UltraSPARC T2, UltraSPARC T2 Plus, and SPARC T3 systems. Some speedups are only be available for guest domains running Solaris 11.2 SRU 8 or later, and will not be available on Solaris 10. Solaris 10 guests must run Solaris 10/09 or later, as that release introduced code for cooperative live migration that works with the hypervisor.

Live migration security

Oracle VM Server for SPARC 3.2 improves live migration security by adding certificate-based authentication and supporting the FIPS 140-2 standard.

Certificate based authentication

Live migration requires mutual authentication between the source and target servers. The simplest way to initiate live migration is to issue an "ldm migrate" command on the source system specifying an adminstrator password on the target system, or point to a root-readable file containing the target system's password. That is cumbersome, and not ideal for security. Oracle VM Server for SPARC 3.2 adds a secure, scalable way to permit password-less live migration using certificates that prevents man-in-the-middle attacks.

This is accomplished by using SSL certificates to establish a trust relationship between different server's control domainss as described at Configuring SSL Certificates for Migration. In brief, a certificate is securely copied from the remote system's /var/opt/SUNWldm/server.crt to the local system's /var/opt/SUNWldm/trust and a symbolic is made from certificate in the ldmd trusted certificate directory to /etc/certs/CA. After the certificate and ldmd services are restarted, the two control domains can securely communicate with one another without passwords. This enhancement is available on all servers supporting Oracle VM Server for SPARC, using either Solaris 10 or Solaris 11.

FIPS 140-2 Mode

The Oracle VM Server for SPARC Logical Domains Manager can be configured to perform domain migrations using the Oracle Solaris FIPS 140-2 certified OpenSSL libraries as described at When this is in effect, migrations are conformant with this standard, and can only done between servers that are all in FIPS 140-2 mode.

For more information, please see Using a FIPS 140 Enabled System in Oracle® Solaris 11.2. This enhancement requires that the control domain run Oracle Solaris 11.2 SRU 8.4 or later.

Where to get more information

For additional resources about Oracle VM Server for SPARC 3.2, please see the documentation at, especially the What's New page, the Release Notes and the Administration Guide

Friday Oct 17, 2014

Oracle VM Server for SPARC Best Practices White Paper

I'm very pleased to announce a new white paper has been published: Oracle VM Server for SPARC Best Practices.

This paper shows how to configure to meet demanding performance and availability requirements. Topics include:

  • Oracle VM Server for SPARC definitions, concepts and deployment options.
  • Software, hardware, and firmware requirements.
  • Best Practices for optimal performance.
  • Best Practices for resiliency and availability.

The paper includes specific recommendations, describes the reasons behind them, and illustrates them with examples taken from actual systems.

Tuesday Sep 23, 2014

Oracle VM Server for SPARC Released

A new maintenance release to Oracle VM Server for SPARC has been released, providing several enhancements described in the What's New page. This update adds support for private VLANs and relieves virtual I/O scalability constraints. This was already announced in the Virtualization Blog, but the I/O scalability improvement deserves further discussion.

Previous blog entries have described scalability improvements that improve virtual disk and network I/O performance. This new update adds scalability in a different context, by increasing the number of virtual I/O devices a domain can have.

Every virtual I/O device requires a Logical Domain Channel (LDC) endpoint. Previous product versions had a limit of 768 LDCs (or 512 on UltraSPARC T2 systems) per domain (not per system) that constrained growth. This set a maximum number of virtual I/O devices in a domain, which impeded migration of large configurations that might have hundreds of disk devices or network connections. While this could be addressed in a number of ways, such as using physical I/O or consolidating many small LUNs onto fewer large LUNs, it was an impediment to adopting Oracle VM Server for SPARC. It especially affected how service domains could be used, since each service domain has LDC endpoints for each of the virtual devices it provides to guests.

With this new update, and with associated system firmware levels, LDC endpoints are arranged into a large pool which can be shared among domains. As described in Using Logical Domain Channels, each domain can have 1,984 LDC endpoints on SPARC T4, SPARC T5, M5, and M6 systems, out of a pool of 98,304 LDC endpoints in total. The required system firmware to support the LDC endpoint pool is 8.5.1.b for SPARC T4 and 9.2.1.b for SPARC T5, SPARC M5, and SPARC M6.

This more than doubles the number of I/O devices available to a guest domain, and can be implemented by installing the current firmware and moving to the Oracle VM Server for SPARC update.

Wednesday Aug 27, 2014

Best Practices for Oracle Solaris Network Performance with Oracle VM Server for SPARC

A new document has been published on OTN: "How to Get the Best Performance from Oracle VM Server for SPARC" by Jon Anderson, Pradhap Devarajan, Darrin Johnson, Narayana Janga, Raghuram Kothakota, Justin Hatch, Ravi Nallan, and Jeff Savit.

Wednesday Jul 02, 2014

Announcing Oracle VM 3.3

Oracle VM 3.3.1 was announced today, with many product improvements. This article highlights some of those, in particular the ones most of interest to SPARC customers. Bonus - a video![Read More]

Wednesday May 21, 2014

Virtual Disk Performance Improvement for Oracle VM Server for SPARC

A new Solaris update dramatically improves performance for virtual disks on Oracle VM Server for SPARC, providing near-native I/O performance. With this change, virtual I/O is suitable for the most demanding I/O intensive applications under Oracle VM Server for SPARC. Read the full article to see more details.[Read More]

Friday Mar 28, 2014

Best Practices - Top Ten Tuning Tips Updated

Oracle VM Server for SPARC can be configured to provide optimal CPU and I/O performance - this blog entry updates a previous version to reflect improvements and new capabilities introduced into the product.[Read More]

Thursday Mar 20, 2014

Oracle VM Server for SPARC 3.1.1 Now Available

Oracle VM Server for SPARC 3.1.1 is now available. This release extends SR-IOV function to Fibre Channel devices, and adds support for controlling guest network bandwidth.[Read More]

Thursday Nov 21, 2013

Disk dynamic reconfiguration in Oracle VM Server for SPARC

Oracle VM Server for SPARC lets you add and remove I/O devices in guest domains (virtual machines) while they are running. This blog entry shows a simple example of adding and removing a disk while the guest runs.[Read More]

Thursday Oct 24, 2013

Best Practices - updated: which domain types should be used to run applications

Continued enhancements to SPARC servers and Oracle VM Server for SPARC open up options for deploying applications with the highest performance requirements. This blog updates an article originally published in October 2012, and reviews domain roles and describes how to choose when to use guest domains with virtual I/O, and I/O domains that use physical I/O.[Read More]

Sunday Oct 13, 2013

Availability Best Practices - using a mirrored ZFS pool with virtual disks

Previous Availability Best Practices blog entries emphasized service domain and path resiliency, rather than redundancy for disk media. This blog shows how to configure a mirrored ZFS pool with virtual disks from different service domains and virtual disk timeouts to provide media and service domain resiliency.[Read More]



« April 2016