Cross-CPU migration in LDoms 3.1

Oracle VM Server for SPARC (aka LDoms) 3.1 is now available. You can read all about it over at the Virtualization Blog. I want to focus on the enhancements we've made to cross-CPU migration.

Cross-CPU migration is the ability to migrate a running virtual machine (VM) from a SPARC server with one type of CPU chip to another of a different type. Prior to LDoms 3.1, the only way to perform such a migration was to specify a "cpu-arch" property of "generic" when first defining the VM. Doing so allowed VMs to be migrated between systems based on the T2, T2+, T3 and T4 processors. However, specifying this property results in the VM running without taking advantage of all the features of the underlying CPU, negatively impacting performance. The most serious impact was loss of hardware-based crypto acceleration on the T4 processors. This was unfortunately necessary, due to major changes in HW crypto acceleration introduced in T4.

With the release of our T5 and M5 processors and systems, we didn't want our customers to continue to suffer this penalty when performing cross-cpu migrations amongst these newer platforms. In order to accomplish that, we have defined a new migration "class" (i.e. another value for the "cpu-arch" property), named "migration-class1". With LDoms 3.1, you can now specify "cpu-arch=migration-class1" on your T4, T5, and M5 based systems, allowing those VMs to be migrated across systems based on those 3 processor families. And most important of all, your VMs will run with full HW crypto capabilities (not to mention supporting the larger page sizes available in these processors)!

Ideally we would have provided this new migration class at the time T5 and M5 were first made available, but for various reasons that was not feasible. It is only available now, with LDoms 3.1. And as with any new LDoms release, there are minimum Solaris and firmware versions required to get all the features of the release; please consult the relevant documentation for details.

Note that the "migration-class1" family does _not_ include the T2, T2+ or T3 processors. To perform cross-CPU migration between those processor families (or between them and T4), you must continue to use the "generic" migration class. In addition, live migrations from T2/T2+/T3 systems to T5/M5 systems (and vice-versa) are not supported. Now, you probably noticed that T4 is in both the "generic" and "migration-class1" families, so you might be tempted to migrate from (say) T2 to T5 in 2 hops, T2->T4 then T4->T5. The problem with that approach is that you'll need to switch the migration class of the VM after it is migrated to the T4 platform. Doing so requires the VM to be restarted, negating the benefit of live migration. For this reason, we recommend cold migration in this scenario.

Update: In a comment below, I state that "it is not possible to migrate from an M5 or T5 platform to an M10 platform (migrating the other way does work)." I am happy to report that live migration from T5/M5 to M10 is supported with M10 firmware XCP2210 or later.


What about M10 servers from Fujitsu? Is it possible to migrate from T4/M5 towards M10 and vice versa?

Posted by steinweo on September 01, 2013 at 04:25 PM EDT #

Hi steinweo, yes it is possible to migrate between Oracle SPARC servers and the SPARC M10 servers from Fujitsu (with one specific exception; see below). In order to allow such migrations, the cpu-arch property for the domain must be set to "generic". As noted above, this will have some negative impacts on performance due to not being able to take advantage of all the capabilities of the underlying platform.

Also, due to a known incompatibility between the current versions of firmware between the platforms, it is not possible to migrate from an M5 or T5 platform to an M10 platform (migrating the other way does work). This is because T5 & M5 are based on a newer firmware base than is M10. If and when M10 migrates to the newer base, migrations from T5/M5 to M10 will work.

Posted by guest on September 03, 2013 at 03:31 PM EDT #

Also, keep in mind that the "generic" cpu-arch setting supports platforms back to T2, so that could mean a substantial performance penalty on the newer platforms, making migrations between T4/T5/M5 & M10 simply not worth the performance tradeoff.

Posted by Eric on September 03, 2013 at 05:10 PM EDT #

Post a Comment:
Comments are closed for this entry.

I work on the Oracle VM Server for SPARC (nee LDoms) team.

View Eric Sharakan's profile on LinkedIn


« March 2015