Cross-CPU migration in LDoms 3.1
By Eric S on Aug 20, 2013
Oracle VM Server for SPARC (aka LDoms) 3.1 is now available. You can read all about it over at the Virtualization Blog. I want to focus on the enhancements we've made to cross-CPU migration.
Cross-CPU migration is the ability to migrate a running virtual machine (VM) from a SPARC server with one type of CPU chip to another of a different type. Prior to LDoms 3.1, the only way to perform such a migration was to specify a "cpu-arch" property of "generic" when first defining the VM. Doing so allowed VMs to be migrated between systems based on the T2, T2+, T3 and T4 processors. However, specifying this property results in the VM running without taking advantage of all the features of the underlying CPU, negatively impacting performance. The most serious impact was loss of hardware-based crypto acceleration on the T4 processors. This was unfortunately necessary, due to major changes in HW crypto acceleration introduced in T4.
With the release of our T5 and M5 processors and systems, we didn't want our customers to continue to suffer this penalty when performing cross-cpu migrations amongst these newer platforms. In order to accomplish that, we have defined a new migration "class" (i.e. another value for the "cpu-arch" property), named "migration-class1". With LDoms 3.1, you can now specify "cpu-arch=migration-class1" on your T4, T5, and M5 based systems, allowing those VMs to be migrated across systems based on those 3 processor families. And most important of all, your VMs will run with full HW crypto capabilities (not to mention supporting the larger page sizes available in these processors)!
Ideally we would have provided this new migration class at the time T5 and M5 were first made available, but for various reasons that was not feasible. It is only available now, with LDoms 3.1. And as with any new LDoms release, there are minimum Solaris and firmware versions required to get all the features of the release; please consult the relevant documentation for details.
Note that the "migration-class1" family does _not_ include the T2, T2+ or T3 processors. To perform cross-CPU migration between those processor families (or between them and T4), you must continue to use the "generic" migration class. In addition, live migrations from T2/T2+/T3 systems to T5/M5 systems (and vice-versa) are not supported. Now, you probably noticed that T4 is in both the "generic" and "migration-class1" families, so you might be tempted to migrate from (say) T2 to T5 in 2 hops, T2->T4 then T4->T5. The problem with that approach is that you'll need to switch the migration class of the VM after it is migrated to the T4 platform. Doing so requires the VM to be restarted, negating the benefit of live migration. For this reason, we recommend cold migration in this scenario.
Update: In a comment below, I state that "it is not possible to migrate from an M5 or T5 platform to an M10 platform (migrating the other way does work)." I am happy to report that live migration from T5/M5 to M10 is supported with M10 firmware XCP2210 or later.