Control domain reconfiguration in LDoms 1.0.1
By Eric S on Nov 01, 2007
- The control domain could only be reconfigured when running in the "factory-default" configuration. Once reconfigured, if subsequent changes were desired, one had to revert back to factory-default and re-apply the initial changes as well as any new ones.
- The only way to instantiate a newly reconfigured control domain was by downloading the new configuration to the SP, and then power-cycling the box.
When reconfiguring the control domain under LDoms 1.0.1, the LDom Manager enters "delayed reconfiguration" mode the first time it's asked to do something that can't be immediately instantiated (i.e. just about anything other than cpu DR and adding or removing disk volumes). Once in this mode, all subsequent operations are pended in the hypervisor until the control domain reboots. This delayed reconfiguration capability actually existed in LDoms 1.0, but could not be applied to the control domain because rebooting the control domain required a full powercycle of the system to make sure the I/O subsystem was properly reset, causing the loss of any pending operations queued up in the hypervisor.
LDoms 1.0.1 introduces the ability to soft reset the I/O subsystem, allowing the control domain (or any I/O domain) to reboot while the rest of the system stays up. This in turn allows delayed reconfiguration to work on the control domain.
Utilizing delayed reconfiguration mode for control domain reconfiguration also means the reconfiguration can take place at any time, not just when running in the factory-default configuration. This allows the control domain to be reconfigured as many times as needed without having to revert to the factory-default configuration and start over each time.
To facilitate control domain reconfiguration under LDoms 1.0, the LDom Manager ran in a special "config mode" when in the factory-default configuration. In this mode, all reconfiguration requests were simply queued up within the LDom Manager, so that the new config could be downloaded to the SP when ready, and instantiated by power-cycling the box. This mode is still utilized in Ldoms 1.0.1 on UltraSPARC T1 based platforms (when booted into its factory-default configuration). This is to support non-LDoms legacy compatibility mode, since these platforms initially shipped before the advent of LDoms technology. On these systems, the first control domain reconfiguration has to be done the same way it was for 1.0.
All subsequent control domain reconfigurations (and all control domain reconfigurations on T2 based and all future LDoms-supported platforms, as they do not utilize config mode) can be accomplished via delayed reconfig operations followed by a simple reboot of the control domain.
In summary, under LDoms 1.0.1, you can now reconfigure your control domain whenever you want, as many times as you need, without having to revert to factory-default and re-apply all your previous changes, without having to save the configuration to the SP, and without having to power-cycle the box (except for the first-ever reconfiguration on a T1 based platform).
The ability to reboot the control domain without the box power-cycling (aided with some magic I'll leave to Narayan to describe) deserves a little elaboration: it means you can truly reconfigure your control domain even with active guest domains running! The guest domains stay up during the ensuing control domain reboot; VIO services are pended & automatically re-established as the control domain comes back up.
One very important note about saving your domain configuration to the SP: just because you no longer _need_ to save the new configuration to the SP before rebooting the control domain to affect a reconfiguration, doesn't mean you _shouldn't_ save it; you absolutely should! It's _strongly_ recommend to always save any new configuration you create to the SP, otherwise if the system were to lose power, it would revert to a previously saved state (or to factory-default), which is almost certainly _not_ what you want. BTW, you can safely save your configuration even if there are delayed reconfig operations pending; in this case, the configuration that gets saved is the pending one.