Server Virtualization - Creating IO Domains on T5440
By pallab on Oct 13, 2008
If we refer to the System Topology diagram in my previous blog,
we find that the internal disks of T5440 are connected to PCIe-0. Hence
it is not possible to remove the PCIe-0 from the Primary (or Control)
Domain. However it is possible to remove PCIe-1, PCIe-2 and PCIe-3 from
the Primary Domain and allocate them to IO Domains.
In order to create a IO-domain using PCIe-1, it has to be removed from Primary Domain. This would cause the Primary Domain to lose its primary network interface if it has been using the On-board NICs. However if there was a network card available on PCIe-0, then the primary network for Primary Domain can be switched to the ports on the network card before removing PCIe-1 from Primary Domain. If an additional network card is not available, it should still be possible to remove PCIe-1 from Primary Domain and create a IO domain (let us call it Secondary Domain) managing devices off PCIe-1. In such a case, the Primary Domain would provide the boot-disk service to the Secondary Domain and the Secondary Domain would provide the primary network service for the Primary Domain. The Pseudo-steps below outlines how this can be done.
- In the Primary Domain
- set the number of VCPUs to 8 (this is just an example number of VCPUs)
- set the memory to 8GB (just an example size of memory)
- create a vdisk-server
- remove PCIe-1 from its control
- This would cause the Primary Domain to lose its network after reboot
- Reboot the Primary Domain and log back into the Primary Domain from Console
- To cause VCPUs for Secondary Domain to be allocated from T1 (refer to the Topology Diagaram), create a dummy domain with the rest of 56 VCPUs from T0. Bind the dummy domain.
- Associate a vdiskserverdevice as the boot-device for Secondary Domain
- Create the Secondary Domain
- set the number of VCPUs to 8
- set the memory to 8GB
- add PCIe-1 to it
- add the vdiskserverdevice as the vdisk for this domain
- Bind, install-OS and boot the domain
- Create a vswitch-device on the Secondary Domain
- Reboot the Secondary Domain
- Create a vnet-device for the Primary Domain associated with the above vswitch-device
- Plumb and configure the vnet device on the Primary Domain (assumingthe On-Board network ports are connected to the primary network of the Data Center) Now the Primary Domain should have the primary network available.
- Remove the dummy domain and proceed with creating other domains.
With the above technique, when the Primary Domain is rebooted, the Secondary Domain may seem to pause until the Primary Domain boots back. Similarly when the Secondary Domain is rebooted, the Primary Domain's primary network may appear to freeze until the Secondary Domain comes back online. But that is far better than losing all the domains and the applications running in those domains.