Tuesday Aug 20, 2013

OVM Server for SPARC 3.1 introduces Dynamic SR-IOV feature

OVM Server for SPARC 3.1 introduces a great enhancement to the PCIe SR-IOV feature. Until now, creating/destroying SR-IOV Virtual Functions(VFs) in a static way, that is it requires a reboot of the root domain and adding and removing VFs requires the Guest domain to be stopped. Rebooting root domains can be disruptive as it impacts the I/O domains that depend on it. Stopping Guest domain to add or remove a VF is also disruptive to the applications running in it.  OVM Server for SPARC 3.1 enhances the PCIe SR-IOV feature with Dynamic SR-IOV for Ethernet SR-IOV devices that removes these restrictions. That is now we can create/destroy VFs while a root domain is running and we can also add/remove VFs from a Guest domain while it is running without impacting applications. Note, OVM Server for SPARC 3.1 introduces another feature named "Non-Primary Root domains" which extends the PCIe SR-IOV and Dynamic SR-IOV features to all Root domains along with Primary domain. That is, you can perform all Dynamic SR-IOV operations on Non-Primary Root domains also.

This feature is supported only when OVM Server for SPARC 3.1 LDoms manager, corresponding System Firmware and the supported OS version. You can refer to the OVM Server for SPARC 3.1 release notes for the exact information the supported OS version and System Firmware versions. The dynamic IOV is enabled for a given logical domain only if all of the s/w components are installed and other configuration requirements are met. If not, you can always use the static method to accomplish your changes.

There is still one operation that we could not make it dynamic yet, that is enabling IO virtualization for a given PCIe bus. For now, IO virtualization for a given PCIe bus needs to be enabled ahead of time. That is, the following need to be done ahead of time. Note, if you are planning to create VFs at the same time, it is a good idea to create VFs at the same time as well as you would be rebooting any way. The following steps enables IO virtualization for a PCIe bus. Note, this needs to be done only once for a PCIe bus, of course while it is assigned to a Root domain.

Enable I/O Virtualization for a PCIe bus:

  • If the PCIe bus is already assigned to a domain: 
    • # ldm start-reconf <root-domain-name>
    • # ldm set-io iov=on <PCIe bus name>
    • # reboot  
  • You can also enable it while adding a PCIe bus a logical domain:
    • # ldm add-io iov=on <PCIe bus name> <domain-name>
  • You can check if the IOV is enabled for a given PCIe bus with "ldm list-io <PCIe bus name>".

Dynamically create or destroy VFs:

  • Once the IOV is enabled for a given PCIe bus, you can create or destroy VFs with create-vf and destroy-vf without requiring the delayed reconfiguration and a reboot. But this requires the Physical Function network device to be either not plumbed or in a multi-path configuration. The dynamic create/destroy VF operations perform hotplug offline and online operations on the PF device, that causes the PF to detached and re-attached. As a result the  PF device need to be either not plumbed(that is not in use) or in a multipathed(IPMP or aggr) configuration so that hotplug offline/online operations are successful. If that is not the case, the dynamic create/destroy operations will fail reporting that the device is  busy, in such case you can use the static method to create/destroy VFs
    • # ldm create-vf <pf-name>
    • # ldm destroy-vf <vf-name>

Dynamically add or remove VFs:

  • You can now  dynamically add and remove VFs to/from a logical domain. All you would need to run is 'add-io' and 'remove-io' commands without stopping the Guest domain.
    • # ldm add-io <vf-name>  <domain-name>
    • # ldm remove-io <vf-name> <domain-name>

Trouble shooting:

  • The dynamic SR-IOV operations are disabled:
    • Check if the System Firmware that is released with OVM Server for SPARC 3.1 is installed on your system.
    • Check if the OVM Server for SPARC 3.1 LDoms manager is installed.
    • Check if the given domain has the required OS version supported. Check the OVM Server for SPARC 3.1 release notes for this information.
    • Check if IOV is enabled for the given PCIe bus, use "ldm ls-io <PCIe bus name>" to check this.
  • The create-vf or destroy-vf  failed to dynamically perform the operation:
    • Verify if the network device that maps to the PF is not plumbed or in a multipath(Aggr or IPMP). 
      • You can obtain the device path for the PF using 'ldm ls-io -l <pf-name>" and then map it to the corresponding device in the root domain by grepping for it in /etc/path_to_inst. Then use "dladm show-phys" command map that to the netX device name.
  • Dynamically removing a VF from a Guest domain fails
    • Ensure the VF is not in use or in a multi-path configuration.


Raghuram Kothakota


« July 2016