By user12611315 on Mar 29, 2014
1. Install the required software:
NOTE: Both Qlogic and Emulex cards are supported now, but only Ganymede class(16G) adapters support this feature.
Ensure the required Firmware installed in your platform. You can find the Firmware support information in the release notes at the following URL:
Ensure that the LDoms manager 3.1.1 software installed in the control domain. Note, the LDoms manager 3.1.1 software is automatically installed if the Solaris11.1 SRU17 or later installed in the control domain. You can verify this with "ldm -V" command.
NOTE: If you are using Qlogic adapter, then you need S11.2 installed in the Root domain that owns the corresponding PCIe bus.
Support for Solaris10 for both Root domain and IO domains is also available, please check with your support contact on this information. Using different OS versions(either S11+ or S10) in Root domain and in IO domains(either S11+ or S10) is supported as well. A very common config that we expect is the Root domain is installed with an S11 version(that supports the adapter you are using) and S10 in an IO domain with patches that support the VF driver for the adapter that you are using.
2. Update the FC HBA adapter Firmware
Update the FC HBA adapter Firmware to the version that supports FC SR-IOV.
The firmware for Emulex 16Gb HBAs can be found at:
Firmware for Qlogic 16Gb HBAs can be found at:
It is important to power cycle the system(actually the HBA that needs the power cycle) after updating the Firmware.
3. FC Connection requirements
Ensure that the FC SR-IOV HBA is connected to a compatible FC Switch that supports NPIV. It is very important ensure that this condition is met. This feature is not supported if the FC port is directly connected to the storage.
4. Verify FC Physical Functions
Each FC HBA port shows up as one Physical Function. The FC PFs are named with "IOVFC.PFx" name, you can identify them in the output of "ldm list-io" command. The following is an example output:
# ldm ls-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ pci_0 BUS pci_0 primary IOV pci_1 BUS pci_1 primary IOV niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary EMP /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 primary EMP /SYS/MB/PCIE3 PCIE pci_1 primary OCC /SYS/MB/PCIE5 PCIE pci_1 primary OCC /SYS/MB/PCIE7 PCIE pci_1 primary OCC /SYS/MB/PCIE9 PCIE pci_1 primary OCC /SYS/MB/NET2 PCIE pci_1 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 primary /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 primary /SYS/MB/PCIE7/IOVFC.PF0 PF pci_1 primary /SYS/MB/PCIE7/IOVFC.PF1 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_1 primary
5. Understand the capabilities of each FC Physical Function
FC Physical functions have only one detail that is, the maximum number of VFs supported by it. You can use the "ldm list -l <pf-name>" to find this information. For example:
# ldm list-io -l /SYS/MB/PCIE7/IOVFC.PF0 /SYS/MB/PCIE7/IOVFC.PF0 PF pci_1 primary [pci@500/pci@1/pci@0/pci@6/SUNW,emlxs@0] maxvfs = 8
6. Create the Virtual Functions(VFs)
We recommend to create all VFs in one step, this is an optimized method of creating the VFs and use them as needed. There is no performance penalty if some VFs are not used. You can use "ldm create-vf" command to accomplish this. For example:
# ldm create-vf -n max /SYS/MB/PCIE7/IOVFC.PF0 Created new vf: /SYS/MB/PCIE7/IOVFC.PF0.VF0 Created new vf: /SYS/MB/PCIE7/IOVFC.PF0.VF1 Created new vf: /SYS/MB/PCIE7/IOVFC.PF0.VF2 Created new vf: /SYS/MB/PCIE7/IOVFC.PF0.VF3 Created new vf: /SYS/MB/PCIE7/IOVFC.PF0.VF4 Created new vf: /SYS/MB/PCIE7/IOVFC.PF0.VF5 Created new vf: /SYS/MB/PCIE7/IOVFC.PF0.VF6 Created new vf: /SYS/MB/PCIE7/IOVFC.PF0.VF7
# ldm start-reconf <root domain> # ldm set-io iov=on pci_X # ldm create-vf -n max <PF-name> # <reboot the root domain to effect the changes>
7. Understand VF WWNs assignment
The LDoms manager assigns a Port-WWN and a Node-WWN automatically to each FC VF. The auto-allocated WWNs are unique only if all of SPARC systems connected to a given SAN fabric are also connected in the same ethernet multicast domain. If not, they won't be unique. Also, if you ever destroy and recreate the VFs, theymay not get the same WWNs. As you may use these WWNs for Zoning or Lun masking, we would recommend using manual WWN allocation. See the following from admin guide for more details.
# ldm set-io port-wwn=<Port WWN> node-wwn=<Node WWN> <vf-name>
8. Configuration of SAN Storage
Configure your SAN storage to assign LUNs to each VFs. It is highly recommend to use LUN masking such that the LUNs are visible only the VF to which they are assigned. This is no different from how LUNs are assigned to different HBAs on different systems. NOTE: One important point to notice here is that you can assign LUNs IO domains such that they are not even visible in the Root domain. This produces equally same level of secure access that you get with different set of HBAs. This is not possible with virtual I/O methods.
9. Assigning VFs to Logical Domains
You can now assign VFs to Logical domains using the "add-io" command. For example, the following commands assign VFs to three different domain.
# ldm add-io /SYS/MB/PCIE7/IOVFC.PF0.VF0 ldg0 # ldm add-io /SYS/MB/PCIE7/IOVFC.PF0.VF1 ldg1 # ldm add-io /SYS/MB/PCIE7/IOVFC.PF0.VF2 ldg2
Make sure to setup the failure-policy settings to handle any unexpected Root domain reboot/crash cases. The following failure-policy will cause the IO domains(here ldg0, ldg1, ldg2) to be reset along with the Root domain.
# ldm set-domain failure-policy=reset primary # ldm set-domain master=primary ldg0 # ldm set-domain master=primary ldg1 # ldm set-domain master=primary ldg2
NOTE: You can dynamically assign VFs too. That is, if the given Logical domain is running the required OS version, you can simply run the same commands to dynamically add the VFs. The IO domain OS requirements are same as what is mentioned for the Root domains, which you can find at:
10. Using the VFs in IO domains
If you statically added the VFs, then you can now start the IO domains and use the VFs like any FC HBA. For example, at OBP prompt of the IO domain, you can perform the "probe-scsi-all" to see all LUNs visible to that VF. Installing the OS and booting from a LUN is fully supported. Features like MPxIO are full supported. For example, you can assign VFs from different Root Domains and configure MPxIO.
You can find the detailed documentation in the OVM Server for SPARC 3.1.1 Admin Guide.
FC SR-IOV Limitations
There are a few limitations that needs to be understood. These are:
- No support when the HBA is connected to the storage directly.
- No NPIV support on top of Virtual functions. NPIV on the physical function is supported as usual.