Using NPIV with Oracle VM Server for SPARC
By Alexandre Chartre on Apr 19, 2012
Oracle Solaris 11 provides native support for NPIV. Although NPIV is not directly integrated with Oracle VM Server SPARC (that is LDoms), you can still take advantage of NPIV with LDoms.
NPIV means N_Port ID Virtualization, and is a feature provided with Fiber Channel (FC) Storage Area Network (SAN). The exact definition, from the Technical Committee T11, is that "NPIV provides a Fiber Channel facility for sharing a single physical N_Port among multiple N_Port IDs, thereby allowing multiple initiators, each with its own N_Port ID, to share the N_Port."
More simply put, in a FC SAN environment, a Fabric, such as a SAN switch, is connected to some nodes; for example, a node can be a Host Bus Adapter (HBA), or a storage array. Each node has a N_Port (Node Port) that provides an attachment point to connect with the fabric. Each N_Port has a fabric unique identifier, the N_Port ID, by which an N_Port is uniquely known. NPIV allows a HBA to have multiple N_Port ID associated with a physical port of the HBA. That way, a HBA can have multiple virtual ports instead of a single physical port.
The benefit of NPIV is that you can then configure your SAN environment (such as zoning, LUN masking or QoS) using virtual ports instead of physical ports.This provides a lot of flexibility as you can then easily transfer a configuration from one physical HBA to another by just configuring a new HBA with the appropriate N_Port ID.
In a virtual environment, it can be convenient to associate a virtual port with a virtual machine, and to have all LUNs visible through this virtual port assigned to the virtual machine. By doing so, you can then manage the association of LUNs and virtual machines through the configuration of your SAN.
As mentioned before, NPIV is a feature provided with FC SAN. In order to use NPIV, you need to have the appropriate hardware and software that supports NPIV; in particular you will need:
- A FC SAN switch that supports NPIV — The switch has to be configured with a switched-fabric topology (FC-SW). NPIV is not available with arbitrary-loop topology (FC-AL).
- A FC HBA that supports NPIV — The HBA has to be connected to a SAN switch that supports NPIV. You can not use NPIV if the HBA is directly connected to a FC storage array.
- Oracle Solaris 11 — To use NPIV with LDoms, Solaris 11 has to be installed on the service domain with the FC HBA. Other domains can run Solaris 10 or Solaris 11.
Using NPIV with LDoms
Here are the different steps to use NPIV with LDoms, and to how to assign LUNs visible through a NPIV port to a LDoms guest domain.
Identify Existing Physical FC HBA Ports
The first step is to identify the different HBA ports available on the LDoms service domain, and select the physical port you want to create a virtual port on. Use the fcinfo command to get information about FC HBA ports. The command will provide information about physical and virtual ports.
# fcinfo hba-port HBA Port WWN: 210000e08b899a79 Port Mode: Initiator Port ID: 10400 OS Device Name: /dev/cfg/c4 Manufacturer: QLogic Corp. Model: 375-3356-01 Firmware Version: 05.04.03 FCode/BIOS Version: BIOS: 1.04; fcode: 1.11; EFI: 1.00; Serial Number: RFC0612G20954 Driver Name: qlc Driver Version: 20110321-3.05 Type: N-port State: online Supported Speeds: 1Gb 2Gb 4Gb Current Speed: 4Gb Node WWN: 200000e08b899a79 Max NPIV Ports: 127 NPIV port list: HBA Port WWN: 210100e08ba99a79 Port Mode: Initiator Port ID: 0 OS Device Name: /dev/cfg/c5 Manufacturer: QLogic Corp. Model: 375-3356-01 Firmware Version: 05.04.03 FCode/BIOS Version: BIOS: 1.04; fcode: 1.11; EFI: 1.00; Serial Number: RFC0612G20954 Driver Name: qlc Driver Version: 20110321-3.05 Type: unknown State: offline Supported Speeds: 1Gb 2Gb 4Gb Current Speed: not established Node WWN: 200100e08ba99a79 Max NPIV Ports: 127 NPIV port list:
In this example, we have two physical ports (the system is configured with a dual channel HBA card), and each port has no virtual port ("NPIV port list" is empty). We are going to create a virtual port on the physical port identified with the World-Wide Number (WWN) 210000e08b899a79.
Create a NPIV Port
Once you have identified the physical port where you want to create a virtual port on, use the fcadm command to create a new virtual port with NPIV.
# fcadm create-npiv-port 210000e08b899a79
Then use fcinfo to look at the port that was created:
# fcinfo hba-port 210000e08b899a79 HBA Port WWN: 210000e08b899a79 Port Mode: Initiator Port ID: 10400 OS Device Name: /dev/cfg/c4 Manufacturer: QLogic Corp. Model: 375-3356-01 Firmware Version: 05.04.03 FCode/BIOS Version: BIOS: 1.04; fcode: 1.11; EFI: 1.00; Serial Number: RFC0612G20954 Driver Name: qlc Driver Version: 20110321-3.05 Type: N-port State: online Supported Speeds: 1Gb 2Gb 4Gb Current Speed: 4Gb Node WWN: 200000e08b899a79 Max NPIV Ports: 127 NPIV port list: Virtual Port1: Node WWN: c0007d39993d1d61 Port WWN: c0007d39993d1d60
As you can see, we now have one virtual port.The virtual port has a node and port WWN. By default, these WWN are randomly generated. However if you want to use a specific WWN, you can specify the node WWN with the -n option, and the port WWN with the -p option. The general syntax is:
fcadm create-npiv-port [-p Virtual_Port_WWN] [-n Virtual_Node_WWN] Physical_Port_WWN
Identify LUNs Visible through the NPIV Port
After the virtual port has been created, you have to make sure that your SAN is properly configured (with zoning or LUN masking) so that the new virtual port can have access to some target devices (LUNs, storage array...). Then you can identify which LUNs after visible through the new virtual port using the fcinfo command:
# fcinfo remote-port -s -p c0007d39993d1d60 Remote Port WWN: 256000c0ffc089d5 Active FC4 Types: SCSI SCSI Target: yes Port Symbolic Name: Node WWN: 206000c0ff0089d5 LUN: 0 Vendor: SUN Product: StorEdge 3511 OS Device Name: /dev/rdsk/c1t256000C0FFC089D5d0s2 LUN: 1 Vendor: SUN Product: StorEdge 3511 OS Device Name: /dev/rdsk/c1t256000C0FFC089D5d1s2
In our example, two LUNs are visible through the virtual port: c1t256000C0FFC089D5d0s2 and c1t256000C0FFC089D5d1s2.
Virtualize LUNs Visible through the NPIV Port
Once you have identified which LUNs are visible through the NPIV port, you can virtualize and assign them to a LDoms guest domain using regular LDoms commands (ldm add-vdsdev and ldm add-vdisk). For example:
# ldm add-vdsdev /dev/rdsk/c1t256000C0FFC089D5d0s2 lun0@primary-vds0 # ldm add-vdsdev /dev/rdsk/c1t256000C0FFC089D5d1s2 lun1@primary-vds0 # ldm add-vdisk vdisk0 lun0@primary-vds0 myguest # ldm add-vdisk vdisk1 lun1@primary-vds0 myguest