Thursday Apr 19, 2012

Using NPIV with Oracle VM Server for SPARC

Oracle Solaris 11 provides native support for NPIV. Although NPIV is not directly integrated with Oracle VM Server SPARC (that is LDoms), you can still take advantage of NPIV with LDoms.

NPIV Overview

NPIV means N_Port ID Virtualization, and is a feature provided with Fiber Channel (FC) Storage Area Network (SAN). The exact definition, from the Technical Committee T11, is that "NPIV provides a Fiber Channel facility for sharing a single physical N_Port among multiple N_Port IDs, thereby allowing multiple initiators, each with its own N_Port ID, to share the N_Port."

More simply put, in a FC SAN environment, a Fabric, such as a SAN switch, is connected to some nodes; for example, a node can be a Host Bus Adapter (HBA), or a storage array. Each node has a N_Port (Node Port) that provides an attachment point to connect with the fabric. Each N_Port has a fabric unique identifier, the N_Port ID, by which an N_Port is uniquely known. NPIV allows a HBA to have multiple N_Port ID associated with a physical port of the HBA. That way, a HBA can have multiple virtual ports instead of a single physical port.

The benefit of NPIV is that you can then configure your SAN environment (such as zoning, LUN masking or QoS) using virtual ports instead of physical ports.This provides a lot of flexibility as you can then easily transfer a configuration from one physical HBA to another by just configuring a new HBA with the appropriate N_Port ID.

In a virtual environment, it can be convenient to associate a virtual port with a virtual machine, and to have all LUNs visible through this virtual port assigned to the virtual machine. By doing so, you can then manage the association of LUNs and virtual machines through the configuration of your SAN.

NPIV Requirements

As mentioned before, NPIV is a feature provided with FC SAN. In order to use NPIV, you need to have the appropriate hardware and software that supports NPIV; in particular you will need:

  • A FC SAN switch that supports NPIV — The switch has to be configured with a switched-fabric topology (FC-SW). NPIV is not available with arbitrary-loop topology (FC-AL).
  • A FC HBA that supports NPIV — The HBA has to be connected to a SAN switch that supports NPIV. You can not use NPIV if the HBA is directly connected to a FC storage array.
  • Oracle Solaris 11 — To use NPIV with LDoms, Solaris 11 has to be installed on the service domain with the FC HBA. Other domains can run Solaris 10 or Solaris 11.

Using NPIV with LDoms

Here are the different steps to use NPIV with LDoms, and to how to assign LUNs visible through a NPIV port to a LDoms guest domain.

Identify Existing Physical FC HBA Ports

The first step is to identify the different HBA ports available on the LDoms service domain, and select the physical port you want to create a virtual port on. Use the fcinfo command to get information about FC HBA ports. The command will provide information about physical and virtual ports.

# fcinfo hba-port
HBA Port WWN: 210000e08b899a79
        Port Mode: Initiator
        Port ID: 10400
        OS Device Name: /dev/cfg/c4
        Manufacturer: QLogic Corp.
        Model: 375-3356-01
        Firmware Version: 05.04.03
        FCode/BIOS Version:  BIOS: 1.04; fcode: 1.11; EFI: 1.00;
        Serial Number: RFC0612G20954
        Driver Name: qlc
        Driver Version: 20110321-3.05
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 200000e08b899a79
        Max NPIV Ports: 127
        NPIV port list:
HBA Port WWN: 210100e08ba99a79
        Port Mode: Initiator
        Port ID: 0
        OS Device Name: /dev/cfg/c5
        Manufacturer: QLogic Corp.
        Model: 375-3356-01
        Firmware Version: 05.04.03
        FCode/BIOS Version:  BIOS: 1.04; fcode: 1.11; EFI: 1.00;
        Serial Number: RFC0612G20954
        Driver Name: qlc
        Driver Version: 20110321-3.05
        Type: unknown
        State: offline
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: not established
        Node WWN: 200100e08ba99a79
        Max NPIV Ports: 127
        NPIV port list:

In this example, we have two physical ports (the system is configured with a dual channel HBA card), and each port has no virtual port ("NPIV port list" is empty). We are going to create a virtual port on the physical port identified with the World-Wide Number (WWN) 210000e08b899a79.

Create a NPIV Port

Once you have identified the physical port where you want to create a virtual port on, use the fcadm command to create a new virtual port with NPIV.

# fcadm create-npiv-port 210000e08b899a79

Then use fcinfo to look at the port that was created:

# fcinfo hba-port 210000e08b899a79
HBA Port WWN: 210000e08b899a79
        Port Mode: Initiator
        Port ID: 10400
        OS Device Name: /dev/cfg/c4
        Manufacturer: QLogic Corp.
        Model: 375-3356-01
        Firmware Version: 05.04.03
        FCode/BIOS Version:  BIOS: 1.04; fcode: 1.11; EFI: 1.00;
        Serial Number: RFC0612G20954
        Driver Name: qlc
        Driver Version: 20110321-3.05
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 200000e08b899a79
        Max NPIV Ports: 127
        NPIV port list:
          Virtual Port1:
                Node WWN: c0007d39993d1d61
                Port WWN: c0007d39993d1d60

As you can see, we now have one virtual port.The virtual port has a node and port WWN. By default, these WWN are randomly generated. However if you want to use a specific WWN, you can specify the node WWN with the -n option, and the port WWN with the -p option. The general syntax is:

 fcadm create-npiv-port [-p Virtual_Port_WWN] [-n Virtual_Node_WWN] Physical_Port_WWN

Identify LUNs Visible through the NPIV Port

After the virtual port has been created, you have to make sure that your SAN is properly configured (with zoning or LUN masking) so that the new virtual port can have access to some target devices (LUNs, storage array...).  Then you can identify which LUNs after visible through the new virtual port using the fcinfo command:

# fcinfo remote-port -s -p c0007d39993d1d60
Remote Port WWN: 256000c0ffc089d5
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port Symbolic Name:
        Node WWN: 206000c0ff0089d5
        LUN: 0
          Vendor: SUN
          Product: StorEdge 3511
          OS Device Name: /dev/rdsk/c1t256000C0FFC089D5d0s2
        LUN: 1
          Vendor: SUN
          Product: StorEdge 3511
          OS Device Name: /dev/rdsk/c1t256000C0FFC089D5d1s2

In our example, two LUNs are visible through the virtual port: c1t256000C0FFC089D5d0s2 and c1t256000C0FFC089D5d1s2.

Virtualize LUNs Visible through the NPIV Port

Once you have identified which LUNs are visible through the NPIV port, you can virtualize and assign them to a LDoms guest domain using regular LDoms commands (ldm add-vdsdev and ldm add-vdisk). For example:

# ldm add-vdsdev /dev/rdsk/c1t256000C0FFC089D5d0s2 lun0@primary-vds0
# ldm add-vdsdev /dev/rdsk/c1t256000C0FFC089D5d1s2 lun1@primary-vds0

# ldm add-vdisk vdisk0 lun0@primary-vds0 myguest
# ldm add-vdisk vdisk1 lun1@primary-vds0 myguest

References

Monday Jun 13, 2011

Device Validation with Oracle VM Server for SPARC 2.1

Oracle VM Server for SPARC 2.1 (aka LDoms 2.1) has been recently released. One of the major new features of this release is live migration; you can find more information about this cool feature on Liam's blog, and some experiment on Jeff's blog. Here, I would like to talk about a much more modest improvement but which is actually very useful: the device validation.

Device Misconfiguration and Previous Versions of LDoms

Previous versions of LDoms are not very friendly for handling some obvious configuration mistakes, like a misspelled path to designate a virtual disk backend. For example, let say I have a virtual disk with a Solaris 11 installation which is on the ZFS volume ldoms/vdisk/solaris_11. If I want to use this ZFS volume as a virtual disk then I have to add the device /dev/zvol/rdsk/ldoms/vdisk/solaris_11 to the virtual device server (vds) with the following command:
  # ldm add-vdsdev /dev/zvol/rdsk/ldoms/vdisk/solaris_11 s11@primary-vds0
If I make a mistake in the path of the device, for example I forget the 's' at the end of 'ldoms', then the system will not complain and I will be able to successfully create and configure my domain:
  # ldm create ldg1
  # ldm set-vcpu 8 ldg1
  # ldm set-mem 8G ldg1
  # ldm add-vnet vnet0 primary-vsw0 ldg1
  # ldm add-vdsdev /dev/zvol/rdsk/ldom/vdisk/solaris_11 solaris_11@primary-vds0
  # ldm add-vdisk vdisk0 solaris_11@primary-vds0 ldg1
  # ldm bind ldg1
  # ldm start ldg1
However, if I access the console of my guest domain ldg1 and try to boot it, I am getting some obscure messages but my domain is definitively not booting:
  # telnet 0 5001
  Trying 0.0.0.0...
  Connected to 0.
  Escape character is '^]'.

  {0} ok boot vdisk0
  Boot device: /virtual-devices@100/channel-devices@200/disk@0  File and args: 
  WARNING: /virtual-devices@100/channel-devices@200/disk@0: Receiving packet from LDC
  but LDC is Not Up!
  WARNING: /virtual-devices@100/channel-devices@200/disk@0: Communication error with
  Virtual Disk Server using Port 0. Retrying.
  ...
  ERROR: /virtual-devices@100/channel-devices@200/disk@0: boot-read fail

 Can't open boot device
Understanding the problem from these error messages is not very straightforward, but if you have some experiences with LDoms, you can conclude that there is a problem with the virtual disk used for booting. Also, you probably know that a good place to look for hints about the problem is the /var/adm/messages file:
  # cat /var/adm/messages
  ...
  Jun 10 22:55:03 dt92-416 vds: [ID 877446 kern.info] vd_setup_vd():
  /dev/zvol/rdsk/ldom/vdisk/solaris_11 is currently inaccessible (error 2)
So here, we find a message from the virtual disk server (vds), about /dev/zvol/rdsk/ldom/vdisk/solaris_11 being inaccessible. In addition, "error 2" means "No such file or directory" (ENOENT). So we can check the device path and notice that it is incorrect because a 's' is missing:
  # ls -l /dev/zvol/rdsk/ldom/vdisk/solaris_11
  ls: cannot access /dev/zvol/rdsk/ldom/vdisk/solaris_11: No such file or directory

  # ls -l /dev/zvol/rdsk/ldoms/vdisk/solaris_11
  lrwxrwxrwx 1 root root 0 May 12 14:58 /dev/zvol/rdsk/ldoms/vdisk/solaris_11 ->
  ../../../../..//devices/pseudo/zfs@0:3,raw
Now, we just have to reconfigure the virtual disk server with the right device path. But it is quite a long trip to identify such an obvious problem!

Improvement with Oracle VM Server for SPARC 2.1

The good news is that with Oracle VM Server for SPARC 2.1, the system will immediately notice such a problem and give you a clear error message. Let's try the same sequence again with Oracle VM Server for SPARC 2.1:
  # ldm create ldg1
  # ldm set-vcpu 8 ldg1
  # ldm set-mem 8G ldg1
  # ldm add-vnet vnet0 primary-vsw0 ldg1
  # ldm add-vdsdev /dev/zvol/rdsk/ldom/vdisk/solaris_11 solaris_11@primary-vds0
  Path /dev/zvol/rdsk/ldom/vdisk/solaris_11 is not valid on service domain primary
As you can see, I get an error message saying that the path is not valid; so I can immediately notice and correct my mistake.

However, there might be some cases where you know that the path you are indicating is not valid, for example because the device does not exist yet or because the service domain providing that device is not currently up. For these situations, the ldm add-vdsdev command has a -q option to quickly add the device without checking if it is valid:

   # ldm add-vdsdev -q /dev/zvol/rdsk/ldom/vdisk/solaris_11 solaris_11@primary-vds0
   # ldm add-vdisk vdisk0 solaris_11@primary-vds0 ldg1
With the -q option, the device is not checked so no error is returned. This can be particularly useful when provisioning some domains for a future usage and when it does not really matter if the associated devices effectively exist yet. Or, if you just want to avoid the overhead of the validation because you know that your path is correct.

After you have configured your domain, virtual devices will be checked anyway when you bind the domain:

  # ldm bind ldg1
  Path /dev/zvol/rdsk/ldom/vdisk/solaris_11 is not valid on service domain primary
That way, you receive an error when you are about to use a domain which is incorrectly configured. At this point, you really need your virtual devices to be properly configured because you are effectively going to use them (note that the ldm bind command also has a -q option to disable the device validation, if you really want to bind your domain anyway).

This example has shown the device validation for virtual disks, but device validation also occurs with the ldm add-vsw command to validate that the physical network device (net-dev) associated with a virtual switch is valid.

Backward Compatibility Mode

Although this should not be very frequent, there might be some cases where you don't want any device validation to occur. For example, because you have some custom scripts and you never want the add-vsw, add-vdsdev or bind commands to fail. For such a case, it is possible to completely disable the device validation and to go back to the same behavior as before Oracle VM Server for SPARC 2.1.

Device validation can be disabled by setting the device_validation SMF property of the Logical Domain manager service to 0.

  # svccfg -s ldmd setprop ldmd/device_validation=0
  # svcadm refresh ldmd
  # svcadm restart ldmd
This setting will entirely disable the device validation. Device validation can be restored by setting the device_validation property back to -1:
  # svccfg -s ldmd setprop ldmd/device_validation=-1
  # svcadm refresh ldmd
  # svcadm restart ldmd
About

Alexandre Chartre is a senior principal engineer in the Oracle Virtualization Engineering organization. He is co-architect for Oracle VM Server for SPARC (LDoms).

Search

Top Tags
Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today