X

News, tips, partners, and perspectives for Oracle’s virtualization offerings

Virtual HBA in Oracle VM Server for SPARC

Jeff Savit
Product Management Senior Manager
Oracle VM Server for SPARC 3.3 was released on October 26 during Oracle OpenWorld. This release added an important new feature, virtual HBA (vHBA), which adds flexibility and relieves prior limitations of virtual I/O without sacrificing performance.


Important note: A lot of functionality has been added to vHBA support in Solaris 11.3 updates. It's very important to be on a recent Solaris 11.3 SRU to get the best results.

Background

Oracle VM Server for SPARC supports both virtual I/O and physical I/O Physical I/O, described in the Administration Guide chapter I/O domains, gives domains direct ownership and access to I/O devices via PCIe busses, a PCIe endpoint device, or a PCIe SR-IOV virtual function. This is ideal when native performance or full access to device features is needed, but comes with restrictions. In particular, it is limited to the number of physical devices that can be assigned to domains, and is incompatible with live migration.

Virtual I/O is more commonly used, as it provides more operational flexibility for virtual networks and virtual disks. Virtual I/O supports live migration, and can provide near native performance when correctly configured. However, virtual I/O also has restrictions. It supports network and disk devices, but not tape or other device types. Virtual disks have good performance, but are limited to active-passive virtual disk multipathing using mpgroups, which cannot be used in conjunction with SCSI reservation. Ideally, a guest domain would be a "full participant" in the SAN infrastructure used by enterprise customers.

Introducing vHBA

Virtual HBA (vHBA) is a new capability of Oracle VM Server for SPARC that lets guest domains have virtual SCSI HBAs.

 


In the vHBA model, a physical host bus adapter is mapped onto a virtual SAN (vsan) in the service domain, and a virtual HBA in the guest domain is associated with the vsan. SCSA protocol is used to communicate between the physical HBA, the virtual SAN, and the virtual HBA.
The physical LUNs addressed by the physical HBA are visible within the guest as virtual LUNs, and are managed within the guest domain using the regular Solaris device drivers, just as in a non-virtual environment.

vHBA provides multiple benefits:

  • Device functionality: - Solaris in the domain uses native device drivers, and can use full functionality for SCSI reserveration and MPxIO for path management
  • Device generality: - Any device Solaris supports on a physical HBA is supported in the virtual HBA. For example, the guest domain will be able to use tape devices for backup.
  • Command scalability - instead of requiring two commands per virtual disk (an ldm add-vdsdev and an

     

    ldm add-vdisk, all of the devices on the physical HBA are presented to the guest in the same commands. This reduces the effort needed by the domain administrator.
  • Logical Domain Channel scalability - Normally, a virtual device consumes an LDC in the guest and in the service domain, and since there are limits on the number of LDCs per system and per domain, this can be a constraint on the number of devices that domains can have. With vHBA, the entire HBA is represented by a single LDC in the guest and service domain, regardless of the number of LUNs.

Implementing vHBA

Let's show an example of vHBA. It requires Oracle VM Server for SPARC 3.3, which is delivered with Oracle Solaris 11.3 in the control domain. The guest domains using vHBA must run Solaris 11, as Solaris 10 does not have the SCSA interface described above. There are no special hardware requirements other than having a physical HBA with LUNs. It runs on any supported SPARC server - in this example on a SPARC T2.

First, let's display the physical HBA that's available on the server:

primary# ldm ls-hba  primary

 

# alias is ldm list-hba
NAME VSAN
---- ----
MB/SASHBA/HBA0
MB/RISER2/PCIE2/HBA0/PORT0,0
MB/RISER2/PCIE2/HBA0,1/PORT0,0
primary# ldm ls-hba -p primary
HBA
|alias=MB/SASHBA/HBA0|dev=/pci@0/pci@0/pci@2/scsi@0|disks=2
|alias=MB/RISER2/PCIE2/HBA0/PORT0,0|dev=/pci@0/pci@0/pci@9/SUNW,emlxs@0/fp@0,0|disks=4
|alias=MB/RISER2/PCIE2/HBA0,1/PORT0,0|dev=/pci@0/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0|disks=4
primary# ldm ls-hba -t primary
NAME VSAN
---- ----
MB/SASHBA/HBA0
init-port 50800200005ab000
Transport Protocol SAS
MB/RISER2/PCIE2/HBA0/PORT0,0
init-port 10000000c9b09b3c
Transport Protocol FABRIC
MB/RISER2/PCIE2/HBA0,1/PORT0,0
init-port 10000000c9b09b3d
Transport Protocol FABRIC

We have a physical adapter, and can even see the LUNs under it. Let's show some output formats for details.

# ldm ls-hba -d primary
NAME                                                 VSAN
----                                                 ----
MB/SASHBA/HBA0                                      
  c3t0d0s0                                          
  c3t1d0s0                                          
MB/RISER2/PCIE2/HBA0/PORT0,0                        
  c4t21000024FF2D4C83d2s0                           
  c4t21000024FF2D4C83d0s0                           
  c4t21000024FF2D4C82d2s0                           
  c4t21000024FF2D4C82d0s0                           
MB/RISER2/PCIE2/HBA0,1/PORT0,0                      
  c5t21000024FF2D4C83d2s0                           
  c5t21000024FF2D4C83d0s0                           
  c5t21000024FF2D4C82d2s0                           
  c5t21000024FF2D4C82d0s0 
# ldm ls-hba -l primary
NAME                                                 VSAN
----                                                 ----
MB/SASHBA/HBA0                                      
[/pci@0/pci@0/pci@2/scsi@0]                       
MB/RISER2/PCIE2/HBA0/PORT0,0                        
[/pci@0/pci@0/pci@9/SUNW,emlxs@0/fp@0,0]          
MB/RISER2/PCIE2/HBA0,1/PORT0,0                      
[/pci@0/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0]  

Now that I've identified the physical resource, I'll create the vsan against it, and the vHBA for my guest domain:

primary# ldm add-vsan MB/RISER2/PCIE2/HBA0,1/PORT0,0 jeff-vsan primary
MB/RISER2/PCIE2/HBA0,1/PORT0,0 resolved to device: /pci@0/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0
primary# ldm add-vhba jeff-vhba jeff-vsan ldom3

That was really all there was to it. The guest now has the vHBA and sees its LUNs. I'll show that in a bit.First, I'll show the virtual services created in the control domain:
 
primary# ldm list-services
... snip ...
VSAN
    NAME             LDOM             TYPE   DEVICE  IPORT                      
    jeff-vsan        primary          VSAN   vsan@0  [/pci@0/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0]
VHBA
    NAME             VSAN                        DEVICE  TOUT SERVER        
    jeff-vhba        jeff-vsan                   vhba@0  0    primary

We now have a vsan and a vHBA I've creatively named for myself. I can inspect the configuration
using the commands I used in the beginning:

primary# ldm ls-hba -d primary
NAME                                                 VSAN
----                                                 ----
MB/SASHBA/HBA0                                      
  c3t0d0s0                                          
  c3t1d0s0                                          
MB/RISER2/PCIE2/HBA0/PORT0,0                        
  c4t21000024FF2D4C83d2s0                           
  c4t21000024FF2D4C83d0s0                           
  c4t21000024FF2D4C82d2s0                           
  c4t21000024FF2D4C82d0s0                           
MB/RISER2/PCIE2/HBA0,1/PORT0,0                       jeff-vsan
  c5t21000024FF2D4C83d2s0                            jeff-vsan
  c5t21000024FF2D4C83d0s0                            jeff-vsan
  c5t21000024FF2D4C82d2s0                            jeff-vsan
  c5t21000024FF2D4C82d0s0                            jeff-vsan
primary# ldm ls-hba -l primary
NAME                                                 VSAN
----                                                 ----
MB/SASHBA/HBA0                                      
[/pci@0/pci@0/pci@2/scsi@0]                       
MB/RISER2/PCIE2/HBA0/PORT0,0                        
[/pci@0/pci@0/pci@9/SUNW,emlxs@0/fp@0,0]          
MB/RISER2/PCIE2/HBA0,1/PORT0,0                       jeff-vsan
[/pci@0/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0]        
primary# ldm ls-hba -t primary
NAME                                                 VSAN
----                                                 ----
MB/SASHBA/HBA0                                      
        init-port 50800200005ab000
        Transport Protocol SAS
MB/RISER2/PCIE2/HBA0/PORT0,0                        
        init-port 10000000c9b09b3c
        Transport Protocol FABRIC
MB/RISER2/PCIE2/HBA0,1/PORT0,0                       jeff-vsan
        init-port 10000000c9b09b3d
        Transport Protocol FABRIC
primary# ldm ls -o san,hba
NAME             
primary          
VSAN
    NAME             TYPE   DEVICE  IPORT                      
    jeff-vsan        VSAN   vsan@0  [/pci@0/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0]
... snip ...
NAME             
ldom3            
VHBA
    NAME             VSAN                        DEVICE  TOUT SERVER        
    jeff-vhba        jeff-vsan                   vhba@0  0    primary   

Not counting the commands to list the environment, it only took two commands in the control domain
(ldm add-vsan, ldm add-vhba) to do the actual work.

vHBA devices viewed from the guest domain

The guest domain was running while the above commands were issues (showing that this works with guest domain dynamic reconfiguration). I thought it would be interesting to see what dmesg reported for the dynamic reconfiguration events, so I tailed it and saw the following interesting events:

root@ldom3:/# dmesg|tail 
Jul 20 16:40:54 ldom3 scsi: [ID 583861 kern.info] sd10 at scsi_vhci0: unit-address g600144f0ede50676000055a815160019: f_tpgs
Jul 20 16:40:54 ldom3 genunix: [ID 936769 kern.info] sd10 is /scsi_vhci/disk@g600144f0ede50676000055a815160019
Jul 20 16:40:54 ldom3 cmlb: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk@g600144f0ede50676000055a815160019 (sd10):
Jul 20 16:40:54 ldom3  Corrupt label; wrong magic number
Jul 20 16:40:54 ldom3 genunix: [ID 408114 kern.info] /scsi_vhci/disk@g600144f0ede50676000055a815160019 (sd10) online
Jul 20 16:40:54 ldom3 genunix: [ID 483743 kern.info] /scsi_vhci/disk@g600144f0ede50676000055a815160019 (sd10) multipath status: degraded: path 1 vhba1/disk@w21000024ff2d4c83,0 is online
Jul 20 16:40:54 ldom3 genunix: [ID 530209 kern.info] /scsi_vhci/disk@g600144f0ede50676000055a815160019 (sd10) multipath status: optimal: path 2 vhba1/disk@w21000024ff2d4c82,0 is online: Load balancing: round-robin
Jul 20 16:40:54 ldom3 genunix: [ID 408114 kern.info] /virtual-devices@100/channel-devices@200/scsi@0/iport@0/probe@w21000024ff2d4c83,2 (nulldriver1) online
Jul 20 16:40:55 ldom3 scsi: [ID 583861 kern.info] sd11 at scsi_vhci0: unit-address g600144f0ede50676000055a81bb2001a: f_tpgs
Jul 20 16:40:55 ldom3 genunix: [ID 936769 kern.info] sd11 is /scsi_vhci/disk@g600144f0ede50676000055a81bb2001a
Jul 20 16:40:55 ldom3 genunix: [ID 408114 kern.info] /scsi_vhci/disk@g600144f0ede50676000055a81bb2001a (sd11) online
Jul 20 16:40:55 ldom3 genunix: [ID 483743 kern.info] /scsi_vhci/disk@g600144f0ede50676000055a81bb2001a (sd11) multipath status: degraded: path 3 vhba1/disk@w21000024ff2d4c83,2 is online
Jul 20 16:40:55 ldom3 genunix: [ID 530209 kern.info] /scsi_vhci/disk@g600144f0ede50676000055a81bb2001a (sd11) multipath status: optimal: path 4 vhba1/disk@w21000024ff2d4c82,2 is online: Load balancing: round-robin

Next, I used format to show the disk devices:

root@ldom3:/# format
Searching for disks...done
c0t600144F0EDE50676000055A815160019d0: configured with capacity of 23.93GB
AVAILABLE DISK SELECTIONS:
       0. c0t600144F0EDE50676000055A81BB2001Ad0 
          /scsi_vhci/disk@g600144f0ede50676000055a81bb2001a
       1. c0t600144F0EDE50676000055A815160019d0 
          /scsi_vhci/disk@g600144f0ede50676000055a815160019
       2. c1d0 
          /virtual-devices@100/channel-devices@200/disk@0
Specify disk (enter its number): ^C

Note the long device names for the LUNs coming from a ZFS storage appliance - those are the ones I've just picked up. You can see that it's using the native device driver, instead of the 'virtual-devices' driver used with a standard vdisk. I even created a ZFS pool on one of the LUNs on another host accessing the physical SAN, so I can now import it:

root@ldom3:/# zpool import
  pool: aux36
    id: 10749927192920141180
 state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
        aux36                                    ONLINE
          c0t600144F0EDE50676000055A81BB2001Ad0  ONLINE
root@ldom3:/# zpool import aux36
root@ldom3:/# zpool status -v
  pool: aux36
 state: ONLINE
  scan: none requested
config:
        NAME                                     STATE     READ WRITE CKSUM
        aux36                                    ONLINE       0     0     0
          c0t600144F0EDE50676000055A81BB2001Ad0  ONLINE       0     0     0
errors: No known data errors

At this point, if I had more devices I could use them too, and I could use all the device features Solaris supports on bare-metal, like SCSI reservation or MPxIO. Now, before you ask: I did some trivial performance tests with I/O workload tools, and it seemed to perform as well as regular vdisks on LUNs.

Summary


vHBA adds I/O capability that increases the generality and function of virtual I/O. Guest domains can make use of tape and other non-disk devices on a SAN, and can use them with native features. This increases the functionality of Oracle VM Server for SPARC without sacrificing performance or flexibility. The existing virtual disk framework doesn't go away, and they can freely coexist with vHBAs. Our recommendation is that customers try it out to see if they find value in the new capabilities (we expect they will), gain experience, and pass their feedback to us. We want to know how it works for you.
 

Oracle VM Server for SPARC 3.3 introduces important and useful new function, and demonstrates Oracle's committment to investing in this growing platform. For other new features, see the What's New

Join the discussion

Comments ( 15 )
  • Julien Gabel Thursday, November 12, 2015

    Hi,

    Thank you for this post. I get a question though: we are interested in trying this, but we didn't understand one thing.

    Since it is all about mutualization, I understand it will possible to map multiple vsan to one physical FC port (if not, what is the benefit vs. direct ownership and access to I/O devices via PCIe busses, etc.). If yes, what about the visibility of the physical LUN in this case (since all pLUNs are presented to... all vsan?)?

    Thank you!

    --

    Best regards,

    julien Gabel.


  • Jeff Thursday, November 12, 2015

    Hi Julien,

    Thanks very much for the comment! The advantage for the domain using vHBA is that it can still be live migrated, which is not possible for domains using physical I/O. The advantage for the system overall is that this doesn't require qualifying specific HBAs or other hardware the way SR-IOV can, and isn't limited to the number of SR-IOV virtual functions or available PCIe buses the way physical I/O would be. So, you get functionality and scale without losing flexibility.

    I hope that helps, and thanks again for the comment.


  • Juanfran Friday, November 13, 2015

    Hi Jeff:

    Thanks for posting this, it's very helpful. Just to clarify: With vHBA, I can have a single FC physical HBA, and we can virtualize it and present it to ONLY one guest domain, right? And by doing this, the guest domain could access a direct-attached tape device, correct?

    And a last one: Do this work with any of the FC HBAs supported by SPARC/Solaris?

    Thanks,

    Juanfran


  • Julien Gabel Friday, November 13, 2015

    Hi Jeff,

    Thank you for your quick reply: I think I get the points thanks.

    In fact, my concern is we are comtemplating using Lan-Free-Backup in our LDOM hosting Oracle RDBMS. And as far as I unsterdand, using either physical I/O or the new Virtual HBA capability, I will ended to need one FC port (at least) dedicated on each LDOM... which is not really feasible (because of added FC cost and "bad" scalability). Am I right? Or is there a better to implement such thing?

    Thanks again.

    --

    Best regards,

    julien Gabel.


  • Jeff Friday, November 13, 2015

    Hi Juanfran,

    You're welcome! Your understanding is correct: you could make a virtual HBA, present it to one guest and then that guest could have a tape device. I would use slightly different language: it's still a virtual device - we wouldn't want to confuse it with physical I/O). As far as the guest is concerned, you're right - the guest would just see the tape and use it. Two the second question: the answer is yes, and this will work with any FC HBAs supported by the SPARC server and Solaris.

    regards, Jeff


  • Jeff Friday, November 13, 2015

    Hi Julien,

    (This is for your second comment) vHBA will let you access the same physical HBAs LUNs from multiple guest domains so they all share the same HBA card in the PCIe bus slot. Different domains will need to use different LUN targets on that HBA. However, you may be able to automate switching the targets between domains, so that (for example) one domain could use the tape drive, then a different one. All the domains will see the targets.

    I hope that helps, Jeff


  • Thomas Wednesday, November 18, 2015

    Hi Jeff,

    This was very informative and much awaited post. I would like to get a clarification on the Target usage. For eg: as mentioned in some early comments, we want to use one 16G FC HBA for backup purpose. More than 10 Ldoms should use the same HBA port to connect to HP Data Protector. Will each ldoms assigned with a vHBA wwn or only one wwn per HBA port? If only single wwn how we can use the Data Protector for backup? Also these ldoms should be able to live migrated to another Physical Server? Is this feature available in VM Manager and Server Pools?

    Thanks

    Thomas


  • Jeff Wednesday, November 18, 2015

    Hi Thomas,

    I know literally nothing about HP Data Protector, so can't answer that part - sorry! My understanding is that there is a single WWN per physical HBA.

    You can live migrate domains using vHBA to another server, subject to the usual rules for live migration (compatible firmware and LDoms version, virtual devices only) and in this case both source and target systems have to be a Oracle VM Manager 3.3 level to handle vHBA feature. At this time, we do not have vHBA support in Oracle VM Manager.

    Thanks for the comment and question, and I hope this is helpful.

    Jeff


  • Ranjaka Thursday, November 19, 2015

    Hi Jeff,

    Can you use a n-port virtualised interface or SR-IOV FC interface to create a vSAN?

    Thanks,

    Ranjaka


  • Jeff Thursday, November 19, 2015

    Hi Ranjaka,

    Thanks for the post! I don't know if you can use NPIV for this, and my expectation is that you don't use an SR-IOV FC interface for this either: you need an HBA, and the way to use SR-IOV is to create virtual functions and assign them to domains. I'll pass the question on to engineers and see if I can get a more definite answer.

    regards, Jeff


  • Sachin Thursday, November 19, 2015

    Hi Jeff,

    Good write-up simplifying the vHBA feature. I have just one question, would you by any chance know whether Symantec NetBackup Enterprise Agent (SAN Agent) now supports installation in a Guest LDOM with vHBA for LAN Free backup?

    Regards,

    Sachin


  • Jeff Thursday, November 19, 2015

    Hi Sachin,

    Thanks for the post! I can't comment on what Symantec supports, because only they can answer that, but I don't see any reason they couldn't. Presumably they would want to do a qualification and testing effort before they made an official statement. I'll see if I can find out if they have. I know we work with partners a lot to make sure our products work together, but it's not my area so I don't know the specifics.

    thanks again - great question. Jeff


  • Jeff Friday, November 20, 2015

    A followup: NPIV has been tested and works, including with live migration. Checking with storage partners for the other questions. Cheers, Jeff


  • Tim Thursday, December 17, 2015

    Hi Jeff,

    Thanks for the article it was very interesting. I have been asked to create a solution for a customer with 30 LDOMs and where each LDOM has about 5-6 disks, as they also want Live Migration the only method I can think of is to configure 150+ disks on the primary and secondary domains and then map the relevant disks to the relevant guest domains. This seems to be a very messy way of doing it and prone to error. I have thought about setting the disks to exclusive to try and hide the mess but I believe that breaks multipathing.

    In an ideal world I would be able to somehow virtualise the HBA into 30 virtual HBAs and map the relevant LUNs directly to the Guest LDOMs but I believe I cannot do that with IO-SRV as it will break Live Migration and after your comment that NPIV has been tested and works that sounded very tempting but from what I have read you cannot boot off a NPIV device.

    Please let me know if I am missing a solution or whether my original plan was correct.

    Thanks,

    Tim


  • Tim Thursday, December 17, 2015

    Thanks Jeff, that was very helpful, with the many different methods of virtualising the storage it is sometime difficult not to get tempted away from the simpliest solution :)

    I will take a look at the Oracle VM Manager and see if the customer can spare a Linux VM somewhere on their estate for me to have a play :)


Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha
Oracle

Integrated Cloud Applications & Platform Services