Playing with SR IOV in Oracle VM 2.2.1

I happen to have a motherboard at home that has an intel 82576 1Gb ethernet controller (or rather 2) on board and the board is capable of VT-d and VT-x (both features enabled in the bios).

This server runs Oracle VM 2.2.1 with the latest xen and dom0 kernels from ULN.

The 82576 device is an SR IOV capable ethernet controller and I wanted to describe the simple steps to enable this in Oracle VM all the way to the guest OS (OL5.5).

1) verify that you have such a card

In dom0 just simply type

$ **lspci | grep 82576**

it should return something like

01:00.0 Ethernet Controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

2) change /etc/grub.conf and /etc/modprobe.conf to enable Virtual Functions for the device and let the hypervisor do the right thing :

**/etc/grub.conf** should have an entry like so (add the values in bold) :

title Oracle VM server (2.6.18-128.2.1.4.27.el5xen)
root (hd0,0)
kernel /xen-64bit.gz dom0_mem=563M **iommu=1 msi=1**
module /vmlinuz-2.6.18-128.2.1.4.27.el5xen ro root=UUID=af3901ee-28c0-4a4a-a259-117c26b9e053 **pci_pt_e820_access=on**
module /initrd-2.6.18-128.2.1.4.27.el5xen.img

**/etc/module.conf** add :

**options igb max_vfs=7**
# number of virtual functions to create
**blacklist igbvf** # don't load the virtual function driver in dom0 (if it is shipped)

3) reboot the server

4) when dom0 is back up you should be able to see the virtual functions

$ **lspci |grep 82576**

01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
01:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
01:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
01:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
01:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
01:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

at this point it shows you are all set.

5) create an hvm guest (Oracle Linux 5.5)

6) to pass through a virtual function to the guest do the following :

a) $ **modprobe pciback** # ensure the pciback module is loaded

b) pick a virtual function device. ie 01:10.0 and make it available

$ **cd /sys/bus/pci/drivers/pciback**
$ **echo -n "0000:01:10.0" > new_slot**
$ **echo -n "0000:01:10.0" > bind**

do this for all the virtual functions you intend to pass through to guests

c) at this point this should show up when you run
$ **xm pci-list-assignable-devices**

d) assign a pci device to the guest

2 options
1) assign directly in the **vm.cfg** , add a line
**pci = [ '01:10.0' ]**
2) do it dynamically (in this case you have to enable hotplug pci in the guest)
$ **xm pci-attach 0000:01:10.0**

7) at this point inside the HVM OL5.5 guest it should show the same device
$ **lspci |grep 82576**
00:03.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

8) linux should have loaded the igbvf driver and you should see an ethX device
igbvf 0000:00:03.0: Link is Up 1000 Mbps Full Duplex



To enable pci hotplug in the guest, please do the following :
in the guest you have to load both pciehp and acpiphp

you can do this easily by doing :

**echo "modprobe pciehp" >> /etc/rc.modules
echo "modprobe acpiphp" >> /etc/rc.modules
chmod 755 /etc/rc.modules**

* note - this is for testing purposes only. We do not currently have official support in the product for this.

Comments:

Wim, how much of a performance boost would you estimate one might be able to achieve by using SR-IOV and those VF drivers vs. paravirtual drivers? Are we talking >=10% gain in the guest nic performance? Do the guest vm's see separate mac's? Would the Virtual Function SR-IOV nics still be able to be used if I am setting the physical nics in the dom0 as trunk devices when connecting to the physical switches?

Posted by Joe Hoot on October 12, 2010 at 05:46 AM PDT #

I am going to do some performance measurements next and post the results :) One thing with VF's is that you cannot do live migrate. However one of my todos is to create a bonding device in the VM on a vif + VF and then on live migrate we should be able to keep going and then a new pci device gets hotplugged in on the other side that can then recover the lost VF pci device during live migrate. re : trunk devices - that is a very good question - I am not certain and I will see if I can try it out. I think I can do that at home ;) I need to double check if my switch is capable

Posted by wim.coekaerts on October 12, 2010 at 05:51 AM PDT #

Hello Wim, Is it okay, if I add you blogs URL to my blog on the side with relevant links (blog: http://drsalbertspijkers.blogspot.com/ ) . I am an official Oracle Unbreakable Linux reseller now. Not however I have to save a little money before I can connect to the ULN network to upgrade to the Unbreakable Kernel. Economy is extremely bad for me so far. But hope to sell lots of support contracts for Oracle. Friendly greetings, Drs. Albert Spijkers DBA Consulting web: www.dbaconsulting.nl blog: http://drsalbertspijkers.blogspot.com/ profile: http://nl.linkedin.com/pub/drs-albert-spijkers/13/b4a/7a8albertspijkers@dbaconsulting.nl email: albertspijkers@dbaconsulting.nl info@dbaconsulting.nl

Posted by Albert Spijkers on October 20, 2010 at 11:10 PM PDT #

Hi Wim, Very interesting post. I have a few questions for you. 1. Assume that this could be done with any SR-IOV capable adapter? I know that such adapters (CNAs) are being developed by the major manufacturers and should be available on he market in 2011. 2. Do you see any issue to interoperate with 802.1Qbh port extender standard? So the bottom line solution would be to use SR-IOV capable CNA + Cisco's pre-standard 802.1Qbh (aka VN-Link). Such functionality is currently available for VMWare hypervisor running on UCS platform where Cisco CNA (VIC) can be used for SR-IOV. VMWare then uses VMDirectPath to connect to the vNIC, which then uses VN-Link (802.1Qbh) towards the Fabric Interconnects on the UCS. So I wonder if the same would hold true for Oracle's VM.... Cisco and VMWare have a special collaboration, however with SR-IOV and VN-Link support on the CNAs, the advantages can be realized with other hypervisor platforms, as long as they can work with those CNAs/technology... Thank you! David

Posted by David Klebanov on November 09, 2010 at 05:03 AM PST #

Hi David - yes on (1) and likely on (2) but I am not certain until we can test it

Posted by wim.coekaerts on November 11, 2010 at 03:40 AM PST #

Hi Wim, Let me ask you further question, after you had performed your testing and tied the VM to the virtual NIC on the physical adapter, what did you test further? Did you test network connectivity from this VM to anything outside the physical server this VM was on? Thanks, David

Posted by David Klebanov on November 11, 2010 at 07:30 AM PST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Wim Coekaerts is the Senior Vice President of Linux and Virtualization Engineering for Oracle. He is responsible for Oracle's complete desktop to data center virtualization product line and the Oracle Linux support program.

You can follow him on Twitter at @wimcoekaerts

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today