X

News, tips, partners, and perspectives for the Oracle Solaris operating system

PV IPoIB in Kernel Zones in Solaris 11.3

The Paravirtualization of IP over Infiniband (IPoIB) in kernel zones is a 
new feature in S11.3 enhancing the network virtualization offering in Solaris.
This allows for existing IP applications in the guest to run over Infiniband
fabrics. Features such as Kernel zone Live Migration and IPMP are supported
with the Paravirtualized IPoIB datalinks making it an appealing option.
Moreover, the device management of these guest datalinks are similar to their
Ethernet counterparts making it straightforward to configure and manage. Zonecfg
is used in the host to configure the kernel zone's automatic network interface
(anet) to select the link of the IB HCA port to paravirtualize and assign as the
lower-link, the Partition Key (P_Key) wthin the IB fabric and the possible
link mode to choose from which could either be IPoIB-CM or IPoIB-UD.
The PV IPoIB datalink is a front end guest driver emulating a IPoIB VNIC
in the host created over a physical IB partition datalink per P_Key and port.
To create a PV IPoIB datalink in a kernel zone the configuration is fairly
simple. Here is an example showing how to create a PV IPoIB datalink in a
kernel zone.
1. Find the IB datalink in the host to paravirtualize.
I am selecting net7 for this example.
# ibadm
HCA TYPE STATE IOV ZONE
hermon0 physical online off global
# dladm show-ib
LINK HCAGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS
net5 21280001A0D220 21280001A0D222 2 up -- -- 8001,FFFF
net7 21280001A0D220 21280001A0D221 1 up -- -- 8001,FFFF
# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net0 Ethernet up 1000 full igb0
net2 Ethernet unknown 0 unknown igb2
net3 Ethernet unknown 0 unknown igb3
net1 Ethernet unknown 0 unknown igb1
net4 Ethernet up 10 full usbecm0
net5 Infiniband up 32000 full ibp1
net7 Infiniband up 32000 full ibp0
2. Create an IPoIB PV datalinks to a kernel zone.
To add an IPoIB PV interface to a kernel zone say tzone1 add an anet
and specify a lower-link and pkey which are mandatory properties using
zonecfg. If not specified IPoIB-CM is the default link mode.
# zonecfg -z tzone1
zonecfg:kzone0> add anet
zonecfg:kzone0:anet> set lower-link=net7
zonecfg:kzone0:anet> set pkey=0xffff
zonecfg:kzone0:anet> info
anet 1:
lower-link: net7
...
pkey: 0xffff
linkmode not specified
evs not specified
vport not specified
iov: off
lro: auto
id: 1
...
zonecfg:tzone1>exit
#
3. Additional IPoIB PV datalinks to the kernel zone.
Additional IPoIB PV interfaces to a kernel zone with a lower-link and pkey
can be added as indicated above. These datalinks can be used exclusively
to host native zones within the kernel zones.
4. The PV IPoIB datalinks appear within the kernel zone on boot.
root@tzone1:~# dladm
LINK CLASS MTU STATE OVER
net1 phys 65520 up --
net0 phys 65520 up --
root@tzone1:~# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
lo0 loopback ok -- --
lo0/v4 static ok -- 127.0.0.1/8
lo0/v6 static ok -- ::1/128
net0 ip ok -- --
net0/v4 static ok -- 1.1.1.190/24
net1 ip ok -- --
net1/v4 static ok -- 2.2.2.190/24
Virtual NICs (VNICs) tzone1/net0 and tzone1/net1 are created in the
host kernel which are the backend of the PV interface.
# dladm show-vnic
LINK OVER SPEED MACADDRESS MACADDRTYPE IDS
tzone1/net1 net7 32000 80:0:0:4d:fe:.. fixed PKEY:0xffff
tzone1/net0 net7 32000 80:0:0:4e:fe:.. fixed PKEY:0xffff

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.