Wednesday Jun 08, 2016

How to configure IP over Infiniband (IPoIB) on Oracle Solaris and SPARC servers

Recently we worked with an ISV who wanted to certify their application with Oracle SuperCluster in order to support their customers who are using this SPARC and Solaris based engineered system. Their application has a kernel module which needed to be tested with the infiniband driver.

We connected 2 SPARC T5-2 servers using 2 IB cards and an IB switch for this project. This simple configuration can simulate a 2 node connection of an Oracle SuperCluster.

The following steps  re needed to make such an IPoIB configuration work. It is very important to first be sure that the switch is configured properly:

For the IB switch :

define it into DNS
then ssh root@switchIP
passwd is "changeme" ( usually this is the password from the manufacture)
enablesm ( to enable the master )
getmaster ( to check the master )

If the switch master is not working, the IB interface status will show as “down” on the server even if all the cables and cards are connected to the switch correctly.

On the Solaris server (T5 server in this case):

Check the physical network interfaces available on the server: dladm show-phys

LINK MEDIA STATE SPEED DUPLEX DEVICE
net1 Ethernet unknown 0 unknown ixgbe1
net2 Ethernet unknown 0 unknown ixgbe2
net0 Ethernet up 1000 full ixgbe0
net3 Ethernet unknown 0 unknown ixgbe3
net6 Ethernet up 1000 full vsw0
net9 Infiniband down 0 unknown ibp1
net5 Infiniband down 0 unknown ibp0

net4 Ethernet up 10 full usbecm2
net10 Ethernet up 40000 unknown vsw2
net11 Ethernet up 40000 unknown vsw1

Check only the IB cards: dladm show-ib

LINK HCAGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS
net9 10E000015A7460 10E000015A7462 2 down -- -- FFFF
net5 10E000015A7460 10E000015A7461 1 down -- -- FFFF

The following 2 commands are not mandatory, but will help you to easier identify the IB

dladm rename-link net9 ibp1

dladm rename-link net5 ibp0

Check renaming: dladm show-phys

LINK MEDIA STATE SPEED DUPLEX DEVICE
net1 Ethernet unknown 0 unknown ixgbe1
net2 Ethernet unknown 0 unknown ixgbe2
net0 Ethernet up 1000 full ixgbe0
net3 Ethernet unknown 0 unknown ixgbe3
net6 Ethernet up 1000 full vsw0
ibp1 Infiniband down 0 unknown ibp1
ibp0 Infiniband down 0 unknown ibp0
net4 Ethernet up 10 full usbecm2
net10 Ethernet up 40000 unknown vsw2
net11 Ethernet up 40000 unknown vsw1

dladm show-ib

LINK HCAGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS
ibp0 10E000015A7380 10E000015A7381 1 up -- -- FFFF
ibp1 10E000015A7380 10E000015A7382 2 down -- -- FFFF

Create IB default partition:

(the name is under GWPORT column resulted from previous command/)

dladm create-part -l ibp0 -P ffff ffff.ibp0

dladm show-part

LINK PKEY OVER STATE FLAGS
ffff.ibp0 FFFF ibp0 unknown ----

Create and assign the IP address :

ipadm create-ip ffff.ib0
ipadm create-addr -T static -a 10.1.10.11/24 ffff.ib0/v4


Important commands to check the status of the connection:
dladm show-ib
ibhosts
ibstat
ibswitches
iblinkinfo.pl –R

grep pciex15b3 /etc/path_to_inst


Should you need further help with your IPoIB configurations on SPARC servers, you can contact us at isvsupport_ww@oracle.com.


Friday Apr 20, 2012

Replay available: Modern Software Packaging For Enterprise Developers (IPS)

Oracle University is offering the replay of the webinar "Modern Software Packaging for Enterprise Developers (IPS)". Eric Reid's webinar replay is available for free to all OTN members. 

IPS

Don't forget to register for the next webinar April 24, 2012 "Managing Application Services, Using SMF manifests in Oracle Solaris 11" by Matthew Hosanee. 

About

Application tuning, sizing, monitoring, porting on Solaris

Search

Categories
Archives
« June 2016
SunMonTueWedThuFriSat
   
1
2
3
4
5
6
9
10
11
12
13
14
16
17
18
19
21
22
23
24
25
26
27
28
29
30
  
       
Today