X

News, tips, partners, and perspectives for the Oracle Solaris operating system

How to configure IP over Infiniband (IPoIB) on Oracle Solaris and SPARC servers

Parnian Taidi
Product Marketing Manager

Recently we worked with an ISV who wanted to certify their
application with Oracle SuperCluster in order to support their customers who
are using this SPARC and Solaris based engineered system. Their application has
a kernel module which needed to be tested with the infiniband driver.

We connected 2 SPARC T5-2
servers using 2 IB cards and an IB switch for this project. This simple
configuration can simulate a 2 node connection of an Oracle SuperCluster.

The following steps  re needed to make such an IPoIB configuration
work. It is very important to first be sure that the switch is
configured properly:

For the IB switch :

define it into DNS

then ssh root@switchIP

passwd is "changeme" ( usually
this is the password from the manufacture)

enablesm ( to enable the master )

getmaster ( to check the master )

If the switch master is not working, the IB interface status
will show as “down” on the server even if all the cables and cards are connected to the switch correctly.

On the Solaris server (T5 server in this case):

Check the physical network interfaces available on the
server: dladm show-phys

LINK MEDIA STATE SPEED DUPLEX DEVICE

net1 Ethernet unknown 0 unknown ixgbe1

net2 Ethernet unknown 0 unknown ixgbe2

net0 Ethernet up 1000 full ixgbe0

net3 Ethernet unknown 0 unknown ixgbe3

net6 Ethernet up 1000 full vsw0
net9 Infiniband down 0 unknown ibp1

net5 Infiniband down 0 unknown ibp0


net4 Ethernet up 10 full usbecm2

net10 Ethernet up 40000 unknown vsw2

net11 Ethernet up 40000 unknown vsw1

Check only the IB cards:
dladm show-ib

LINK HCAGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS

net9 10E000015A7460 10E000015A7462
2 down -- -- FFFF

net5 10E000015A7460 10E000015A7461
1 down -- -- FFFF

The following 2 commands are not mandatory, but will help
you to easier identify the IB

dladm rename-link net9 ibp1

dladm rename-link
net5 ibp0

Check renaming: dladm show-phys

LINK MEDIA STATE SPEED DUPLEX DEVICE

net1 Ethernet unknown 0 unknown ixgbe1

net2 Ethernet unknown 0 unknown ixgbe2

net0 Ethernet up 1000 full ixgbe0

net3 Ethernet unknown 0 unknown ixgbe3

net6 Ethernet up 1000 full vsw0

ibp1 Infiniband down 0 unknown ibp1

ibp0 Infiniband down 0 unknown ibp0

net4 Ethernet up 10 full usbecm2

net10 Ethernet up 40000 unknown vsw2

net11 Ethernet up 40000 unknown vsw1

dladm show-ib

LINK HCAGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS

ibp0 10E000015A7380 10E000015A7381
1 up -- -- FFFF

ibp1 10E000015A7380 10E000015A7382
2 down -- -- FFFF

Create IB default partition:

(the name is under
GWPORT column resulted from previous command/)

dladm create-part -l ibp0 -P ffff ffff.ibp0

dladm show-part

LINK PKEY OVER STATE FLAGS

ffff.ibp0 FFFF ibp0 unknown ----

Create and assign the IP address :

ipadm create-ip
ffff.ib0

ipadm create-addr -T static -a 10.1.10.11/24 ffff.ib0/v4



Important commands to check the status of the connection:
dladm show-ib

ibhosts

ibstat

ibswitches

iblinkinfo.pl –R

grep pciex15b3 /etc/path_to_inst



Should you need further help with your IPoIB configurations on SPARC servers,
you can contact us at isvsupport_ww@oracle.com.


Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.