Recently we worked with an ISV who wanted to certify their
application with Oracle SuperCluster in order to support their customers who
are using this SPARC and Solaris based engineered system. Their application has
a kernel module which needed to be tested with the infiniband driver.
We connected 2 SPARC T5-2
servers using 2 IB cards and an IB switch for this project. This simple
configuration can simulate a 2 node connection of an Oracle SuperCluster.
The following steps re needed to make such an IPoIB configuration
work. It is very important to first be sure that the switch is
configured properly:
For the IB switch :
define it into DNS
then ssh root@switchIP
passwd is “changeme” ( usually
this is the password from the manufacture)
enablesm ( to enable the master )
getmaster ( to check the master )
If the switch master is not working, the IB interface status
will show as “down” on the server even if all the cables and cards are connected to the switch correctly.
On the Solaris server (T5 server in this case):
Check the physical network interfaces available on the
server: dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net1 Ethernet unknown 0 unknown ixgbe1
net2 Ethernet unknown 0 unknown ixgbe2
net0 Ethernet up 1000 full ixgbe0
net3 Ethernet unknown 0 unknown ixgbe3
net6 Ethernet up 1000 full vsw0
net9 Infiniband down 0 unknown ibp1
net5 Infiniband down 0 unknown ibp0
net4 Ethernet up 10 full usbecm2
net10 Ethernet up 40000 unknown vsw2
net11 Ethernet up 40000 unknown vsw1
Check only the IB cards:
dladm show-ib
LINK HCAGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS
net9 10E000015A7460 10E000015A7462
2 down — — FFFF
net5 10E000015A7460 10E000015A7461
1 down — — FFFF
The following 2 commands are not mandatory, but will help
you to easier identify the IB
dladm rename-link net9 ibp1
dladm rename-link
net5 ibp0
Check renaming: dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net1 Ethernet unknown 0 unknown ixgbe1
net2 Ethernet unknown 0 unknown ixgbe2
net0 Ethernet up 1000 full ixgbe0
net3 Ethernet unknown 0 unknown ixgbe3
net6 Ethernet up 1000 full vsw0
ibp1 Infiniband down 0 unknown ibp1
ibp0 Infiniband down 0 unknown ibp0
net4 Ethernet up 10 full usbecm2
net10 Ethernet up 40000 unknown vsw2
net11 Ethernet up 40000 unknown vsw1
dladm show-ib
LINK HCAGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS
ibp0 10E000015A7380 10E000015A7381
1 up — — FFFF
ibp1 10E000015A7380 10E000015A7382
2 down — — FFFF
Create IB default partition:
(the name is under
GWPORT column resulted from previous command/)
dladm create-part -l ibp0 -P ffff ffff.ibp0
dladm show-part
LINK PKEY OVER STATE FLAGS
ffff.ibp0 FFFF ibp0 unknown —-
Create and assign the IP address :
ipadm create-ip
ffff.ib0
ipadm create-addr -T static -a 10.1.10.11/24 ffff.ib0/v4
Important commands to check the status of the connection:
dladm show-ib
ibhosts
ibstat
ibswitches
iblinkinfo.pl –R
grep pciex15b3 /etc/path_to_inst
Should you need further help with your IPoIB configurations on SPARC servers,
you can contact us at isvsupport_ww@oracle.com.