IB Updates 3 (OFUV UD)

Finally a timely announcement! Yesterday, Solaris InfiniBand Updates 3 was released. The big addition over the previous Updates 2.1 is the first increment of Open Fabrics User Verbs or OFUV.

What's OFUV you ask? Well, here is the story. Quite a while ago, we delivered uDAPL into Solaris. But over time, with the growing popularity of the Linux OFED InfiniBand releases, OFUV became the pre-dominant OS Bypass InfiniBand framework.

Now the question which comes up from time to time is, "why don't you just port OFED?" The simplified explanation is that it's quite hard to do a straight port for the kernel parts, unless you can completely emulate Linux kernel internals. So instead for the kernel-based InfiniBand services, we are trying to provide equivalent stuff but not necessarily a straight port. But on the userland side of things, it's much more practical to do a port.

So with this new software, we have done a port of userland OFUV APIs on Solaris. Now it's a big project, so this first increment has the Unreliable Datagram (UD) support (based on OFED 1.3 verbs and rdma_cm) and associated drivers (for hermon and tavor), utilities, etc. While the verbs exist for reliable connected (RC) support, the rdma_cm support is not yet in there. Following phases will round out the offering with rdma_cm support for RC and other stuff. And of course, sometime we have to update to OFED 1.4. But finally we have the first part.

So what do you do with userland UD? Lately, it has been used to do high volume, low latency messaging systems on top of InfiniBand, especially in financial services. So stay tuned ...

Technorati Tags:

Comments:

Doesn't seem to work in OpenSolaris. I just tried with 2009.06 (snv_109), with a Mellanox Infinihost III Lx card, but I get this error:

Mar 24 07:51:05 rob-036 arbel: [ID 405241 kern.notice] arbel_attach: ibc_attach failed
Mar 24 07:51:07 rob-036 arbel: [ID 913231 kern.notice] NOTICE: arbel0: driver failed to attach: attach_ibcattach_fail

With an older build (snv_94), I'd been running this card for several months, so it appears that something in the OpenSolaris updates has caused problems with the Infiniband drivers.

Posted by Ross on March 23, 2009 at 05:55 PM PDT #

Hmm, I think I might have spotted part of the problem. From the readme:

"for "MemFree" InfiniBand HCA based on Mellanox MT25204 or MT25208; each of these devices will appear in the Solaris device tree as one of these entries:
pciex15b3,5e8d@<pci-e-address>
pciex15b3,6274@<pci-e-address>
pciex15b3,6282@<pci-e-address>"

However, prtconf shows that I have:
pci15b3,6274 (driver not attached)

So it looks like this card is showing up as "pci15b3" instead of "pciex15b3".

Posted by Ross on March 23, 2009 at 06:14 PM PDT #

Ok, there's a chance I messed up the installation. I missed the step in the instructions where you have to manually create the /opt/SUNW... directory.

Having done that, the installation completed without any warnings, but after a reboot I still have no infiniband, and a new set of errors:

Mar 24 10:27:43 rob-036 genunix: [ID 819705 kern.notice] /kernel/drv/amd64/ib: undefined symbol
Mar 24 10:27:43 rob-036 genunix: [ID 826211 kern.notice] 'ibdm_ibnex_get_waittime'
Mar 24 10:27:43 rob-036 genunix: [ID 472681 kern.notice] WARNING: mod_load: cannot load module 'ib'
Mar 24 10:27:49 rob-036 pcplusmp: [ID 803547 kern.info] pcplusmp: pciex15b3,6274 (arbel) instance 0 vector 0x38 ioapic 0xff intin 0xff is bound to cpu 2
Mar 24 10:27:49 rob-036 arbel: [ID 405241 kern.notice] arbel_attach: ibc_attach failed
Mar 24 10:27:51 rob-036 arbel: [ID 913231 kern.notice] NOTICE: arbel0: driver failed to attach: attach_ibcattach_fail

Posted by Ross on March 23, 2009 at 08:29 PM PDT #

Hi Ross,

Just to update others on the related email I have seen about this issue internally...

I think you we need an OpenSolaris (not Solaris 10) driver for Arbel in your case. FWIW, it's been discussed internally, though I guess it's not yet resolved.

Posted by Ted Kim on April 15, 2009 at 04:50 AM PDT #

Post a Comment:
Comments are closed for this entry.
About

user12611518

Search

Archives
« March 2015
SunMonTueWedThuFriSat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    
       
Today