Monday Jan 14, 2008

OpenSolaris and iWARP

Not much has been posted yet, but I should point out that there is now an OpenSolaris project for iWARP.

Look here: RDMA Offload Framework (ROF)

Technorati Tags:

Wednesday Nov 15, 2006


A while ago, I described RNICPI. I would have liked to have said it's taking over the world. But alas, that is not to be. I really did want RNICPI to succeed. After all, I am the co-chair of the group that authored it. Now, that is not to say there won't be some RNICPI implementations. I wish them luck.

But as far as Sun is concerned, we have decided that the industry momentum is increasingly behind the Open Fabrics Alliance (OFA). Now OFA was originally called OpenIB because of it's InfiniBand roots. And most of its software is still heavily influenced by that. But it also has iWARP support under development. There are some significant milestones still to come before OFA iWARP support can be considered mature, but it's coming along.

Now if you have been reading my blog, you know I am a Solaris developer. Folks familiar with OFA know it's based around Linux (and there is a Windows version too!). So there are various issues involved with leveraging OFA stuff for iWARP on Solaris and making it all work together. But that is what we are doing. And since it's SC06 time, I guess I shouldn't delay blogging about this any longer.

So there you have it. We are trying to take advantage of the growing interest in iWARP based on OFA code and interfaces. Many of the iWARP IHVs have OFA code internally or under development. We feel that OFA is going to become the de facto standard for iWARP software.

More on this story to come ...

Now I would be remiss if I didn't offer all the standard disclaimers. YMMV. I don't time travel. But who knows, I might still be right. :-)

Technorati Tags:

Tuesday Oct 11, 2005


It's long overdue for me to talk about RNICPI (pronounced ARE-NICK-PIE). So let's start from the begining. RNICPI is a working group in ICSC (Interconnect Software Consortium) in OpenGroup. People know OpenGroup mostly for UNIX related standards. OpenGroup also hosts consortium activity to do other standards work. ICSC was formed to deal with software standards arising out of InfiniBand. But the charter includes fast interconnect software standards in general and is not restricted to just InfiniBand.

RNICPI (RDMA-Capable NIC Programming Interface) is a working group under the ICSC. I am currently the co-chair of this group. Essentially, RNICPI was tasked with writing a standard verb level programming interface for RDMA transports. RNICPI 1.0 was released on 21 September 2005. RNICPI is designed to work with both InfiniBand and iWARP, but tries to be as independent of the specific differences of the transports as much as possible. In principle, other future RDMA transports could be added later to RNICPI without disturbing the existing transports. Further, RNICPI is also aiming at being OS-independent and hardware vendor independent. And RNICPI covers both kernel level and user level (typically OS bypass) programming interfaces.

The value of having such a standard means that the same RDMA software stack can exist in multiple OSes, using different transports or different adapters for any particular transport. At the OS/IHV interface, the same interface can be used through out the industry. At least that is the vision. Just so you get some idea of the participation in RNICPI, we had IBM, HP, Sun and 7 other companies (primarily iWARP IHVs) involved in RNICPI.

Now, what is the reality? Well, RNICPI is primarily applicable to non-Microsoft OSes. Microsoft typically retains control of their APIs, so they will probably have their own API. At least Sun and HP officially endorse RNICPI 1.0 as per the presentation at the RAIT 2005 conference. Here at Sun, Solaris uses RNICPI 1.0 as the interface to IHV code for iWARP implementations. We developed a stable InfiniBand interface years ago and probably won't change that anytime soon. On the other hand, RNICPI does give us an option to use for InfiniBand in the future.

In the Linux world, the picture is not clear. First of all, the Linux community has its own independent process to decide what they are going to use. Second, sometime after we started the RNICPI effort, OpenIB announced that they would try to extend their OpenIB InfiniBand programming interfaces to cover iWARP. OpenRDMA is another Linux oriented community which has declared that they will use RNICPI. So it would seem two different groups are potentially trying to be the standard Linux iWARP framework. In my own personal opinion, RNICPI is the superior technical approach. But OpenIB has already been accepted into the Linux kernel, so their follow-on work is likely to get in much easier than OpenRDMA. The two groups have talked. There has been some talk about convergence using the best features of both. But for now the outcome is still not clear in the Linux world.

Technorati Tags:

Thursday Oct 06, 2005

iWARP vs. IB

People often ask me what is Sun's position on iWARP (RDMA on IP, particularly TCP/IP/Ethernet) versus InfiniBand (IB). Of course, the people asking me would really like me to give a clear "this is better" answer (mostly because they have a stake in one of the two contestants). They may point out that Sun is a member of the IB Trade Association, has made public statements about it's IB support, has actual IB products and continues to invest in it. Others ask about our participation in RNIC-PI, our membership in the iWARP interoperability consortium, our current iWARP development in Solaris and our future plans.

Then, there are some who try to convince us which is better. Neither camp is immune to hype. But of course, it is not a simple comparison (as reality often is). Though it went through a tortuous history, InfiniBand is shipping today. Also, IB does have an established and expanding presence in the clustering market. iWARP is much younger. So while you can get it in some forms, its market presence is not really here yet. iWARP's big appeal is the spectre of Ethernet volume. Certainly, Ethernet has a history of defeating other networking alternatives and delivering big volume at low cost. However, the slow progress in standardizing 10G-BaseT (i.e. the low cost port/cable for 10Gb Ethernet) has delayed the real ramping on 10Gb Ethernet.

Currently, IB latency seems to be better than iWARP. It has already a path to higher bandwidth than what 10Gb Ethernet can deliver through DDR/QDR speeds and 12x cables. But when 10Gb Ethernet comes in earnest, it probably will have bigger volume (i.e. lower cost). On the other hand, IBTA is already hatching a plan to run IB over 10G-BaseT cables. That may help with physical layer costs, but that is only one variable. Both camps are laboring mightily to try other tricks to squeeze out cost (e.g. memory free cards). When iWARP really arrives, IB may try to position itself as being slightly faster and bigger bandwidth to justify any cost differences.

Anyway back to the original question ... which one is Sun supporting? In my view, the answer is both. In the end, we are not about to tell a customer that he cannot have what he wants. (Other vendors can try that approach and send any disagreeable customers to us!) If the customer likes InfiniBand, we give it to him. If customers want iWARP, we will offer that too. Obviously, there are going to be cases where one or the other might be more appropriate. We will give advice. But we aren't going to try to change people's minds.

Is this a cop out? No. It just makes business sense. InfiniBand is here today and important enough, so we need to sell it now. iWARP is not yet here, but has great promise. So we need to get it going as soon as possible. Can we afford to pass up either one? No. In both cases, there are significant opportunities. We think this is the best way to go both for us and the customer -- the customer gets to choose.

Since I said all this (plus I am just a lowly engineer), I guess I should offer all the standard disclaimers. Hey, I at least said "In my view ...". But officially, these rantings are not necessarily anyone's views but my own. And in a lame attempt at humor: some assembly required, void where prohibited, batteries not included, sold by weight not volume, do not try this at home, I misspoke (is that actually a word?), plausible deniability, actual mileage my vary, fitness for any particular purpose is disclaimed, weight loss examples not typical, etc.

Technorati Tags: ,

Monday Oct 11, 2004

What is iWARP?

There is way too much to write about and not enough time. Nevertheless, a message on this topic is way overdue. So let's start off with the most basic stuff.

What is iWARP anyway? It's a rather stretched acronym for "Internet Wide Area RDMA Protocol" or more simply Remote DMA on Internet Protocol. (My apologies if you are wondering what "Remote DMA" is, but that will be a topic for another time.) If you are still with me, the heart of this work is actually a set of protocols being standardized in the IETF RDDP working group. The original submissions for this stuff came from the RDMAC.

Back to the protocols. Now just to confuse things, one of the protocols is actually called the "RDMA" protocol and the other "DDP" (Direct Data Placement). These protocols layer on top of framed reliable transports. In particular, this stuff is designed to run over SCTP or TCP with the "MPA" (Marker PDU Aligned Framing) protocol. (TCP needs MPA, because TCP is really a stream protocol, not a message oriented one.) All of this rides on IP itself.

Since IP is the foundation of everything here, this whole stack could run over anything that can do IP. But frankly what drives the whole commercial interest in iWARP is 1 or 10Gb Ethernet TCP/IP with hardware offload to do the protocol processing. Ethernet TCP/IP has the volume. The Gigabit speeds make hardware offload more compelling (otherwise a lot of CPU is wholy consumed by network protocol processing). The potential market is enough to get some big system/OS vendors, NIC vendors and a host of startups excited.

To complete the picture, the UNH-IOL has setup an iWARP Consortium to work on Interoperability. Also, the ICSC has the RNIC-PI working group to work on standardized standard programming interfaces. Also, there is an OpenRDMA group is in the process of getting organized.

iWARP also has something called the verbs. The verbs are an abstract programming interface to iWARP. The verbs are needed because the programming model for using RDMA efficiently is not the same as what most socket based network programming currently uses. Verbs are not a true API, hence the need for groups like RNIC-PI.

iWARP bears many similarities to InfiniBand, especially in its use of RDMA and the verbs. But the main difference is in the transport implementation choices. iWARP was designed to ride on IP allowing it try to ride on the coattails of the whole IP world and on the volume of Ethernet. Some folks have issues with whether this whole stack is rather awkward and ungainly, especially on TCP. But frankly the commercial side ignores all of that because the Ethernet TCP/IP base is key to the perceived market opportunity.

More to come ...

Technorati Tags:




« August 2016