• February 10, 2012

InfiniBand Vocabulary

Before I talk more about InfiniBand, I thought its a good idea to just have a blog space with collection of all terms and abbreviations we use. The InfiniBand Jargon, per se. The first one is easy... the list is long... so lets keep it live and add more as we progress.

IB - InfiniBand

IBTA - InfiniBand Trade Association. They deserve the top position here being the founder and maintainers of InfiniBand Specifications since 1999.

OFED - Open Fabrics Enterprise Distribution. This is an open source community driven software package for InifniBand. The group is also active in interoperability, workshops, architectures and protocol development.

OSI - Open Systems Interconnection. A standard for communication systems and modern day networking based on seven layers.

HCA - Host Channel Adapter. This is a piece of hardware that gets installed inside an end point participating in IB network. Similar to network interface card (NIC) that you see in Ethernet world. In OSI model, this enables Layer 1.

MAC - Media Access Control. An entity is worthless without an identity. MAC address provides identity of an end point in the network at hardware layer. In OSI model, this enables Layer 2.

GUID - Globally Unique Identifier. To keep it simple, I would say this is a fixed hardware address of an end point participating in IB network. Conceptually it is similar to MAC address but has longer address length.

LID - Local Identifier. This is a 16-bit address assigned to end points dynamically in an operational IB network. In OSI model, this enables Layer 2 switching. You must be wondering that if MAC and GUID addresses are similar to each other then why IB introduces another number called LID ? Well, this is an IB implementation to ensure sequential and simplified addressing within a network. Who would like to remember those long hex format GUIDs :) On flip side, the limitation is that we can not have more than 2^16 end points.

SLID - Source LID. In messaging sequence, this will be the originator of packet.

DLID - Destination LID. And this will be the destination of packet.

SM - Subnet Manager. This is a software implementation that takes care of IB network management. I will have more on this topic later but in short, SM is responsible to monitor the connected network periodically for any changes, assign LIDs to end points, create switching tables, manage quality of service (QoS) etc.

IPoIB - Internet protocol over InfiniBand. We are moving up the OSI layered stack now. Remember, in my last blog post I mentioned that our socket based messaging still works in IB. IPoIB is the first step in enabling that. What it means is that we simply assign a Layer 3 IP address to an underlying IB device. The address can be IPv4 or IPv6.

When it comes to networks, we talk about numbers like 10/100/1000/10000Mbps (Ethernet), 11/54/150/300Mbps (WiFi) etc. Its about the signalling rate of bits. As a standard, I will use small letter 'b' for bits and capital 'B' for bytes.

SDR - Single Data Rate. This is baseline at 2.5 Gbps

DDR - Double Data Rate. Next level from SDR. 5.0 Gbps

QDR - Quad Data Rate. Nest level from DDR. 10.0 Gbps

SFP - Small Form-Factor (Hot) Pluggable Transceiver.  These
are special connectors at the end of cables that we use for connecting
network equipment.'+' is added for enhanced version which is capable of 10Gbps signalling

QSFP - Add Quad to the above mentioned SFP and here we get
four SFP links into a single cable. '+' version is also available for
enhanced and this is what gives us 4X 10Gbps.

Sometimes you will also see a splitter cable which has QSFP+ on one end and four SFP+ on other. 

Hmm.. everybody talks about InfiniBand capable to provide 40Gbps. How do we get that then ? Basically, its done at hardware level by the product manufacturers. IB specification allows link aggregation at hardware layer. Two possibilities are 4X and 12X. I am sure you already got it by now.

4X QDR InfiniBand provides us with a 40Gbps link. But can you actually see it from your application's perspective ? Short answer is "No". Lets see why.

I told you earlier that these numbers are signalling rates for bits - On the Wire ! Let me introduce you to yet another term.

8B/10B - 8 bits over 10 bits. This is an encoding mechanism at OSI Link Layer-2. For every 10 bits transmitted on wire, 8 bits carry actual data. The other two bits are not waste but they are used to control the overall signalling. Math is simple here. We use only 80% of the signalling rate. 80% of 40Gbps = 32Gbps.

Okay! So can we see 32Gbps throughput then ? Hmm... may be not.

Without going into too much details, I will simplify it for you. There are a few factors which further reduces the realistic throughput to an end user's application(s).

First one is the hardware capabilities. Compute machines interface with IB HCA through PCI or PCI-E bus. This has certain limitations hence reduces our throughput.

The second one is communication protocol overheads. We know that there may be a few layers between our IB hardware and actual application based on OSI's 7-Layer implementation. Each layer will introduce some processing overhead. So, overall our throughput is reduced from 40Gbps. This is not a fixed number.

Lets think of InfiniBand network as a high speed freeway. Its upto you - what and how you want to drive on this, but watch for the speed limit :)

More next time.. Thanks for your time.

Join the discussion

Comments ( 3 )
  • jason arneil Friday, February 10, 2012

    Hello Neeraj ,

    This could be useful at filling quite a gap in peoples knowledge.

    Some troubleshooting/monitoring tips would be excellent.


  • Neeraj Friday, February 10, 2012

    Jason, I will be posting a lot more here shortly. Just running out of time as you can imagine :)


  • Manjinder Singh Tuesday, March 6, 2012

    Hi Neeraj,

    Wonderful and simple explanations.

    I am loving it please continue the good work.

    I really appreciate it.

    I also understand that not every thing can be explained in a single post and above all in simplistic style.


    Manjinder Singh

Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.