Recent Posts by Neeraj Gupta

When it comes to InfiniBand Partitions, we often hear two terms: Partition and Pkey. Let me try to make a distincton between these two terms for the purpose of easier understanding and its usage in Oracle's Engineered Systems. On Oracle's InfiniBand switch, the partition administration syntax (smpartition) allows us to define custom partition IDs from 0x0002 to 0x7FFE.  0x7FFF (last) is reserved for default partition and 0x0001 (first) is reserved for internal use. After putting...

I know many of you have been working on Oracle's Exalogic and other Engineered Systems. With partitions enabled now, things have gone multi dimension. But its fun. Isn't it ? While you have some EoIB configurations together with InfiniBand partitions, the VNICs are not coming up and staying in WAIT-VHUB state ?  Chances are that you have forgot to add InfiniBand Gateway switches' Bridge-X port GUIDs to your partition. These must be added as FULL members for EoIB to work...

When you want to bring up a compute server in your environment and need InfiniBand connectivity, usually you go through various installation steps. This could involve operating systems like Linux, followed by a compatible InfiniBand software distribution, associated dependencies and configurations. What if you just want to run some InfiniBand diagnostics or troubleshooting tools from a test machine ? What if something happened to your primary machine and while recovering in...

So this is not for the faint heart. I am going to give you a roller coaster ride going at 40 Gb/s. ;) Used some shark teeth and cut the InfiniBand packets apart to show you some screenshots here. Just kidding, dont worry. This post is about how to capture InfiniBand packets and then view them in latest Wireshark. Mellanox has generously released a nice and powerful utility called ibdump. You can visit their website and get a copy till supplies last. So hurry up ! I suggest...

Earlier we discussed about a host participating in different networks or subnets, referred to as multi-homed host. Here I am going to talk about how to handle layer 3 routing. I will break down my discussion into two scenarios - Simple and Not-So-Simple. Simple Scenario Lets assume a host with four network interfaces connected to unique layer 3 subnets respectively. Three subnets are private LANs and the fourth one is a bigger one - WAN. The smaller networks could be for your...

I have been talking to many people who are working hard towards migrating an existing Ethernet based infrastructure or starting from grounds up to deploy their services and applications on InfiniBand based computing environments. At the end of the talk, it all looks very simple.. right ? But where to start.. that's the fun part. Lets give some insight into these type of discussions. Recall one of my previous blog where I focused on a few keywords - Consolidation, Isolation and...

Quick Introduction to Linux Bonding As the name suggests, bonding driver creates a logical network interface by using multiple physical network interfaces underneath. There are various reasons to do so, including link aggregation for higher bandwidth, redundancy, high availability etc. Upper layers communicate through the logical bond interface which has an IP address but eventually the active physical interface(s) communicate to lower layer 2. It also provides transparency to...

Continuing from my last blog about InfiniBand building blocks, now lets review the network switches used inside Oracle's Engineered Systems a little bit in detail. This will help you in understanding the overall integration, network design, architecture and troubleshooting in later articles. There are total two category of network switches used to prepare computing environment inside the rack. InfiniBand Switches - two models used depending on requirements Sun Oracle 36-port...

Virtualization is always a heavily talked subject among infrastructure owners. Someone may ask why ? Often times, at any given time, the capacity of hardware equipment used to deploy services may be more than the services can actually utilize them. Data center real estate and power is expensive. So, what do we want to do ? Consolidation. Hmm.. sure but what about security ? Someone will not like the fact that they are running on a shared set of resources with everything...