Here is a quick look under the covers of the new Sun Datacenter Switch 3x24, the new InfiniBand switch
just announced by Sun at ISC 2008 in Dresden. First some photos and then an explanation of
how this switch is used as a Sun Constellation System component to build clusters with up to
First, the photos:
|Nano Magnum's three heat sinks sit atop Mellanox 24-port InfiniBand 4x switch chips. The
purple object is an air plenum that guides air past the sinks from the rear of the unit.|
|Looking down on the Nano, you can see the three heat sinks that cover the switch chips
and the InfiniBand connectors along the bottom of the photo. The unit has two rows of twelve connectors with the bottom row somewhat
visible under the top row in this photo.|
|The Nano Magnum is in the foreground. The unit sitting on top of Nano's rear deck for display
an InfiniBand NEM. See text for more information.|
You might assume NanoMagnum is either a simple 24-port InfiniBand switch or, if you know that each connector actually carries
three separate InfiniBand 4X connections, a simple 72-port switch. In fact, it is neither. NanoMagnum is a core switch and none of
the three InfiniBand switch chips is connected to the others. Since it isn't intuitive how a box containing three unconnected switch chips can be used to create single, fully-connected clusters, let's look
in detail at how this is done. I've created two diagrams that I hope will make the wiring configurations clear.
Before getting into cluster details, I should explain that a NEM, or Network Express Module, is an assembly that plugs into the back of each of the four shelves in a Sun Blade 6048 chassis.
In the case of an InfiniBand NEM, it contains the InfiniBand HCA logic needed for each blade as well as two InfiniBand leaf switch
elements that are used to tie the shelves into cluster configurations. You can see a photo of a NEM above.
The first diagram (below) illustrates how any blade in a shelf can reach any blade in any other shelf connected to a NanoMagnum
switch. There are a few important points to note. First, all three switch chips in the NanoMagnum are connected to every switch
port, which means that regardless of which switch chip your signal enters, it can be routed to any other port in the switch. Second,
you will notice that only one switch chip in the NEM is being used. The second is used only for creating redundant configurations
and the cool thing about that is that from an incremental cost perspective, one need only buy additional cables and additional
switches--the leaf switching elements are already included in the configuration.
If the above convinced you that any blade can reach any other blade connected to the same switch, the rest is easy. The diagram
below shows the components and connections needed to build a 288-node Sun Constellation System using four NanoMagnums.
Clusters of smaller size can be built in a similar way, as can clusters that are over-subscribed (i.e. not non-blocking.)