The Most Beautiful Supercomputer in the World

Sorry everyone, but I'm not talking about Ranger, Sun's 500 TFLOP system which is still a-birthing at TACC in Texas. More on Ranger and Sun's Constellation System in the second half of this entry...

With my supercomputer-as-art background from Thinking Machines, I'm perhaps pre-disposed to appreciate nicely done computers and computer installations. I was blown away when I saw the photos below of MareNostrum, currently the 9th largest supercomputer in the world and the largest in Europe, which is installed in a former chapel at the Barcelona Supercomputing Center (BSC). Wow.

[marenostrum at the barcelona supercomputing center, view1]

[marenostrum at the barcelona supercomputing center, view2]

As beautiful as this is, the site is a bit more typical when you look deeper. Here are a few shots:

[marenostrum at the barcelona supercomputing center, under floor]

[marenostrum at the barcelona supercomputing center, rear view]

MareNostrum's Myrinet interconnect fabric requires four cabinets. Myricom did a nice job of reducing the cable count as much as they could by using quad-link ribbon cables between their switch elements. But with 12 separate switch elements in the fabric that means they still have a lot of cables and more cables means more connectors, more points of potential failure. They also use one cable per compute node, as is typically done in cluster configurations. Lots more cables, lots more connectors.

Contrast this with Sun's Constellation System approach, which uses one large, ultra-dense InfiniBand switch (therefore no inter-switch cabling) and node connections that are bundled three per-cable and connected directly to Constellation's ultra-dense blade chassis. This takes complexity management to the next level and makes petascale computing systems an achievable goal.

For a cogent description of Constellation and its value proposition, including some nice diagrams, check out Jonathan's blog.


The Myrinet crossbars in this fabric have 32 ports. So, it would be possible to build a 8192 ports switch with a diameter 5 like TACC. However, it's not cost effective, because the vast majority of clusters out there have less than 256 nodes, which is the port count on a single Myrinet 2000 switch.

Sun's primary motivation to build Magnum was definitively not to make a profitable business.

Posted by guest on August 11, 2007 at 05:34 PM EDT #

Thanks for your comment.

I did not mean to imply that Myrinet could not be used to build large interconnect fabrics. My primary point was that using increasing numbers of small switch elements to achieve that size introduces significant cabling and reliability issues. This is not a Myrinet-specific observation--it is true iof InfiniBand fabrics as well when built from smaller components.

A few observations relative to your comment about Magnum (Sun's switch.) First, I generally agree with your implied point that business at the top of the HPC market does not generally make a lot of money for a vendor. Those high-end customers will argue that "solving their problem solves the problem for all customers with less stringent requirements." While, strictly speaking, this is a true statement, it is also generally true to say (going back to your point about the vast majority of customer
requirements) that if we solve problems at perhaps two orders of magnitude BELOW their requirements, we have still solved the problem for the majority of the market (I am especially thinking of software when I say this.)

Having said that, there ARE ways for a full system vendor like Sun to find a reasonable business case for these systems when you look beyond just the compute nodes and the interconnect fabric. (And even the interconnect in this case offers something valued by customers.)

Last point. HPC is very often the incubator space for new technologies that eventually become important for the much larger commercial, non-technical datacenter. Developing expertise in very large, scalable machine room interconnect fabrics seems a good thing from that perspective, at least that's my view.

Josh Simons

Posted by Josh Simons on August 12, 2007 at 12:49 AM EDT #

That is so sweet. I wish my chapel had one of these ...
Max ... Out! - free online operating system

Posted by Max on August 24, 2007 at 05:26 PM EDT #

wooow :O

Posted by ARIAN ONLINE on August 27, 2007 at 11:01 PM EDT #


Posted by ebrahim on August 28, 2007 at 02:08 AM EDT #

this supercomputer uses the IBM PPC970 microprocessor and I was on that design team. Nice to see people appreciate the machine!

Posted by John Sargis on August 28, 2007 at 05:32 AM EDT #

I studied at UPC, the university that hosts the supercomputer in Barcelona and got to visit it as part of the program I was with. They had monastic chanting in the background as we walked in, and some of the guys I was with got down and prayed at the thing. They had some of there old nodes on display as well, some of them looked straight out of Kubrick. If your ever in Barcelona and you've got some connections, this is a really cool place to visit.

Posted by trey on August 29, 2007 at 04:21 PM EDT #

Why is it in a chapel again? No one questions that at all?

Posted by Travis on December 15, 2007 at 10:45 PM EST #

It's funny that despite all these awesome technological advances that we are enjoying today, when you want to build a super-computer you still need to use miles of cable to hook the damn thing up!

Posted by Matthias on December 16, 2007 at 01:37 AM EST #

Looks realy fantastic!

Posted by myxomop on December 16, 2007 at 04:17 AM EST #

Putting a supercomputer in a former chapel - this reminds me of something James Watson wrote in "The Double Helix":
The book I poked open the most was Francis' copy of The Nature of the Chemical Bond. Increasingly often, when Francis needed it to look up a crucial bond length, it would turn up on the quarter bench of lab space that John had given me for experimental work. Somewhere in Pauling's masterpiece I hoped the real secret would lie. Thus Francis' gift to me of a second copy was a good omen. On the flyleaf was the inscription, "To Jim from Francis - Christmas '51." The remnants of Christianity were indeed useful.

Posted by JMR on December 16, 2007 at 04:47 AM EST #

I'm pretty sure you misused the word "cogent" in the last sentence.

Posted by lawstud on December 16, 2007 at 06:33 AM EST #


Posted by eSpy on December 16, 2007 at 08:21 AM EST #

Awesome, I can really play Crysis on high now.

Posted by fugazzi on December 16, 2007 at 08:27 AM EST #

I'm sorry but this compy looks absolutely nothing like GlaDOS.

Posted by Singe on December 18, 2007 at 12:09 PM EST #

It's really impressive. I study at the UPC, very close to it. It has a surreal feeling... very nice.
/trey: no one really cares, it wasn't used anyway.

Posted by Snale on December 23, 2007 at 08:06 PM EST #

Posted by mirc on November 21, 2008 at 10:57 AM EST #

as beautiful as it's performance...combining technology with art... Let's Think Out of The Box Portal Lifestyle On The Net Tricks and Tips Big Conspiracies

Posted by Portal Lifestyle On The Net on January 12, 2010 at 04:26 AM EST #

pozri si toto :))

Posted by Peter on February 25, 2010 at 05:13 PM EST #

Post a Comment:
  • HTML Syntax: NOT allowed

Josh Simons


« July 2016