Thursday Mar 24, 2011

catching up: Oracle and Mellanox

Back in October 2010, Oracle had this press release. The announcement was titled "Oracle Makes Strategic Investment in Mellanox Technologies, Ltd. -- Acquires 10.2% of Ordinary Shares". The message goes on about the importance of InfiniBand and the partnership between the companies.

But since I blog mostly about Solaris and InfiniBand, I would like to point out this part: Mellanox intends to make Oracle Solaris a core supported operating system platform to help meet increasing customer demand.

As always, this blog should only be regarded as my personal opinion. If some official statement contradicts me, I am probably wrong.

Wednesday Mar 23, 2011

catching up: Solaris IB in Supercluster

In December 2010, Oracle announced SPARC Supercluster with some new record setting benchmark results. Supercluster was also mentioned in a press release about the overall refresh of the SPARC product line. So what is a "Supercluster"?

Supercluster is listed under "Oracle Optimized Solutions" on the website with a datasheet for the T3-2 version and an overall FAQ. These solutions are for running Oracle Databases and other applications using SPARC machines (so far M5000, T3-2 and T3-4 are mentioned), Solaris, InfiniBand, FlashFire (flash cache) and ZFS Storage Appliances. The T3-2 datasheet mentions using using Solaris 10 9/10 (Update 9) or update 8 with patches and the 7420 ZFS Storage Appliance with InfiniBand.

So with this solution, Solaris 10 InfiniBand on SPARC is being used. Plus, under the covers Solaris IB is also being used in the ZFS Storage appliance.

As always, this blog should only be regarded as my personal opinion. If some official statement contradicts me, I am probably wrong.

Tuesday Mar 22, 2011

catching up: Solaris IB in Exadata & Exalogic

Currently, one of the biggest stories at Oracle is the Exadata product. The current generation of this product is known as "X2" with a version using 2-socket x86 servers (X2-2) and one using 8-socket x86 servers (X2-8). It is well known that InfiniBand is the communication fabric at the heart of these products.

What is perhaps not as well known is that Solaris has been announced as an option for this product family. The Solaris 11 Express announcement mentions that Oracle Solaris 11 Express will also power the Oracle Exadata X2-2 and X2-8 Database Machines, as well as the Oracle Exalogic Elastic Cloud machine.

The Exalogic product family also uses an InfiniBand communication fabric with an option to use Solaris 11 Express. The Exalogic X2-2 product uses x86 2-socket servers. As mentioned in the previous link, Exalogic has been optimized for integrated use together with Exadata. Now you can have InfiniBand's high bandwidth and low latency operating through all your internal data center tiers.

The Exalogic T3-1B version uses SPARC processors with Solaris 11 Express. The announcement link also mentions Solaris 10 Zones as a way to migrate from Solaris 10 to this platform.

For further information on the new IB features in Solaris 11 Express, please see my previous post.

As always, this blog should only be regarded as my personal opinion. If some official statement contradicts me, I am probably wrong.

Monday Mar 21, 2011

catching up: IB in the ZFS Storage Appliance

It's well known that the ZFS Storage Appliance uses Solaris under the covers. I am not the expert in this area, but I do know that Solaris InfiniBand content has been steadily increasing with each new release. Folks like Cindi are the real experts in this area. For full details, go to the FishWorks Software Updates wiki.

InfiniBand support first came in the 2009.Q3 release with support for IPoIB, which of course allows any supported IP based protocol (e.g. NFS or iSCSI) to be run over IB links. This release also had support for NFS over RDMA, which uses IB's RDMA capabilities with NFS.

In the 2010.Q1 release, iSER (iSCSI version with RDMA) and SRP (SCSI RDMA Protocol) were added. Also in 2010, the 2010.Q3 version added Connected Mode for IPoIB to boost performance and a new IPoIB administrative model, which parallels the dladm improvements in the "Nevada" development train.

Of course that isn't the end of the story, so stay tuned to see what happens in future releases.

As always, this blog should only be regarded as my personal opinion. If some official statement contradicts me, I am probably wrong.

Friday Mar 18, 2011

catching up: IB in Solaris 11 Express

The "Nevada" development train produced various OpenSolaris releases which added various new IB features. Then, in last November Solaris 11 Express released, which in turn is leading to Solaris 11.

When you compare Solaris 10 to Solaris 11 Express, you will notice some new IB features. In storage, iSER (iSCSI Enhanced for RDMA) and the SRP (SCSI RDMA Protocol) target are now available (these had released previously under OpenSolaris). Also RDS (Reliable Datagram Sockets) version 3 is now available to improve Oracle RAC support. SDP (Sockets Direct Protocol) got a make over to drastically improve performance. Further, the SDP interposition library (libsdp) is now available to provide transparent redirection from TCP to SDP when desired. IPoIB also got a new dladm administrative interface to realign it with the rest of networking.

Finally, Open Fabrics User Verbs (OFUV) was added in Solaris 11 Express. Many folks regard the OFED Linux distribution as the de facto standard for InfiniBand. OFUV is a port of most of the userland libraries and commands of OFED. Now Linux folks can recompile their Linux InfiniBand apps on Solaris 11 Express and use many of the familiar InfiniBand administration commands that they are used to.

And of course, more is coming for Solaris 11.

As always, this blog should only be regarded as my personal opinion. If some official statement contradicts me, I am probably wrong.

Thursday Mar 17, 2011

catching up: IB in Solaris 10

It has been such a long time since I posted here, but I am hoping to quickly catch up. First up is the topic of Solaris 10. It looks like I have not done an update on that topic since Solaris 10 5/09 (Update 7).

In Solaris 10 10/09 (Update 8), there were lots of enhancements and bug fixes trickling down from our "Nevada" development. I will just mention some of the more notable ones. In general, we started using Reserved L_Key and PCI Relaxed Ordering which was a big boost in performance. ConnectX also added FMA and PCI Relaxed Ordering support. IPoIB moved to GLDv3 and also had a lot of performance work done on it.

In Solaris 10 9/10 (Update 9), the story is much the same with another wad of stuff going in. This time ConnectX got support for 4K IB MTU. IPoIB had more performance improvements added, including the option to use "Connected Mode" which uses the IB RC transport type allowing MTUs up to nearly 64K. NFS over RDMA also got a big modernization update to bring it in line with the Nevada train. Plus, there were lots of general robustness improvements making more use of IB subnet manager events.

And of course, more is coming for Solaris 10.

As always, this blog should only be regarded as my personal opinion. If some official statement contradicts me, I am probably wrong.

Monday Jul 21, 2008

IB Updates 2

Last week we released Solaris InfiniBand Updates 2 through the Sun Downloads site. This replaces the earlier Solaris InfiniBand Updates 1. The new version supports the Mellanox ConnectX HCA as well as the cards supported in the earlier version. The new update is based on Solaris 10 5/08, which is the latest update of Solaris 10. Support for flashing the HCA firmware and for uDAPL (used by Sun's ClusterTools MPI) is also included. Sun options X4217A-Z, X4216A-Z and X5196A-Z are all based on the ConnectX HCA as well as other IHV cards.

Note that further down the line, various components of the download release will make their way into the mainline Solaris releases. But this download release makes them available now for those who are eager to try them out. Whenever components go back into mainline Solaris, they will be removed from this download release series.

Download Solaris InfiniBand Updates 2 here. And, of course, the software is free.

Technorati Tags:

Friday Jun 27, 2008

Sun IB News Round-Up 06.2008

So just in case you missed these...

  • The Ranger system at University of Texas using the Sun Constellation system clocked in as number 4 on the Top 500 for June 2008.

  • Sun announced a smaller NanoMagnum switch for the Sun Constellation system.

  • Solaris 10 5/08 released with Sockets Direct Protocol (SDP) support and the fwflash utility for updating HCA firmware. Both are also in OpenSolaris 05.2008. And you can get either version of Solaris for free.

  • ClusterTools 8 Early Access 2 released. ClusterTools is Sun's version of OpenMPI and of course has IB support. This version has both Solaris and Linux support. And of course it's free.

Technorati Tags:

HCA Line-up

To state the obvious, I have not blogged in a while. But here is an article I have wanted to do for some time. Occasionally, people wonder which HCAs are offered by Sun. So here is a summary. Yes there are numerous IHV cards out there. But these HCAs are the ones you can get off the Sun price list.

Sun OptionForm FactorChipIO BusIB Ports
(per HCA)
Card MemorySolaris Driver
PCI-X 133Dual x4 SDR256 MBtavor
InfiniHost III Ex
PCIe x8Dual x4 SDR256 MBtavor
X1288A-ZExpress Module
X1289A-ZSunBlade 8000 NEM
(10 HCAs)
InfiniHost III Lx
PCIe x8One x4 DDRMemfreearbel
X4217A-ZCardConnectX-IBPCIe x8Dual x4 DDRMemfreehermon
X4216A-ZExpress Module
X5196A-ZSunBlade 6048 NEM
(12 HCAs + 2 switches)

Now all the notes and explanation: The Express Module is the PCIe hot-plug form factor used by Sun blade servers. NEM is our Network Express Module form factor which plugs into all the blades in a shelf. And because the NEMs have multiple HCAs in them, you have to note that the IB Ports column is listing the ports per HCA. The term Memfree means it uses host memory instead of using memory on the card. The arbel driver is currently available through our IB Updates 1. The hermon driver is NOT out yet, but it's coming; and before you ask I can't say any more than that. The X5196A-Z has the final stage of the switch setup for a Sun Constellation system built into the NEM.

If you are trying to use third party cards, the thing that might confuse you is that two different drivers are used with InfiniHost III Ex cards. If there is card memory, the Solaris tavor driver is used. On the other hand, Memfree cards use the Solaris arbel driver.

And of course, if there is any contradiction, I am sure the official stuff takes precedence over anything I say.

Technorati Tags:

Thursday Feb 07, 2008

Using IB for world records

Here are some recent blog entries featuring Sun hardware using InfiniBand in action. In each case, InfiniBand is being used with Sun hardware to challenge for world records.

  • More Ranger Facts and Figures gives a run down on the stats for the huge TACC installation which includes 62,976 CPU cores in the system. We won't know how it places on the Top500 until June.

  • A trio of stories about world record benchmarks using Sun gear connected by InfiniBand: ABAQUS v6.7,   EXA PowerFLOW,   Fluent.

Technorati Tags:

DDR Patch

Back when we came out with our first Solaris HCA driver, InfiniBand only had what is now called single data rate (SDR). Now the situation is different. Not only are double (DDR) and quad (QDR) rates spec'ed for IB, DDR is common out in the customer base. To address this, we have fixes that are going to be released in the next update of Solaris 10, which means they are also in OpenSolaris already. But to get it out faster, the fixes are also being pushed out in official Solaris patches.

So if you have a DDR capable HCA that runs our "tavor" driver, try out the patch (sparc: 126647-02, x86: 126648-02). Of course, remember to be sure your switch is configured for DDR too. Folks using Sun Connection Update Manager should have seen these come out already in the patch stream. BTW, if you are wondering about the "arbel" driver, these fixes are already in our recently released InfiniBand Updates 1.

I should also mention that this patch contains some other fixes which improve our uDAPL performance (used by Sun's MPI implementation in ClusterTools) as well as other fixes which might be of interest to our Solaris IB users. Check it out.

Technorati Tags:

Friday Jan 25, 2008

Sun Downloads for InfiniBand

Recently, some parts of the Sun download site were reorganized. Now, it is a bit easier to find some InfiniBand items, since they are now in the InfiniBand section under the Hardware Drivers category.

Currently, we have some Linux stuff there as well as these add-ons to the base InfiniBand support in Solaris:

  • Solaris InfiniBand Updates 1 - which I talk more about here

  • SCSI RDMA Protocol (SRP) Driver

  • Firmware Flash Update Tool - to update your HCA firmware when necessary

Of course, this is not the complete list of downloads with InfiniBand content. I don't totally understand the classification system, but I guess it depends on whether they are primarily InfiniBand or more generic than that.

So for example, there is HPC ClusterTools 7.1 which Sun's MPI product which also has InfiniBand support and can be downloaded. But it's in a different section. I think there are some others like this, but you get the idea.

Oh yeah, did I mention that all the downloads I mentioned above are all FREE.

Technorati Tags:

Monday Jan 14, 2008

OpenSolaris and IB

Just wanted to point out some OpenSolaris projects for InfiniBand ...

IP on InfiniBand - Of course, IPonIB is already in Solaris, but this OpenSolaris project is about futures.

iSCSI Extensions for RDMA (iSER) - This project includes both initiator and target.

NFS over RDMA - Of course, this is both client and server.

OpenFabrics User Verbs - This project is a port of the OpenFabric's userland verb interface to Solaris. Once this is done, many apps which run on OFED could then be ported to run on Solaris.

Technorati Tags:

Friday Jan 11, 2008

Solaris InfiniBand Updates 1

Like many other bloggers, I have to simply own up to the reality that blogging is bursty. Often it's a particular event rather than a lifestyle habit that motivates posting.

Well, this time I am motivated by the release of Solaris InfiniBand Updates 1. If you want cut to the chase, then get it here. But if you have no idea what this is about, then read on ...

This product is a FREE software download containing additional software for Solaris 10 InfiniBand. To run it, you need Solaris 10 8 / 07 (Update 4). From the name, you can probably guess that it will not be the only release in this series. You may wonder if this software is also going back into the other Solaris release trains. The answer is YES. It is our intention to eventually fold all this stuff back into the mainstream Solaris releases. So it is going to end up there. But for now, to satisfy the demands of our rapidly growing InfiniBand market in both HPC and the Enterprise, it's coming out NOW to minimize delay in getting to the market.

So what's in this release? Let me be the first to admit the name of the product is something only an accountant would like. The name really does not tell you what's in it. This product contains support for MemFree InfiniBand HCAs based on the Mellanox MT25204 and MT25208 chips (these PCI IDs: 15b3,5e8d   15b3,6274   15b3,6828). For the non-IB crowd, "MemFree" means that the HCA uses host memory instead of memory on the I/O adapter card for the HCA state (so it's LESS EXPENSIVE). So in particular the Sun 10-Port 4x DDR IB NEM for the SunBlade 8000/8000P (option X1289A-Z) is supported (see here, link to a drawing of the NEM). Of course, there are also numerous third party cards also using these Mellanox chips. Note that this software is only for the "MemFree" cards, other Sun software should be used for the cards (and Express Modules) with on-board memory.

In other Sun InfiniBand news, I should also point out that ClusterTools 7.1 (Sun's MPI based on OpenMPI 1.2.4) has been released. See this article.

Technorati Tags:

Wednesday Jun 27, 2007

A big switch ...

Others have mentioned it, but in case you missed it ...

Folks may want to check out the YouTube video of the unveiling of Sun's 3456 port InfiniBand switch.

If you are wondering what you do with such a big switch, you build an HPC setup something like this.

Technorati Tags:




« July 2016