Thursday Mar 24, 2011

catching up: Oracle and Mellanox

Back in October 2010, Oracle had this press release. The announcement was titled "Oracle Makes Strategic Investment in Mellanox Technologies, Ltd. -- Acquires 10.2% of Ordinary Shares". The message goes on about the importance of InfiniBand and the partnership between the companies.

But since I blog mostly about Solaris and InfiniBand, I would like to point out this part: Mellanox intends to make Oracle Solaris a core supported operating system platform to help meet increasing customer demand.

As always, this blog should only be regarded as my personal opinion. If some official statement contradicts me, I am probably wrong.

Wednesday Mar 23, 2011

catching up: Solaris IB in Supercluster

In December 2010, Oracle announced SPARC Supercluster with some new record setting benchmark results. Supercluster was also mentioned in a press release about the overall refresh of the SPARC product line. So what is a "Supercluster"?

Supercluster is listed under "Oracle Optimized Solutions" on the website with a datasheet for the T3-2 version and an overall FAQ. These solutions are for running Oracle Databases and other applications using SPARC machines (so far M5000, T3-2 and T3-4 are mentioned), Solaris, InfiniBand, FlashFire (flash cache) and ZFS Storage Appliances. The T3-2 datasheet mentions using using Solaris 10 9/10 (Update 9) or update 8 with patches and the 7420 ZFS Storage Appliance with InfiniBand.

So with this solution, Solaris 10 InfiniBand on SPARC is being used. Plus, under the covers Solaris IB is also being used in the ZFS Storage appliance.

As always, this blog should only be regarded as my personal opinion. If some official statement contradicts me, I am probably wrong.

Tuesday Mar 22, 2011

catching up: Solaris IB in Exadata & Exalogic

Currently, one of the biggest stories at Oracle is the Exadata product. The current generation of this product is known as "X2" with a version using 2-socket x86 servers (X2-2) and one using 8-socket x86 servers (X2-8). It is well known that InfiniBand is the communication fabric at the heart of these products.

What is perhaps not as well known is that Solaris has been announced as an option for this product family. The Solaris 11 Express announcement mentions that Oracle Solaris 11 Express will also power the Oracle Exadata X2-2 and X2-8 Database Machines, as well as the Oracle Exalogic Elastic Cloud machine.

The Exalogic product family also uses an InfiniBand communication fabric with an option to use Solaris 11 Express. The Exalogic X2-2 product uses x86 2-socket servers. As mentioned in the previous link, Exalogic has been optimized for integrated use together with Exadata. Now you can have InfiniBand's high bandwidth and low latency operating through all your internal data center tiers.

The Exalogic T3-1B version uses SPARC processors with Solaris 11 Express. The announcement link also mentions Solaris 10 Zones as a way to migrate from Solaris 10 to this platform.

For further information on the new IB features in Solaris 11 Express, please see my previous post.

As always, this blog should only be regarded as my personal opinion. If some official statement contradicts me, I am probably wrong.

Monday Mar 21, 2011

catching up: IB in the ZFS Storage Appliance

It's well known that the ZFS Storage Appliance uses Solaris under the covers. I am not the expert in this area, but I do know that Solaris InfiniBand content has been steadily increasing with each new release. Folks like Cindi are the real experts in this area. For full details, go to the FishWorks Software Updates wiki.

InfiniBand support first came in the 2009.Q3 release with support for IPoIB, which of course allows any supported IP based protocol (e.g. NFS or iSCSI) to be run over IB links. This release also had support for NFS over RDMA, which uses IB's RDMA capabilities with NFS.

In the 2010.Q1 release, iSER (iSCSI version with RDMA) and SRP (SCSI RDMA Protocol) were added. Also in 2010, the 2010.Q3 version added Connected Mode for IPoIB to boost performance and a new IPoIB administrative model, which parallels the dladm improvements in the "Nevada" development train.

Of course that isn't the end of the story, so stay tuned to see what happens in future releases.

As always, this blog should only be regarded as my personal opinion. If some official statement contradicts me, I am probably wrong.

Friday Mar 18, 2011

catching up: IB in Solaris 11 Express

The "Nevada" development train produced various OpenSolaris releases which added various new IB features. Then, in last November Solaris 11 Express released, which in turn is leading to Solaris 11.

When you compare Solaris 10 to Solaris 11 Express, you will notice some new IB features. In storage, iSER (iSCSI Enhanced for RDMA) and the SRP (SCSI RDMA Protocol) target are now available (these had released previously under OpenSolaris). Also RDS (Reliable Datagram Sockets) version 3 is now available to improve Oracle RAC support. SDP (Sockets Direct Protocol) got a make over to drastically improve performance. Further, the SDP interposition library (libsdp) is now available to provide transparent redirection from TCP to SDP when desired. IPoIB also got a new dladm administrative interface to realign it with the rest of networking.

Finally, Open Fabrics User Verbs (OFUV) was added in Solaris 11 Express. Many folks regard the OFED Linux distribution as the de facto standard for InfiniBand. OFUV is a port of most of the userland libraries and commands of OFED. Now Linux folks can recompile their Linux InfiniBand apps on Solaris 11 Express and use many of the familiar InfiniBand administration commands that they are used to.

And of course, more is coming for Solaris 11.

As always, this blog should only be regarded as my personal opinion. If some official statement contradicts me, I am probably wrong.

Thursday Mar 17, 2011

catching up: IB in Solaris 10

It has been such a long time since I posted here, but I am hoping to quickly catch up. First up is the topic of Solaris 10. It looks like I have not done an update on that topic since Solaris 10 5/09 (Update 7).

In Solaris 10 10/09 (Update 8), there were lots of enhancements and bug fixes trickling down from our "Nevada" development. I will just mention some of the more notable ones. In general, we started using Reserved L_Key and PCI Relaxed Ordering which was a big boost in performance. ConnectX also added FMA and PCI Relaxed Ordering support. IPoIB moved to GLDv3 and also had a lot of performance work done on it.

In Solaris 10 9/10 (Update 9), the story is much the same with another wad of stuff going in. This time ConnectX got support for 4K IB MTU. IPoIB had more performance improvements added, including the option to use "Connected Mode" which uses the IB RC transport type allowing MTUs up to nearly 64K. NFS over RDMA also got a big modernization update to bring it in line with the Nevada train. Plus, there were lots of general robustness improvements making more use of IB subnet manager events.

And of course, more is coming for Solaris 10.

As always, this blog should only be regarded as my personal opinion. If some official statement contradicts me, I am probably wrong.

Friday Oct 09, 2009

InfiniBand on OpenStorage, etc.

Some exciting news: InfiniBand has been added to the OpenStorage offerings starting with the Q3.2009. Check out Cindi's blog here.

Of course, there have improvements for InfiniBand support in Solaris 10 10/09 release migrating over from the stuff in Nevada.

I have been told the X4239A DDR FEM HCA is really for the X6450 blade.

And there have been more Sun QDR switch models now too.

And of course, if what I say is contradicted elsewhere by official stuff, I am probably wrong!

Technorati Tags:

Friday May 29, 2009

Solaris/IB and Open Source

OpenSolaris is known for being open source. In the InfiniBand area, the framework and most Upper Level Protocols (ULPs) were open source right from the start of OpenSolaris. However, we had to keep the Host Channel Adapter (HCA) drivers in closed source because of prior legal agreements. Well this took a while to sort out, and then finally make the changes to the source tree.

So I am happy to announce that the HCA drivers are now open source. You can look at them here.

There is still part of one ULP (Sockets Direct Protocol) which is in closed source, but that is the only remaining Solaris/IB item in closed source. Unfortunately, it is not likely to change any time soon, because of the way SDP was licensed. For similar reasons, even Linux/OFED has some problems in this area.

Technorati Tags:

Wednesday May 27, 2009

SRP target

As noted here, the SRP target for COMSTAR just went back to Solaris Nevada, which is of course the base for future OpenSolaris releases. So now, Solaris can be an SRP target as well as an iSER target or an NFS/RDMA server. So you can get your blocks and files through all the popular flavors of RDMA protocols on IB.

Technorati Tags:

Thursday May 14, 2009

Java can use SDP

Here is an item that was recently pointed out to me. You can now configure Java to use Sockets Direct Protocol (SDP) on InfiniBand, see this article.

Technorati Tags:

Monday May 11, 2009

DDR FEM

Along with all the other Nehalem announcements related to IB was one other IB related item which I haven't talked about yet. That's the X4239A DDR Fabric Expansion Module (FEM). It's an HCA with PCIe 2.0 I/O interfaces and dual DDR 4x IB ports for connection to the in-chasis IB fabrics.

Now the x6275 blade has a QDR HCAs built in. So what would you do with this item? After looking here, it appears that you use it with the x6270 blade to connect them to the QNEM.

Technorati Tags:

Friday May 01, 2009

ConnectX driver in Solaris 10

With the release of Solaris 10 5/09 (aka Solaris 10 Update 7), the ConnectX HCA driver (called "hermon") finally makes it's appearance in Solaris 10. It has been available in OpenSolaris since Nevada build 100. But now it finally makes its way to the Solaris 10. Download Solaris 10 5/09 here. And of course it's free.

Technorati Tags:

Tuesday Apr 14, 2009

iSER

As other folks have mentioned, iSCSI Extensions for RDMA (iSER) was recently integrated into build 112 of OpenSolaris. Congratulations to the iSER team! The general background on iSER is here with the Solaris architecture specifics here. But the general idea is iSCSI with InfiniBand RDMA to drive up performance.

Technorati Tags:

IB QDR and Nehalem (updated)

In general, InfiniBand DDR HCAs were already limited by the IO bus under PCIe 1.X. With today's Nehalem (Xeon 5500 series) related announcements come a bunch of PCIe 2.0 servers (including x2270, x4170, x4270, x4275 servers plus x6270 and x6275 blades). Now x4 DDR should be able to operate at full speed with a PCIe 2.0 x8 slot and x4 QDR is now realistic (though probably still IO bus limited). The x6275 blade even has on-board QDR HCAs installed. For some great shots of the boards, look here.

So in addition to earlier HCA offerings, we now have this QDR HCA story:

Sun OptionForm FactorChipIO BusIB Ports
(per HCA)
Card MemorySolaris Driver
X4237ACardConnectX-IBPCIe 2.0 x8Two x4 QDRMemfreehermon
X4219A-ZExpress Module

Edit: I think I might finally understand this now! The X5500A SunBlade 6048 QDR NEM has switches, but no HCAs. This design is different from how the DDR NEM works, which has the HCAs in it too. Instead, the QDR NEM is meant to be used with on-board HCAs like with the x6275 blade.

But both IB NEMs for the SunBlade 6048 are meant to be used as part of an overall Sun Constellation system with the final stage of the fat-tree (or torus for the QDR one) switch network incorporated directly in the NEM itself.

And of course, if there is any contradiction, I am sure the official stuff takes precedence over anything I say.

Technorati Tags:

Friday Feb 06, 2009

IB Updates 3 (OFUV UD)

Finally a timely announcement! Yesterday, Solaris InfiniBand Updates 3 was released. The big addition over the previous Updates 2.1 is the first increment of Open Fabrics User Verbs or OFUV.

What's OFUV you ask? Well, here is the story. Quite a while ago, we delivered uDAPL into Solaris. But over time, with the growing popularity of the Linux OFED InfiniBand releases, OFUV became the pre-dominant OS Bypass InfiniBand framework.

Now the question which comes up from time to time is, "why don't you just port OFED?" The simplified explanation is that it's quite hard to do a straight port for the kernel parts, unless you can completely emulate Linux kernel internals. So instead for the kernel-based InfiniBand services, we are trying to provide equivalent stuff but not necessarily a straight port. But on the userland side of things, it's much more practical to do a port.

So with this new software, we have done a port of userland OFUV APIs on Solaris. Now it's a big project, so this first increment has the Unreliable Datagram (UD) support (based on OFED 1.3 verbs and rdma_cm) and associated drivers (for hermon and tavor), utilities, etc. While the verbs exist for reliable connected (RC) support, the rdma_cm support is not yet in there. Following phases will round out the offering with rdma_cm support for RC and other stuff. And of course, sometime we have to update to OFED 1.4. But finally we have the first part.

So what do you do with userland UD? Lately, it has been used to do high volume, low latency messaging systems on top of InfiniBand, especially in financial services. So stay tuned ...

Technorati Tags:

About

user12611518

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today