Tuesday Sep 22, 2009

New Image for Old Blog

Yesterday, I posted the wrong image for the sequential read experiment.  That's been corrected and the words now match the image. :)

Infiniband for Q3.2009

The lastest release of the SS7000 software includes support for Infiniband HCAs.  Each controller may be configured with a Sun Dual Port Quad Rate HCA (Sun option x4237) in designated slots.  The slot configuration varies by product with up to three HCAs on the Sun Storage 7410.  The initial Infiniband upper level protcols (ULP) include IPoIB and early adopter access for NFS/RDMA.  The same file and block based data protocols (NFS, CIFS, FTP, SFTP, HTTP/WebDav, and iSCSI) we support over ethernet are also supported over the IPoIB ULP.  OpenSolaris, Linux, and Windows clients with support for the OpenFabrics Enterprise Distribution (OFED 1.2, OFED 1.3, and OFED 1.4) have been tested and validated for IPoIB. NFS/RDMA is offered for early adopters of the technology for Linux distributions that run with the 2.6.30 kernel and greater.

Infiniband Configuration

Infiniband IPoIB datalink partition and IP interface configuration is easy and painless using the same network BUI page or CLI contexts as ethernet. Port GUID information is available for configured partitions on the network page as shown below. This makes it easy to add SS7000 HCA ports to a partition table on a subnet manager.  Once a port has been added to a partition on the subnet manager, the IPoIB device will automatically appear in the network configuration.  At this point, the device may be used to configure partition datalinks and then interfaces.  If desired, IP network multi-pathing (IPMP) can be employed by creating multi-pathing groups for the IPoIB interfaces.

HCA and port GUID and status information may also be found on the hardware slot location.  Navigate to the Maintenance->Hardware->Slots for the controller and click on the slot information icon to get see firmware, GUID, status and speeds associated with the HCA and ports.

Performance Preview

So how does Infiniband perform in the SS7000 family?  Well, it really depends upon the workload and a adequately configured system.  Here, I'll demonstrate two simple workloads on a base SS7410.


  • Sun Storage 7410
  • Software release Q3.2009
  • 2 x quad core AMD Opteron 2.3 GHz CPUs
  • 64GBytes DRAM
  • 1 JBOD (23 disk, each 750 GB) configured for mirroring
  • 2 logzillas configured for striping
  • 2 Sun Dual Port DDR Infiniband HCA, one port each configured across two separate subnets


  • 8 x blade servers, each containing:
  • 2 x quad core Intel Xeon 1.6 GHz CPUs
  • 3GBystes DRAM
  • 1 Sun Dual Port Infiniband HCA EM
  • Filesystems mounted using NFSv4, forcedirectio
  • Solaris Nevada build 118
  • 8 Solaris Nevada (build 118) clients: 4 clients connected to subnet 1 and 4 clients connected to subnet 2


  • Sun 3x24 Infiniband Data Switch, switches 0 (subnet 1) and 2 (subnet 2) configured across server and clients
  • 2 OFED 1.4 OpenSM subnet managers operating as masters for switch 1 and 2

The SS7410 is really under-powered (2 CPUs, 64G memory, 1 JBOD, 2 logzillas, DDR HCAs) and no where near its operational limitations. 

Sequential Reads

In this experiment, I used a total of 8 clients with up to 5 threads each performing sequential reads from a 10GB file in a slightly modified version of Brendan's seqread.pl script.  The clients are evenly assigned to each of the HCA ports.  More than 5 threads per-client did not yield any significant gain as I hit the maximum amount I could get from the CPU.  I ran the experiment twice: once for NFSv4 over IPoIB and once for NFSv4/RDMA. As expected, IPoIB yields better results with smaller block sizes but I was surprised to see IPoIB outperform NFS/RDMA with 64K transfer block sizes and stay in the running with every size in between.

I'm using the default quality of service (QOS) on the subnet manager and clients that are evenly assigned to each of the HCA ports.  As a result, we can see a nice even distribution of network throughput across each of the devices and IOPS per-client. 

Synchronous Writes

In the read experiment, I was able to hit an upper bound on CPU utilization at about 8 clients x 5 threads.  What will it take to reach a maximum limit for synchronous writes?  To help answer that question, I'll use a stepping approach to the single synchronous write experiment above.  Looping through my 8 clients at one minute intervals, I'll add a 4K synchronous write thread every second until the number of IOPS levels.  At about 10 threads per client, we start to see the the number of IOPS reach a maximum.  This time CPU utilization is below its maximum (35%) but latency turns into a lake-effect ice storm.  We eventually top out at 38961 write IOPS for our 80 client threads.

As a sanity check, I also captured the per-device network throughput.  If I account for the additional NFSv4 operations and packet overhead, 93.1MB/sec seems reasonable.  I ran this experiment with NFS/RDMA and discovered a marked drop-off (30%) in IOPS when run for a long period.  Until then, NFS/RDMA performed as well as IPoIB.  Something to investigate.


I have a baseline for my woefully underpowered SS7410.  For sequential reads, I quickly bumped into CPU utilization limits at  40 client threads.  With the synchronous write workload, I top out 38691 IOPS due to increased disk latency.  But all is not lost, the SS7410 is far from its configurable hardware limitations.  The next round of experiments will include:

  • Buff up my 7410: give it two more CPUs and double the memory to help with reads
  • Add more JBODS and logzillas to help with writes
  • Configure system into a QDR fabric to help the overall throughput

Tuesday Sep 08, 2009

Better Late, Than Never

I've been remiss in posting and completely missed reporting on a couple of new Q2 features for the Fishworks software stack. In Q2, we introduced three new secure data protocols: HTTPS, SFTP, and FTPS.  Dave Pacheco covers HTTPS in his blog so I'll highlight SFTP and FTPS here.


Our FTP server is built from the proFTPD server software stack.  In Q2, we updated the server to version 1.3.2 to take in a number of critical bug fixes and add support for FTP over SSL/TLS (FTPS).  The proFTPD server implements FTP over SSL/TLS in accordance with the FTP Security Extensions defined by RFC 2228.  Not all FTP clients support the FTP security extensions but a list of clients that do may be found here.

Enabling FTPS on a Fishworks appliance is very simple.  From the FTP service configuration BUI page or CLI context, an administrator may optionally select to turn on FTPS for the default port or an alternate port.  If FTPS is enabled for a port, the FTP server will use TLSv1 for its authentication and data channels. 


The SSH File Transfer Protocol (SFTP) is a network protocol that provides file transfer over SSH. SFTP is a protocol designed by the IETF SECSH working group.  SFTP does not itself provide authentication and security but rather delegates this to the underlying protocol. For example, the SFTP server used on the SS7000 is implemented as subsystem of OpenSSH software suite.  The SFTP server is responsible for interpreting and implementing the SFTP command set but authentication and securing the communication channels over which the server transfers data is the responsibility of the underlying OpenSSH software.  SFTP should not be confused with:

  • FTP over SSL/TLS (FTPS)
  • FTP over SSH
  • Simple File Transfer Protocol
  • Secure Copy (SCP)

Configuration of the SFTP service is very similar to SSH.  The default port for SFTP is set to 218.  This port was selected as it does not conflict with any other ports in a Fishworks appliance and does not interfere with SSH communication for administration (port 22).  As with FTP and HTTP, shares may be exported for SFTP access by selecting read-only or read-write access from the Shares->Protocol BUI page or CLI context.

Battle of the SSL All-Stars

If you're an administrator pondering which secure protocol (HTTPS, FTPS, or SFTP) to choose, you're main consideration will be client support.  Not all clients support all protocols.  FTPS is limited in client adoption and the SFTP IETF specification has yet to be finalized.  Secondary to client support will likely be performance.  Using a simple file upload workload of a 10GB file, we can easily compare the three protocols.  All three protocols use OpenSSL for encryption and decryption, so we would expect each protocol to be impacted pretty much the same for secure transfers.  In the following image, we see from top to bottom, the raw network throughput for SFTP, FTPS, and HTTPS.

To be fair to FTPS and HTTPS, I used curl(1) and the native version of sftp(1) on the Solaris client (the Solaris version of curl did not support the SFTP protocol).  Even so, HTTPS transfer rates are clearly lagging at almost 50% of FTPS.

Not surprising, CPU utilization increases by as much as 50% on the client and 10% on the server for a FTPS upload as compared to its non-SSL counterpart.    Thanks to Bryan and the new FTP (and SFTP) analytics, we can see the difference in FTP (top pane) vs FTPS (bottom pane) data rates.  FTPS can be as much as 84% slower than FTP.  Ouch!

Your mileage may vary with your workload but its nice to have the tools at-hand to get a accurate assessment of each protocol and SSL implementations.




Top Tags
« September 2009 »