Monday Jan 04, 2010

OpenSolaris, ZFS, USB 2.0 + 30 Watts = Home Fileserver


A little project I'd had in the back of my mind for some time was to replace the family fileserver over the Holidays. I'd been using a Linksys NSLU2 for several years, but we were about to outgrow our 500GB capacity, and response time was never all that great. I'd installed Unslung early on, which gave me SMB, NFS, FTP, etc., but the poor little Intel XScale chip inside struggled just to drive one server, let alone several simultaneous access methods.

The NSLU2 was most recently running with two 500GB USB 2.0 drives attached. The only redundancy provided is the ability to backup from the disk to an identical disk once a day - not very robust. I'd been thinking about it ever since I saw Zhong's really good writeup of creating an OpenSolaris fileserver about a year ago. The factors driving my decision were as follows:

  1. Green: be as energy-efficient as possible; reuse existing pieces if possible
  2. Robust and Redundant: Provide more storage and more robust storage (RAID Z or better)
  3. Compatible: Provide the SMB/CIFS (for family) and NFS (for me) access we were accustomed to at home
  4. Familiar: Leverage technologies I know:

Well, when I saw the rollout for Shuttle's X27D dual-core Atom-based SFF, I knew that was what I wanted. I waited, I planned. Last month the stars aligned, the prices came down, and I acted.

Old Setup 

Best Current Price
Linksys NSLU2
US$43 refurbished; US$86 new
USB2 disk I/F; 100Mbps net I/F; running Unslung 6.8
2 x Simpletech SimpleDrive External Disk drives
US$88 refurbished; US$51 used (EBay)

3.5" form; USB2 disk I/F; 524GB raw storage each

This setup provides:

  • 490GB formatted storage
  • Nightly automatic backup onto clone disk
  • SMB/CIFS access via LAN
  • Boots in around 15 minutes
  • NFS access via LAN (only via reflashing with Unslung)
  • ~4.5MB/s write (NFS), ~3MB/s write (SMB)
  • ~5MB/s read (NFS), ~4MB/s read (SMB), ~150KB/s (scp, sftp or rsync) - note: any encrypted transfers will be CPU-limited by that poor XScale processor)

New Setup

I decided to reuse my existing external disk drives, as well as an existing laptop drive. This meant I only needed to purchase the X27D itself, plus memory. No DVD/CD - I use the network and bootable USB devices instead. To allow for larger, more robust, and more robust storage, I was able to find a used Simpletech drive (same access times, same total raw storage capacity) on EBay.

Price Paid
Shuttle X27D SFF
US$139 refurbished
Builtin dual-core Atom 330 CPU @ 1.6GHz; GigE networking

US$33 new
2GB DDR2 memory

Existing disk
2.5" 120GB SATA disk
3 x Simpletech SimpleDrive External Disk drives
Two existing USB disks plus a third from EBay - US$51

3.5" form; USB2 disk I/F; 524GB raw storage each

This setup provides:

  • 957GB formatted storage via 3-disk RAID-Z pool
  • SMB/CIFS, SSH, NFS protocols supported
  • Boots in around 75 seconds
  • Server consumes 26W idle, 32W under load; disks consume power only when accessed, and spin down after 20 minutes
  • ~14MB/s write (NFS)
  • ~18MB/s read (NFS)
  • Same disk latency (due to disk spin-up) as original solution
  • More onboard processing 'headroom' for future services

Configuring OpenSolaris and ZFS

I downloaded the USB Boot Image of Build 129 of OpenSolaris from here, placing the image on a 1GB USB Flash drive using the usbcopy command. After assembling the server, I set the BIOS to boot from USB first, then booted from the 'Live USB'. Installation just worked, and everything seems to be properly supported (haven't tried audio, as this is a headless system).

I'd rsync'ed the contents of the old fileserver to another server the day before (slooowwwww from an NSLU2 - when one of your systems is CPU-limited, make sure you rsync with the lightest-weight encryption possible [I used arcfour]). Setting up the powerful, performant, redundant ZFS filesystem required only a few steps:

# zpool create fs raidz c4t0d0 c6t0d0 c7t0d0   # RAIDZ pool (2+1)
# zfs set sharenfs=on fs # NFS share
# zfs set sharesmb=on fs # Samba share

Still to do:

  • Investigate adding cache to the zpool via part of the internal SATA disk (or possibly eventually an SSD, when prices come down)
  • More rigorous and apples-to-apples performance comparisions
  • Optimizing the OpenSolaris settings and services

Thursday Jul 23, 2009

Open, Robust and Performant Preservation/Archival - First Steps

As part of the Sun Open Archive for Digital Repositories effort, Sun has been working closely with Fedora Commons to develop and characterize performance of Fedora on Sun (OpenSolaris, ZFS, Sun Storage 7000 appliances, Sun Storage JBOD storage).

The first phase of performance testing and configuration validation was completed recently, with encouraging performance observed from Fedora, running on OpenSolaris 2008.11 and using RAID0 ZFS Pools. Much, much more testing is needed (and is underway at this writing), but much of what we've seen so far bests any Fedora performance testing done to date.

There's quite a compelling case to be made with the combination of Sun hardware and open source OS and application stack, so we're pretty excited about this work.

Check out the attached documents:

  • Poster presented by Sun and Fedora Commons at Open Repositories '09
  • FInal version of this first performance testing document released this month - ingest and access performance of Fedora on OpenSolaris, ZFS and JBODs

Tuesday Jun 02, 2009

Community One West 2009

Pulling into San Francisco this weekend with a beefed-up laptop (all the OpenSolaris virtualization goodness requires lots-o-memory - I'm up to 4GB now on my Tecra M2) and a mysterious case of poison ivy (I blame my cat), I set my sights on the Community One West and Java One conferences. Perhaps the last time the 'Sun Microsystems' name is attached to these conferences, and as such a mixture of excitement, trepidation and sadness.

First impression: just as many developer attendings as ever, as just as much energy as ever. Community One is a one-day developer frenzy before JavaOne itself starts tomorrow. Open Source developers. OpenSolaris mavens. Java afficienados. PHP and Perl and Python and Apache and MySQL and Postgres. Something for everyone.

Some notes from the talks I attended Monday:

  • Keynote address (Dave Douglas and John Fowler)
    • OpenSolaris 2009.06 is out!
    • Crossbow 'virtual networking' in Solaris/OpenSolaris - the entire networking stack was done from the bottom up
    • The coming Computing Cloud offering from Sun - impressive!
    • Open Storage (which includes the Open Archive effort I blogged about a couple weeks back)

  • Becoming a ZFS Ninja
    • It's amazing that after four years how many folks still don't know about ZFS
    • It's still easy as pie, performant and bulletproof (my observation, not his)

  • DTrace for Applications

    • Delivered by my ISV-E peer Angelo Rajadurai
    • Great demonstration of using basic and PHP DTrace probes to instrument Drupal (watch this space for his scripts later this week)

  • Source Juicer for Open Solaris
    • Has come a long way in just a few months, to provide packages into OpenSolaris's /pending and /contrib packages repos
    • Coolest new feature: repositories and Source Juicer mechanism itself now have 'install' one-click package install buttons


Main Sequence: 1) an astronomy term denoting the lifecycle of a majority of known stars. 2) err @ Sun/Oracle: long-time (since 1988) Sun/Oracle veteran, still shining in an ever-changing high-tech universe


« April 2014