Monday Jan 04, 2010

OpenSolaris, ZFS, USB 2.0 + 30 Watts = Home Fileserver


A little project I'd had in the back of my mind for some time was to replace the family fileserver over the Holidays. I'd been using a Linksys NSLU2 for several years, but we were about to outgrow our 500GB capacity, and response time was never all that great. I'd installed Unslung early on, which gave me SMB, NFS, FTP, etc., but the poor little Intel XScale chip inside struggled just to drive one server, let alone several simultaneous access methods.

The NSLU2 was most recently running with two 500GB USB 2.0 drives attached. The only redundancy provided is the ability to backup from the disk to an identical disk once a day - not very robust. I'd been thinking about it ever since I saw Zhong's really good writeup of creating an OpenSolaris fileserver about a year ago. The factors driving my decision were as follows:

  1. Green: be as energy-efficient as possible; reuse existing pieces if possible
  2. Robust and Redundant: Provide more storage and more robust storage (RAID Z or better)
  3. Compatible: Provide the SMB/CIFS (for family) and NFS (for me) access we were accustomed to at home
  4. Familiar: Leverage technologies I know:

Well, when I saw the rollout for Shuttle's X27D dual-core Atom-based SFF, I knew that was what I wanted. I waited, I planned. Last month the stars aligned, the prices came down, and I acted.

Old Setup 

Best Current Price
Linksys NSLU2
US$43 refurbished; US$86 new
USB2 disk I/F; 100Mbps net I/F; running Unslung 6.8
2 x Simpletech SimpleDrive External Disk drives
US$88 refurbished; US$51 used (EBay)

3.5" form; USB2 disk I/F; 524GB raw storage each

This setup provides:

  • 490GB formatted storage
  • Nightly automatic backup onto clone disk
  • SMB/CIFS access via LAN
  • Boots in around 15 minutes
  • NFS access via LAN (only via reflashing with Unslung)
  • ~4.5MB/s write (NFS), ~3MB/s write (SMB)
  • ~5MB/s read (NFS), ~4MB/s read (SMB), ~150KB/s (scp, sftp or rsync) - note: any encrypted transfers will be CPU-limited by that poor XScale processor)

New Setup

I decided to reuse my existing external disk drives, as well as an existing laptop drive. This meant I only needed to purchase the X27D itself, plus memory. No DVD/CD - I use the network and bootable USB devices instead. To allow for larger, more robust, and more robust storage, I was able to find a used Simpletech drive (same access times, same total raw storage capacity) on EBay.

Price Paid
Shuttle X27D SFF
US$139 refurbished
Builtin dual-core Atom 330 CPU @ 1.6GHz; GigE networking

US$33 new
2GB DDR2 memory

Existing disk
2.5" 120GB SATA disk
3 x Simpletech SimpleDrive External Disk drives
Two existing USB disks plus a third from EBay - US$51

3.5" form; USB2 disk I/F; 524GB raw storage each

This setup provides:

  • 957GB formatted storage via 3-disk RAID-Z pool
  • SMB/CIFS, SSH, NFS protocols supported
  • Boots in around 75 seconds
  • Server consumes 26W idle, 32W under load; disks consume power only when accessed, and spin down after 20 minutes
  • ~14MB/s write (NFS)
  • ~18MB/s read (NFS)
  • Same disk latency (due to disk spin-up) as original solution
  • More onboard processing 'headroom' for future services

Configuring OpenSolaris and ZFS

I downloaded the USB Boot Image of Build 129 of OpenSolaris from here, placing the image on a 1GB USB Flash drive using the usbcopy command. After assembling the server, I set the BIOS to boot from USB first, then booted from the 'Live USB'. Installation just worked, and everything seems to be properly supported (haven't tried audio, as this is a headless system).

I'd rsync'ed the contents of the old fileserver to another server the day before (slooowwwww from an NSLU2 - when one of your systems is CPU-limited, make sure you rsync with the lightest-weight encryption possible [I used arcfour]). Setting up the powerful, performant, redundant ZFS filesystem required only a few steps:

# zpool create fs raidz c4t0d0 c6t0d0 c7t0d0   # RAIDZ pool (2+1)
# zfs set sharenfs=on fs # NFS share
# zfs set sharesmb=on fs # Samba share

Still to do:

  • Investigate adding cache to the zpool via part of the internal SATA disk (or possibly eventually an SSD, when prices come down)
  • More rigorous and apples-to-apples performance comparisions
  • Optimizing the OpenSolaris settings and services

Saturday Oct 17, 2009

Drupal-on-a-Stick: now in easy-to-take download form!

At Bay Area DrupalCamp today, Sun is handing out USB Flash sticks that boot any USB-bootable X86/X64 box into OpenSolaris with a running Drupal 6 instance. For those of you not at BADCamp this weekend, we've also made the USB image available for download here. It's a little less than 1GB, and fits on 1GB flash drives.

To burn the image to your own USB flash drive:

  • From a Windows box: see here
  • From an OpenSolaris box: install and use /usr/bin/usbcopy from SUNWdistro-const package

For those who'd rather try this from a "Live DVD", the .iso image is here.

Update 19 October: thanks to alert readers who spotted my ill-formed DVD link above. It's fixed now.druplicon

Monday Sep 28, 2009

Open Source SysAdmins: curious about Sun Storage 7000 series boxes?

A quick note that our friends at Sun BigAdmin have published a short paper of mine called "Joomla! Deployment Guide for Sun Storage 7000 Unified Storage Systems". We wanted to give a more hands-on constructive view of these very cool products from the sysadmin and open-source perspectives. These boxes work, work well, work fast, and bring intelligent network-based storage roaring into the Enterprise.

Friday Sep 25, 2009

Oracle Enterprise Linux Certification - Yes, You Heard Right...

What to do during those... erm... 'uncertain' times? Why, branch out of course... and that's what lead this intrepid blogger to consider questions like "what's fun?", "what haven't I done in a while?" and "what might be relevant to our imminent New Overlords?".

Relax, Sun Blog Police, that's all Imma say about Snorcle, OK?

I picked 'Oracle Enterprise Linux Certification'. Just cuz. Just cuz I enjoy Linux almost as much as I do OpenSolaris. Just cuz I went and gots me a Red Hat Certified Technician certificate a few years back, also "Just Cuz."  And just cuz it might even be applicable in future endeavors.

For those of you whose knowledge of OEL (sometimes referred to by the overarching 'Unbreakable Linux' moniker) rivals that of Solaris -- Oracle takes the Red Hat code base, strips out any Red Hat identity, slaps on Oracle veneers, and supports the snot out of it. That allows Mssr. Ellison to raise the hackles of yet another rival. Which is good. I'm thinkin'...

Anywho, OEL 5 is RHEL 5 with a different shade o' red paint. If you know RHEL, you know OEL.

An OEL Administration Associate certificate requires the passing of two two-hour multiple-choice exams. If you have a comparable certifications already, you might be given a pass on the first exam - indeed, my RHCT certificate allowed me to test out of the first exam. The second, a more in-depth sysadmin exam, is 113 questions over two hours. Not hands-on, as Red Hat exams are, but also with a lot of 'choose all th apply' questions. Far from trivial stuff, but quite doable if you prepare adequately.

So, as of today, I add the OEL Admin Associate designation to my list of... umm... things. Right along side the RHCT and Solaris and Java Certifications. And my collection of Wired Magazines -- can't forget those!

Cuz, you know, this just seemed like the Thing To Do...

Thursday Jul 23, 2009

Joomla! and recent OpenSolaris distros - workaround

Though documented elsewhere, it's a good idea to propagate for the benefit of the next person.

Problem: Installation of Joomla! 1.5.x on recent versions of OpenSolaris (2009.06 and later) using MySQL fail with errors after install

Symptom: After otherwise successful install of config on MySQL on OpenSolaris, the home page comes up in the browser with numerous MySQL errors, and the MySQL server restarts:

   Error loading Modules:MySQL server has gone away SQL=SELECT id, title, module, position, 
   content, showtitle, control, params FROM jos_modules AS m LEFT JOIN jos_modules_menu 
   AS mm ON mm.moduleid = WHERE m.published = 1 AND m.access =  0 AND m.client_id = 0 
   AND ( mm.menuid = 1 OR mm.menuid = 0 ) ORDER BY position, ordering

Diagnosis: known issue with MySQL 32-bit server as delivered from

Workaround: use the 64-bit version of the MySQL server (/usr/mysql/<version>/bin/64/mysqld_safe, et al)


Open, Robust and Performant Preservation/Archival - First Steps

As part of the Sun Open Archive for Digital Repositories effort, Sun has been working closely with Fedora Commons to develop and characterize performance of Fedora on Sun (OpenSolaris, ZFS, Sun Storage 7000 appliances, Sun Storage JBOD storage).

The first phase of performance testing and configuration validation was completed recently, with encouraging performance observed from Fedora, running on OpenSolaris 2008.11 and using RAID0 ZFS Pools. Much, much more testing is needed (and is underway at this writing), but much of what we've seen so far bests any Fedora performance testing done to date.

There's quite a compelling case to be made with the combination of Sun hardware and open source OS and application stack, so we're pretty excited about this work.

Check out the attached documents:

  • Poster presented by Sun and Fedora Commons at Open Repositories '09
  • FInal version of this first performance testing document released this month - ingest and access performance of Fedora on OpenSolaris, ZFS and JBODs

Tuesday Jul 21, 2009

When your 'pkg install' is slow on Virtualbox

I like VirtualBox. It does what I need -- often times it does many different things via multiple clients on the same server.

Recently, I've been creating OpenSolaris VirtualBox clients, and, again, they do what I need. Except when it comes to accessing OpenSolaris repositories. With default NAT networking, an OpenSolaris client tends to have slower networking, and at times when I'm installing several packages from,  I'll get:

 # pkg install amp
 pkg: Maximum number of network retries exceeded during download. Details follow:

Frustrating, to be sure, but there is a way to put up with all the waiting:

  # export PKG_CLIENT_TIMEOUT=300   # Seconds (default=30)
  # export PKG_TIMEOUT_MAX=10       # Attempts (default=4)

Now, although your pkg commands won't be any faster without tweaking the VirtualBox networking stack, they will eventually finish. This works outside of VirtualBox as well.

Monday Jun 15, 2009

Archival and Preservation: Open Source and Open Archive

One of the more exciting and interesting projects I've been involved with this year centers around Sun's Open Archive systems for Digital Repositories - the use of Open Source software and Sun off-the-shelf hardware to implement world-class data preservation/archival systems. For those of you who remember Sun's Honeycomb archival hardware, this new architecture represents a truly open alternative approach - based on Fedora Commons' offering, OpenSolaris, Sun's X64 servers, JBOD disk arrays and (soon) Unified Storage Systems.

One of the first efforts Sun ISV Engineering contributed to involved sizing and performance testing of Fedora atop OpenSolaris, ZFS and JBODs. The results are both encouraging and impressive, and provide myriad follow-on testing opportunities. I've attached the first draft of test report here.

Tuesday Jun 02, 2009

Dear JavaOne Organizers...

Just what \*is\* the deal with the wireless coverage @ Moscone Center? My Tecra M2 with brand new OpenSolaris 2009.06 kept the up-down-up-down-up-down all day... never a consistent connection. Anyone have any tips or clues?

Community One West 2009

Pulling into San Francisco this weekend with a beefed-up laptop (all the OpenSolaris virtualization goodness requires lots-o-memory - I'm up to 4GB now on my Tecra M2) and a mysterious case of poison ivy (I blame my cat), I set my sights on the Community One West and Java One conferences. Perhaps the last time the 'Sun Microsystems' name is attached to these conferences, and as such a mixture of excitement, trepidation and sadness.

First impression: just as many developer attendings as ever, as just as much energy as ever. Community One is a one-day developer frenzy before JavaOne itself starts tomorrow. Open Source developers. OpenSolaris mavens. Java afficienados. PHP and Perl and Python and Apache and MySQL and Postgres. Something for everyone.

Some notes from the talks I attended Monday:

  • Keynote address (Dave Douglas and John Fowler)
    • OpenSolaris 2009.06 is out!
    • Crossbow 'virtual networking' in Solaris/OpenSolaris - the entire networking stack was done from the bottom up
    • The coming Computing Cloud offering from Sun - impressive!
    • Open Storage (which includes the Open Archive effort I blogged about a couple weeks back)

  • Becoming a ZFS Ninja
    • It's amazing that after four years how many folks still don't know about ZFS
    • It's still easy as pie, performant and bulletproof (my observation, not his)

  • DTrace for Applications

    • Delivered by my ISV-E peer Angelo Rajadurai
    • Great demonstration of using basic and PHP DTrace probes to instrument Drupal (watch this space for his scripts later this week)

  • Source Juicer for Open Solaris
    • Has come a long way in just a few months, to provide packages into OpenSolaris's /pending and /contrib packages repos
    • Coolest new feature: repositories and Source Juicer mechanism itself now have 'install' one-click package install buttons


Main Sequence: 1) an astronomy term denoting the lifecycle of a majority of known stars. 2) err @ Sun/Oracle: long-time (since 1988) Sun/Oracle veteran, still shining in an ever-changing high-tech universe


« November 2015