OpenSolaris, ZFS, USB 2.0 + 30 Watts = Home Fileserver

Background

A little project I'd had in the back of my mind for some time was to replace the family fileserver over the Holidays. I'd been using a Linksys NSLU2 for several years, but we were about to outgrow our 500GB capacity, and response time was never all that great. I'd installed Unslung early on, which gave me SMB, NFS, FTP, etc., but the poor little Intel XScale chip inside struggled just to drive one server, let alone several simultaneous access methods.

The NSLU2 was most recently running with two 500GB USB 2.0 drives attached. The only redundancy provided is the ability to backup from the disk to an identical disk once a day - not very robust. I'd been thinking about it ever since I saw Zhong's really good writeup of creating an OpenSolaris fileserver about a year ago. The factors driving my decision were as follows:

  1. Green: be as energy-efficient as possible; reuse existing pieces if possible
  2. Robust and Redundant: Provide more storage and more robust storage (RAID Z or better)
  3. Compatible: Provide the SMB/CIFS (for family) and NFS (for me) access we were accustomed to at home
  4. Familiar: Leverage technologies I know:

Well, when I saw the rollout for Shuttle's X27D dual-core Atom-based SFF, I knew that was what I wanted. I waited, I planned. Last month the stars aligned, the prices came down, and I acted.

Old Setup 


Make/Model
Best Current Price
Specs
Server
Linksys NSLU2
US$43 refurbished; US$86 new
USB2 disk I/F; 100Mbps net I/F; running Unslung 6.8
Storage
2 x Simpletech SimpleDrive External Disk drives
US$88 refurbished; US$51 used (EBay)

3.5" form; USB2 disk I/F; 524GB raw storage each

This setup provides:

  • 490GB formatted storage
  • Nightly automatic backup onto clone disk
  • SMB/CIFS access via LAN
  • Boots in around 15 minutes
  • NFS access via LAN (only via reflashing with Unslung)
  • ~4.5MB/s write (NFS), ~3MB/s write (SMB)
  • ~5MB/s read (NFS), ~4MB/s read (SMB), ~150KB/s (scp, sftp or rsync) - note: any encrypted transfers will be CPU-limited by that poor XScale processor)

New Setup

I decided to reuse my existing external disk drives, as well as an existing laptop drive. This meant I only needed to purchase the X27D itself, plus memory. No DVD/CD - I use the network and bootable USB devices instead. To allow for larger, more robust, and more robust storage, I was able to find a used Simpletech drive (same access times, same total raw storage capacity) on EBay.


Make/Model
Price Paid
Specs
Server
Shuttle X27D SFF
US$139 refurbished
Builtin dual-core Atom 330 CPU @ 1.6GHz; GigE networking


US$33 new
2GB DDR2 memory


Existing disk
2.5" 120GB SATA disk
Storage
3 x Simpletech SimpleDrive External Disk drives
Two existing USB disks plus a third from EBay - US$51

3.5" form; USB2 disk I/F; 524GB raw storage each

This setup provides:

  • 957GB formatted storage via 3-disk RAID-Z pool
  • SMB/CIFS, SSH, NFS protocols supported
  • Boots in around 75 seconds
  • Server consumes 26W idle, 32W under load; disks consume power only when accessed, and spin down after 20 minutes
  • ~14MB/s write (NFS)
  • ~18MB/s read (NFS)
  • Same disk latency (due to disk spin-up) as original solution
  • More onboard processing 'headroom' for future services

Configuring OpenSolaris and ZFS

I downloaded the USB Boot Image of Build 129 of OpenSolaris from here, placing the image on a 1GB USB Flash drive using the usbcopy command. After assembling the server, I set the BIOS to boot from USB first, then booted from the 'Live USB'. Installation just worked, and everything seems to be properly supported (haven't tried audio, as this is a headless system).

I'd rsync'ed the contents of the old fileserver to another server the day before (slooowwwww from an NSLU2 - when one of your systems is CPU-limited, make sure you rsync with the lightest-weight encryption possible [I used arcfour]). Setting up the powerful, performant, redundant ZFS filesystem required only a few steps:

# zpool create fs raidz c4t0d0 c6t0d0 c7t0d0   # RAIDZ pool (2+1)
# zfs set sharenfs=on fs # NFS share
# zfs set sharesmb=on fs # Samba share

Still to do:

  • Investigate adding cache to the zpool via part of the internal SATA disk (or possibly eventually an SSD, when prices come down)
  • More rigorous and apples-to-apples performance comparisions
  • Optimizing the OpenSolaris settings and services


Comments:

Hi Eric,

thanks for sharing! How did you motivate the disks to spin down with OpenSolaris? I re-used my EeePC for a comparable setup ( http://blogs.sun.com/rolfk/entry/an_opensolaris_eeepc_homeserver ) but didn't manage to spin down the disks so far...

Thanks
Rolf

Posted by Rolf on January 05, 2010 at 04:04 AM EST #

Hi Eric,

I have been looking for information on low power ZFS based fileserver so this blog entry is very welcomed.I have a few questions though...

1) Maybe I don't know atom cpu's well enough but it appeasr the one you chose is not a "Pine Trail" based one. Why did you wait for this specific system ? and why did you pick it over the just released pine trail.

2) how come you chose USB drives with single disks over multibay units eg."RaidSonic ICY Dock" which is a 4 bay unit ?

3) how come you didn't base your server on a bigger cabinet+ atom motherboard and Sata connected disks (even if it only 2)

4) could you explain about the caching on SATA disk you talked about ?

5) How much headroom do you think the system have, how well do you think it can eq also run a squeezebox server and a glassfish (hudson+nexus) ? My current asus EEE box b202 does this, but on linux and glassfish v2 can get a bit slow.

Or do you think I should upgrade to either a dell zino or mac mini for that ?

Posted by Henrik Lynggaard on January 05, 2010 at 08:28 AM EST #

@ Rolf

The disks spin up and down on their own - the server doesn't control them.

@ Henrik
1) Wanted low price, didn't need to wait for latest Atom - this
met my needs just fine

2) Mainly wanted to leverage existing disks - I could go from 500GB
w/backup to 1TB RAID-Z by adding only one more external drive

3) Didn't need cabinet space for cards (it's headless), and wanted to
keep power consumption low;

4) One can add ZILs or Caches to ZPools, but I'm not sure if this will
help unless it's a dedicated internal SATA HDD or SSD

5) I can't imagine this box will have enough headroom for all the things
you want to do, but again mine was only intended as a fileserver - I'd
go for more than Atom for what you want

Posted by Eric Reid on January 05, 2010 at 08:54 AM EST #

Hi Again

Regarding question 5 I had... I am a bit puzzled d by your answer. My current setup is a asus eee b202 (Atom N270) with 2 GB ram. Right now it runs normal ubuntu with ssh,subversion,squeezebox and glassfish (hudson+nexus). The build times are a somewhat slow but at least it is able to build without the music skipping.

Does opensolaris+zfs really add that much overhead that it wont run those things on a new 64bit atom cpu, compared to the old ubuntu+ext3 ?

The problem preventing me from trying the performance of Opensolaris + ZFS + the rest is that the cpu is only 32bit, and as far as I have understood ZFS does not work with 32bit, or at least there are some limitations.

Am I correct that 32bit and ZFS don't mix ?

Posted by Henrik on January 07, 2010 at 09:15 AM EST #

@ Henrik

>> Does opensolaris+zfs really add that much overhead that it wont run those things >> on a new 64bit atom cpu, compared to the old ubuntu+ext3 ?

No, not a lot of overhead -- I'm just unsure how much total dual-core Atom + OpenSolaris gives for anything 'substantial' in the first place.

>> Am I correct that 32bit and ZFS don't mix ?

Works, but not recommended - a 32-bit CPU will use a lot of cycles managing a 128-bit filesystem.

Posted by Eric Reid on January 14, 2010 at 03:24 AM EST #

Post a Comment:
Comments are closed for this entry.
About

Main Sequence: 1) an astronomy term denoting the lifecycle of a majority of known stars. 2) err @ Sun/Oracle: long-time (since 1988) Sun/Oracle veteran, still shining in an ever-changing high-tech universe

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today