Sunday Nov 22, 2009

My Home Media Server using OpenSolaris, ZFS, and free software

A while ago, I wrote several blog entries about what I did to set up a media server at home.  I'm writing this blog entry to wrap things up with some details about how much it all cost, and the software I'm running on the computer now.


I decided to design a "media server" for home that would be the main data storage for our family's music, photos, recorded TV shows and movies, and personal documents and backups of our home directories on the computers we commonly use at home.  I had a few objectives for the media server:
  1. All of this data would be in a single computer that we could grab and stick in the car in case of emergency.  Friends of ours lost their house, but they had the foresight of having all of their personal data (over a terabytes' worth) on a single file server, so when they evacuated the house, they didn't lose any personal data.  Seemed like a great idea to me.
  2. The media server would store data reliably; i.e., I assumed that a disk would fail, and I wanted the media server to be able to continue working in case of failure.
  3. Reduce noise in the house as much as possible.  That meant putting the server in the garage where it could make as much fan noise as it needed to, but we wouldn't hear a thing in the home office or where we watch TV.
  4. Keep the cost down to a reasonable amount.  This was not the primary factor, but it was important enough for me to pay attention to when shopping for the components.

What I Purchased

Here's the build of materials for the computer I put together (I'm sure prices on these will have gone down since I bought them in October 2008).  Total price of the system: $742.  If you take out the cost of the disks, the rest of the computer cost $345.
  • Motherboard (, $62.99) : ECS Elitegroup A780VM-M2 Micro ATX AMD Motherboard.  Supported the cheap AMD processor I wanted, and had plenty of on-board SATA ports (6).  Remember, I want all the disk to be in a single enclosure for easy, snatch-and-grab transportation.
  • Memory (, $69.99): GSkill 4GB (2 x 2GB) DDR2 SDRAM (PC2 6400).  No reason to get greedy, and it keeps power draw and price down for the system.
  • CPU (, $59.50): AMD Athlon 64 X2 5000 Brisbane 2.6GHz Socket AM2 65W Dual-Core Processor.  Nice price for a processor that does everything I need in a media server that can also run some apps.  More on the apps in a bit.
  • Disks (, $396.80): 2 Seagate Barracuda 1.5TB 7200 RPM SATA internal disks.  Prices have gone down considerably for the storage, but this was a fair enough price last year.
  • Case (Fry's Electronics, $130): Antec Sonata III Mid Tower ATX Case.  Got good reviews, came with a 500W power supply which is plenty for what I was putting in it, and has plenty of ports and internal drive bays for the storage.

How I Set Up The Media Server

Here is the first blog entry I wrote that describes my thought process about using ZFS on OpenSolaris to store our home's data.

Here is the second blog entry I wrote, giving all the details about what I needed to do to format the disks and set up the ZFS pools and filesystems.  This was based on the OpenSolaris 2008.11 release; I plan to upgrade to a more recent release (probably an OpenSolaris build after the June 2009 release; I'd love to try the new ZFS deduplication feature).  Everything has been running fine, with the exception of the TimeSlider feature for doing automated ZFS snapshots.  That was fixed in the June 2009 release.

What I'm Running On The Media Server

The media server is doing a few things for us at home; here are the services it's providing:
  • iTunes music storage.  We have a Mac mini that we use for iTunes; all of our music is in iTunes, and the Mac mini NFS-mounts a filesystem from the media server.  I tried both CIFS (Samba) and NFS.  I preferred NFS, but it seemed to have troubles with the Mac as an NFS client.  After reading James Gosling's blog entry on what he observed with Macs and Solaris and NFS, I made a tweak on the Mac mini and everything has been working smoothly ever since.  We make this library shareable to our other Macs on the home network.
  • iPhoto storage.  The same Mac mini also stores all of our photos.
  • TiVo media backup and playback.  There is a great open source Java application called Galleon that uses the TiVo Home Media Engine (HME) API; the API lets you write your own Java apps that show up as part of the TiVo's on-screen menu system, and it lets you talk to the TiVo to grab shows off the TiVo's disk and put shows on there, among other functions.  Primarily, we use it as a way to keep backups of shows that we don't want to lose in case the TiVo's disk crashes and loses data.  This has happened several times, mostly due to unforeseen loss of power.
  • Personal finances (via Intuit's Quicken software).  To do this, I run the free, open source VirtualBox software.  I used to use VMware, but VMware isn't free and it doesn't run on any version of Solaris.  VirtualBox runs on every operating system I use, it's free, and it's improving much faster than VMware is.  So, I run Windows XP as a guest OS inside VirtualBox, and use that guest OS to run Quicken.  When I'm feeling like the guest OS is getting slow or am worried that it's collected viruses, I just blow away that guest image and go back to an earlier snapshot.

My Wish List

Are any of you doing something similar with your home setups?  If so, maybe you have done some of the things I'd like to set up but haven't gotten done yet.  Here's my wish list of apps and features I'd like to add, and I'd like to do it all natively in OpenSolaris (in other words, none of these services would need to run in a guest OS like Windows or Linux under VirtualBox):
  • DLNA server running natively in OpenSolaris.  DLNA will give me the ability to share and stream my media (photos, music, video) to the PlayStation 3.  I've played with several solutions; more on this in a moment.
  • Live streaming of TiVo content from the media server back through the TiVo.  Galleon lets me transfer shows to and from the TiVo, but not play them live.  This is a feature that the ReplayTV product had (I still miss that product, and am sorry they went out of business), and their Java app was superior in several ways to Galleon.  But, it looks like somebody has written software to let you play TiVo shows from your media server straight through the TiVo.  I just haven't gotten it running on the media server yet.

Powered by ScribeFire.

Using OpenSolaris as a TiVo backup device

OpenSolaris logo TiVo logo

Sometimes my TiVo gets confused after a power outage, and it takes forever to reboot (I think it's trying to recover its files which takes a while when you have over 1TB of storage connected to the TiVo).  Sometimes, I lose some of my shows, and that makes me sad.

So I decided to find some software I could get running on OpenSolaris so I could use my home media server as a backup device for my TiVo shows.

I did it, using the Galleon open source Java application; here's how.

Now, if somebody smarter than me would just port the TiVo software to OpenSolaris, we'd have the benefits of ZFS on a TiVo box.  That would be fantastic.

Powered by ScribeFire.

Friday Aug 14, 2009

Terracotta on Sun's CMT and x64 Solaris servers

In our ISV Engineering organization, we do some pretty cool work with a variety of software companies built around open source business models; here are just a few of our more strategic open source partners.  This week, we published some work we've been doing with Terracotta for the last few months to help them optimize their technology on Sun's products.  The 4-page document provides an overview of the business benefits of Terracotta for Java developers, plus some results of testing we did with Terracotta on both x64 and CMT servers.  We also ran their same tests on RedHat Enterprise Linux to see how we did.  We did great.

I really like what Terracotta's done; my overly-simplistic explanation of what they do is to hook into a Java Virtual Machine (JVM) and link it with other JVMs working together, so that a cluster of JVMs look like a single, big JVM to the Java developer.  The significance is this: if you're a Java developer and you want to easily scale up your application so it can take on more load, Terracotta makes it really easy for you to do that.

In the document that we published, we showed the results of tests we did with sample workloads that Terracotta created to demonstrate what it can do for some common Java application scenarios.  (one scenario models an online test-taking application where many people login concurrently to take their tests, maybe leave the test midway through, come back where they left off, etc.)  If you look at the results table, you'll see a couple of results that I find interesting:
  1. Performance of Terracotta on Solaris vs. RedHat.  Everything else was the same: same JVM, same physical hardware.  But Terracotta on Solaris performed much better, making more efficient use of the compute resources.  You leave less of your computing budget on the table with Terracotta on Solaris, is what this says to me.
  2. Terracotta performance on CMT.  On the T5240 CoolThreads server, we didn't get the top result, but we had plenty of headroom to go (using 9% of the CPU resource available), which means we could launch more copies of Terracotta, or the Java application itself.  Our tests with Terracotta show us we can use CMT to get massive scaling; the results table clearly reflects that.
Once we started scaling up with Terracotta on CMT, we started to notice that their persistence mechanism was becoming a bottleneck (if you read more about Terracotta, you find that they make your cluster of JVMs reliable because Terracotta keeps track of Java objects that change, and persists those changes to its local disk).  So we introduced Terracotta to our solid state disk (SSD) products and configured the Terracotta server to persist its data to the SSDs instead of spinning disk.  That essentially gave us reliability at in-memory speeds which means that you don't have to make the tradeoff of performance vs. reliability.  It's very cool.

We've had a blast working with Terracotta; they're sharp people, and they create a product that I think is hugely valuable to Java developers, especially those trying to write apps that work at large scales on the web.  If you're such a developer, you should check them out.  Their software is available as open source and it works.

Powered by ScribeFire.


The views expressed on this blog are my own and do not necessarily reflect the views of Oracle. What more do you need to know, really?


« July 2016