A Small and Energy-Efficient OpenSolaris Home Server

In an earlier entry, I outlined my most important requirements for an optimal OpenSolaris Home Server. It should:

  1. Run OpenSolaris in order to fully leverage ZFS,
  2. Support ECC memory, so data is protected at all times,
  3. Be power-efficient, to help the environment and control costs,
  4. Use a moderate amount of space and be quiet, for some extra WAF points.

So I went shopping and did some research on possible components. Here's what I came up with:

Choosing a Platform: AMD or Intel?

Disclosure: My wife works for AMD, so I may be slightly biased. But I think the following points are still very valid.

Intel is currently going through a significant change in architecture: The older Core 2 microarchitecture was based on the Front Side Bus (FSB), where the CPU connects to the Northbridge which contains the memory controller and connects to the memory, while also connecting to the Southbridge which connects to I/O.

Now, they are switching to the new Nehalem microarchitecture which has a memory-controller built into the CPU and a scalable I/O bus called Quickpath Interconnect that connects CPUs with other CPUs and/or IO.

Unfortunately, none of these architectures seem to support ECC memory at consumer level pricing. The cheapest Intel-based ECC-motherboard I could find had still more than double the cost of an AMD-based one. Even though the new Intel Core i7 series is based on Nehalem and thus could support ECC memory easily in theory, Intel somehow chose to not expose this feature. In addition, Core i7 CPUs are relatively new and there are not yet any power efficient versions available.

The Intel Atom processor series may be interesting for a home server from a pure power-saving perspective, but again, Atom motherboards don't support ECC and once your workload becomes a little more demanding (like transcoding or some heavier compiling), you'll miss the performance of a more powerful CPU.

AMD on the other hand has a number of attractive points for the home server builder:

  • AMD consumer CPUs use the same microarchitecture than their professional CPUs (currently, it's the K10 design). They only vary by number of cores, cache size, number of HT channels, TDP and frequency, which are all results of the manufacturing process. All other microarchitecture features are the same. When using an AMD consumer CPU, you essentially get a "smaller brother" of their high end CPUs.
  • This means you'll also get a built-in memory-controller that supports ECC.
  • This also means less chips to build a system (no Northbridge needed) and thus lower power-consumption.
  • AMD has been using the HyperTransport Interconnect for quite a while now. This is a fast, scaleable interconnect technology that has been on the market for quite a while so chipsets are widely available, proven and low-cost.

So it was no suprise that even low-cost AMD motherboards at EUR 60 or below are perfectly capable of supporting ECC memory which gives you an important server feature at economic cost.

My platform conclusion: Due to ECC support, low power consumption and good HyperTransport performance at low cost, AMD is an excellent platform for building a home server.

AMD Athlon II X2 240e: A Great Home Server CPU

An AMD Athlon II X2 240e CPU

While I was shopping around for AMD Athlon CPUs and just before I was about to decide for an AMD Athlon II X2 variant, AMD offered me one of their brand new AMD Athlon II X2 240e for testing, provided that I blogged about it. Thank you, AMD!

Introduced in October 20th, this CPU is part of the newest energy-efficient range of consumer CPUs from AMD. It has 2 cores (hence X2), snazzy 2.8 GHz and a 2 MB L2 cache. What's most important: The TDP for this CPU is only 45W, meaning that even under the highest stress, this CPU will never exceed 45W of power consumption. Including the memory controller. As you've guessed already, the "e" in the model number stands for "efficient".

There's an important trade-off to consider for home server CPUs: For instance, the AMD Phenom II series would have been more powerful because it has an additional L3 cache, but their TDP is at 65W minimum. While big caches (both with AMD and Intel) are good for compute-intensive operations and games, they can't help much in a home server context: Home servers spend most of their non-idle time transferring data from A to B (files, videos, music) and a cache doesn't help much here, cause it's just another stop between I/O and CPU to pass by. Transferred data hardly gets re-used.

Instead, for home servers, sacrificing the L3 cache for lower power consumption makes a lot of sense: You pay less for the CPU and you pay less for your power bill without sacrificing too much (if any) server relevant performance.

My CPU conclusion: For home servers, AMD Athlon II "e" series are perfect, because they save power and money and do the job very well. For games you might choose a more powerful Phenom II processor, which delivers better compute power at a slightly higher power bill.

Finding the Right Motherboard

After nailing the Platform and CPU question, I needed a motherboard. This can be a confusing process: For each CPU there are different chipsets, then there are different vendors offering motherboards based on these chipset, and then they offer different variants with different features. What should a good home server motherboard offer?

  • OpenSolaris support: Most motherboards "just work" with OpenSolaris, especially if they've been available for some time and/or use some well-known chipsets. To remove any doubts, consult the OpenSolaris Hardware Compatibility List.
  • ECC: Yes, AMD's ECC support is in the CPU, but just in case, we want to make sure that the motherboard exposes this feature to the user.
  • Integrated graphics: We only need a graphics card for installation, BIOS-settings/updates or debugging. The graphics shouldn't consume much power and doesn't need to deliver a lot of performance. Therefore, integrated graphics is just fine for a home server, and it saves some precious slot space, too.
  • Enough SATA ports: You need two ports for a mirrored root pool, another two for a mirrored data pool and then some more for hot-spare or if you want to spread your data pool across 4 or more disks. Many motherboards come with 6 SATA ports, which is ok.
  • Stability: If you have the choice between different variants, try to go for the "business" or "quality" version. It will likely have slightly better components and be better tuned towards 24/7 operation than the "performance", "gaming" or "overclocker" type boards which try to squeeze out more performance at the expense of durability.

Here's a very useful email thread on the OpenSolaris ZFS-discuss mailing list about CPU and motherboard options, pros and cons and user experiences. In this discussion, F.Wessels recommended the M3A78 series from Asus so I went for the M3A78-CM motherboard, which is their "business class" variant, priced at around 60 Euros and it has 6 SATA and 12(!) USB ports.

My motherboard conclusion: The Asus M3A78-CM motherboard has everything I need for a home server at a very low cost, and it's proven to run OpenSolaris just fine.

The Case: Antec NSK-1380

I won't go into much details about the case. My goal was to find one that can support at least 4 disks while being as compact as possible. The Antec NSK-1380 was the smallest case I could find that supports 4 disks. It comes with a built-in power supply, an extra fan, some features to help with keeping noise down and it looked ok for a PC case.

Miscellaneous Tips&Tricks

While putting everything together, I ran into some smaller issues here and there. Here's what I came up with to solve them:

  • CPU cooler: My CPU being a gift from AMD, it came without a cooler. I chose the Zalman CNPS 7000C AL-CU because it was the cheapest cooler that was less than 6cm high, which is a limit imposed by the case. Unfortunately, it collides with one of the 4 harddisk slots by a mere couple of millimeters, so I need to figure out how to best cut or bend the blades to make room for the disk without hurting the cooler too much. I'm not very familiar with PC CPU coolers, but I suspect that with 45W TDP one could even get away with a passive cooler. The Zalman came with a throttle circuit which I set to the minimum speed. This seems to be more than enough and it makes the system really silent, but I need some more thorough testing to confirm. Drop me a comment if you are familiar with passive cooling of 45W TDP CPUs.
  • Boot disks: We need something to boot our home server from, so boot disks are unavoidable. I say "disks", because they should always be mirrored. But other than providing a way of booting, they tend to get in the way, especially when space is a precious resource in a small case. Some people therefore boot from CF cards or other small flash media. This can be a nice solution, especially combined with an IDE-to-CF adapter, but consumer level flash media is either very slow (a typical OS install can take many hours!) or very expensive. While looking for alternatives, I found a nice solution: The Scythe Slot Rafter fits into an unused PCI slot (taking up the breadth of two) and provides space for mounting four 2.5" disks at just EUR 5. These disks are cheap, good enough and I had an unused one lying around anyway, so that was a perfect solution for me.
  • Extra NIC: The Asus M3A78-CM comes with a Realtek NIC and some people complained about driver issues with OpenSolaris. So I followed the advice on the aforementioned Email thread and bought an Intel NIC which is well supported, just in case.
  • USB boot: I couldn't get the M3A78-CM to boot from USB at all. I tried a USB stick and a USB disk and different boot settings in the BIOS to no avail. I gave up and built in a DVD-ROM drive from my old server just to install OpenSolaris, then built it out again. Jan Brosowski has the same motherboard and he found out that you need to update to the latest BIOS revision and use the USB port that's right below the Ethernet port for USB boot to work. YMMV :).

The Result

And now for the most important part: How much power does the system consume? I did some testing with one boot disk and 4GB of ECC RAM and measured about 45W idle. While stressing CPU cores, RAM and the disk with multiple instances of sysbench, I could not get the system to consume more than 80W. All in all, I'm very pleased with the numbers, which are about half of what my old system used to consume. I didn't do any detailed performance tests yet, but I can say that the system feels very responsive and compile runs just rush along the screen. CPU temperature won't go beyond the low 50Cs on a hot day, despite using the lowest fan speed, so cooling seems to work well, too.

I just started full 24/7 operation of my new home server this weekend, so I hope I'll have some more long-term experience about performance and stability in a few months. Meanwhile, I'm in the middle of configuring the system, installing some services and implementing a new way of managing my home server. But that's probably the topic of another blog post...

Do you agree with the home server conclusions I reached in this post? Or would you suggest alternatives? Do you have experiences to share with the mentioned components? Or do you have suggestions and tips on how to get the most out of them? Let me know by posting a comment here!

Many thanks go to Michael Schmid of AMD for sending me the AMD Athlon II X2 240e CPU.

Comments:

With the combination of the M3A78-CM and 240e, what RAM speed did you end up picking. With that board, it seems that it depends if the CPU is AM2+ or AM3. I couldn't find a review that told me which the CPU was.

Also, something you didn't mention that I am unsure of is the choice of WD green or black drives. Assuming you used Wd. From your requirements of low power, I would assume green, but not sure if they play nice with OpenSolaris.

Great post.

Greg

Posted by Gregory Gee on October 25, 2009 at 07:31 PM CET #

Very useful info, thanks a whole bunch!

I know there are a lot of people, myself included, who are always interested in building a better--and more energy efficient--home server.

Because of the ZFS, as you correctly concluded, OpenSolaris is in everyone's mind.

If it is possible, please continue to post follow-ups. Many will benefit, and, probably more important at this time, OpenSolaris/Sun will also benefit.

Posted by W. Wayne Liauh on October 26, 2009 at 12:00 AM CET #

Please don't get me wrong, but a device - sitting there 24/7 doing nothing than serving files occasionally - draining 45 Watt is not power efficient in any way. Compared to x4150 this might be true, but not compared to a NSLU2 for example.

If you're looking for small energy efficient system a look at http://www.teo-systems.de might be worthy (not affiliated in any way).

Posted by Andreas on October 26, 2009 at 01:42 AM CET #

Hi everybody, thanks for reading my blog!

Greg: You're right, I should have added a section on RAM picking :). The 240e is a socket AM3 processor: http://www.amd.com/us/products/desktop/processors/athlon-ii-x2/Pages/AMD-athlon-ii-x2-processor-model-numbers-feature-comparison.aspx
AM3 processors are backward compatible with AM2+ motherboards, so you get to choose what motherboard you want: http://en.wikipedia.org/wiki/AM3
Since performance is not my main goal, but power efficiency is, I picked the slowest ECC RAM that's supported by the CPU, a pair of Kingston 2048MB ValueRAM DDR2 533MHz CL4. Lower MHz in the RAM saves some power and budget, too and as long as the RAM is faster than the network and the disks (which it always is :) ), it's sufficient for my home server needs.
Regarding drives, I was originally going to pick the WD green drives because they have more than enough performance for streaming and save some power. But then I found that the Samsung SpinPoint 1.5TB EcoGreen F2 drives were even less expensive so I ordered a pair of those. In hindsight, I should have considered mixing a 1.5 TB drive from Samsung with one 1.5 TB from WD to minimize the likeliness of a 2-drive failure in the mirror. I'm now going to wait a while, then order a 2TB drive when it's cheap enough and configure as a hot spare. This gives me 2 advantages: When the first drive starts failing, I'll use the occasion to upgrade the size of my data pool (and order the 2nd 2TB drive then), and I'll avoid the hassle of accidentally getting a replacement drive that is slightly too small.

Wayne: Thanks for the encouragement, I'll certainly continue this series. Next up, I'll write about some scripting in order to make the configuration more automated.

Andreas: Yes, I'm not after 100% power-efficiency. The goal is still to have a server that is 100% available, then, as a second requirement, it should consume as little as possible. However, I am considering to power the server down periodically (every night, when not at home etc.) but OTOH, this may pose a bigger stress to the hardware than letting it run. Not quite sure at the moment. I plan to use the server for a number of things like regular downloads or maybe a CMS that pushes its stuff to an web hoster etc.
The TEO systems machines look interesting, but they are not servers: There's no space for more than one disk (and I wanted to get away from USB disks) and they don't support ECC RAM. That's the wrong tradeoff if you want to have a reliable home server machine. But they look nice as energy efficient client machines. Check out the Sun Rays, which are even nicer, they consume only 3-4W and they reduce the local client support hassle to 0: http://www.sun.com/sunray/sunray2/index.xml

Thanks, and looking forward to some more comments!

Constantin

Posted by Constantin Gonzalez on October 26, 2009 at 03:44 AM CET #

Hey Constantin, congratulations on building your new OpenSolaris ZFS file server!

I read it all with great interest, as I'm always interested in seeing more energy efficient devices.

I agree with your AMD v Intel description. I was considering a new build using a low-end Nehalem 55xx with mobo and the price put me off, so I will also look for another AMD solution for a future build.

Also, I had a few questions:

1. Does suspend & resume mode work. If so, what is the power consumed when it is suspended?

2. Which CPU frequencies are supported by this processor, and does OpenSolaris have built-in support for these different frequencies, like with the Intel Nehalem processors?

3. Which PSU did you use?

4. Which model Intel NIC did you use and what average transfer speed do you get across this NIC when sending/receiving large files?

Nice post and I look forward to your next posts on this subject!

Cheers,
Simon

Posted by Simon Breden on October 26, 2009 at 08:39 AM CET #

Oh, and I forgot to say that the power consumption figures that you quote are quite impressive. I have 8 drives in my system (6 x 750GB WD drives + 2 SSDs), plus a graphics card and an 8-port AOC-USAS-L8i SATA controller card and it's drawing about 160 W of power, so I'm very impressed with your figures!

I think if I replaced my WD 750GB drives with larger capacity WD green drives and put them on the mobo/CPU you are using, I should see quite a drop in power consumption. I turn my system off when not in use.

Posted by Simon Breden on October 26, 2009 at 08:45 AM CET #

Hi Simon,

glad you liked the entry!

To your questions:

1. I haven't tried much suspend/resume yet. Since the IGP is a Radeon type one, the display will prevent suspend from working. You can work around that by using a workaround described here: http://opensolaris.org/jive/thread.jspa?messageID=337602
I tried it out and it seems to suspend, but I couldn't get to wake up quickly over Wake-on-LAN (the server is in the basement). I gave up went down to power-cycle and found it running. The logs say it suspended/resumed but apparently resume took more time than I expected. I'll do some more testing.

2. PowerTOP reports the following:
Cn Avg residency P-states (frequencies)
C0 (cpu running) (10.4%) 800 Mhz 100.0%
C1 2.2ms (89.6%) 1600 Mhz 0.0%
2100 Mhz 0.0%
2800 Mhz 0.0%
When idle, I see the system operating at 800 MHz and only 10% in C0 (running) state. When under load, it immediately jumps to 2.8 GHz. Not sure whether it should be using the frequencies in between or not, but these are really two dimensions of power saving and I assume getting the job done quickly, then going back to sleep is a good strategy.

3. I used the PSU that came as part of the Antec Case. At 350W, it is grossly oversized and 80% efficiency is better than nothing, but not optimal. Essentially, this means that up to 9W of my 45 W may be just burned away from the PSU. I'm considering switching to a 100W PSU, but right now, it may be too much hassle to do that.

4. I got an Intel EtherExpress Pro/1000PT at a local computer store. It's expensive (EUR 70) but I needed the quick gratification :). You can probably find a much cheaper Intel NIC that is supported by the e1000g driver.

5. Thanks for the power consumption comments. The 45 are just with one 2.5" boot disk. After adding the second root pool disk and the two 1.5 TB Samsung EcoGreen drives I'm probably consuming more. But OTOH, I've configured all drives to power themselved off when not in use, so I may still go down to 45W on idle, I just haven't measured again. I assume that having the SATA controller in the chipset and less drives helps quote a bit.

Posted by Constantin Gonzalez on October 26, 2009 at 09:51 AM CET #

Thanks for the answers. Would be interesting to hear if you get any success with the suspend/resume stuff after more investigation, and also with WOL.

Seems like the CPU frequency scaling is working, and good to see lowest frequency in use when the NAS is idle.

If you measure the power later with all drives connected, it would be interesting to hear the consumption figures. But already, with the CPU frequency scaling working and idle draw of around 45W (with one 2.5" drive), it sounds very good...

Posted by Simon Breden on October 26, 2009 at 05:21 PM CET #

Interesting article, Constantin. I'm building an fileserver with Opensolaris now and i'm wondering, if there are any great benefits (flexibility?) from using extra mirrored 2.5" small bootdisks, when you aren't using fast ssd disks or low power CF Cards for OS? You could deploy Opensolaris directly on the big data disks and you will save a little bit more energy.

Beside that, i'll test for my fileserver the Gigabyte GA-MA770-UD3 motherboard for Opensolaris.

Greetings, Oliver

Posted by Oliver on October 29, 2009 at 02:13 PM CET #

Hi Oliver,

thanks for reading my blog!

There are a couple of reasons for keeping boot disks and data disks separate:

1. Boot disks and data disks have different access patterns. Keeping them separate enables them to work independently. For instance, data disks can shut down completely for power saving while the (smaller, more energy-efficient) boot disks may still occasionaly do something (log files etc.).
2. If anything breaks, you can tear out the data disks and quickly import them into another system without the rpool naming etc. colliding with the second system's rpool.
3. For larger systems, you may want a different pool geometry for data and for boot. For instance, you can only mirror the boot pool, but you may want to choose RAID-Z for the data disks.

In my case, I wanted to go for the flexibility of having boot and data pools separate. Also, I like to minimize the individuality of the boot pool and put everything else (config scripts, users, apps, data etc.) on the data pool. That way, the data pool becomes a portable representation of my system.

All the best for your project and let us know how your Gigabyte board tests turned out!

Thanks,
Constantin

Posted by Constantin Gonzalez on October 30, 2009 at 02:47 AM CET #

Hey,
That was a very good post. I recently built a opensolaris machine as a file server to store my media files. The configurations is Intel atom gclf2 board with 2GB memory, Western digital 1 TB HD, Rosewill mini itx case. The total cost came around 220$. I recently added a USB wireless card too. And the machine is so small, no noise and just works good.

Regards,
Balaje Sankar

Posted by Balaje Sankar on October 30, 2009 at 03:14 PM CET #

Hi Balaje,

thank you for sharing your configuration. This sound like a cool and cost-effective mini-server for OpenSolaris!

Cheers,
Constantin

Posted by Constantin Gonzalez on November 02, 2009 at 09:19 AM CET #

Hi,

although I used plenty of AMD CPU, in order to re-use core2duo and quad CPU I have lying around, I went to look for Intel ECC chipset.

looking at wikipedia, http://en.wikipedia.org/wiki/List_of_Intel_chipsets, I found that X38 and X48 do support ECC for DDR-2.

So, I got a special-sales Gigabyte EX38-DS4 (
6 SATA-2 on board, 2 PCI-Express and 2 GigE network port)
with 4 Gig of ECC RAM
and a CPU CoreDuo E2180.

The fully loaded setup (1 HD boot +6 HD in raid-z2 +2 DVD writer+PCI graphic card + DELL hd controller) is at 115W.

I may replace the 2 sata dvd writer for one IDE and get rid of the dell HD controller

Posted by nicolasc on November 05, 2009 at 08:22 AM CET #

Hi Nicolas,

thanks for sharing your config and glad you were able to find an Intel ECC motherboard. I looked up its price and it's at EUR 150 in Europe. I hope you got a better deal. That confirms what I've been finding so far: ECC with Intel starts at EUR 150 for the motherboard. But then, I assume the quality of the Motherboard is better than that of my cheaper one.

115W is a bit hefty for 24/7. Did you test with powertop whether power management is actually running on the server? Also, make sure to set the disks to shut themselves off when idle to save 4-5 W per disk.

You can find more information on power saving with Solaris here: http://hub.opensolaris.org/bin/view/Community+Group+pm/FAQ

Posted by Constantin Gonzalez on November 05, 2009 at 08:51 AM CET #

Post a Comment:
Comments are closed for this entry.
About

Tune in and find out useful stuff about Sun Solaris, CPU and System Technology, Web 2.0 - and have a little fun, too!

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Bookmarks
TopEntries
Blogroll
OldTopEntries