Eco responsible power friendly small home server (part 2)
By user13366129 on Okt 05, 2009
A long time ago, I did write about my small new home server. You can read all about that at: http://blogs.sun.com/pfuetz/entry/eco_responsible_small_home_server
Meanwhile some changes have been done to the system. Yes, I needed to buy the 90W PSU (although I now know how much Watt the system really uses, I'm curious, why that was really needed, but it sometimes did crash with the 60W PSU and that's gone, now, that I have the 90W PSU, so I assume, it must have had to do with the 60W PSU. Or on that specific PSU, but the dealer refused to change it, as it is running. More on Wattage later).
I also did change the passive heat sink on the CPU to also be a ZALMAN, safe is safe (see image). And with that, I now have the case put on the small side, which in the image is below, so that the heat can exit through the two case openings, which are now at the top. No fans any more in the case... Quiet!
More importantly, I did also replace the CF card by a real disk, as that only uses 1.8W at load, less, if not in use. I did buy a 1.8 inch Toshiba disk (those, that are used inside the iPods, in my case a TOSHIBA MK2006GA) plus two converters (3.5 -> 2.5 and 2.5 -> 1.8) and placed the disk inside the cabinet (No part numbers here, and no distributor, as I bought them on german eBay. Overall disk + two converters were around 40 EURO). The main reason for this change is the fact, that the CF card was really slow, and that 8 GB weren't enough to perform Live Upgrade (and now maintain different Boot Environments) (I now have 30 G in the rpool, for swap, crash and all the rest).
I also updated the OS..
Now, I did install OpenSolaris 2009.06 from an image on a usbstick, that went really smooth, no changes needed. And, after that, I upgraded to the actual dev build, which this weekend was build 124...
One more change hardware-wise: I also added two more 1 TB disks, because I had a nasty failure of BOTH external power supplies of the WD disks over the same weekend some months ago... Rendering me without storage... So, I decided to mirror across different types of disks, so I bought two no-name 1 TB external USB disks (which have Hitachi SATA disks in it). WD quickly sent new PSUs, great service, and they admitted, that those PSUs were faulty... They had many such failures...
With the new setup of the disks, I had a different problem: The new disks did not do power-save by themselves, whereas the WD disks did fall into power save. ZFS does not have any clue about power management of the disks, it assumes, that that is done by the underlying elements. The problem here resulted in a DEGRADED zpool, as one disk of the mirror was not available...
The solution is simple: Have an entry for every disk in /etc/power.conf so that the powerd takes care of that. So here's my /etc/power.conf:
device-dependency-property removable-media /dev/fb autopm enable autoS3 default system-threshold 900s cpu-threshold 1s # Auto-Shutdown Idle(min) Start/Finish(hh:mm) Behavior autoshutdown 30 9:00 9:00 noshutdown device-thresholds /pci@0,0/pci8086,464c@1d,7/storage@1/disk@0,0 900s device-thresholds /pci@0,0/pci8086,464c@1d,7/storage@2/disk@0,0 900s device-thresholds /pci@0,0/pci8086,464c@1d,7/storage@3/disk@0,0 900s device-thresholds /pci@0,0/pci8086,464c@1d,7/storage@4/disk@0,0 900s device-thresholds /pci@0,0/pci-ide@1f,1/ide@0/cmdk@0,0 900s cpupm enable
Then there was the problem, that the WD disks did not reveal different devid strings (that problem had been brought to my attention by a colleague of mine), the string was the same for all WD disks. You can check that by running (as root):
bash-3.2# echo '::spa -c' | mdb -k
and compare the lines that contain devid. In my case that was:
devid='id1,sd@TWD______10EAVS_External_/a' devid='id1,sd@f004fa5244aa68827000dc26e0004/a' devid='id1,sd@TWD______10EAVS_External_/a' devid='id1,sd@f004fa5244aa76c85000a3a89000a/a'
So I did open a bug, and got a pre-fix, that will come to OpenSolaris and Solaris soon. The fix fixes the way the scsa2usb driver handles disks that do not correctly report page 80 or page 83 scsi inquiries (which many USB don't do right!). So, now with the new scsa2usb driver the output looks like:
devid='id1,sd@f000665644ab34259000aa2b30004/a' devid='id1,sd@f004fa5244aa68827000dc26e0004/a' devid='id1,sd@f004fa52449bea5b4000f1f610000/a' devid='id1,sd@f004fa5244aa76c85000a3a89000a/a'
That makes for four uniquely identifiable USB disks. Which is a good thing, as now the ZFS stack can in case additional infos get lost still uniquely address the disks, and not treat one for the other! That would render the data corrupted, in case all other infos might get lost. Rest assured, ZFS not only checks for the devid string, but the devid string is the last resort if all else fails...
Until the final fix (bug is in fix-delivered state!) will be out, there's a workaround. Quoted from the workaround section of bug 6881590:
For the issue of devid not unique due to page 83 data, it can be a workaround by setting the similar lines as below in sd.conf, which enforces sd to ignore the vpd page data and fabricate a devid. for x86: sd-config-list="VendorID+ProductID","sd-ver1-tst-data";sd-ver1-tst-data= 1,0x00004,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0; for sparc: sd-config-list="VendorID+ProductID","sd-ver1-tst-data";sd-ver1-tst-data= 1,0x00008,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0;
For rare cases, if I would need to reinstall my system, I keep a log of all steps performed, so if I would need to install from scratch, I would end up at the same stage. Here's that script:
# Install OpenSolaris 2009.06 from media (CD, USB stick, whatever) # After install, perform: pfexec su - svcadm disable nwam svcadm enable network/physical:default svcadm disable sendmail vi /etc/hosts vi /etc/defaultrouter vi /etc/hostname.rge0 vi /etc/ethers vi /etc/power.conf vi /etc/inet/ntp.conf svcadm enable ntp crontab -e # (to add: 0 3 \* \* 3 /usr/sbin/zpool scrub rpool) pkg install SUNWsmba SUNWsmbfskr SUNWsmbs SUNWsmbskr svcadm enable smb/server vi /etc/nsswitch.conf hosts: files dns mdns ipnodes: files dns mdns smbadm join -w PFUETZNER vi /etc/pam.conf # Manually added for kernel CIFS server in workgroup mode other password required pam_smb_passwd.so.1 nowarn passwd pfuetz passwd root svccfg -s idmap setprop config/default_domain = astring: PFUETZNER svccfg -s idmap setprop config/ds_name_mapping_enabled=boolean: false svcadm refresh idmap # install new scsa2usb driver \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* # Then, for the upgrade to work, I prefer to do it by hand, so: # ONCE: pkg set-publisher -O http://pkg.opensolaris.org/dev opensolaris.org pkg set-property flush-content-cache-on-success true # Every Update: pkg refresh --full pkg install SUNWipkg pkg image-update --be-name buildXXX beadm activate buildXXX
And for the setup of the CIFS/NFS shares on ZFS, here's what needs to be done (assuming, you have a zpool named "pfuetz" ):
zfs create -o casesensitivity=mixed pfuetz/smb-share zfs set email@example.com pfuetz/smb-share zfs set sharesmb=on pfuetz/smb-share zfs set sharesmb=name=smb-share pfuetz/smb-share
I've been testing the power consumption of the system alone, with no external USB disks. It uses 39 W. With the external four 1 TB disks the power usage goes to 72W, when the disks are awake, but not doing anything. If the disks are under load, the power consumption goes up to 80W. If the disks fall asleep, the power consumption goes down to 63 W.
Lessons learned so far:
There's one additional lesson: The "zpool scrub" for the 2 TB mirrored disks takes 16 hours. So it might be a good idea, just for power savings to really use SATA disks internal to the server, and not external. That also would solve the devid problem. And also would solve the power management problem. And even safe money, as the scrub would be finished in way less time.
So here, finally the image of the "actual interior":
I hope, these additional infos might help you in determining, what to build as a small home server. You might also check Jan's or Constantin's blog, they also have some insights!