Monday Mar 14, 2011

Blog migration

More than a year ago, I migrated my blog to its new home:

Constant Thinking

And I'm very happy to see that the new incarnation has more RSS subscribers and daily visitors than this old one, and it's growing nicely. Thank you!

Still, there are a lot of people that visit this blog, probably by finding an old article through a search engine or so. And there are still hundreds of subscribers to the old RSS feed. Again, thank you, but you may be missing something!

So here's a short reminder:

  1. Please check out my new blog Constant Thinking now. I'm putting a lot of effort and energy into it to make is as useful and interesting to you as possible. Of course, your feedback is very welcome!
  2. If you like it, stay in touch by adding the new RSS feed to your feed reader.
  3. If you don't like/use/know RSS, or if you want an extra look behind the scenes, there's a brand new option now: The Constant Thinking Newsletter will send you timely updates about new articles, plus some extra stuff like background information, more tips and the occasional inner circle type goodie. Sign up now!

Meanwhile, this blog will still remain for reference and archival purposes.

Thanks,

   Constantin


Tuesday Feb 23, 2010

Six Weeks of Constant Thinking

As you know, I've moved this blog into an independent webspace and relaunched it as "Constant Thinking" about six weeks ago. Check it out, it's at constantin.glez.de. Here's a short "status report" after about 6 weeks of independent blogging:

One of my goals for my new blog is to achieve 500 RSS subscribers by summer (Yes, you're invited, too :)). You see, as a blogger, my lifebread is the number of people who regularly read my blog. That's how we bloggers feel rewarded for what we do: Every single RSS subscriber, every comment, every piece of feedback is like a pat on our shoulder, a drop of endorphine in our blood, a carrot... Well, you get the picture. Little things please little minds :).

In this case, my old blog has about 550 subscribers and I want you all to show up on my new blog as well. No, I won't just redirect the feed, that would feel like cheating and I'd never know if I deserved all those subscribers for my new blog or not (after all, you might just follow this blog just because it's a Sun blog, not because of is content).

So first of all, a big "Thank You!" to the more than 160 people who chose to follow my new blog's RSS feed. This is greatly appreciated!

To the other 390 or so, here's what you've probably been missing:

I hope you'll find these articles interesting and useful. Feel free to get more Technology Thoughts and add "Constant Thinking" to your RSS reader. I'm looking forward to your comments and feedback!

Sunday Jan 17, 2010

A New Home For My Blog

A screenshot of my new blog First of all, a big "Thank You!" to you, my dear readers and the more than 500 subscribers (according to Feedburner) to this blog!

After having blogged here, on my Sun blog, for more than 5 years, I thought that the year 2010 is the right time to start running my own, independent blog. This is a thought that I've been having in my mind for a long time, and it is also something like the next logical step in developing myself as a blogger. Also, to me as a geek, it is a great excuse to play more deeply with some new blogging/CMS software.

New Year, New Blog

During the Christmas weeks (and before), I've been digging around different blogging systems and CMS and decided to use Drupal as my blogging platform, which is very powerful and flexible while offering endless possibilities to expand my blog in the future. It is hosted at Strato, which is a great Sun customer in Germany, and I'm happy to see that they use UltraSPARC T based systems to run my blog, too :).

So, without further ado, I encourage you to check out my new blog called: "Constant Thinking". You can reach it at constantin.glez.de or just type constantthinking.org into your browser. I'm looking forward to your feedback and suggestions: What do you like, what improvements do you suggest, what subjects do you want me to blog about?

Future Plans

I expect to continue writing about OpenSolaris, Technology, Home Servers, Podcasting, and some occasionaly useful diversions, like you're accustomed to. As a result of building my new blog, I've dug deeper into the Drupal Web Content Management System, so expect a few articles around that, too. Finally, I've also developed an interest in ebooks, readers and the EPUB format and I plan to share some experience around that as well in future blog articles.

Stay In Touch!

I hope you like the new concept and design and I'd like to encourage you to subscribe to the new Constant Thinking RSS Feed, so we can stay in touch at least the next 5 years to come!

Thursday Dec 03, 2009

OSDevCon 2009 Paper: Implementing a simple ZFS Auto-Scrub Service with SMF, RBAC, IPS and Visual Panels Integration - Lessons learned

A while ago, I wrote a little tool that helps you keep your ZFS pools clean by automatically running regular scrubs, similar to what the OpenSolaris auto-snapshot service does.

The lessons I learned during development of this service went into an OSDevCon 2009 paper that was presented in September 2009 in Dresden. It is a nice summary of things to keep in mind when developing SMF services of your own and it includes a tutorial on writing a GUI based on the OpenSolaris Visual Panels project.

Check out the Whitepaper here, the slides here, the SMF service here and if you want to take a peek at the Service's Visual Panels Java code, you'll find it here.

Monday Nov 16, 2009

Fun With DTrace: The Windows-Key Prank

The current episode of the German HELDENFunk podcast features an interview with Chris Gerhard about one of his favourite subjects: DTrace (in English, beginning at 14:58):

After the interview, we hear a guy called "Konteener Kalle" express his love (in German) for DTrace by playing a prank on his boss: Whenever he presses the Windows key (on an OpenSolaris system, mind you), he's punished by watching the XScreensaver BSOD hack (of course not knowing that it's just a screensaver).

That little joke challenged me to actually implement this prank. Here's how to do it.

The Idea

The idea of this prank is to start the XScreensaver Blue-Screen-of-Death screensaver (which simulates a Windows crash experience) on an OpenSolaris system whenever the user presses a certain key a certain number of times. This could be the Windows-Key (which doesn't have any real use on an OpenSolaris machine) or any other key. We count the number of key presses and only execute the BSOD after a certain number of key presses in order to make the prank less obvious.

Step 1: Identify the Windows (or any other) Key

If you have a Windows-Keyboard, this is easy: Run xev and press the Windows-Key. Take note of the keycode displayed in the xev output. Of course you can use any other key as well to play this prank. In this case, I'm using the left Control-Key, because I don't have a Windows-Key on the system I'm working on. The Control key has the keycode 37.

Step 2: Configure XScreensaver for BSOD

XScreensaver comes with a great collection of "hacks" that do interesting stuff on the screen when the screensaver activates. Check out the /usr/lib/xscreensaver/hacks directory. Each hack can be run individually, but then it will only execute inside a new window. For the BSOD illusion to be realistic, we want to execute the BSOD hack in full-screen.

This can be achieved by telling XScreensaver to demo the BSOD hack for us. It will then create a full-screen window and execute the BSOD hack inside the new window. The following command will tell XScreensaver to run a hack for us:

xscreensaver-command -demo <number>

The <number> part is a little complicated: XScreensaver looks at its config file ~/.xscreensaver where it stores a list of programs and arguments after the keyword "programs:". <number> simply refers to the number of the hack on that list. Therefore, we must create an entry in our admin user's .xscreensaver file that starts bsod(6) with the right parameters and that gives us a known number to call xscreensaver-command with.

Let's put our entry at the top of the list so we can simply use the number "1" to execute the BSOD screensaver. Somewhere in our .xscreensaver, the programs section should look like this:

  ...
  textFile:       /etc/motd
  textProgram:    date
  textURL:        http://blogs.sun.com/roller/rss

  programs:                                                                     \\
  -               "BSOD Windoze"  bsod -root -only nt         \\n\\
  -                "Qix (solid)"  qix -root -solid -segments 100              \\n\\
  -          "Qix (transparent)"  qix -root -count 4 -solid -transparent      \\n\\
  ...

You can test this by running xscreensaver-command -demo 1.

Step 3: Write a DTrace Script That Sets Up the Trap

Now it gets more interesting. How do we use DTrace to find out when a user presses a certain key? All we know is that the Xorg server processes the keystrokes for us. So let's start by watching Xorg in action. The following DTrace command will trace all function calls within Xorg:

pfexec dtrace -n pid`pgrep Xorg`:::entry'{ @func[probefunc] = count(); }'

Let's start it, press the desired key 10 times, then stop it with CTRL-C. You'll see a long list of Xorg functions, sorted by the number of times they've been called. Since we pressed the key 10 times, it's a good idea to look for functions that have been called ca. 10 times. And here, we seem to be lucky:

  ...
  miUnionO                                                          8
  DeviceFocusInEvents                                               9
  CommonAncestor                                                   10
  ComputeFreezes                                                   10
  CoreLeaveNotifies                                                10
  key_is_down                                                      11
  FreeScratchPixmapHeader                                          12
  GetScratchPixmapHeader                                           12
  LookupIDByType                                                   12
  ProcShmDispatch                                                  12
  ProcShmPutImage                                                  12
  ...

The key_is_down function looks like exactly the function we're looking for! In fact, some googling tells us that this function's 2nd argument is the keycode of the key that is down when the function is called.

Why do we see "11" and not "10" function calls to key_is_down? Because it also counted my pressing of the Ctrl-Key when I stopped the DTrace script through Ctrl-C :).

This gives us enough knowledge to create the following DTrace script:

  #!/usr/sbin/dtrace -s

  /\*
   \* BSODKey.d
   \*/

  /\*
   \* This D script will monitor a certain key in the system. When this key is
   \* pressed, a shell script will be executed that simulates a BSOD.
   \*
   \* The script needs the process id of the Xorg server to tap into as its
   \* first argument.
   \*
   \* One example of using this script is to punish a user pressing the
   \* Windows key on an OpenSolaris system by launching the BSOD screen saver.
   \*/

  #pragma D option quiet
  #pragma D option destructive

  BEGIN
  {
          ctrlcount = 0;
          keycode=37
  }

  pid$1::key_is_down:entry
  /arg1 == keycode/
  {
          ctrlcount ++;
  }

  pid$1::key_is_down:return
  /ctrlcount == 10/
  {
          ctrlcount = 0;
          system("/usr/bin/xscreensaver-command -demo 1");
  }

First, we need to enable DTrace's destructive mode (ever heard of a "constructive prank"?) otherwise we can't call the system-command at the end. The script uses the pid provider to tap into Xorg. Therefore, we need to give it the PID of the Xorg server as an argument:

pfexec ./BSODKey.d `pgrep Xorg`

It then sets up a probe that fires whenever key_is_down is called with our keycode and counts the key presses. At the end of the key_is_down function call, it checks whether we reached 10 keypresses, then executes the BSOD screen saver and resets the counter. You may need to make sure that the DISPLAY variable is set correctly for the BSOD program to show up on the victim's screen when starting this script.

After hitting the Control-Key 10 times, we're rewarded with our beloved BSOD:

Conclusion

That wasn't too difficult, was it? Yes, one could have done the same thing by writing a regular script that taps into /dev/kbd or something similar. But the beauty of DTrace lies in the simplicity of this script (Tap into the right function while it's running) and in the fact that it now can be modified very easily to fire BSODs at any kind of event, including the user hitting a certain area of the screen with his mouse or selecting a particular text or whatever you choose it to be.

So, have fun with this script and let me know in the comments what kind of pranks (or helpful actions) you can imagine with DTrace!

Sunday Oct 25, 2009

A Small and Energy-Efficient OpenSolaris Home Server

In an earlier entry, I outlined my most important requirements for an optimal OpenSolaris Home Server. It should:

  1. Run OpenSolaris in order to fully leverage ZFS,
  2. Support ECC memory, so data is protected at all times,
  3. Be power-efficient, to help the environment and control costs,
  4. Use a moderate amount of space and be quiet, for some extra WAF points.

So I went shopping and did some research on possible components. Here's what I came up with:

Choosing a Platform: AMD or Intel?

Disclosure: My wife works for AMD, so I may be slightly biased. But I think the following points are still very valid.

Intel is currently going through a significant change in architecture: The older Core 2 microarchitecture was based on the Front Side Bus (FSB), where the CPU connects to the Northbridge which contains the memory controller and connects to the memory, while also connecting to the Southbridge which connects to I/O.

Now, they are switching to the new Nehalem microarchitecture which has a memory-controller built into the CPU and a scalable I/O bus called Quickpath Interconnect that connects CPUs with other CPUs and/or IO.

Unfortunately, none of these architectures seem to support ECC memory at consumer level pricing. The cheapest Intel-based ECC-motherboard I could find had still more than double the cost of an AMD-based one. Even though the new Intel Core i7 series is based on Nehalem and thus could support ECC memory easily in theory, Intel somehow chose to not expose this feature. In addition, Core i7 CPUs are relatively new and there are not yet any power efficient versions available.

The Intel Atom processor series may be interesting for a home server from a pure power-saving perspective, but again, Atom motherboards don't support ECC and once your workload becomes a little more demanding (like transcoding or some heavier compiling), you'll miss the performance of a more powerful CPU.

AMD on the other hand has a number of attractive points for the home server builder:

  • AMD consumer CPUs use the same microarchitecture than their professional CPUs (currently, it's the K10 design). They only vary by number of cores, cache size, number of HT channels, TDP and frequency, which are all results of the manufacturing process. All other microarchitecture features are the same. When using an AMD consumer CPU, you essentially get a "smaller brother" of their high end CPUs.
  • This means you'll also get a built-in memory-controller that supports ECC.
  • This also means less chips to build a system (no Northbridge needed) and thus lower power-consumption.
  • AMD has been using the HyperTransport Interconnect for quite a while now. This is a fast, scaleable interconnect technology that has been on the market for quite a while so chipsets are widely available, proven and low-cost.

So it was no suprise that even low-cost AMD motherboards at EUR 60 or below are perfectly capable of supporting ECC memory which gives you an important server feature at economic cost.

My platform conclusion: Due to ECC support, low power consumption and good HyperTransport performance at low cost, AMD is an excellent platform for building a home server.

AMD Athlon II X2 240e: A Great Home Server CPU

An AMD Athlon II X2 240e CPU

While I was shopping around for AMD Athlon CPUs and just before I was about to decide for an AMD Athlon II X2 variant, AMD offered me one of their brand new AMD Athlon II X2 240e for testing, provided that I blogged about it. Thank you, AMD!

Introduced in October 20th, this CPU is part of the newest energy-efficient range of consumer CPUs from AMD. It has 2 cores (hence X2), snazzy 2.8 GHz and a 2 MB L2 cache. What's most important: The TDP for this CPU is only 45W, meaning that even under the highest stress, this CPU will never exceed 45W of power consumption. Including the memory controller. As you've guessed already, the "e" in the model number stands for "efficient".

There's an important trade-off to consider for home server CPUs: For instance, the AMD Phenom II series would have been more powerful because it has an additional L3 cache, but their TDP is at 65W minimum. While big caches (both with AMD and Intel) are good for compute-intensive operations and games, they can't help much in a home server context: Home servers spend most of their non-idle time transferring data from A to B (files, videos, music) and a cache doesn't help much here, cause it's just another stop between I/O and CPU to pass by. Transferred data hardly gets re-used.

Instead, for home servers, sacrificing the L3 cache for lower power consumption makes a lot of sense: You pay less for the CPU and you pay less for your power bill without sacrificing too much (if any) server relevant performance.

My CPU conclusion: For home servers, AMD Athlon II "e" series are perfect, because they save power and money and do the job very well. For games you might choose a more powerful Phenom II processor, which delivers better compute power at a slightly higher power bill.

Finding the Right Motherboard

After nailing the Platform and CPU question, I needed a motherboard. This can be a confusing process: For each CPU there are different chipsets, then there are different vendors offering motherboards based on these chipset, and then they offer different variants with different features. What should a good home server motherboard offer?

  • OpenSolaris support: Most motherboards "just work" with OpenSolaris, especially if they've been available for some time and/or use some well-known chipsets. To remove any doubts, consult the OpenSolaris Hardware Compatibility List.
  • ECC: Yes, AMD's ECC support is in the CPU, but just in case, we want to make sure that the motherboard exposes this feature to the user.
  • Integrated graphics: We only need a graphics card for installation, BIOS-settings/updates or debugging. The graphics shouldn't consume much power and doesn't need to deliver a lot of performance. Therefore, integrated graphics is just fine for a home server, and it saves some precious slot space, too.
  • Enough SATA ports: You need two ports for a mirrored root pool, another two for a mirrored data pool and then some more for hot-spare or if you want to spread your data pool across 4 or more disks. Many motherboards come with 6 SATA ports, which is ok.
  • Stability: If you have the choice between different variants, try to go for the "business" or "quality" version. It will likely have slightly better components and be better tuned towards 24/7 operation than the "performance", "gaming" or "overclocker" type boards which try to squeeze out more performance at the expense of durability.

Here's a very useful email thread on the OpenSolaris ZFS-discuss mailing list about CPU and motherboard options, pros and cons and user experiences. In this discussion, F.Wessels recommended the M3A78 series from Asus so I went for the M3A78-CM motherboard, which is their "business class" variant, priced at around 60 Euros and it has 6 SATA and 12(!) USB ports.

My motherboard conclusion: The Asus M3A78-CM motherboard has everything I need for a home server at a very low cost, and it's proven to run OpenSolaris just fine.

The Case: Antec NSK-1380

I won't go into much details about the case. My goal was to find one that can support at least 4 disks while being as compact as possible. The Antec NSK-1380 was the smallest case I could find that supports 4 disks. It comes with a built-in power supply, an extra fan, some features to help with keeping noise down and it looked ok for a PC case.

Miscellaneous Tips&Tricks

While putting everything together, I ran into some smaller issues here and there. Here's what I came up with to solve them:

  • CPU cooler: My CPU being a gift from AMD, it came without a cooler. I chose the Zalman CNPS 7000C AL-CU because it was the cheapest cooler that was less than 6cm high, which is a limit imposed by the case. Unfortunately, it collides with one of the 4 harddisk slots by a mere couple of millimeters, so I need to figure out how to best cut or bend the blades to make room for the disk without hurting the cooler too much. I'm not very familiar with PC CPU coolers, but I suspect that with 45W TDP one could even get away with a passive cooler. The Zalman came with a throttle circuit which I set to the minimum speed. This seems to be more than enough and it makes the system really silent, but I need some more thorough testing to confirm. Drop me a comment if you are familiar with passive cooling of 45W TDP CPUs.
  • Boot disks: We need something to boot our home server from, so boot disks are unavoidable. I say "disks", because they should always be mirrored. But other than providing a way of booting, they tend to get in the way, especially when space is a precious resource in a small case. Some people therefore boot from CF cards or other small flash media. This can be a nice solution, especially combined with an IDE-to-CF adapter, but consumer level flash media is either very slow (a typical OS install can take many hours!) or very expensive. While looking for alternatives, I found a nice solution: The Scythe Slot Rafter fits into an unused PCI slot (taking up the breadth of two) and provides space for mounting four 2.5" disks at just EUR 5. These disks are cheap, good enough and I had an unused one lying around anyway, so that was a perfect solution for me.
  • Extra NIC: The Asus M3A78-CM comes with a Realtek NIC and some people complained about driver issues with OpenSolaris. So I followed the advice on the aforementioned Email thread and bought an Intel NIC which is well supported, just in case.
  • USB boot: I couldn't get the M3A78-CM to boot from USB at all. I tried a USB stick and a USB disk and different boot settings in the BIOS to no avail. I gave up and built in a DVD-ROM drive from my old server just to install OpenSolaris, then built it out again. Jan Brosowski has the same motherboard and he found out that you need to update to the latest BIOS revision and use the USB port that's right below the Ethernet port for USB boot to work. YMMV :).

The Result

And now for the most important part: How much power does the system consume? I did some testing with one boot disk and 4GB of ECC RAM and measured about 45W idle. While stressing CPU cores, RAM and the disk with multiple instances of sysbench, I could not get the system to consume more than 80W. All in all, I'm very pleased with the numbers, which are about half of what my old system used to consume. I didn't do any detailed performance tests yet, but I can say that the system feels very responsive and compile runs just rush along the screen. CPU temperature won't go beyond the low 50Cs on a hot day, despite using the lowest fan speed, so cooling seems to work well, too.

I just started full 24/7 operation of my new home server this weekend, so I hope I'll have some more long-term experience about performance and stability in a few months. Meanwhile, I'm in the middle of configuring the system, installing some services and implementing a new way of managing my home server. But that's probably the topic of another blog post...

Do you agree with the home server conclusions I reached in this post? Or would you suggest alternatives? Do you have experiences to share with the mentioned components? Or do you have suggestions and tips on how to get the most out of them? Let me know by posting a comment here!

Many thanks go to Michael Schmid of AMD for sending me the AMD Athlon II X2 240e CPU.

Friday Oct 02, 2009

Gonzalez Goes GeekAndPoke: My First Co-Authored Webcomic

A while ago, I highlighted a few of my favourite web comics. Little did I know then, that today I was going to be part of one. Here's the story:

Yesterday, @moellus complained yet again about his eternal nemesis, the NT admin, by saying something like: "Damn, can't unfollow the NT admin because he doesn't twitter - here's why - that's the sh\*# he sends me!" (Original Tweet)

That reminded me of the famous insult by the late Douglas Adams from "The Hitchhiker's Guide to the Galaxy", in which Arthur Dent says about the Vogons: "I wish I had a daughter so I could forbid her to marry one..."

So I twittered back that "I wish you were on Twitter so I could unfollow you!" must be the new worst insult you can do in 2009 and asked Oliver whether that would make a nice Geek And Poke cartoon.

Today, I'm proud to be part of the "Geek And Poke"-uversum, here's the cartoon:

Geek And Poke, October 1st, 2009: Post 2.0 Insulting (Guest Writer Cartoon)

I guess, to be "GeekAndPoked" is the new "Slashdotted" :). Thanks, Oliver!


P.S.: @moellus: Actually, to a geek, in 2009, sending someone a box of Windows 7 is like the medieval slapping with a glove. Don't take it lightly and get your LART-whip ready!

Saturday Sep 19, 2009

New OpenSolaris ZFS Home Server: Requirements

Old OpenSolaris Home Server with blinkenlights USB drives

A few months ago, I decided it was time for a new home server. The old one (see picture) is now more than 3 years old (the hardware is 2 years older), so it was time to plan ahead for the inevitable hardware failure. Curiously enough, my old server started to refuse working with some of my external USB disks only a few weeks ago, which confirmed my need for a new system. This is the beginning of a series of blog articles around building a new OpenSolaris home server.

Home Server Goals

Let's go over some goals for a home server to help us decide on the hardware. IMHO, a good home server should:

  1. Run OpenSolaris. This means I don't want an appliance because this is too limiting and I'd end up hacking it to make it run something it doesn't do by itself anyway. It should therfore use a real OS.
    It also means I don't want to use Linux, because quite frankly the whole Linux landscape is too unstable, confused and de-focused (don't get me wrong: It's nice for experimentation and as a hobbyist-OS, but I want something more serious to guard my data).
    Windows is out of the question because it delivers too little for too high a price.
    I like BSD (and used to run NetBSD on my Amiga 4000 back then in the mid nineties), but it seems to be more oriented to some (albeit interesting) niches for my taste.
    Right now I prefer OpenSolaris because it's rock-solid, clean, well-documented, well-designed and it has lots of advanced features that other OSes only dream of. Yes, I'd still write and do the same if I weren't a Sun employee.
  2. Leverage ZFS. This should be a no-brainer, but I just wanted to point out that any system that is serious about its data should absolutely run ZFS because of the end-to-end-integrity. Period. And then there are many useful features such as compression, send/receive, snapshots, ease of administration, no fscks and much more. Oh, and I'm looking forward to leveraging encryption and de-duplication at home in the near future, too!
  3. Use ECC Memory: What's the use of having end-to-end data integrity with ZFS if your data is corrupted before ZFS can create it's checksum? That's why you need ECC Memory. Simply put: Use ECC memory and kiss those unexpected, unexplicable system crashes and broken data surprises good bye.
  4. Be Power Efficient: Think 1.5 Euros of electricity bill per Watt per Year for a system running 24/7. The difference between your typical gaming PC and a power-efficient home server can easily be 50W or more when idle, so you're looking at an extra 75 Euros and more of free cash if you just pick your components more carefully. Notice that I'm not saying "Low-Power". There are a lot of compromises when trying to reach absolute low-powerness. Like many optimization problems, squeezing the last few Watts out of your system means investing a lot of money and effort while sacrificing important features. So I want this system to be power-efficient, but without too many sacrifices.
  5. Use a Moderate Amount of Space: While my home server sits in the basement, form doesn't matter. But I may move into a new apartment where networking to the basement is not an option. Then the server needs to be living-room capable and have a decent WAF. Which also brings us to:
  6. Be quiet: A power-efficient server needs less cooling which helps with being quiet. Again, we don't want to stretch the limits of quietness at all costs, but we want to make sure we don't do any obvious mistakes here that sacrifice the living-room capabilities of the system

What's Next

In the next blog entry, we'll discuss a few processor and platform considerations and reveal a cool, yet powerful option that presented itself to me. Meanwhile, feel free to check out other home server resources, such as Simon Breden's blog, Matthias Pfuetzner's blog, Jan Brosowski's Blog (German) or one of the many home server discussions on the zfs-discuss mailing list.

What are your requirements for a good home server? What do you currently use at home to fulfill your home server needs? What would you add to the above list of home server requirements? Feel free to add a comment below!

Thursday Sep 17, 2009

New OpenSolaris ZFS Auto-Scrub Service Helps You Keep Proper Pool Hygiene

A harddisk that is being scrubbed

One of the most important features of ZFS is the ability to detect data corruption through the use of end-to-end checksums. In redundant ZFS pools (pools that are either mirrored or use a variant of RAID-Z), this can be used to fix broken data blocks by using the redundancy of the pool to reconstruct the data. This is often called self-healing.

This mechanism works whenever ZFS accesses any data, because it will always verify the checksum after reading a block of data. Unfortunately, this does not work if you don't regularly look at your data: Bit rot happens and with every broken block that is not checked (and therefore not corrected), the probability increases that even the redundant copy will be affected by bit rot too, resulting in data corruption.

Therefore, zpool(1M) provides the useful scrub sub-command which will systematically go through each data block on the pool and verify its checksum. On redundant pools, it will automatically fix any broken blocks and make sure your data is healthy and clean.

It should now be clear that every system should regularly scrub their pools to take full advantage of the ZFS self-healing feature. But you know how it is: You set up your server and often those little things get overlooked and that cron(1M) job you wanted to set up for regular pool scrubbing fell off your radar etc.

Introducing the ZFS Auto-Scrub SMF Service

Here's a service that is easy to install and configure that will make sure all of your pools will be scrubbed at least once a month. Advanced users can set up individualized schedules per pool with different scrubbing periods. It is implemented as an SMF service which means it can be easily managed using svcadm(1M) and customized using svccfg(1M).

The service borrows heavily from Tim Foster's ZFS Auto-Snapshot Service. This is not just coding laziness, it also helps minimize bugs in common tasks (such as setting up periodic cron jobs) and provides better consistency across multiple similar services. Plus: Why invent the wheel twice?

Requirements

The ZFS Auto-Scrub service assumes it is running on OpenSolaris. It should run on any recent distribution of OpenSolaris without problems.

More specifically, it uses the -d switch of the GNU variant of date(1) to parse human-readable date values. Make sure that /usr/gnu/bin/date is available (which is the default in OpenSolaris).

Right now, this service does not work on Solaris 10 out of the box (unless you install GNU date in /usr/gnu/bin). A future version of this script will work around this issue to make it easily usable on Solaris 10 systems as well.

Download and Installation

You can download Version 0.5b of the ZFS Auto-Scrub Service here. The included README file explains everything you need to know to make it work:

After unpacking the archive, start the install script as a privileged user:

pfexec ./install.sh

The script will copy three SMF method scripts into /lib/svc/method, import three SMF manifests and start a service that creates a new Solaris role for managing the service's privileges while it is running. It also installs the OpenSolaris Visual Panels package and adds a simple GUI to manage this service.

ZFS Auto-Scrub GUI

After installation, you need to activate the service. This can be done easily with:

svcadm enable auto-scrub:monthly

or by running the GUI with:

vp zfs-auto-scrub

This will activate a pre-defined instance of the service that makes sure each of your pools is scrubbed at least once a month.

This is all you need to do to make sure all your pools are regularly scrubbed.

If your pools haven't been scrubbed before or if the time or their last scrub is unknown, the script will proceed and start scrubbing. Keep in mind that scrubbing consumes a significant amount of system resources, so if you feel that a currently running scrub slows your system too much, you can interrupt it by saying:

pfexec zpool scrub -s <pool name>

In this case, don't worry, you can always start a manual scrub at a more suitable time or wait until the service kicks in by itself during the next scheduled scrubbing period.

Should you want to get rid of this service, use:

pfexec ./install.sh -d

The script will then disable any instances of the service, remove the manifests from the SMF repository, delete the scripts from /lib/svc/method, remove the special role and the authorizations the service created and finally remove the GUI. Notice that it will not remove the OpenSolaris Visual Panels package in case you want to use it for other purposes. Should you want to get rid of this as well, you can do so by saying:

pkg uninstall OSOLvpanels

Advanced Use

You can create your own instances of this service for individual pools at specified intervals. Here's an example:

  constant@fridolin:~$ svccfg
  svc:> select auto-scrub
  svc:/system/filesystem/zfs/auto-scrub> add mypool-weekly
  svc:/system/filesystem/zfs/auto-scrub> select mypool-weekly
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> addpg zfs application
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/pool-name=mypool
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/interval=days 
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/period=7
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/offset=0
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/verbose=false
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> end
  constant@fridolin:~$ svcadm enable auto-scrub:mypool-weekly

This example will create and activate a service instance that makes sure the pool "mypool" is scrubbed once a week.

Check out the zfs-auto-scrub.xml file to learn more about how these properties work.

Implementation Details

Here are some interesting aspects of this service that I came across while writing it:

  • The service comes with its own Solaris role zfsscrub under which the script runs. The role has just the authorizations and profiles necessary to carry out its job, following the Solaris Role-Based Access Control philosophy. It comes with its own SMF service that takes care of creating the role if necessary, then disables itself. This makes a future deployment of this service with pkg(1) easier, which does not allow any scripts to be started during installation, but does allow activation of newly installed SMF services.
  • While zpool(1M) status can show you the last time a pool has been scrubbed, this information is not stored persistently. Every time you reboot or export/import the pool, ZFS loses track of when the last scrub of this pool occurred. This has been filed as CR 6878281. Until that has been resolved, we need to take care of remembering the time of last scrub ourselves. This is done by introducing another SMF service that periodically checks the scrub status, then records the completion date/time of the scrub in a custom ZFS property called org.opensolaris.auto-scrub:lastscrub in the pool's root filesystem when finished. We call this service whenever a scrub is started and it deactivates itself once it's job is done.
  • As mentioned above, the GUI is based on the OpenSolaris Visual Panels project. Many thanks to the people on its discussion list to help me get going. More about creating a visual panels GUI in a future blog entry.

Lessons learned

It's funny how a very simple task like "Write an SMF service that takes care of regular zpool scrubbing" can develop into a moderately complex thing. It grew into three different services instead of one, each with their own scripts and SMF manifests. It required an extra RBAC role to make it more secure. I ran into some zpool(1M) limitations which I now feel are worthy of RFEs and working around them made the whole thing slightly more complex. Add an install and de-install script and some minor quirks like using GNU date(1) instead of the regular one to have a reliable parser for human-readable date strings, not to mention a GUI and you cover quite a lot of ground even with a service as seemingly simple as this.

But this is what made this project interesting to me: I learned a lot about RBAC and SMF (of course), some new scripting hacks from the existing ZFS Auto-Snapshot service, found a few minor bugs (in the ZFS Auto-Snapshot service) and RFEs, programmed some Java including the use of the NetBeans GUI builder and had some fun with scripting, finding solutions and making sure stuff is more or less cleanly implemented.

I'd like to encourage everyone to write their own SMF services for whatever tools they install or write for themselves. It helps you think your stuff through, make it easy to install and manage, and you get a better feel of how Solaris and its subsystems work. And you can have some fun too. The easiest way to get started is by looking at what others have done. You'll find a lot of SMF scripts in /lib/svc/method and you can extract the manifests of already installed services using svccfg export. Find an SMF service that is similar to the one you want to implement, check out how it works and start adapting it to your needs until your own service is alive and kicking.

If you happen to be in Dresden for OSDevCon 2009, check out my session on "Implementing a simple SMF Service: Lessons learned" where I'll share more of the details behind implementing this service including the Visual Panels part.

Edit (Sep. 21st) Changed the link to CR 6878281 to the externally visible OpenSolaris bug database version, added a link to the session details on OSDevCon.

Edit (Jun. 27th, 2011) As the Mediacast service was decommissioned, I have re-hosted the archive in my new blog and updated the download link. Since vpanels has changed a lot lately, the vpanels integration doesn't work any more, but the SMF service still does.

Monday Jul 06, 2009

How to Fix OpenSolaris Keyboard Irregularities with Virtual Box

Virtual Box is great: It allows you to install OS A on OS B for impressively large sets of A and B OSes and their permutations. Almost everything works smoothly and seamlessly between host and guest: Cut&Paste, File sharing, networking, USB pass-through, even seamless windows are supported.

But there's one little glitch that is still a little annoying, but apparently not annoying enough for someone else to have blogged about this before: Keyboard remapping on Mac OS X hosts.

The Problem

Simple problem: Macs are different than PCs (phew), but they have slightly different keyboard mappings (oops). Most notably, on my German keyboard, the "<" key at the bottom left on the Mac will yield "\^" on OpenSolaris and vice versa. Same thing goes for "@", which is Right-Alt-L on the Mac, but Right-Alt-Q on PCs. Similar difficulties are encountered if you try to create a "|" pipe symbol or angular/curved brackets ("[]" and "{}" respectively).

Pressing the Right Keys

Usually no big deal. Close your eyes and blindly type what you would type on a PC and that'll give you a good hint at where the right keystrokes are. That works because Virtual Box actually maps the physical locations of the keys between host and guest, but not what's painted on them. So, with a little practice, you should be fine. But what happens if you can't quite remember what that PC keyboard looked like?

Last Friday I had an hour or so left and the playfulness of the problem got the better of me, so I decided to see if this can be fixed the Unix way. It's actually quite easy.

Searching for a cure

There are some helpful hints on the net, most notably Petr Hruska's entry on "Switching Keyboard Layout in Solaris", but it only deals with internationalization issues. What if you have the keyboard nationalities right, but individual keys are still different as in the Mac/PC case? Here's a step-by-step guide to help you with any keyboard remapping problem, plus a bonus table for OpenSolaris on Macbook users to get you started:

Xmodmap to the rescue

  1. We're going to use xmodmap(1) to remap the keys on our keyboard. Check out the man-page to familiarize yourself with how it works.
  2. See the keystrokes as OpenSolaris sees them: Use xev(6) to find out what keycodes belong to the keys you want to correct.
  3. Check out what OpenSolaris is thinking about your problematic keys, either by testing them in a terminal or by checking your version of the standard Sun USB keyboard layouts.
  4. Before you start modifying the current keyboard mapping, get the currently active one by saying something like:
    xmakemap > ~/.xmodmaprc.current
    Caution: There seems to be a bug in xmakemap that corrupts some of the entires. So, please use this only for reference but do not feed this file back into xmodmap (see later) or you'll likely make your keyboard unusable (until this bug is resolved).
  5. Start editing your own remapping script for xmodmap:
    vi ~/.xmodmaprc
  6. For each key you want to remap, copy it's keycode entry from the xmakemap output into your own remapping table and modify to taste. Be careful, some entries from xmakemap are broken, but you should be able to figure those out. Here's my current .xmodmaprc file as a reference:
    !
    ! Set up keys for a MacBook Pro running OpenSolaris on VirtualBox
    !
    !       Key   Unshifted       Shifted         AltGraph        AltGraph-Shifted
    !       ---   --------------- --------------- --------------- ----------------
    
    keycode  49 = less            greater
    keycode  94 = asciicircum     degree          asciicircum     degree
    keycode  14 = 5               percent         bracketleft
    keycode  15 = 6               ampersand       bracketright
    keycode  16 = 7               slash           bar             backslash
    keycode  17 = 8               parenleft       braceleft
    keycode  18 = 9               parenright      braceright
    keycode  24 = q               Q               q               Q
    keycode  46 = l               L               at
    keycode  57 = n               N               asciitilde
    
    This works well on my MacBook Pro, your mileage may vary.
  7. You can activate your remapping by saying something like:
    xmodmap ~/.xmodmaprc
  8. In case something goes wrong and you render your keyboard useless, you can restart your X server by pressing Ctrl-Alt-Backspace twice.
  9. If you're happy with your remapping, you can automatically activate it on every login by using the System->Preferences->Sessions panel and adding an entry for the above xmodmap command there.

Conclusion

I hope this little exercise in some lesser known X-Windows commands (Hi Jörg) was useful for you, now you shouldn't need to worry too much about keyboard mapping inconsistencies any more.

If you want to learn a little more about modifying your keyboard, check out this section of the OpenSolaris docs.

The example keymap modifications above work well for me, but I'm sure I've forgotten a key or two. What other keys did you remap and why? Feel free to leave me a comment below.

About

Tune in and find out useful stuff about Sun Solaris, CPU and System Technology, Web 2.0 - and have a little fun, too!

Search

Categories
  • General
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Bookmarks
TopEntries
Blogroll
OldTopEntries