Wednesday Jun 02, 2010

Goodbye my Sun

Goodbye my Sun.

(I don't eat fish and the Douglas Adams quote is overused, anyway.)

I joined Sun Microsystems on March 10, 1997 so I was just about to complete thirteen years at Sun when the acquisition happened. It may have had its highs and lows but all in all it was a wonderful ride at a legendary company. Having been part of Sun was a dream fulfilled.

I'll always remember Sun for the relentless innovation and the highest standards for technical excellence. Sun was a place where engineering reason prevailed over bureaucrats, assumptions were questioned and critical thinking stood over mere paperwork. We may not have marketed it very well but it sure was the best technology on Earth.

I worked on many projects over the years but the longest running and most special one was the Web Server [iPlanet|SunONE|JES|SJS - nobody loved continuous product renaming like Sun!] Always a small team but with the highest passion for producing the best code on Earth, even against all odds. Thanks for the good memories, Web Server team!

All wonderful things come to an end, I suppose, so did Sun and so does this blog now. Hopefully the articles will continue to be here for future reference but if not, I have also made them available on my own site at

Interesting challenges have fallen my way so it is time to pursue them.

You know where to find me. Keep in touch!

Saturday Mar 27, 2010

Joining the ZFS Revolution

For a long time now I've been meaning to migrate my home file storage over to a ZFS server but the project kept getting postponed due to other priorities. Finally it's alive!

For the last ten years or so my home fileserver has been through the general purpose debian box in the garage. It has three disks, one for the system and home directories, a larger one which gets exported over NFS and the largest one which backs up the other two (nightly rsync). It has been an adequate solution, in as far as I've never lost data. But whenever a disk dies I always have several days of downtime and have to scramble to restore from backups and maybe reinstall.

There are many articles about this topic that make for good reading if you're considering the same. My goals were:

1. Data reliability, above all.
Initially I had visions of maximizing space, mainly for the geek value of having many terabytes of home storage. But in the end, I don't really need that much. The NFS export drive on my debian box was currently only 500GB and that was used not only by the shared data (pictures, mostly, and documents) but also for MythTV storage. Since I wasn't planning on moving the MythTV data to the ZFS pool, even 500GB would be plenty adequate for some time.

2. Low power consumption.
Since this is another server that'll need to run 24/7, I wanted to keep an eye on the power it uses.

3. But useful for general computing.
Since this will be the only permanent (24/7) OpenSolaris box on my home network, I also wanted to be able to use it for general purpose development work and testing whenever needed. So despite the goal of low power consumption, I didn't want to go all out with the lowest possible power setup, needed a compromise.

Here's the final setup:

CPU: AMD Phenom II X4 (quad core) 925. Reasonable power consumption and the quad cores give me something fun to play with.

Memory: 8GB ECC memory. Since I'm going primarily for data reliability, might as well go with ECC.

ZFS pool: 3 x 1TB drives. These are in a mirror setup, so total storage is just 1TB. That's still about three times as much as I really need right now. With three drives, even if two fail before I get to replace them I should be ok. I got each of the three drives from a different manufacturer, hopefully that'll make them fail at different times.

        NAME        STATE     READ WRITE CKSUM
        represa     ONLINE       0     0     0
          mirror    ONLINE       0     0     0
            c8d0    ONLINE       0     0     0
            c8d1    ONLINE       0     0     0
            c9d0    ONLINE       0     0     0

System disk: I expected to just use one older drive I had on the shelf, but after installing it I found it was running hot. Maybe it is ok but decided to do a two-way mirror of the rpool as well, maybe it'll save me some time down the road. I don't need much space here so found the cheapest drive I could get ($40) to add to the rpool. At that price, might as well mirror!

        NAME         STATE     READ WRITE CKSUM
        rpool        ONLINE       0     0     0
          mirror     ONLINE       0     0     0
            c9d1s0   ONLINE       0     0     0
            c10d1s0  ONLINE       0     0     0

Total power consumption for the box hovers around 78-80W most of the time.

Thursday Jan 21, 2010


So here I sit, at the very end of Sun Microsystems.

(oblink to James Gosling's entry)

Who would've thought!

Close to twenty years ago the university received a shiny new SPARCserver/390. Sure we had other hardware from HP (HP-UX, ugh) and IBM (AIX, even worse!) but that 390 with SunOS was special. I cajoled my way into being the sysadmin for that lab mainly so I could get unlimited play time with it.

Later after finishing grad school I ended up elsewhere but Sun was still the coolest company on Earth. I quickly "found" (not by accident) myself with a SPARCstation10 which later became a 20 and so on... Today my 'desktop' is a SunFire server but since it is insanely noisy I keep it in a lab and display through a SunRay in my office.

Inevitably, I later ended up here at Sun (coincidentally, right when Bellcore got acquired) and the engineering culture was as great inside as the products were cool from a customer perspective (as to the management side of the company, the less said the better I suppose). So here we are, at the Sunset of it all. Now, a very red sunrise.

So, what's next for Sun Web Server?

Wednesday Dec 09, 2009

Where Did The Time Go?

Back in June I posted my initial entry on email time management with my intention of allocating time windows for email instead of spending all day on that alone. In August I posted some observations and a pie chart of the data so far.

As 2009 winds down, I now have about seven months of data so seems like a good time to revisit the numbers. I'll start with the bottom line, how much time went to what?

So the email timesuck has continued to be the problem it ever was, with 45% of the time between May to December spent on that alone. No matter how I slice it, that's really bad. My observations from the August entry are as valid as then. The relentless email firehose is hard to shut down so keeping it at a reasonable percentage of time is seemingly impossible. But at least segregating email reading into well-defined time windows during the day does help a great deal.

The really sad story in that pie chart is that the two slices which represent Real Work - the fun part of software engineering - add up to less than 10% of the total! The blue slice is Web Stack engineering work (8%) and the green slice is Web Server engineering work ( ploticus obscured the text but it's about 1.5%). So there you have it, about 90% of the time of a product architect/senior engineer at Sun is spent on email and overhead.

Put that way, those numbers are appalling beyond words. Clearly this must change, so I'm going to change it. Starting this week I will cut down email time down to 90-120 minutes a day (I've been doing well over three hours minimum) and start to prioritize Real Work higher on my task list above TPS report-kind of work.

Can this plan succeed? Well I guess you can read my update in a few months to see. I'm looking forward to a much more reasonably balanced pie chart...

Tuesday Aug 18, 2009

Time Allocation

Last week I wrote about the time spent dealing with email. While writing that entry I though it'd be nice to also visualize where all the time went, not just how much was spent on email. So tonight I went over the data to generate the following pie chart, showing relative allocation of working hours from early May until today, split into high level categories:

The 'Email' slice is self evident.. 'ARC' is the time I've spent in my role in Sun's Architecture Review Committee. 'Communications' includes conferences, presentation, blog entries, articles and other related work. 'Administrivia' is a catch-all category for all kinds of mindless unproductive overhead. Finally, 'Engineering' represents the time spent doing "real work".

About the only thing I can add is that this is about as concise a representation as we can get on why very large companies have trouble competing with agile startups. Part of my goal in this exercise is to find ways to grow that nice blue pie slice, but I realize there's a limit to what can be achieved in this environment. All those TPS\^H\^H\^HPTL reports needs to be filed, after all.

Wednesday Aug 12, 2009

Let Me Check My Email

About two months ago I posted on attempting to keep email in check so it's a good time to review some statistics and results...

The following graph shows the percentage of time I spent reading email each day:

The average over the past three months is about 45% Wow.. So over the last quarter I've spent just under half of all working hours reading (and answering) email. No wonder it is hard to get concrete work done!

This is somewhat higher than the 37.5% (three hours a day out of eight) that I had predicted in the previous article a couple months ago. This is largely explained due to the recent release of Web Stack 1.5. Due to the impending release I found myself having to check email more often than scheduled to keep on top of last minute pre-release activities.

A few points worth noting out of the experiment so far...

  • It is not easy to limit email activity to the scheduled two or three hours a day. Ideally the graph above should be mostly flat. While part of this is inevitably due to the release activities, I'll try harder going forward to stick to the scheduled email times.
  • While the total times may have fluctuated more than I wanted, I did (mostly) manage to contain my email activities to bounded windows of time within the day, instead of checking emails every three minutes all day long. This has helped a great deal. Even while spending nearly half my hours on email, I've managed to get many other non-email tasks done more productively than before. This part has been a success and I highly recommend it. Shut down that email client!
  • I found myself doing three (or even four) email sessions per day. This is too many. I need to more strictly limit myself to reading email only twice a day, at the beginning and end of the day. If these sessions need to be longer it is better to make them longer but stick with only two. Whenever I started inserting email tasks in the middle of the day, it fragmented my concentration too much, making the day less productive.
  • I'm convinced the ideal arrangement is to do one single email session per day, at the end of the day. That way all the concentration disruption occurs after the days work is done, so it does no harm. The end of the day is also a good time to be entering new tasks into the to-do list so they'll be there tomorrow. Given our distributed time zones it is difficult to do only one email session per day, but that would be ideal. Maybe I'll try that at some point.

As a longer term goal I need to think of ways of reducing the time spent on email. Not sure how to do that yet but spending 45% or even "only" 37% of all working hours on email is totally insane. I suppose email overload is inevitable at a large company with tens of thousands of employees (all of whom, it seems at times, are emailing me) but there has to be a better way. I suppose I could cap my email time to an hour a day and let whatever goes unread just go unread. I'm sure people will be unhappy but will that unhappiness be greater than my productivity gain at doing real work? It's all about tradeoffs, after all. Hard to say what's worse.

Monday Jun 15, 2009

Email Backlog

It's no secret that email overload is a problem these days, here's just a few of many articles on the topic:

A quote from the second article above is particularly interesting (or scary):

   In this study Dr Jackson found that it takes an average of 
   64 seconds to recover your train of thought after interruption 
   by email. So people who check their email every five minutes 
   waste 8 and 1/2 hours a week figuring out what they were 
   doing moments before.

In nearly 20 years (wow!) of reading lots of email daily this has never been much of a problem for me though. I always managed to keep my inbox almost empty from day to day (I long considered 100 emails to be the maximum threshhold to ever have pending in the inbox).

Thinking back, I'd say historically the bulk of my incoming email has been either

  • Administrivia (meeting announcements and such): quickly dispatched without thought or mental interruptions
  • Engineering content, directly related to whatever I'm working on that day: these take time to read and process but since the emails are relevant to the current project they don't cause a mental context switch and may even help further the project at hand so there is a net win

As resources get tighter and I find myself filling more and more roles simultaneously the dynamic has changed in the last 6-9 months or so. From a perpetually clean inbox I've now gone to a significant backlog. Even more annoying is that I find there are many days where all I get to do is read email!

After some months it is clear this is not a temporary crunch, so I need to change strategies from what has worked in the past. I spent some time monitoring my email activity to figure out what is different. It's not really quantity, I've always received lots of email but it hasn't been a problem. The key difference appears to be that now I'm involved in many projects each one with many unrelated trains of discussion.

As emails arrive, each one is more often than not unrelated to the previous ones and also unrelated to what I'm actually trying to get done at the moment. And thus, I find myself facing the case made in the Dr. Jackson study quoted above.

As each email arrives I read it and start thinking of that particular project/problem for a few moments (a few seconds to a minute or two). It is not enough time to solve or address the issue, just enough to get distracted. Hoping to get back to the real work I was doing instead of spending more time on this new train of though, I don't actually process the email, so it remains in the queue.

By then, several other emails have arrived so I repeat the cycle with each one. By the time I finally get back to what I was actually working on, that project is so many mental context switches behind I no longer have any idea what I was doing and need to spend several minutes getting back into it. By which time, of course, ten more emails have arrived... and the cycle repeats all day.

So I need to address the interruption and context switch problem. A few weeks ago I started to allocate limited time to email. Specifically:

  • Only read email in specific blocks of time preallocated for email on that day.
  • If I can answer or resolve the issue in less than 5 minutes, do so right then, within the time allocated for email handling.
  • If it's going to take any longer than that, add a task to the bottom of my to-do list and move on.
  • The rest of the time, quit mutt and resist all urges to go look at email.

I started by allocating two hours a day to email, one in the morning and one in the afternoon. Quickly it became apparent this is not enough to keep up, so I increased it to three hours. I'll gather more data before settling on the final timing but looks like it'll have to be a bit over three hours a day for email processing.

Here's a graph, showing only a few days from last month. I'll post another one with much more data once some more time goes by so I have more numbers. The yellow area is my current email backlog and the blue line is the number of minutes a day spent processing email.




Top Tags
« August 2016