Thursday Sep 03, 2009


I don't consider myself an academic, but I did manage a "Desmond" honours degree in Microelectronics and Computing from the University of Wales, Aberystwyth. It was there that I first fell in love with BSD UNIX (on a VAX 11/750), and there that I saw my first Sun workstation (a 2/120, although I was never allowed to use it).

I have since maintained an active interest in the education market, because I feel it is a natural recruiting ground for future Sun employees and customers. Indeed, I have recruited at least three people into Sun from Aberystwyth alone. Over the years I have worked with the Universities of Aberystwyth, Bangor, Bradford, Dundee, Durham, Leeds, Liverpool, Manchester, Oxford, St Andrews, Salford, Warwick and York, most recently on Sysadmin day conferences in Aberystwyth and Manchester.

Job rotations

An invaluable part of my personal development at Sun has been the SE Job Rotation programme, which was run by Barabara Hill (or "Mom" as she was affectionately known by many of us "on rotation") ...

  • Siebel scalability and load balancing (MDE, 2 weeks)
  • Oracle 7 on Solaris x86 and Windows NT (OPG, 2 weeks)
  • BaaN scalability (MDE, 9 weeks)
  • Many users project (PAE, 4 weeks)
These job rotations gave me the opportunity to learn alongside thought leaders such as Adrian Cockcroft, Allan Packer, Bob Larson, Brian Wong, Dan Powers, Jim Mauro, Mike Briggs and Richard McDougall. They also gave me my first real contact with Solaris engineering, and paved the way for some of the jobs I've moved through over the years.

The GORB (Giga Object Request Broker)

One of the first solutions designed to make full use of the 64 CPUs and 64GB of the Enterprise 10000 ... in a single process ... using threads. This was a collaborative R&D project with a large telco, which resulted in at least one patent being filed. My role was to deliver the multithreading expertise needed to make this fly.

Solving solitaire with a sledgehammer

I'm only including this example because it was so outrageous! A government agency was investigating various hardware platforms for (what I guess was) HPC cryptographic applications. Obviously, they were not sharing any of their actual code, but instead specified a number of number crunching challenges for large scale multiprocessors. I took up the challenge of Solitaire with a 16-way E6000 and more than 8GB of RAM.

    o o o           \* \* \*           o o o           5 6 7          5 6 7
    o o o           \* \* \*           o o o           2 3 4          2 3 4
o o o o o o o   \* \* \* \* \* \* \*   o o o o o o o       0 1        7 4 0 1 0 2 5
o o o o o o o   \* \* \* o \* \* \*   o o o \* o o o                  6 3 1 x 1 3 6
o o o o o o o   \* \* \* \* \* \* \*   o o o o o o o                  5 2 0 1 0 4 7
    o o o           \* \* \*           o o o                          4 3 2
    o o o           \* \* \*           o o o                          7 6 5

    Fig.1           Fig.2           Fig.3           Fig.4          Fig.5

In Figs.1-3 a "o" represents a hole, and a "\*" a marble in a hole. Thus, Fig.1 shows the empty board, Fig.2 the staring position (32 marbles), and Fig.3 the target end position. My solution was to encode each board position as 33 bits, with 0 for a hole and 1 for a marble. Threads in a worker pool took known board positions from a work pile and found all possible news moves, which were then added to the work pile. Exploiting rotational and reflectional symmetry, each new board position becomes up to eight possible board positions.

By coding the 33 bits as shown in Figs.4-5 I was able to make rotation a simple byte swap. Reflection is harder, but only needs to be done once (since the remaining three reflections and be achieved by rotating the first reflection). The really extravagant part was adding an 8GB char array to record board positions that had already been seen (and which, therefore didn't require to be explored again).

The result was a solution in just 2 seconds, with all possible board positions being found within 30 seconds. Sun hardware and my expertise in multithreading has moved on a lot in the intervening years. I'm now itching to try the exercise again on an T5220!

The WOT (Wall Of Terminals)

The Wall Of Terminals was a high-tech response to something a competitor was doing in their benchmarking centre with 50 dumb terminals, namely showing the state of simulated users during large scale multiuser testing.

My design consisted of the following hardware:

  • a custom built piece of furniture
  • six dual-headed SPARCstation 5 workstations
  • two triple-headed SPARCstation 10 workstations
  • eighteen premium 21 inch Sun monitors
and the following software:
  • scripts for building cloned diskless boot environments for the eigt workstations
  • a CDE application which multiplexed up to 48 DtTerm widgets in one window
  • a multithreaded application routing up to 864 pseudo terminals across 18 instances of the above
  • a Java applet to reconfigure the number of DtTerm widgets displayed on the fly, and to select one to
  • zoom

The WOT was instrumental in winning a huge MRP deal in the aerospace industry, but also proved very useful as a collection of eighteen X11 screen for displaying just about any benchmark data. My WOT also won an innovation award at the second Sun Technical Symposium in San Francisco.

Oracle, poll() and nanosleep()

Whilst on my first SE Job Rotation to Mountain View, I noticed that Oracle was using poll(0,0,10) as a "cheap" sleep mechanism (it is, after all, cheaper than the usleep() SIGALRM dance). However, at that time Solaris had a clunky implementation of poll() which did not scale well (see below). My solution was to write an interposing shared library (something very new in those days) which mapped poll() onto nanosleep() for the sleep only case. We told our Oracle engineering contacts about what I had done, but heard nothing. About a year later I was running truss on Oracle, and noticed that they had implemented my recommendation.


One of the early challenges for systems like the SPARCcenter2000 was that Solaris only supported 48 TELNET users by default. But as this limit was removed, Solaris would die from heavy lock contention triggered by in.telnetd in poll(). Whilst on my first SE Job Rotation, it seemed to me that the obvious solution was to recode the TELNET service using a STREAMS module. At the time I didn't have the kernel skills to do this, but I knew someone who did. However, they needed a business case to do the work. I implemented a poll()-free TELNET service using the newly available user-level threads library (i.e. one process handling hundred of TELNET sessions). My data provided the business case, and in.telnetd was given its STREAMS implementation.

The Liverpool University Benchmark

The second SPARCcenter2000 in the UK went to Liverpool University. They had the full complement of 20 CPUs, and were running Solaris 2.2 (which was not certified beyond 8 CPUs). Their acceptance tests consisted of simulated TELNET traffic from a number of PCs replaying various shell user scenarios (ls, vi, cc, f77, etc). Needless to say, the system didn't stay up for more than about 10 minutes at a time. I ported their test hardness to EMPOWER so that it could be used by engineering, and it is still used to this day).

Project Cascade and Jumpstart for Solaris 1

Engineering insurers, National Vulcan placed a large order for IPX workstations. I devised a simple, but elegant system for installing and configuring the systems that leveraged Solaris 2's JumpStart and NIS netgroups (the clever bit is that the target system was Solaris 1). This was then also rolled out a number of other customers (including some universities).

Sun User Group and Solaris 2.0 for System Administrators

Over the years I was a regular speaker at Sun User Group, and wrote several book reviews for their magazine. My first real speaking engagement ever was a 2 hour slot introducing Solaris 2.0 to system administrators at a joint SUG / UKUUG meeting at Oxford University. I'm not sure what scared me most: the 200+ people in the room, or the fact that Rob Gingell sat through it all, grinning. This launched my public speaking career (and I still prefer to have at least 2 hours)! I have been back to Oxford University many times since, but nothing quite compares with that first "baptism of fire".

Realtime satelite data acquisition with SunOS 4

Perhaps the most rewarding work I've ever done. The Dundee Satelite Receiving Station had a PDP11 system to capture data from circumpolar satelites, and had recently upgraded to a 4/330 with a DR11W VME board. Unfortunately, this interface had no buffering, and we needed to capture 22K of data at 6Hz. I devised a multiprocess, non-blocking implementation which did the job admirably. My reward was seeing stunning live pictures of the earth. Utterly beautiful!

Changing the rules at Scottish Courts

This is probably the biggest and longest running single project I've ever undertaken at Sun, and it was enough to convince me that I would prefer shorter "flog-it and leg-it" projects in future. DEC (remember them?) gone a long way with the customer, and a fully functional VMS/Oracle proof of concept implementation was in place, complete with VT100 screen dumps and hundreds of DCL scripts. It also made extensive use of VMS batch queues and print spooling. Somehow we won the business, and my job was to architect the final implementation on SunOS/Oracle. My solution was to convert all the DCL to Bourne Shell, and to write my own batch/print queueing system (which was network extensible with NFS ... Sun Grid Engine eat your heart out!) plus tape management system. The system went live in over 50 sheriffs' courts (some with just a single SPARCserver 1, but larger sites having a Sun 4/330 server and multiple SPARCserver 1 clients). This was my first exposure to Oracle and SQL\*NET. I have no idea if or when the system was replaced (except that I was asked to advise on the Solaris 2 port a few year later).

APT at LIFFE on the SPARCstation 1

This was an early instance of holistic system performance work. Automated Pit Trading was a world first. Instead of chaps in blazers waving at eachother and throwing orders on the floor (to be collected by sweepers), APT provided a SunView GUI to traders across London. Fairness was guaranteed by providing everyone with exactly the same hardware platform. The customer was not at all happy when we introduced the SPARCstation 1+, because it meant everyone had to be upgraded. RAM was always at a premium, and I had been asked to go in to pare down the kernel and system services to a minimum. While I was there, I was asked to investigate an application hang. It turned out that the printer (which was not spooled) had run out of paper, and that the application was blocking on a write to the printer port!


Probably the first serious code I wrote at Sun was my solution to the "rtelnet" problem (but it was much better than the utility of that name that came with certain TELNET terminal servers). The pty.telnet service allowed system administrators to setup robust, fixed mappings from named pseudo terminal devices to specific ports on specific terminal servers. It became a "UK Consulting Special", which meant people paid real money for it. I alter ported it to Solaris 2, where the dynamically allocated, clone psuedo terminal devices required a dance with symlinks to provide the same functionality. I've got the code still somewhere, if anyone is interested!

Update: it's moved, but it's official

On Tuesday (September 1st, 2009) I had the long-awaited phone call from my manager, and am now able to report that I will be leaving Sun on September 30th, after 20 years and 7 months. Yes, this differs from my previous post, and it's not the first change of plan, but it's hard to soar like eagles... Although I've been waiting for this for some time, I'm still quite numb. Most of the last 20 years has been a lot of fun, but I'm not leaving the company I joined. However, there are still some very special people at Sun, and I will miss being their colleague. I hope to post a few more things here before taking up "garden leave", and to rehost the blog elsewhere so that I can continue to share a few morsels from time to time. I'm not sure what the future holds, but I know who holds the future, and that He has a plan.




« April 2014