Tuesday Apr 29, 2014

My own private crash-n-burn farm: using kernel zones for speedy testing

I've spent most of the last two years working on a complete rewrite of the ON consolidation build system for Solaris 12. (We called it 'Project Lullaby' because we were putting nightly to sleep). This was a massive effort for our team of 5, and when I pushed the changes at the end of February we wound up with about 121k lines of change over close to 6000 files. Most of those were Makefiles (so you can understand why I'm now scarred!).

We had to do an incredible amount of testing for this project. Introducing new patterns and paradigms for the complete Makefile hierarchy meant that we had to be very careful to ensure that we didn't break the OS. To accomplish this we used (and overloaded) the resources of the DIY test group and also made use of a feature which is now available in 11.2 - kernel zones.

Kernel zones are a type-2 hypervisor, so you can run a separate kernel in them. If you've used non-global zones (ngz) on Solaris in the past, you'll recall the niggle of having to have those ngz in sync with the global when it comes to SRUs and releases.

Using kernel zones offered several advantages to us: I could run tests whenever I wanted on my desktop system (a newish quad-core Intel Core i5 system with 32gb ram), I could quickly test updates of the newly built bits, I could keep the zone at the same revision while booting the global zone with a new build, and (this is my favourite) I could suspend the zone while rebooting the global zone.

Our testing of Lullaby in kernel zones had two components: #1 does it actually boot? and #2 assuming I can boot the kz with Lullaby-built bits, can I then build the workspace in the kz and then boot those new bits in that same kernel zone?

Creating a kernel zone is very, very easy:

limoncello: # zonecfg -z crashs12 create -t SYSsolaris-kz
limoncello: # zoneadm -z crashs12 install -x install-size=40g
limoncello: # zoneadm -z crashs12 boot

I could have used one of the example templates (eg /usr/share/auto_install/sc_profiles/sc_sample.xml) but for this use-case I just logged in and created the necessary users, groups, automount entries and installed compilers by hand. (Meaning pkg install rather than tar xf).

To start with, I ensured that crashs12 was running the same development build as my global zone, but I removed the various hardware drivers I had no need for.

The very first test I ran in crashs12 was a test of libc and the linker subsystem. Building libc is rather tricky from a make(1s) point of view, due to having several generated (rather than source-controlled) files as part of the base. The linker is even more complex - there's a reason that we refer to Rod and Ali as the 'linker aliens'! Once I had my fresh kz configured appropriately, I created a new BE, mounted it, then blatted the linker and libc bits onto it and rebooted. I was really, really happy to see the kz come up and give me a login prompt.

Several weeks after that we got to the point of successful full builds, so I installed the Lullaby-built bits and rebooted:

root@crashs12:~# pkg publisher
nightly origin online F file:///net/limoncello/space/builds/jmcp/lul-jmcp/packages/i386/nightly-nd//repo.osnet/
extra (non-sticky, disabled) origin online F file:///space/builds/jmcp/test-lul-lul/packages/i386/nightly/repo.osnet-internal/
solaris (non-sticky) origin online T http://internal/repo
root@crashs12:~# pkg update --be-name lul-test-1
root@crashs12:~# reboot

This booted, too, but I couldn't get any network-related tests to work. Couldn't ssh in or out. Couldn't for the life of me work out what I'd done wrong in the build, so I asked the linker aliens and Roger for help - they were quick to realise that in my changes to the libsocket Makefiles, I'd missed the filter option. Once I fixed that, things were back on track.

Now that Lullaby is in the gate and I'm working on my next project, I'm still using crashs12 for spinning up a quick test "system" and I'm migrating my 11.1 Virtualbox environment to an 11.2 kernel zone. The 11.2 zone, incidentally, was configured and installed in about 4 minutes using an example AI profile (see above) and a unified archive.

Kernel zones: you know you want them.

Sunday Aug 02, 2009

Presentations and flickrstream of KCA2009 available

I've finally pulled the finger out to get my photos from KCA2009 up on flickr. Have a squiz at these.

More importantly, we've got almost all of the presentations from the conference up on the website. As time goes by we'll get the others (Brendan, Stewart, Jeff and Bill).

Monday Jul 20, 2009

KCA2009 - winding down #1

Now that I've had a weekend to start recovering from Kernel Conference Australia, it's time to start catch up on a few blog entries. There'll be several over the next few weeks as I work through everything I want to mention.


A massive thankyou to our volunteers on the ground (James Lever, Greg Black, Nikolai Lusan, Daniel Dawson) who all did a stellar job and made it possible for me to be the shiny happy face of the conference :-)

Thankyou to the review committee (Jake Carroll, Andre van Eyssen, David Gwynne and John Sonnenschein) who helped put the program together.

Thankyou to our speakers, without whom there would not have been any conference to attend.

Thankyou to Deirdre Straughan who got us live streamed sessions on ustream and recorded many hours of video.

Thankyou to Claire Operie, Gabriele Decelis and Diana Pearce who handled the logistics of getting the event off the ground (registrations, the event website, catering, swag etc).

Thankyou to Mitch Roberts for setting up the demo room and who answered hundreds of questions about pretty much every piece of technology Sun sells. He also showed off some really amazing VDI and demonstrated the Fishworks kit. Mmmmmmmmmmmm!

Thankyou to Jake Carroll for setting up the infrastructure we needed at the Queensland Brain Institute. Thankyou especially to QBI's director Professor Perry Bartlett, FAA and Ian Duncan for generously allowing us to use QBI's facilities for the conference.

Thankyou to the Brisbane branch of Sun Microsystems who got behind the idea and provided connections, spread the word and encouraged people to come along.

Thankyou to everybody who attended KCA, who thought it would be something worthwhile. I hope you went away excited, energised and enthused about all the really cool technologies that you can find in Open Source kernels, and even be inspired to contribute to your favourite kernel in the future.

Finally, thankyou to our sponsors - Frontline, OpenSolaris, and Sun Microsystems.

Presentations, Videos, Papers, Proceedings

A number of people have asked me over the last few days whether the presentations will be made available online. The answer is a resounding YES, but not just yet. What the review committee (well, just Andre, and myself) are doing is creating a Proceedings of the conference. Within that we will have the papers or slideware and speakernotes that each presenter wrote. When we have that finished I will announce it on this blog, and I will email all the delegates to provide details on how to obtain a copy. (Hopefully it'll be in the National Library of Australia with proper CiP data too).

The videos that we recorded are being cleaned up and will be uploaded to somewhere with a lot of space. Soon. I don't know when, but when I do know, I'll announce it here.

Will we run this again in 2010?

The feedback I've received from people attending KCA has generally been very positive about the event, and encourages me to organise another KCA for next year. We'll have to wait and see how things pan out following the buyout vote, but I'm very hopeful that we'll be able to make it happen.

Wednesday Jul 08, 2009

KCA starts in one week

I'm really excited now - it's only one week until Kernel Conference Australia kicks off at the Queensland Brain Institute.

The speaker presentations (slideware and copious notes, as well as plain text so we can incorporate them into the forthcoming Proceedings volume) are coming in and are looking really good. It'll take a bit of time after the conference ends before we can get the Proceedings finalised, but the presentations themselves will be available fairly soon afterwards; I'll mention the url here when I get it finalised.

There's still time for you to register and come along to hear, learn from and hang out with some of the finest minds in Open Source today.

Thursday Jun 25, 2009

Late feature addition - to the KCA speaker list

In case you weren't quite convinced about just how cool, uuuuber, and just plain old fantastic an opportunity Kernel Conference Australia is, then you should consider this. We've got one more speaker coming along:

Brendan Gregg, member of the Fishworks team, author of the DTraceToolkit, co-author of Solaris Performance and Tools.

Naturally, he'll be talking about DTrace and all the serious and crazy (and seriously crazy) things he's done with it over the years.

Dates15 - 17 July, 2009
VenueQueensland Brain Institute, University of Queensland
Conference homepagehttp://au.sun.com/sunnews/events/2009/kernel
Full abstractshttp://wikis.sun.com/display/KCA2009/KCA2009+Conference+Agenda
Registration pagehttps://www.conveneit.com/secure/sun/kernel_jul_09
regular price$300

Thursday Jun 11, 2009

Kernel Conference Australia - earlybird price closes \*TOMORROW\*

Just a short reminder that if you want to come to Kernel Conference Australia at the earlybird price of $195, then you've got until the end of this Friday, 12 June, to get your registration happening.

If you're interested in any of these areas:

  • linux kernel crypto services

  • ZFS deduplication

  • cross-architecture OS and driver porting

  • packet filtering and QOS

  • TCP/IP protocol security

  • bug finding tools for OpenSource Operating Systems

  • network virtualisation, or

  • how a DB engine can really hurt your system

then Kernel Conference Australia is definitely for you.

Dates: 15 - 17 July, 2009
Venue: Queensland Brain Institute, University of Queensland

The student price is still $95, too.

For full abstracts please see http://wikis.sun.com/display/KCA2009/KCA2009+Conference+Agenda

For the conference homepage, see http://au.sun.com/sunnews/events/2009/kernel/

And for registration, Go Without Delay To


I look forward to seeing you there.

Sunday May 24, 2009

KCA2009 - earlybird registrations close in 1 week!

It's only one week before the earlybird registration period for Kernel Conference Australia closes.

As a quick reminder, in addition to our excellent keynote speakers Jeff Bonwick, Bill Moore and Max Alt, here are the people who you'll be able to meet, listen and learn from at KCA:

Fernando GontResults of a Security Assessment of Common Implementation Strategies of the TCP and IP Protocols
Henning Brauer (OpenBSD)Faster Packets: Performance Tuning in the OpenBSD Network Stack and PF
Gavin Maltby (Sun Microsystems)Hardware & Software Fault Management Architecture
Pawel Dawidek (FreeBSD)GEOM - The FreeBSD way of handling storage
John Sonnenschein (Sun Microsystems)Driver and Filesystem Development with the Solaris and OpenSolaris DDI/DKI
David Gwynne (University of Queensland)MCLGETI: Effective Network Livelock Mitigation and More
Cristina Cifuentes (Sun Microsystems)Finding Bugs in Open Source Kernels Using Parfait
Sherry Moore (Sun Microsystems)Fast reboot support (and more) for OpenSolaris
Max Bruning (Bruning Systems)Porting USB HID Device Drivers Between Linux and OpenSolaris
James Morris (Red Hat)Linux Kernel Security Overview
Percy Pari-Salas (Bond University)Automated Testing of OpenSolaris
Vivek Joshi (Sun Microsystems)Porting OpenSolaris across architectures
Jayakara Kini (Sun Microsystems)Crossbow for OpenSolaris Developers
Garrett D'Amore (Sun Microsystems)Boomer: the new OpenSolaris audio system
Pramod Batni (Sun Microsystems)Debugging and Diagnosing Interesting Kernel Problems
Stewart Smith (Sun Microsystems)(Ab)use the Kernel: what a database server can do to your kernel

So what are you waiting for? Hurry up and

Friday May 15, 2009

KCA2009 - first media contact

KCA received its first media hit with a query from the editor of www.technologyandbusiness.com.au. An interview (via email) with me! New and exciting :-)

I was nervous about how the answers I gave to her questions might get transformed into an article - I've never dealt with a media enquiry before - so I was relieved to see that the article that popped up just before lunchtime today was true to the answers I'd provided: Sun shines more light on Open Source Kernels.

On the registration front, people are starting to register for the conference (cue sigh of relief) and there's still plenty of time (well, another 2 weeks) to get your seat at the early bird price. So don't be shy, decide that you really do want to meet, learn from and hang out some of the finest minds in Open Source software. You Know You Want To!

Tuesday May 12, 2009

KCA2009 - registrations are NOW LIVE (also, agenda posted)

About 15 minutes ago I was ecstaticly pleased to see that the KCA2009 website had gone live with the confirmed agenda, speaker bios and most importantly, registrations.

Wednesday May 06, 2009

KCA2009 - we've finalised the accepted presentations

After a lot of back-n-forth in the committee, we've finally come up with a list of presentations that we have accepted for the 16 speaking slots at Kernel Conference Australia 2009. Hopefully all the presenters will get back to me asap so I can finalise the schedule and say who they are.

Saturday May 02, 2009

KCA2009 - Call for Papers is now CLOSED

I'm pleased to announce that the official Call for Papers for Kernel Conference Australia 2009 is now closed. Over. Finito!

We've had a great response to the CfP, and the review committee will be meeting in coming days to nut out just which presentations we'll take.

Personally, I'm very happy that given the event is new, and economies around the world are in turmoil, we've had so many presentation proposals (and of a very high quality) submitted.

The review committee will be working hard to ensure that we get the accepted papers finalised as soon as possible, but either way we've still got enough of the schedule mapped out already (keynote speakers and panels) that when registrations open on Monday you'll have a fair idea of how it'll all go.

If you are planning to register and attend, please join our Facebook Event.

Monday Apr 27, 2009

Less than ONE WEEK to go for KCA2009's Call for Papers

I woke up with a shock this morning, realising that there's less than ONE WEEK to go before the KCA2009 Call for Papers closes!


If you're going to submit a presentation proposal, please get it in as soon as possible - you don't want to miss out on being part of this conference.

Here are the links, just link case you'd misplaced them:

Official Conference Website

Call for Papers (all pretty)

Call for Papers (original, dry and technical)

The Facebook event

If you're a potential sponsor, please contact myself, Claire or Gabriele (details on the Official Conference Website).

I've also put together a flyer which you could print out and post around your workplace. Please do - the more the merrier!

Don't forget, registrations open in one week, on 4 May 2009. Save The Date

Friday Feb 27, 2009

Kernel Conference Australia - it's coming!

Over the years it's been a source of frustration for me that the conferences which I wanted to attend were either too expensive (time, travel, registration etc) or not covering topics I was interested in.

Late last year I realised I should stop grumbling about it and fix it myself.

So it is with great pleasure that I can announce that this July 15th to 17th in Brisbane, there will be an Open Source kernel-focused conference: Kernel Conference Australia. We don't have absolutely all of the organisational bits together yet, but here's what we do have:

World class speakers

Fantastic location

Our venue is the Queensland Brain Institute, within the University of Queensland.

Excellent climate

The University is situated in Brisbane, Queensland, Australia. Yes, it will be winter... but winter in Brisbane is a beautiful time to visit.

Call for Papers

The Call for Papers is not quite ready (still a few details to be ironed out), but if you'd like to be considered for a presentation spot you should be thinking about a topic in any of these areas:

  • Cross-architecture kernel development

  • Porting an OS to a new architecture

  • Filesystems

  • System performance visualisation (DTrace, SystemTap?)

  • Image visualisation (GPU kernels)

  • Fault Management

  • (globally) Distributed kernel development - how to make it work

  • Virtualisation

  • Clustering (HPC and High Availability)

  • Distributed systems

  • Kernel Testing - methodologies, interesting problems found

  • Traps and pitfalls found when porting drivers between OSes

  • Realtime performance and scheduling

  • Embedded OSes and drivers (including control systems)

  • Patents and Open Source

  • The state of OS kernel research / what's new / work in progress

Apart from the above, we expect that pretty much any kernel-focused topic for an Open Source licensed OS will be considered by the organising committee.

Target OSes

If you would like to help with sponsoring the conference, please let me know via email (jmcp at my employer . com).


I work at Oracle in the Solaris group. The opinions expressed here are entirely my own, and neither Oracle nor any other party necessarily agrees with them.


« May 2016