Friday Mar 28, 2008

OpenSolaris Elections Disappointment

The results of the OpenSolaris Elections are in.  Congratulations to the new board members!

So why did I entitle this post Elections Disappointment?  My complaint is with the "community priorities" question which was asked; this was a chance for core contributors to indicate what they felt were pressing priorities.  I was not happy with the OGB's choice (which I didn't know about until I voted) to reuse the same set of priorities questions from last year.  One of which was:

  • Deploy a public code review facility on opensolaris.org

Which was subsequently identified by voters in this election as the #3 priority.  Did the OGB believe that we did not accomplish this goal in the past year?  I know that I took the results of the last poll (it was voted #3 in that poll, too) to heart, and worked hard to make that goal a reality, publicized it using the official channels, and have been enhancing it and taking care of it since that time, mostly on my own time.  I felt that my contribution was really undermined by the question.

Second: Who voted for this item as a priority?  If you voted for this as a priority, I would like to hear why you did (anonymously is fine).  Are you unaware that cr.opensolaris.org exists?  Are you unsatisfied with the service it provides?

I hope the new OGB will find a way to reformulate a more cogent poll question about priorities.

Monday Mar 24, 2008

cr.opensolaris.org gets an ATOM feed

For the past couple of weeks, I have been working late at night and on weekends to add an ATOM feed (i.e. blog feed) to cr.opensolaris.org, so that as people post new code reviews, they are automatically discovered and published.  Stephen has been heckling me to do this work for more than a year.  This weekend I managed to finish it, despite the incredibly nice weather in the bay area: I was stuck inside with a nasty cold.

As an aside, I'm looking for help with cr.opensolaris.org.  This is a great opportunity for someone to step up and help out with an important part of our community infrastructure.  Send me mail.

You can check out the results of my hacking on cr.opensolaris.org.  Or you can subscribe to the feed.  If you want to opt-out of publishing reviews, you can create a file called "opt-out" in the same directory as your webrev's index.html file.  Or you can create a file called "opt-out" in your home directory, if you'd like to opt out of all reviews.

Implementation Notes 

This was an interesting learning experience for me, since I had to learn a lot about ATOM in the process.  I also learned the XSLT language along the way as well, and how to process HTML using python.  All in all, I'd say this project took about 20 hours of effort, and resulted in about 500 lines of python code.  The most difficult problems to solve were:

  • I wanted the feed to include some meaningful information about the codereview.  If you subscribe to the feed using your favorite reader, you'll see that a portion of the "index.html" file from each webrev is included.  This is done using a somewhat tricky piece of python code.  In retrospect, using XSL for this might have been a better choice, although I've found that people have a tendency to introduce non-standard HTML artifacts into their webrev index.html files, and I don't know how well XSL would cope with that.

  • ATOM has some rules about generating unique and lasting IDs for things-- this is the contents of the <id> tag in the ATOM specification.  I found a lot of valuable information on dive-into-mark.  For cr.opensolaris.org, this was complicated by the fact that the user might log in and move their codereview around, or might copy one review over another.  In the end, I solved this by remembering the <id> tag in a dot-file which rides along with the codereview.  A cronjob roves around the filesystem looking for new reviews, and adds the special tag-file.  By storing the original <id> tag value, and looking at the modtime of the index.html file, I can correctly compute both the value of the <id> and <updated> fields for each entry.  If a user deletes a codereview, the dot-file will go away with it.

  • Once I had an ATOM feed I needed to transform it back into HTML for display on the home page.  The only problem was that there aren't a lot of good examples of this on the web-- many of the ATOM-to-HTML conversions only work with ATOM 0.3, not the 1.0 specification, and I didn't know the first thing about XPATH or XSL.  In the end, I only needed 25 lines or so of XSLT code.

Future Work 

I think of the current implementation as a "1.0"-- it'll probably last us pretty well for a while.  One thing I'd like to research for a future revision is actually placing the entries into a lightweight blog engine, and letting it do the rest of the work: Using an excellent list from Social Desire I took a quick look at Blosxom, Flatpress, Nanoblogger, and some others.

Tuesday Jan 29, 2008

The joy of 'zpool scrub'

Some days, when it's cold and you're not feeling very motivated (like me, today), it's nice to do a zpool scrub on the machines you manage, and then once it's done:

$ zpool status
  pool: aux
 state: ONLINE
 scrub: scrub completed with 0 errors on Tue Jan 29 15:52:38 2008
config:

        NAME          STATE     READ WRITE CKSUM
        aux           ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c1t0d0s7  ONLINE       0     0     0
            c1t1d0s7  ONLINE       0     0     0

errors: No known data errors
And then relax, knowing that your data is safe.

Tuesday Dec 04, 2007

Podcast on Etude!

Hal Stern invited Joost and I to sit down for a chat about the Solaris 8 Migration Assistant for an episode of his podcast Innovating@Sun.  This was my first podcast experience and it was a lot more fun than I had expected.   Check it out here, or you can download the MP3 audio directly.

Wednesday Nov 14, 2007

A big mess...

Recently I've been thinking about, learning about, and contributing to IPS, the Image Packaging System, a next generation package management solution project which is happening in the OpenSolaris community.  IPS happens to also be the packaging system project Indiana has elected to use.  Stephen has written extensively about his thoughts on his blog.  And Bart has too.  IPS subsumes a lot of existing functionality which appears in various parts of the system today.  But a lot of people seem to be willing to look at it only as a package manager in the sense that "pkgadd" is a package manager.

My problem with this is that "pkgadd" is only a small part of a larger problem.  So, to explain that, I want to distill a series of email posts I made to pkg-discuss last month into a coherent blog entry, since I've referred back to them a couple of times.

My feelings on this topic are pretty nicely summarized by an article by Peter J. Denning which recently appeared in Communications of ACM, entitled "Mastering the Mess."  The whole article is instructive, but see in particular: "Signs of Mess."

If one accepts Dr. Denning's "mess" framework, then the next question is whether we are in what he dubs, "a mess."  I personally think the answer is "yes."   In no particular order (apologies to anything I left out), as a community, we have:

 

  • SVR4 package creation tools
  • SVR4 package deployment tools
  • Sun's patch creation tools
  • Sun's patch application and inventory tools (patchadd, showrev -p)
  • PCA (Patch Check Advanced, a nice open source tool I use)
  • Solaris Patch Manager (smpatch)
  • pfinstall
  • Live Upgrade
  • flash archive creation and deployment tools
  • graphical install (old and dwarf caiman, etc)
  • ttinstall
  • Jumpstart
  • virt-install (from xVM)
  • zones install
  • zones attach/detach logic (which knows how to parse various patch and packaging databases)
  • So-called "toxic" patching for zones
  • Zones support for live upgrade (zulu)
  • BFU/ACR (update part of the system, but violates package metadata)
  • IDR (patches the system, but renders system subsequently unpatchable until IDR is removed and a "real patch" is applied)
  • Solaris Product Registry (I've never really understood what this was for, but you can try it via prodreg(1))
  • Service Tags -- a layer which adds "software RFID tags" in a sense: UUIDs backed by a Sun-maintained ontology; helps to inventory what is on your system.
  • pkg-get
  • Network Repositories (like Blastwave)
  • DVD media & CD media construction tools (several of these, I think)
  • Various other unbundled products which promise to ease "patching pain"
  • Various system minimization tools
  • Layered inventory management tools
  • Numerous hand-rolled and home-grown solutions built on some or all of the above.                                            

Some parts of the mess represent great (from the perspective of the those caught up in the mess) technologies which people have spent a lot of time and effort building.  But a lot of the above represent accreted layers with duplicated functionality.  In some cases, the various layers interact in complex and subtle, and perhaps interface-violating ways. To people outside of the mess (i.e. new users we would like to entice) the mess looks bizarre, and terrible.  Another sign of a "big mess":  In several cases, huge engineering efforts have resulting in only modest improvements.  In some cases, huge engineering efforts have been total failures: Sun attempted a rewrite of the SVR4 packaging system in the early part of this decade, the project basically failed.

It's easy to look at the above list and feel a sense of hopelessness-- how will we \*ever\* improve upon this situation?  Will people keep creating new and different tools which add more layers? 

I'll cite a second source which has helped guide my thinking on this topic: Jeff Bonwick.  Jeff spent years relentlessly seeking out and blowing up duplicated and broken kernel functionality, and then took on the storage stack.  The result was ZFS, which was recently labelled "a rampant layering violation" by a detractor.  Jeff responded this way.  In particular, Jeff said:

"We found that by refactoring the problem a bit -- that is, changing           
where the boundaries are between layers -- we could make the whole thing
much simpler."

Which to me summarizes my thinking about what the \*opportunity\* is here: to rethink the layers and to merge and unmerge them to come to a more complete, efficient, modern
solution.

IPS is heading in this direction: Packaging, patching, upgrade, live upgrade, the mechanisms for Software Delivery, the toolset for delivering packages/patches, and the software-enforced policy decisions seem to be condensing here into a coherent entity-- which means we'll have many fewer layers.  And because the system will be fast, lightweight, redistributable and shared, we should also be able to discard artifacts such as BFU and ACR (in other words, OpenSolaris developers will use the same tools our customers use to update systems).  The huge amount of code which handles zones patch and packaging should be greatly reduced.  Package dependencies will be far more accurate and minimization will be easier, and diskless client support should be far more robust.

What I see with Caiman, IPS and Distro Constructor is the opportunity to do for software delivery and update to OpenSolaris systems what ZFS did for storage management.  I do not think we have all the answers just yet, but I think we can get there.

Monday Oct 22, 2007

Solaris 8 Migration Assistant 1.0 (Project Etude) Ships!

I'm very happy to announce that the Solaris 8 Migration Assistant 1.0 (also known as Project Etude) has shipped!  The product is now officially available from Sun.  Some key links:

In a nutshell, the product provides a migration solution from Solaris 8 to Solaris 10 by creating a bridge between the two operating systems.  You can perform P2V (physical-to-virtual) conversions of existing Solaris 8 systems, and drop those into Solaris 8 containers running on your Solaris 10 host.

Above all, I want to take another chance to thank the many people who worked extremely hard for the past eight months to make this project a reality.  It was sprint from start to finish, which is certainly tough on everyone involved.  But I was amazed and pleased that almost universally, people helped us out with dedication and a good sense of humor.  Thank you very much.

Saturday Oct 13, 2007

cr.opensolaris.org now live, exits beta

On Thursday I posted this announcement to opensolaris-announce, which notes that I have taken cr.opensolaris.org out of beta status. Please take a look, use, and enjoy...

I am pleased to announce that cr.opensolaris.org, the OpenSolaris
Code Review site, is now officially part of the opensolaris.org
infrastructure. I encourage everyone in the community to use it.

In the March 2007 community priorities poll, the community declared:

...
#3 Deploy a public code review facility on opensolaris.org
...

cr.opensolaris.org was designed to meet that goal. It has been in Beta
test for some time, and after some final tweaking[1] it is now ready for
production.

You will need an SSH key linked to your account in order to use it.
For more information on SSH keys, see
http://opensolaris.org/os/project/website/ssh_instructions/.

Once you have a registered key, see http://cr.opensolaris.org for
information about how to gain access to this system. We request
that users review the terms of service before using it. Problems can be
reported to website-discuss at opensolaris dot org.

Finally, if you would like to help out with the coding/maintenance of
this site, please visit the site and read the "Call for Help" section.

Thanks

Dan Price

[1]: Specifically, concerns about who may access the system have been
resolved.

Tuesday Oct 09, 2007

Solaris 8 Migration Assistant (Etude) Launch, live from CEC

I was lucky enough to get to attend CEC (Sun's Customer Engineering Conference) in Las Vegas, although not lucky enough to win anything at the casino.  So now, I'm sitting in the CEC System Launch session, and John Fowler is launching the new T5120 and T5220 systems.  Thankfully there is ample wifi here in the hall.

The best news for me is that John just announced the product I've been working on, the Solaris 8 Migration Assistant (or Project Etude) and showed how it can help with migration and consolidation activities.  A FAQ has been posted over here.  That's not to say that you can download and use it now.  But very soon!  I think that these new systems in combination with the Solaris 8 Migration Assistant will make a very tempting combination for our customers who are still running Solaris 8 on older hardware.  You can upgrade and reduce your service costs, rack space, cooling, power costs, and get greater performance and bandwidth.


Tuesday Oct 02, 2007

Success with USB Audio

I have a home server which I built myself.  I also occasionally use it as a desktop system, especially when I have a long computing session planned, when my laptop would tend to get uncomfortable.  Unfortunately, I've been unable to get Solaris to drive the audio.  I think this is bug 6518184 but despite some hacking with /etc/driver_aliases I couldn't get the onboard audio device to attach.  While I need to get my home machine updated to the last SXDE release in order to really tackle this properly, I found myself really wanting working audio, now.  I happened to stop in at my friendly neighborhood computer shop, and noticed a USB audio gadget for $25 (I actually snagged an open box model for $20), the Vantec NBA-100U. It works like a charm in Solaris thanks to its excellent USB audio support.  Here is what it looks like from 'cfgadm -av':

usb0/2                         connected    configured   ok         Mfg: <undef>  Product: USB Audio  NConfigs: 1  Config: 0  <no cfg str descr>

It's pretty bare bones-- you can see that Vantec didn't even bother to set the Manufacturer string!  Kwality.  Once I plugged it into my USB port and into some cheap speakers, I simply had to start up rhythmbox and everything worked like a charm. The audio quality seems fine to me.  While the device supports 5.1 audio, it does so only under Windows.  Stereo is fine for me.  I should also note that the volume up and down buttons are non-functional both on Solaris and on Macs.  This is no big deal for me as the thing is buried under my desk, and my speakers and Solaris both have volume control.

I realize that cheaper USB audio gadgets can be had, but this one should be easy to find online and seems to work reliably.  It's really nice when it all "just works."

Wednesday Sep 26, 2007

Etude Progress Update

The whole Etude engineering team (except, unfortunately, for Penny) gathered in the Bay Area last week.  We made a ton of progress and we're on our final round of bug fixes.  It has been fun (if a bit nerve wracking) to watch the bug counts drop day by day.

One problem which has been challenging with this project is that we have a number of differently delivered software parts, all of which must undergo some change:

  • The solaris8 brand itself is a piece of software which bridges the gap between Solaris 8 and Solaris 10.  This is something you must add to an Solaris 10 8/07 (also called S10U4) system.  When it is finished, it will be delivered as two packages, SUNWs8brandu and SUNWs8brandr, plus an optional "demo" package.
  • We've got a very few Solaris 8 patches which we require (currently I think there are 6 of them).  Some of these (for example, a fix to the linker, and a fix to ptree(1)) we had to engineer ourselves. And some have been available for a long time and are probably already patched onto most S8 systems out there.  All of these are (or will be) available via the normal patch mechanisms.
  • We needed to add some enhancements to Solaris 10 in order for BrandZ to work solidly on SPARC systems.  While these changes will be automatically included in the next update of Solaris 10, for Solaris 10 8/07 you need to add a patch-- the kernel jumbo patch, in fact (as I mentioned before, this patch is not out just yet).

For our Beta release, we were able to supply workarounds for some of these issues, but for our official release, we need every 't' crossed, and every 'i' dotted. So last week we finally had all of those pieces available (internally) in at least a preliminary form.  Everyone has been busy testing the whole works.  For example, I've done a dry run on a T1000, which looks like this:

  • Install S10 8/07 onto the system (or into an LDOM (logical domain) on the system)
  • Bring system to single user mode
  • Add kernel patch using patchadd
  • Reboot system (or LDOM)
  • Add SUNWs8brandr and SUNWs8brandu packages to the system
  • Configure a Solaris 8 container and install it from an existing system archive
    • This will auto-apply any required Solaris 8 patches to the system
  • Boot S8 zone, and enjoy!

It's nice to see the pieces coming together...

Wednesday Sep 05, 2007

Project Etude, Revealed

Well, it has been a while since I last wrote anything here.  I have been working on a team developing a new project, which we named Etude.  Marc Hamilton, VP of Solaris Marketing, has written about it here.

[As an aside, it's weird to have a code name you selected cross the lips of senior executives several months later] 

In a nutshell, we've built a Solaris Container (or Zone) which is capable of running the Solaris 8 user environment.  We have also created a capability to perform P2V (or Physical-to-Virtual) transformation of existing Solaris 8 systems into containers running on a Solaris 10 host.  This is an enabler for rapid migration of legacy Solaris 8 environments onto modern, environmentally friendly, cost effective hardware.  And onto Solaris 10.  The idea is to break up the upgrade tasks into chunks, allowing the hardware and OS to be upgraded, while continuing to run legacy environments.  Next, the legacy environments can be used until they are retired, or redeployed into Solaris 10 containers, or into logical domains.

I will write more about the why and the how of this project in a future blog entry.  (And Marc does a good job of explaining some of the why in his blog, so go there for more background).  But for now, I want to expand upon the what.  The notion of a Solaris 8 container was not a new one when we began to look at this problem early this year.  But with the completion of the BrandZ project, we had the tools in hand to make a serious attempt to realize this idea.  BrandZ was originally developed so that we could have a "Linux Zone" but the core "brand" framework is really very flexible (and is now more so, thanks to our work).  It allows the development of a variety of OS personalities atop Solaris 10.

So who built Etude? I'm very lucky to be leading a group of seasoned and incredibly talented engineers: Bill, Ed, Jerry, and Steve (no blog), and Penny on documentation.  We worked at something of a breakneck pace to assemble a prototype, which we had in hand by late April. Since then, we have worked to convert the prototype into a production quality offering.  I'll try to say more about that later.  Along the way, we've had a lot of help from many corners of the company-- people too numerous to list.  I especially want to thank Bill Franklin, Jerri-Ann, Joost, Susan, Allan, Richard, Tim and Liane.

Ok, so I'm anxious to show off our creation. First, let's archive a Solaris 8 system using the Flash Archiving tools. If your S8 system is patched up to date, you already have these tools installed. Alternatively, you could use CPIO or ufsdump or some other tool to create a suitable archive:

s8-system # uname -a
SunOS s8-system 5.8 Generic_108528-29 sun4u sparc SUNW,UltraAX-i2
s8-system # flarcreate -S -n s8-system /net/s10system/export/s8-system.flar
Determining which filesystems will be included in the archive...
Creating the archive...
Archive creation complete.
s8-system #

[Note to our beta customers!  In the beta version of Etude, support for Flash Archives is not present, so if you use this blog entry as a substitute for reading the documentation, you will be disappointed.  Please ensure that you carefully read the supplied instructions] 

In the above example, I've simply used NFS to place the flash archive onto my Solaris 10 system, but you could use any method for moving files around. Now, we'll head over to the Solaris 10 system and create a container suitable for use as a Solaris 8 environment...

s10-system # zonecfg -z s8-system
s8-system: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:s8-system> create -t SUNWsolaris8 zonecfg:s8-system> set zonepath=/aux/zones/s8-system zonecfg:s8-system> add net zonecfg:s8-system:net> set address=10.6.62.226/24 zonecfg:s8-system:net> set physical=e1000g0 zonecfg:s8-system:net> end zonecfg:s8-system> add dedicated-cpu zonecfg:s8-system:dedicated-cpu> set ncpus=2 zonecfg:s8-system:dedicated-cpu> end zonecfg:s8-system> commit zonecfg:s8-system> exit s10-system # s10-system # zoneadm list -vc ID NAME STATUS PATH BRAND IP 0 global running / native shared - s8-system configured /aux/zones/s8-system solaris8 shared

The last part (setting ncpus=2) is not strictly necessary, but is a good example of how easy it is to allocate CPU resources to containers starting in Solaris 10 8/2007. Jerry (who led that work) and Jeff Victor have written more about this capability here and here.  (By the way, Solaris 10, 8/07 was launched yesterday, and includes BrandZ and many other enhancements to the containers infrastructure.)

Now that we've configured the zone, we need to install it.  We'll use the flash archive we made to do so:

s10-system # zoneadm -z s8-system install -a /aux/flar/s8-system.flar
      Log File: /var/tmp/s8-system.install.104490.log
        Source: /aux/flar/s8-system.flar
    Installing: This may take several minutes...
Postprocessing: This may take several minutes...

        Result: Installation completed successfully.
      Log File: /aux/zones/s8-system/root/var/log/s8-system.install.104490.log
s10-system # 

That's it! We've now installed a Solaris 8 container! Let's boot it up, and log onto the console, and see what happens:

s10-system # zoneadm -z s8-system boot; zlogin -C s8-system
[Connected to zone 's8-system' console]

SunOS Release 5.8 Version Generic_Virtual 64-bit
Copyright 1983-2000 Sun Microsystems, Inc. All rights reserved

Hostname: s8-system
The system is coming up. Please wait.
NIS domainname is mpklab.sfbay.sun.com
starting rpc services: rpcbind keyserv ypbind done.
syslog service starting.
Print services started.
The system is ready.

s8-system console login: root
Password:
Sep 5 17:55:52 s8-system login: ROOT LOGIN /dev/console
Last login: Wed Sep 5 13:11:37 on console
Sun Microsystems Inc. SunOS 5.8 Generic Patch February 2004
s8-system # uname -a SunOS s8-system 5.8 Generic_Virtual sun4v sparc SUNW,Sun-Fire-T200 s8-system # psrinfo 0 on-line since 08/29/07 13:32:21 1 on-line since 08/29/07 13:32:23

We could have also performed a sys-unconfig(1m) on the Solaris 8 image during the container installation (by passing -u to the installer). In that case, we would have been asked to answer the usual system identity questions. This zone can be cloned, moved around, renamed, attached/detached and manipulated like any other. You can even install it atop a ZFS filesystem, and from the global zone, use DTrace against the applications running inside of it.

Most importantly, you can run real workloads. Building on the Solaris Application Binary Compatibility Guarantee, we have done the difficult work to make sure that your applications will work successfully inside of these containers. This includes software such as databases, application servers, Java programs, web servers, and more. We've also utilized the amazing array of test suites we have available in-house.  You can even do software development inside of Solaris 8 Containers, building binaries which will run on any Solaris 8 (and 9, and 10) systems.

You can also patch these containers using the same tools you use to patch Solaris 8 (and using the same patches). We've even pulled down hundreds of Blastwave packages to test gnome, KDE, and lots of other applications available there. You can run your favorite ancient desktop environment: Solaris 8 was the last version of Solaris to include OpenWindows. So here's the obligatory screenshot from the wayback machine: Netscape 4.x, Java 1.3 and some other miscellaneous stuff all running atop the OpenWindows environment.  Running on emulated Solaris 8, on Solaris 10, on a T2000. A weird blast from the past mixed up with the present day.  (It's interesting to think that today's whizzy desktop applications will in a few years look just as antiquated as OpenWindows and Netscape 4.x do today).  Click the image to zoom in.

 


That's all I have time for tonight.

If you're interested in hearing more about this technology, talk to your local Sun Representative. Or if that fails for any reason, send me an email (my-firstname dot my-lastname @sun.com will do the trick) and I'll try to hook you up with someone helpful (please put "Etude" in the subject line).

Thursday May 17, 2007

cr.opensolaris.org goes to Beta

I am happy to announce that cr.opensolaris.org, the OpenSolaris code review site, is now in Beta. For now, the site automatically grants an account to anyone with a contributor grant or better. For more on the "grants" system, see article III, Section 3.3 of the OpenSolaris Constitution.

For now, new accounts get created periodically by me running a script -- eventually this will update hourly. So if you don't seem to have an account now, let me know when you've got your grant.  Update 5/27/07: I set this up to auto-update every 20 minutes.

So take a look, give it a spin, and see if it works for you-- and please let me know, good or bad. Keep in mind that this is a Beta, so things might be in flux over the next few weeks. Also: if you think you should have an account (because you have a grant) but don't, please let me know.

Happy code reviewing!

Monday May 07, 2007

cr.opensolaris.org, a work in progress

A while back I wrote about online codereview.  And Stephen has been hassling me to move my existing code review site, cr.grommit.com (hosting graciously provided by Steve Lau) over to the OpenSolaris.org infrastructure.  So I have been slowly chipping away at all this, and now have some progress to report.

Gary was kind enough to provision me a zone in the OpenSolaris infrastructure.  As such, I've started to bring up the new cr.opensolaris.org codereview site.  Thus far, I have been working on getting mod_layout working so that we can decorate user supplied content with a little header and footer.  This was a bit of a pain because the Apache bundled with Solaris is compiled with the Sun compilers, which are not installed on the datacenter systems.  Only gcc is available (as it is in general on Solaris 10, under /usr/sfw/bin/gcc).  So, I did the build elsewhere-- but I wonder what customers do when faced with this problem?  I wonder if we could use our "compiler wrapper" technology (which we use when building the OS under both compilers) to help us to bridge this gap?

Once I got that squared away, I brought up Apache with some stubbed out content.  Here is the beginning of the home page, and here is a sample code review with the decorations in place (that bar at the top is being dynamically inserted at page transmit time by mod_layout).  What do you think?  I'm pretty pleased about the way it looks, which was inspired by a similar "link bar" feature in gmail.

There is a bunch of stuff left to do: I'm going to change the account management around so that cr.opensolaris.org participates in the opensolaris.org SSH key management scheme.  My working idea is that anyone with a contributor grant will be able to upload files.  The rssh stuff needs to be moved over from grommit.  Existing user data needs to move over.  The main page needs an overhaul... and probably a bunch of other stuff I haven't thought of yet needs to be done.  Hopefully in another couple of weeks it will be fully up and running.

Thursday Apr 12, 2007

Some work on libMicro; Mercurial transition notes

I decided to help out Bart and Phil by opening the libMicro project. The source for libMicro has been open for a while (hosted in the performance community), and internally, Bart kept it in a Teamware workspace.  But now we've got it officially ensconced in a project of its own, and the source has been moved into a Mercurial repository. As a result, you can now browse the source online!  I've also been attempting to make some improvements to it which you can also examine.  Here are some notes which might be useful to other folks making similar Teamware-to-Mercurial transitions:

  • There are some good directions here for setting up a repository.
  • You'll want to set up a personal ~/.hgrc file because you probably want to associate your email address with your putbacks.  (So that you can be e.g. Joe.Smith@sun.com and not joe@mymachine.eng.sun.com. See hgrc(5). Note that if you've set your repository to send notifications to a list (like xyz-discuss@opensolaris.org) this is especially important. I set the from= and username= directives.
  • You should set up an hgignore(5) file for your repository.  The idea is to tell Mercurial which files it should definitely not care about.  Things like .o files and generated binaries.  .hgignore is just another file which you can check into your repository and version control.
  • You may wish to go through your source files and remove and SCCS detritus (like #pragma ident directives) since Mercurial doesn't use these.

Kudos go to the OpenSolaris infrastructure team: The Mercurial infrastructure just works (perhaps with the exception of the email caveat I mention above).  That said, there are a couple of minor improvements I'd like to see:

  • A link from the project page listing the repositories, including: committers, repository URIs, links to the source browser, etc.
  • In the administration GUI, the repository URIs  are not displayed, and should be.
  • A way to display recent commits on the project page.
All in all, I've had a very positive experience with this new set of tools.  I think the trajectory here looks very, very promising.

Friday Mar 23, 2007

Thinking about online code reviews...

Recently the OpenSolaris community conducted a poll about priorities.  I took note that Deploy a public code review facility on opensolaris.org came in at #3 on the list.  Of course some of you already know that I already run a public code review facility for the opensolaris community: cr.grommit.com.  It turns out that I've learned some valuable lessons from that experience:

  • It's possible to roll out such a facility without a lot of fancy web application infrastructure (cr.grommit.com is "powered by" shell scripts).  I built a useful facility with fairly stone age tools.
  • Running the infrastructure in a zone (graciously donated by Steve Lau) means that I can have some autonomy.  Steve provides me a special ZFS filesystem to which I can dump data for him to backup.
  • Providing rsync access makes posting codereviews an automatable process.
  • Providing public-key-only ssh-only access means that I don't worry much about security.
  • mod_pgheader is really useful for this sort of application.  I use it to add google analytics to all pages on the site (including user content pages).  We might want to use to add some OpenSolaris.org site navigation as well.

Having talked to Stephen and Steve since the poll closed, I think we'll be rolloing cr.grommit.com over to something like cr.opensolaris.org in the next few month.  Being part of the opensolaris.org infrastructure is exciting because we can remove our ssh key management scheme, and use the SSH keys which users provide.  The net effect is that the same key you use for voting will also be useful for posting codereviews.

Of course at the same time I'm very busy on a new project (more on that some other time), enhancing OpenGrok, et cetera.  So it will take some time to get community priority #3 squared away.  I've tentatively set the following goals for the re-work and migration:

  • RSS (or ATOM or whatever) feed of posted codereviews; also a mailing list with the same.
    • To do this we may want some more enhancements to webrev to generate a file of metadata suitable for use by a feed generator (perhaps we need a UUID for each codereview for disambiguation?  What about codereviews which get reposted or updated?   This will be an interesting challenge.)
    • May also want to enhance webrev to be able to rsync things directly, like: webrev -o rsync://cr.opensolaris.org/...
  • Integration with the opensolaris.org SSH key infrastructure
  • Resolution of the spam problem-- today I approve each account request.  If you start posting inappropriate materials, I'll just delete your account.  If we move to a more automated system, we'll need to cope with this somehow.  Perhaps you must have at least a "contributor grant" to post?
  • Reskinned website which fits in with the OpenSolaris site look and feel.

 If you want to help out, most of the above could be broken off into separate tasks.  Since the community has agreed this is a priority, you could volunteer to help out and know that your contribution would be deeply valued.

In the future, I'd like to see a truly collaborative online code review system integrated right in with our RTI system-- something like Mondrian perhaps.  Stephen and I have talked about such a system several times over the past couple of years.  Mostly the issue is that it is a really a big commitment to build such a tool.  If we're lucky, we might be able to integrate some existing pieces together (some bookmarks in this area:  http://www.cenqua.com/crucible has a great demo; http://jcodereview.sourceforge.net/, http://www.notesfromatooluser.com/2006/12/online_code_rev.html thoughts on why these tools "suck", http://cvw.sourceforge.net/cvw/info/docs40/jcvwscreen.php3, http://www.laatuk.com/tools/review_tools.html).

About

Kernel Gardening.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today