Monday May 25, 2009

SourceJuicer: How to contribute a package to OpenSolaris

[UPDATE: A few small errors fixed and some clarifications added. See Comments for details.]

I tried recently to add a package to the OpenSolaris contrib repository, but quickly learned I didn't have enough packaging experience to understand the directions provided at SourceJuicer so I did some homework, asked some questions, and eventually did successfully contribute a package. I've documented in this entry everything I've learned hoping it will be helpful to others who want to build and submit OpenSolaris packages. Specifically, I'll describe how I wrote the spec file for Ploticus (my favorite open source plotting/graphing utility) and how I submitted the package to OpenSolaris.

I used SourceJuicer to submit my package because it is the easiest way for a community member to contribute. Before getting into details, a few words about the overall submission process. Packages are first submitted to the pending repository, which is basically a holding area for packages on their way to the contrib repository, the primary repository for community-contributed packages. Once a package has been validated and successfully built, it can then be moved into /contrib. I'll cover all of this below.

On to the details.

To submit a package to SourceJuicer, you need to supply two files: a text file containing copyright information and a spec file. The spec file contains the information SourceJuicer needs to create a final binary package starting from source code. Ideally the OpenSolaris package will be buildable from the standard, community-released source code without changes, which may require asking the community to adopt changes necessary to build the code for OpenSolaris. In practice, this will often not be necessary since many packages are designed to build on several Unix versions. In cases where changes must be made and those changes have not been accepted by the community, it is possible to specify patches that should be applied to the community source code during the build process. Though not desirable, it is sometimes necessary to do this. I'll supply pointers to information on how to do this below.

Spec files are not an OpenSolaris invention--they have been used for a long time to build RPM packages. This is good news because there are several excellent web resources that document spec files in detail. I recommend Maximum RPM by Edward Bailey as a detailed reference. One complication: It seems that OpenSolaris spec files are not exactly the same as RPM spec files. However, for the purposes of this exercise, don't worry about this -- the Ploticus example below should give you enough information to create a valid OpenSolaris spec file in most cases. However, if you insist on worrying, you can read the information I found here and here. If anyone knows of a better explanation of the differences, let me know and I will include a pointer here.

Okay, lets get to it. I started with a spec file template and created the following file for Ploticus. My commentary includes all of the tips and other information I discovered during the process of writing the spec file for this particular open source package. While I've attempted to give pointers to additional information throughout, this is not meant to be the definitive guide to the full capabilities of spec files. There should, however, be enough information here to allow typical open source apps to be packaged and contributed to OpenSolaris. Consult Maximum RPM for additional details.

spec filecommentary
#
# spec file for package: ploticus
#
# This file and all modifications and additions to the pristine
# package are under the same license as the package itself.
#
# include module(s): ploticus
#
This is all boilerplate commentary. Insert the name of your package twice.

%include Solaris.inc

Required for all OpenSolaris packages. For the curious, the source is here.
Name: ploticus

Once you specify the name of your package, you can use the macro %{name} to refer to it later in the spec file. As you will see below, there are other predefined macros available that you will use to write your spec file. You can also define your own macros using the syntax:

%define macro_name macro definition

Summary: ploticus -- creates plots, charts, and graphics from data

Summary is a one-line description of the package that will be displayed by the OpenSolaris Package Manager.
Version: 2.41
The version number can be referenced as %{version} later in the spec file, which can often be used to generalize file and directory names. In the case of Ploticus the version number string (e.g. "2.41") happens not to be used as part of its filenames (e.g. ploticus241src) so I do not use %{version} in this example, except in one instance of boilerplate.
License: GPLv2
Free text field describing code's open source license. I've seen all of these used: GPL, GPLv2, GPLv3, BSD, LGPLv2.1, New BSD License. If GPL, be explicit if you can: GPLv2 or GPLv3. The "or later" licenses might be appropriate as well, e.g. GPLv2-or-later, GPLv3-or-later, etc. There is a nice discussion here about the pros and cons of "or later" licenses.
Source: http://voxel.dl.sourceforge.net/sourceforge/ploticus/pl241src.tar.gz

The source tag specifies the location of the source-code tarball (possibly gzip'ed) that should be downloaded to build the package. Because Ploticus is hosted on sourceforge I had to specify a manual download URL rather than that of the automated download site (downloads.sourceforge.net.)

Note that the source location can also be specified as an ftp:// address.

URL: http://ploticus.sourceforge.net

The open-source community's web address.
Group:  Applications/Graphics and Imaging
The group tag describes the kind of software in the package and will be used by the OpenSolaris Package Manager to categorize the package hierarchically. I chose a group name based on the package classifications listed here.
Distribution:	OpenSolaris
Vendor: OpenSolaris Community

%include default-depend.inc

Boilerplate.
BuildRequires: SUNWxorg-headers, SUNWzlib, SUNWgcc

These are other OpenSolaris packages that must be available on the build system in order to correctly create the binary package. In this case, I am building Ploticus with X-Windows capabilities, so I need to ensure the X client header files are available. I am also enabling a Ploticus compression option so zlib is needed as well. And, to be safe, I've specified which compiler is required. I could have used Sun Studio, but I know for sure that Ploticus compiles with gcc so I've used that.

You can find these package names by searching in the Package Manager on your local OpenSolaris system.

Requires: SUNWzlib
This section lists packages that must be installed on the end-user system for the software to work correctly. In this case, Ploticus will be dynamically-linked against zlib so I need to make sure the Package Manager knows about this dependency. When the users asks for Ploticus from the repository, the Package Manager will know it also needs to download and install the SUNWzlib package as well.
BuildRoot:      %{_tmppath}/%{name}-%{version}-build
SUNW_Basedir:   %{_basedir}

This is boilerplate. The intent of BuildRoot is to define a user- and application-specific path that can be used as the root of an area in which your package will be installed on the build server, allowing the build server to support simultaneous builds of multiple packages by multiple users without interference. Note, however, that I do not use BuildRoot in this spec file because this conversation indicates that $RPM_BUILD_ROOT is the officially supported way to refer to the top of a package install area. I don't know if this is true in the OpenSolaris world as well, but most spec files I've seen for OpenSolaris use $RPM_BUILD_ROOT so I have opted to use that as well.

Note that while $RPM_BUILD_ROOT (and BuildRoot) refers to the root of the installation area on the build server, the top of the build area itself -- the location where your package will actually be untar'ed and built -- is referred to as %{_builddir}.

I do not know how SUNW_Basedir is used.

SUNW_Copyright: %{name}.copyright
This is the name of the copyright file you will upload to SourceJuicer along with this spec file. It must be named as shown (ploticus.copyright in my case.) You will typically find this copyright file on the community's website and/or included within the community's source tarball. In the case of Ploticus, the tarball contains a file in src called Copyright, which I have copied, renamed to ploticus.copyright and then edited to remove html markup. This is the file I will then upload to SourceJuicer. The original src/Copyright file is ignored by SourceJuicer. Update: The preceding was actually not sufficient for my package to be validated. I was asked to append the file GPL.txt, which was also in the tarball's src directory, to ploticus.copyright so that the actual text of the GPL v2 copyright was in the file. The original version of the copyright file (src/Copyright) only refers to the GPL copyleft, it does not include the copyright itself.
Meta(info.upstream): Steve Grubb <ploticus@yahoogroups.com>
Meta(info.maintainer):  Josh Simons <josh.simons@sun.com>

These fields are specific to OpenSolaris's packaging system. The upstream field contains the name and address of the individual or group that creates and supports the open-source software. The maintainer field contains the name and email address of the individual responsible for the OpenSolaris packaging of the open-source project. The preferred format is as shown in these examples.

Additional info fields that can be included are documented here.

%description
A free, GPL, non-interactive software package for producing plots, 
charts, and graphics from data. It was developed in a Unix/C 
environment and runs on various Unix, Linux, and win32 systems. 
ploticus is good for automated or just-in-time graph generation, 
handles date and time data nicely, and has basic statistical capabilities. 
It allows significant user control over colors, styles, options and details. 
Ploticus is a mature package, available since 1999, and version 2.40 has 
more than 12,000 downloads to date.

A more detailed description of the open source software. This description was taken from the Ploticus web page.
%prep
%setup -q -n pl241src

Now we begin specifying what actions are required to build the software. The %setup macro cd's into the build directory, removes any cruft left over from earlier builds, unzips the source tarball (which will have been downloaded at this point), and then untars the sources into the build directory. It then cd's into the package's top-level directory. All of this is done with %{_builddir} as the root directory as described earlier.

Note that %setup assumes the top-level directory specified in the tarball is named %{name}-%{version}. If this is not true for your package, use the -n option to specify the correct name. For Ploticus, all files in the tarball are in the pl241src directory, so I've used the -n option to specify this.

See this page for more details about the %setup macro. The %patch macro, which can also be used in the %prep phase, can be used to apply patches prior to building the binaries if the standard community source code needs to be modified in some way to build successfully on OpenSolaris. See the same page for %patch information. Note that you should try to have your OpenSolaris changes accepted by the community to avoid having to apply these patches.

I don't know what the -q option does.

%build

cd src
make NOX11= XLIBS='-L/usr/openwin/lib -lX11' XOBJ='x11.o interact.o'  \\
     XINCLUDEDIR=-I/usr/openwin/include WALL= ZLIB=-lz ZFLAG=-DWZ \\
     PREFABS_DIR=/usr/lib/ploticus/prefabs pl

The %build section contains the commands needed to build the package binaries. At the end of the %prep phase we were left sitting in the top-level directory of the source tarball. Since the Ploticus makefile and sources are one level down from this (pl241src/src), I cd into src before invoking the correct make command for OpenSolaris.

Assuming the make ran correctly, we exit this phase with the binaries and other files all built on the build server in a sub-directory under %{_builddir}.

%install

mkdir -p $RPM_BUILD_ROOT%{_mandir}/man1
cp man/man1/pl.1 $RPM_BUILD_ROOT%{_mandir}/man1/pl.1
mkdir -p $RPM_BUILD_ROOT%{_bindir}
cp src/pl $RPM_BUILD_ROOT%{_bindir}
mkdir -p $RPM_BUILD_ROOT%{_libdir}/%{name}
cp -r prefabs $RPM_BUILD_ROOT%{_libdir}/%{name}


In the install phase, we execute a "make install" or equivalent, moving all files that will be included in the binary package to their final installed locations, but relative to $RPM_BUILD_ROOT rather than to "/" to avoid collisions on the build server. Because the Ploticus "make install" action doesn't do exactly what I need, I instead manually move each required file to its final location. For many projects, something similar to "make DESTDIR=$RPM_BUILD_ROOT install" would be appropriate in this phase.

If you are moving files manually, do not assume directories exist -- make them before you use them. And use the predefined directory macros (e.g. %{_mandir} ) to reference standard installation locations. Others are documented here.

%clean
rm -rf $RPM_BUILD_ROOT

This is boilerplate clean-up code. Insert other commands as necessary.
%files

%defattr(-,root,bin)
%attr(0755, root, bin) %dir  %{_bindir}
%attr(0755, root, bin) %dir  %{_mandir}
%attr(0755, root, bin) %dir  %{_mandir}/man1
%attr(0755, root, bin) %dir  %{_libdir}
%attr(0755, root, bin) %dir  %{_libdir}/%{name}
%attr(0755, root, bin) %dir  %{_libdir}/%{name}/prefabs
%{_bindir}/\*
%{_libdir}/%{name}/prefabs/\*
%{_mandir}/\*/\*

This can be a complicated section so I suggest reading the Max RPM %files section.

The %files section specifies the locations and attributes of all files that will be placed onto the end-user's system when the binary package is installed. The %attr directive is used to specify permissions and ownership for files and directories. The %dir directive identifies directories. Multiple directives can be applied to objects by including them on the same line.

The first line specifies default mode, default user ID and default group ID for all files created during the build process. The dash ("-") means that a default is not set explicitly for that field. Note that failure to include this line in your spec file will cause an obscure error to be generated when an end-user tries to install your package. That would be very bad.

The next four lines specify the directories in which Ploticus-related files will reside. The last three ensure that the Ploticus binary, all of the Ploticus prefabs config files, and the man page will be included in the binary package. Note again the use of macros to specify standard installation directories.

%changelog
\* Tues Apr 28 2009 - Josh Simons <josh.simons@sun.com>
- initial version

Add any changelog information you desire here.

Once you've created your spec file, it is time to feed it to SourceJuicer for syntax and other checking and then iterate as necessary until your spec file is correct and has passed validation. The basic flow is shown in the diagram below.

The first step is to submit the spec file to SourceJuicer along with the project's copyright file. To do so, go to the SourceJuicer Submit page (login required.) Assign a descriptive name to your upload (I used 'ploticus') and then specify your spec file. Use 'add another file' to add your copyright file. Add whatever other files you may need (see 'more help' on the Submit page.) Click Submit and you will see a page like this:


The summary page includes an indication that my spec file successfully passed a syntax check. If an error occurs at this point, make the necessary corrections and use the ReSubmit tab (not shown) at the bottom of this page to upload new versions of your copyright and spec files.

Looking under Reviews, I can see my package has not yet been validated, which means my submission hasn't yet been checked by someone to ensure my copyright file is appropriate, that someone else has not already packaged this program for OpenSolaris, etc.

The next day I receive two email messages with comments from reviewers. When I log back into SourceJuicer and look at the Review tab, I see the two comments that were submitted. The fact that the package is still marked as not validated means I have issues to address:

Clicking on the "[review]" link takes me to the page with detailed information about the Ploticus review. I can also view this page by visiting the MyJuicer tab and then clicking on the appropriate link under My Submissions. This second method is better since it can be difficult to find your review on the main Review page. In any case, the page looks like this:

As you can see from Amanda and Christian's comments, I did not use the correct naming convention for the copyright file I uploaded to SourceJuicer. Rather than "Copyright", the file should have been named "ploticus.copyright" (more generally, %{name}.copyright). Also, Amanda hopes I can remove the html that is for some reason embedded in the standard Ploticus copyright file.

Using this same review page, I submit a clarifying question back to the reviewers to ensure I address their issues. I am not clear on the relationship between the copyright file that is submitted manually to SourceJuicer and the copyright file in the source tarball that is described with the "SUNW_Copyright" tag in the spec file.

Now that I understand the copyright issue and have adjusted my spec file and copyright file appropriately (and also updated the spec file and annotations in this blog entry--meaning you never saw that I had initially called my copyright file "Copyright"), I use the same Review page to Resubmit the spec file and copyright file. Use the tab at the bottom of the Review page to do this:

As of this writing, there is no way to remove a file that has been submitted to SourceJuicer so all three files (Copyright, ploticus.copyright, and ploticus.spec) are associated with the project even though Copyright is now extraneous. Until removal is possible, just ignore the extra files. [UPDATE: As of SJ 1.2.0, files can removed by visiting the MyJuicer review page for the appropriate package.]

I resubmitted the files, the package was subsequently validated, and then it was automatically scheduled to be built on the build server. I did not receive a notification when the build attempt occurred so you need to check status periodically (use the MyJuicer tab). When I checked, I saw my build had completed successfully on the first attempt:

Had the build not succeeded, I would have followed the Log link to view the build log, found the problem, fixed the spec file, and then Resubmitted. The package would then be rescheduled for another build automatically with no need for re-validation.

With the Ploticus build successfully completed, it is now very important to verify that the package installs correctly and that the software actually works. Though I don't cover it here, my first Ploticus package did not work correctly on my test system. I had to make changes to my spec file, rebuild the package, and reinstall it. Therefore, please do install and test your software!

To do the test installation, I first added the pending repository as a package authority on my 2008.11 system. Note carefully the location of this repository; I had expected it to be http://pkg.opensolaris.org/pending, but that is not correct:

% pfexec pkg set-authority -O  http://jucr.opensolaris.org/pending pending

I then started the Package Manager, selected the Pending repository and did a search for Ploticus. Voila! The package is available:

After selecting the package and clicking on Install/Update, the installation proceeds smoothly. I then start a terminal window and verify that Ploticus does, in fact, work correctly:

Once you are sure your package installs and runs correctly, send an email to sw-porters-discuss@opensolaris.org requesting that the package be promoted from the pending repository to the contrib repository. Note that you'll need to subscribe to this mailing list before you can post to it. To subscribe, go here.

Once the package is available in contrib, users will be able to install your package on their systems.

FIN!

[See my later blog entry for additional information about SourceJuicer and OpenSolaris improvements that make package contributions even easier.]


Monday Dec 03, 2007

Apple Success Disaster?

I've been accused of being an Apple fanboy and that may be true to some degree, but even this fanboy's patience is being tried by Leopard, Apple's newest OS release.

Ever since upgrading, my MBP's keyboard freezes for 10 seconds or so every few minutes until I either sleep and wake the system or reboot it. Sometimes I don't have the problem for days and then it descends on me again, making the system unusable. Every day more people contribute to this Apple Discussions thread about the problem. Some say Apple is aware of the issue and that a fix is coming soon. Many are getting very, very frustrated.

Time Machine is also having problems, though thankfully it has worked for me without issues so far. After having used Leopard for awhile, my colleague Eric erased his pre-Leopard disk backup and switched to Time Machine. Which then failed to back up his disk, leaving him without a viable backup of his system. When we last talked, he was going to use Carbon Copy Cloner or an equivalent to do an interim backup, but he had run into Leopard compatibility problems with at least one of these tools. This Time Machine problem is apparently also affecting many people, though I don't have a Discussions link for you.

And then there is X11, which I need to run OpenOffice, the GIMP, and Inkscape. To say Apple's transition to the Xorg X server has been less than smooth would be an understatement. I've had to download and install an unofficial version of X11 to work around some of the problems and get the above applications running again.

Another small issue: /usr/bin/emacs was broken on my system after I did an in-place upgrade to Leopard. After some poking, I found that the Leopard installer had left the Tiger version of the emacs executable on my system. Luckily, emacs can be rebuilt in place with a simple command and that fixed the problem. Go here for how to fix this problem.

And then there is the fact that I can't run SecondLife on my machine without starting the program using the OpenGL Profiler utility that ships with Xcode. Failing to do this can wedge your machine so badly that one's only recourse is to power-cycle the system. This is apparently a graphics driver problem and isn't Leopard specific, but I tripped over this at about the same time all the rest of this nonsense was happening and it has contributed to my annoyance with Apple. For details on the SL problem and its workaround, check here.

This morning Google Earth died mysteriously on startup and found I needed to install a new Leopard version. Thankfully, one is now available.

Apple usually fixes some significant issues in a dot release soon after their initial major release. Unfortunately, 10.5.1 did not fix any of these problems.


Sunday Oct 28, 2007

Work from Home

As I sit here in my office in Burlington on a Sunday, I realize how much I depend on my laptop to let me work from anywhere at any time. And how much additional work I do because I have that capability.

I've been without a laptop for about two weeks. Its replacement should arrive in the next two days (I've tracked it from Shanghai to East Boston so far), but not quickly enough to avoid having to drive (40 miles each way) to the office to work on a presentation about code review practices that I'm giving tomorrow.

Aside from that one inconvenience, I've come to realize how much work I usually do in the mornings before driving to work and how much I do in the evenings afterward. It makes a big difference in my ability to keep up with everything that needs doing at work.


Thursday May 03, 2007

Logical Domains for UltraSPARC: Now available!


ldoms architecture diagram

Our group (SPARC Platform Software) has just released V1.0 of Logical Domains (LDoms) for the UltraSPARC T1 processor. With LDoms virtualization, you can run 32 separate Solaris instances on a single UltraSPARC-T1 based system. Customers with Sun Fire T1000 and T2000 systems or with Netra T2000 or CP3060 systems will find the downloadable bits here.

My colleague, Ashley Saulsbury, has more details about LDoms and virtualization on his blog. Or check out the official product page.

Did I mention LDoms software is free? Yes, I guess I did. :-)


Thursday Mar 29, 2007

Protecting Your Project From Poisonous People

I just watched a Google TechTalk by Ben Collins-Sussman and Brian Fitzpatrick, titled How To Protect Your Open Source Project from Poisonous People. It's almost an hour long, but well worth watching for people involved in open source communities, or really any communities at all, technical or not. Their "four stages of protection" against poisonous people (comprehension, fortification, identification, and disinfection) are relevant for dealing with misbehavior in any group.

Their abstract:

Every open source project runs into people who are selfish, uncooperative, and disrespectful. These people can silently poison the atmosphere of a happy developer community. Come learn how to identify these people and peacefully de-fuse them before they derail your project. Told through a series of (often amusing) real-life anecdotes and experiences.

focuses on poisonous people in particular, but their talk also shares some more general wisdom about running effective open source projects. Many of their examples come from their experiences with the Subversion project. And, yes, some of the anecdotes are very amusing.

The video is here.

Thanks to Monty for the pointer.


Tuesday Mar 06, 2007

Virtualizing SPARC

Interested in running multiple, virtualized Solaris instances on your CoolThreads server? Well, now you can: SPARC virtualization is here! Details on Logical Domains (LDoms) 1.0 Early Access are available on Eric Sharakan's blog, Virtuality.

Go team!


Wednesday Feb 21, 2007

Put your least creative engineers on that project, please...

Sun's Systems Group holds regular, two-day internal CTO Review meetings at which the status of pretty much every project currently running within the Systems division is reviewed. These meetings are run by Mike Splain (Systems CTO and Sun's Chief Engineer) with John Fowler (Systems EVP) in attendance as well.

At least week's meeting, John made the following somewhat surprising comment during a presentation about a new software project starting soon in Systems. He said:

"Do me a favor and put your least creative engineers on that project, please."

John was right on target. Why? Because the project is about implementing functionality currently available on some non-Sun systems, and customers who have these systems already understand how to use the existing interfaces and utilities. The correct objective should be to do an excellent job implementing those existing interfaces for our systems. Doing a better or different interface is not always the right answer. In fact, it is often the wrong answer, especially when growing aggressively into new markets and taking business from competitors. In such cases, lowering the switching costs for customers is very important. Common interfaces, especially management interfaces, are an important way to do this.

That's not to say innovation isn't important--it is. But innovate in ways that add value, not in ways that create gratuitous differences between our products and those of the competition.


Saturday Dec 30, 2006

Hueston on Tactical Leadership

Bob Hueston, a Senior Staff Engineer in Sun's Systems Group, is writing about leading software projects on his new blog, Tactical Leadership. He's been wanting for years to write what he calls a field handbook for engineering project leaders and has started to share his thoughts publically.

The content is what I expect from Bob: insightful, thoughtful and well-presented. His introductory entry is here. I especially enjoyed his thoughts on top-down versus bottom-up planning here.

Time to add another Sun blog to my blogroll. This one's a keeper.


Wednesday Dec 13, 2006

DE Promotion Review Meeting

Sun held its biannual Distinguished Engineer promotion review meeting this week. All of the DEs and and Fellows convened in Menlo Park for a 1.5 day meeting to review candidate cases and to recommend promotions to Distinguished Engineer. We reviewed more cases in this session than in any other I've attended.

For those not familiar, the process works like this. First, a candidate's Vice President or Director decides to nominate an employee for consideration. A case is prepared without the knowledge of the candidate, following established guidelines that have been published on the DE website. Case materials are distributed to the DEs and Fellows in advance of the in-person review meeting. At the review meeting, each candidate's Director or Vice President spends about twenty minutes presenting the candidate and then spends about the same amount of time fielding questions from the attendees. Once the questioning is completed and the speaker has left the room, an internal discussion ensues, followed by a formal vote.

As a participant, I form an initial opinion based on the reading of each case prior to the meeting. That opinion can be affected (positively or negatively) by the case presentation and then again (positively or negatively) by the detailed private discussion prior to the vote.

At this session, I entered the meeting with a tentative determination that I would vote Yes on 30% of the cases being considered. After presentations, that percentage had risen to 60%. After discussions, I ultimately voted Yes on 75% of the cases. The presos and the discussions matter a lot in this process, as you can see.

Promotion announcements should begin rolling out soon.


Wednesday Aug 23, 2006

Systems Leadership Summit, Day 2

The second and final day of John Fowler's System Group Leadership Summit began with a presentation by John on his leadership values, which all of the attendees I spoke with later found very valuable. I guess it's always good to know what the boss is thinking and what he values. :-) In this case, the concept of "stewardship" is something that resonates with John. He defined stewardship as having three axes: Responsibility, Service, and Mature Leadership.

The follow-on leadership training session, run by an external trainer, was not very useful in my view. While it worked reasonably well as a teaming session, I didn't gain any real insight into improving my leadership abilities. Others seemed to feel the same way.

The balance of the day was filled with presentations by Mike Lehman, Greg Papadopoulos, and Jonathan Schwartz.

I always appreciate an opportunity to hear Mike speak because he is frank and clear about where we are, what we need to be doing, what we've done so far, and where we are going.

Greg's talk was one I'd heard before, but it reinforced the message for me. I agree with him that there is a real shift happening from delivering applications as bits on local systems towards delivering applications as services in the network. I can see that as a Sun employee, but more important, I can see it as a consumer.

Jonathan's was the last session of the day and it was more of a Q&A session that a set of prepared remarks. He was pleased with the great server market share news that was released today. Since he was talking to the Systems Group, the division responsible for designing and building those servers, he expressed his appreciation to all for the hard work that contributed to these results. He also apologized to John for running late and not being able to pick up a bottle of champagne to open in front of the team. Instead, he simulated the event by throwing a cup of water on John as an homage, which drew a big laugh from the crowd.

Systems Group Leadership Summit

John Fowler, Systems Group EVP, has gathered his VPs and Directors; Fellows and DEs; and his corporate business partners for a two-day, in-person meeting in Santa Clara. This is the first Systems Group Leadership Summit.


[john fowler]
John Fowler, Executive Vice President, Systems Group

Yesterday, John reviewed FY06 accomplishments, presented a business update, and covered his strategy going forward for Systems. In addition, we had two excellent invited talks.

The first was given by Dan Miller, Senior Vice President of Sun's Global Systems Practice. He gave a very good overview/update of GSS (Global Sales and Service) with an emphasis on the Systems Practice, the sales arm most closely associated with the Systems Group. The two organizations work together to ensure we deliver products and solutions of value to Sun's systems customers.

Anil Gadre, Sun's Chief Marketing Officer, gave an engaging and interesting talk on Sun's marketing strategy, including updates on how we did last year in terms of market perception, messaging, exposure, etc. For any Sun employees who get a chance to hear Anil speak, I recommend you take advantage of the opportunity.

Today's agenda includes leadership topics followed by presentations by Mike Lehman, Greg Papadopoulos, and Jonathan Schwartz.


Sunday Jul 30, 2006

Leveling the Playing Field at Sun (and elsewhere)

The Sun Global Technology Conference in Bangalore last week included a session on Global Engineering during which we discussed the challenges around building and running an effective, widely distributed engineering organization. I'd like to share a suggestion I made during this session. It addresses only a specific aspect of the global engineering challenge, but I believe it could deliver real value for Sun, Sun's employees, and ultimately Sun's customers.

In a nutshell: Ban the use of conference rooms at Sun for meetings with multi-site attendees.

It would cost a modest amount to ensure employees have headsets for quality, full-duplex audio, but if we are serious about being a global company, this suggestion should be studied carefully.

For our senior executives, it's probably been awhile since you've been stuck on a phone, dialed into a conference room full of people who you can't hear, who haven't remembered to email the conference materials, and who use whiteboards you can't see. If you can get past all that, now try to actually participate in the conversation instead of just trying to listen. It doesn't work. And better conference room audio systems are not the answer.

As a senior technical person, I find such meetings annoying, but I make it work as best I can. I'm not shy about insisting that the meeting materials be sent out or about being aggressive in inserting myself into the conversation, despite the difficulties of being heard on the phone. But what about our less senior engineers? Or engineers in countries where such aggressiveness isn't part of their culture? Sure, they need to step up and participate in Sun's engineering culture, but why not remove a significant barrier to their full participation? One that we can easily address with a small technology investment.

In terms of how meetings should work, I've been involved for several years in several standing meetings where everyone is required to use a headset even though these meetings involve groups of people on each of the Burlington, Menlo Park, and San Diego campuses. We could use three conference rooms, but we don't. And what a difference it makes:

Conference materials must be sent out to everyone. And the audio playing field is completely leveled, allowing everyone to participate equally. It's true that tools must be used to share online demos and do whiteboarding, but such tools allow everyone to participate, regardless of location.

Some may argue reasonably that this is a lowest common denominator approach which removes the benefits co-located attendees derive from sitting in conference rooms together. While that may be so, there are other ways for co-located employees to replace that in-person meeting interaction, for example through hallway conversations, after-work socializing, cafeteria lunches, etc. That problem is a lot easier to fix than the remote ("remote" is headquarters-speak--there are no remote employees) distributed collaboration problem.

It doesn't take a corporate ban to start levelling the playing field. I don't like big policies like this anyway. If you run a meeting with distributed attendees and are serious about allowing all your team members to contribute equally, make the change. Just do it!


Thursday May 18, 2006

Firefighting in Product Development: Past the Tipping Point

Give an engineer a problem involving interactions between lots of complicated bits of technology and she'll go and solve it---happens all the time. So why is it that we don't turn turn those powers of analysis loose on the complicated system that we call the software development process? Applying some rigor to understanding the dynamics of the development process makes huge sense. Turning the crank--producing high-quality products quickly and efficiently--is the essence of product engineering. Doing it well can get new products to market more quickly with higher quality and save your company millions of dollars in the process. Doing it well makes your customers happy and can create scads of new customers.

Well, good news. A group of MIT researchers have applied system dynamics to create mathematical models of the product development cycle and have published the results of their analyses in several papers. One of my favorites, and the topic of this blog entry, is Past the Tipping Point: The Persistence of Firefighting in Product Development by Repenning, Goncalves, and Black at the MIT Sloan School of Management.

Some of their findings are surprising and counterintuitive.

Consider, for example, firefighting--the unplanned allocation of resources to fix problems late in the development cycle. We all do it. We suffer the short-term scheduling hit on other projects to do the diving save. It hurts and we move on.

But it turns out it isn't that simple. The research shows that these short-term reallocations of resources can, in fact, have serious long-term effects on an organization's ability to turn the product development crank. Think death spiral.

The Model

For the purposes of explication, the authors created a simple model that illustrates their primary findings without getting into the messier details of their more complex, true-to-life models. The model is based on a several assumptions.

First, assume the organization releases a product every 12 months. The development cycle for a product consists of a 12-month concept development phase, followed by a 12-month product design and testing phase. At any given time, the organization is working on two products simultaneously--the design and testing phase for this year's product and the concept phase for next year's product.

Second, assume that the overall resource pool for the organization is fixed. If unanticipated problems arise during the design and test phase for this year's product, resources will be borrowed from next year's concept development work. This is the typical firefighting scenario in which longer term work is either skipped or delayed to deal with shorter term problems.

And, third, the model assumes that upstream concept development work that is skipped (for example, creating clear specifications of customer requirements), will create additional rework in the subsequent design and test phase for that product. There is ample industry evidence to support the reasonableness of this assumption.

The Dynamics

The resulting system dynamics model (see diagram from the paper, below) has two feedback loops. In the Rework loop, more design problems in this year's product leads to more resources being tasked to fix those problems, which in turn reduces the number of design problems in the product. The other, Tipping Loop, is intimately tied to this first loop. In the Tipping Loop, as the number of resources assigned to do rework on this year's product increases, the number of resources available to work on the concept phase for next year's product decreases. This, in turn, results in less work being done on concept development activities for next year's product, which then, after a delay, results in more design problems in the subsequent design and test cycle.

So, what happens when you run the model under various scenarios? The model has a tipping point as illustrated in the diagram below. The tipping point, which is marked with the blue circle, divides the phase space into two regions. The arrows indicate the direction of motion in phase space for each of the two regions. In the bad region, the amount of up-front work completed this year is low enough that it causes significant rework in the following design and test phase. Which pulls resources off of the up-front work being done in next year's concept phase, which in turn creates even more defects in the subsequent design and test phase, etc. Mouse over the diagram to see both the vicious and virtuous cycles illustrated graphically (works under Mozilla and Safari, no guarantees with other browsers.)


[phase plot]

The research shows that product development organizations have a Tipping Point and if you push them too far, you run the risk of pushing the organization into a death spiral of decreasing capability. Even worse, just as airline pilots cannot rely on their innate sense of physical orientation to pilot a plane, engineering management's intuition just doesn't work in dealing with these kinds of organizational problems. Our gut says temporary resource shifts should have only temporary ramifications. At some point, that just ain't so.

Pilots can be trained to fly on instruments. It remains to be seen if the same can be said for engineering management. This research is an attempt to raise awareness of the issues and highlight some of the non-intuitive ramifications of persistent firefighting in development organizations. I recommend reading the paper for a much more complete explanation of the model and its consequences.


Friday Jan 13, 2006

OpenSolaris: A Word of Caution to our Prospective Partners

I've talked to two companies--one a provider of plug-in networking cards and the other a software vendor--who have been confused by Sun's OpenSolaris effort and how it relates to their business. I guess that this is more of an issue for prospective partners who are new to Solaris rather than for those companies already established in our partner programs. We want all of our partners--established, new, and prospective--to have a great Sun experience, so please be careful of the following.

In the hardware vendor's case, they were doing their first Solaris port and had used the OpenSolaris website to download the latest Nevada build as their development baseline. This is incorrect since they would like to sell their solution with Sun's current hardware and software products. Sun's products currently use Solaris 10 and will for some time. Therefore product development and testing should be done under the appropriate version of Solaris 10 rather than Nevada. OpenSolaris is a great venue for participating in an open-source OS development effort, for looking at source code, and for a peek at the future, but please be careful to think about whether it aligns with your immediate business needs or not.

If you are a prospective Sun partner, the best way to avoid issues like this is to get involved with our partner programs. Details can be found here.


Monday Nov 28, 2005

Come Kick Some Butt

I work in Sun's SPARC Platform Software Group. We have a wide set of responsibilities, including the development of platform-specific firmware, service processor software, fault management and system management components, to name several. And if you've been poking around on OpenSolaris.org and wondered about the hypervisor, yeah, we do that, too. We are also very involved in the bring-up of new SPARC chips. Like Niagara. And you may have heard of ROCK. Very cool and innovative technologies.

We are hiring and looking for some really strong, motivated technical people who really want to kick some butt and have fun doing it. Follow the links below for full details on some of our positions and drop me a note at joshua.simons (at) sun.com if you are qualified and interested.


Firmware/bringup:

Member of Technical staff, Burlington MA or Bay Area CA
Member of Technical Staff, California

Solaris kernel development:

Member of Technical Staff, California or Texas
Member of Technical Staff, California or Texas

About

Josh Simons

Search

Archives
« July 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
  
       
Today