Thursday Nov 06, 2008


So it ends. I've been at Sun since March of 1989 (with a couple of years off for good behavior), and for the last 10 years I've worked on internal processes, tools and problem solving related to software development, systems architecture and product delivery. Much of my focus has been on OpenSolaris, in the Architecture community as well as serving on the OGB. Today, I was informed that my position has been eliminated, and I was being laid off.

I intend to remain active in the OpenSolaris community, though I expect to start participating in other areas (Arduino is cool, JMRI is as well, and I want to help make both available in an OpenSolaris repository ASAP)

Since this may well be my last blog post here, the following links should help keep you in touch:

See you around,
   - John

Saturday Dec 31, 2005

Caboose hunting - in the beginning...

I've moved the "caboose hunting" blog to my personal web site since it has little to do with my job at Sun (and because it is an excuse to play around with blogging software :-)...

Please see Trains of Thought on my SP Coast web site for the continuing saga of WP668 - buying and restoring a 90 year old outside braced wooden bay window caboose!

Tuesday Jul 26, 2005

What do the ARCs do?

What is it that we expect the ARCs to do?

The ARCs attempt to maximize the effective output of all developers. Some of the ways it does this are

  • by eliminating/minimizing inter-project friction
  • by ensuring customers can accept present and future releases, and
  • by performing systems analysis

Goals of the ARC process's interactions with the products being developed:

  • Consistently fulfill our customers' expectations of quality, across product lines and over generations of products.
  • Establish the basis for multiple independent engineering projects to develop in parallel
  • Provide architectural training and mentoring for developers

The ARCs do this by reviewing the proposed changes to components. These changes need to be proactively understood and committed to by the community before they are integrated into the codebase. The scope and level of such reviews obviously will vary in relationship to the impact of the change, with simple bugfixes getting the least attention and complex feature additions or changes getting the most.

Thursday Jul 14, 2005

What is an ARC?

The Architecture Review Committees (ARCs) are responsible for the architecture process at Sun. The "Systems Architecture" effort at Sun is a process that establishes the basis for multiple engineering projects to be developed in parallel and then to integrate with each other and the existing system, either at Sun or in the customer's hands.

Sun's Architecture Review Committees (ARCs) review and approve technical aspects of software projects, implementing a process to manage technical change in the architecture of our software systems. These architectural reviews:

  • assure that a technical description of the project exists,
  • assure the developers' plans are technically sound and consistent with current conventions and existing products,
  • collect and archive project specifications, and
  • improve inter-project coordination: decrease the likelihood that projects conflict with (or duplicate) each other.

The goal of the ARC process is to consistently fulfill our customers' expectations of what a Sun product is, across the product line and over generations of products.

Reusable components

This is part of a discussion I'm having on the opensolaris-arc mailing list...

One of the recurring problems that we try to manage at Sun is how to effectively reuse components that should be reusable.

Reused components add risk to complex systems, primarily because incompatible changes to a component will break other parts of the system.

The FOSS/distro world many times deals with this situation by resyncing and recompiling whole subsystems every time some component revs in an incompatible way, giving what I call "source level compatability". Think gentoo and autoconf.

The alternative chosen by Sun and others is to provide "binary compatability", even in the face of incompatible change. This requires that we worry a lot more about changes to the interfaces between components:

  • Exposing too many implementation details makes it difficult to evolve a component without breaking things that depend on it. We call exposed interfaces "public" and implementation details "private".
  • Interfaces have expectations that need to be set and managed. We ask "Can we expect this interface to be stable over releases, or is it more of an experimental thing that will change in the next release?
  • As a component evolves, we ask "what interfaces have you changed, and what expectations did you set when you first exposed them?
  • We use two taxonomies to help us keep track of all this - an interface taxonomy that sets expectations of stability and a release taxonomy that defines the type of release that can contain the changed interface.
    StabilityPublicStable, Unstable, Volatile, "not an interface"
    PrivateConsolidation Private, Project Private
    ReleasesMajor, Minor, Micro, Patch

    There is a relationship between the two:

    Incompatible changes to a Stable interface are only allowed to be made in a Major release of the consolidation that contains that interface.
    Unstable Major or Minor
    Volatile Major or Minor or Micro
    NAI Major or Minor or Micro or Patch
    \* Private Major or Minor or Micro or Patch

As long as these components are not reused by anything else in the system, it is easy to "not care" about how they evolve or change. The collection (which we call a "consolidation") is both low-risk and low-maintenance.

As these components start to get reused as building blocks for other consolidations, the potential cost to the system increases. If a component changes in an incompatible way, all the things that depend on it will need to react; if they don't, things may/will break.

The architectural question here really comes directly from kindergarten: "does it play well with others?"

Friday Jun 03, 2005

Sun's development process - Basic Relationships

Sometimes a picture is worth more than a page of blog text. This diagram illustrates my prior posting about basic architectural definitions and their relationships.

(Click on image to see a larger version)
WorkspacesSun uses Teamware to manage much of its source code. The Teamware term Workspace is used to describe a copy of a source tree that contains its version control history; Teamware can manage the relationships and content flow between workspaces.
I-TeamsImplementation Teams (aka developers)
RERelease Engineering
RealizationsInstances of a product in specific forms
PLC Phase 3The first three phases of Sun's Product Life Cycle are "Propose a Concept", "Commit to a Plan" and "Develop a Product". The reference indicates that (duh!) product development happens in the development phase. Yes, there are some people that need to be told the obvious :-).

Thursday Jun 02, 2005

What kind of ARC review do projects need?

Alan Coopersmith writes about making a simple change to Solaris that has a simple architectural impact. This raises the question of how are changes classified?

Obviously, we want to be proactive and intentional about making changes to our products, since the alternative of being suprised by unexpected changes usually means upset customers and broken products.

The development processes we use tries to formalize this intent by asking two basic questions about every proposed change:

  • Do we want this particular change?
  • Is this the best way to make the change?

The former is primarily a business decision, and the latter, a technical one.

Many times, the want question is obvious - Priority 1 bugs are more "wanted" than Priority 5 ones, competitive pressures may demand certain new features, etc. I won't spend much time discussing this topic, as I want to focus on the technical path.

Architectural Review Committees (ARCs) are made up of senior engineers and other domain experts, and are responsible for the technical review and approval of these changes. The first step in that review process is to determine what kind of review a particular proposal requires:

  • The simplest review is None.
      Projects that
      • have had an earlier proposal approved by an ARC,
      • do not introduce new interfaces visible outside their own project, and
      • do not alter any interface visible outside their own project
      usually qualify for self review and automatic approval.

      Most bugfixes qualify for this level of review.

      The documentation provided in the bug report (evaluation, suggested fix...) combined with the usual peer review is deemed to provide sufficient technical oversight.

  • Next up is Fast-Track
      A lightweight process for simple cases whose architectural impact is obvious, and that are not likely to be controversial.

      Projects suitable for a fast-track review generally apply "common practice" in a frequently-performed change or addition.

      The fast-track process is primarily handled as email discussions and can take as little as a week's time to complete.

    The intent is that 80% or more of the changes being made will be either self reviewed or fast tracked

  • If you don't qualify for any of the above shortcuts, you need a Full ARC Review

      The full ARC review process involves a combination of email, detailed project specifications, and formal meetings. The time required varies considerably, but teams should schedule reviews early - an inception review by mid-prototype stage, with commitment reviews between alpha and beta in order to allow time to incorporate any ARC required changes before integration.

Technorati Tag: OpenSolaris
Technorati Tag: Solaris

Wednesday Jun 01, 2005

Change Management - Operational Definitions

Products are made up of one or more components, which are, in turn, made up of one or more elements. Elements and components evolve over time as Projects are initiated to change them.

By watching these changes as they happen, we have a chance to be proactive and manage risk:

  • Should we make this change at this time?
  • Is this the best way to implement this change?
  • What is the impact of making this change on other parts of the system?
  • How much will this change cost (risk, time, complexity, resources...)?

The focus of change management is, of course, a change. We use the word projectto mean "a proposal to make a change to a component". Proposals always reference a specific plan; they may range all the way from the simple (documentation changes and bug fixes) to the complex (new features, refactoring and redesign of major subsystems).
The formal articulation of a proposal. Plans identify deliverables, schedule, resources, and "completion criteria". If you don't have all of these written down, you don't have a plan, you only have a goal. Plans say "I am going to do this to that by then". Plans can be as simple as an evaluated bug report (evaluation, suggested fix, commit-to-fix release) or as complicated as a business plan with marketing requirements, functional specifications, etc.

A collection of elements that are always delivered together as a self-consistant entity. Components are reusable, sharable things that form the basis for the complicated systems platforms that we deliver. Components are combined to form Releases.

(Think "workspace" or "CVS repository" when you think of components)

Components let us take advantage of independent evolution, decoupling, modularity and reuse - in a very real sense, they are the building blocks that we use to build solutions for our customers.

Changes are made to elements of components. At the beginning, the very first change request is to create a component (i.e., create a new product or feature). However, once a component exists, it spends the rest of its existance being changed. Bugfixes, code restructuring, performance improvements, and the addition of new elements/features all impact the evolution of a component.

The name we use to describe the thing that remains in a component as a result of making a change. The term is not a formal definition, but it exists so that we don't have to overload the term "project" to mean both the ephimeral process of making a change and the thing that remains as the result of making such a change. Examples include the ls command in the Posix Utilities, the IPV6 or DTrace features in Solaris, the spell checker in OpenOffice, etc)
Products or Distributions

Components do not usually live in isolation - they usually depend on other components, or they do something useful so other components come to depend on them. In that sense, components are usually combined with other components into product releases.

This dependency introduces a risk - the risk that something you depend upon might change in the future and break your component. Or, more likely, you may wish to do something to your component that might break components that depend on yours. This risk management is the focus of Sun's development process.

Technorati Tag: OpenSolaris
Technorati Tag: Solaris


And it ought to be remembered that there is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things. Because the innovator has for enemies all those who have done well under the old conditions, and lukewarm defenders in those who may do well under the new. This coolness arises partly from fear of the opponents, who have the laws on their side, and partly from the incredulity of men, who do not readily believe in new things until they have had a long experience of them.

Thus, it happens that wheneever those who are hostile have the opportunity to attack they do it like partisans, whilst the others defend lukewarmly...

The Prince, Chapter 6 by Niccolo Machiavelli

Sun's involvement in community development projects such as OpenSolaris, Gnome and Mozilla is all about change - both process changes and code changes.

In the spirit of trying to avoid the above fate, I will be using this space to present and discuss some of the processes that Sun uses to identify and manage such change.

Technorati Tag: OpenSolaris
Technorati Tag: Solaris



« June 2016