By now I'm sure it's painfully apparent to anyone who has read my blog I am a hard-core geek. In fact, I'm
not just your run-of-the-mill hard-core geek, but a
Aside from our being born socially inept, a common affliction amongst hard-core geeks is an inability to
communicate our thoughts and ideas (and emotions -- but I ain't goin' there!) effectively. While there are
I've met out there who seem to have stumbled upon the
magic cure to this
nasty affliction, the hard-core geek who can communicate effectively
As a professional software engineer, one of my fundamental beliefs is that the dominant factor between
a successful project and an unsuccessful project reduces to the effective dissemination
of key information. As a battle-worn veteran of several software projects, I've
seen other projects flounder because the members of the team didn't understand or buy-into the mission,
major deliverables or objectives weren't clearly defined, team members didn't agree on the requirements,
or team members grew differing perceptions of what the final product would look like!
The impacts of ineffective written communications within a project often are magnified by
conspiring factors such as geographically dispersed teams, rushed schedules (i.e. we don't have the TIME
to define WHAT we're going to deliver!), and in extreme cases, recalcitrant team members!
In this post I examine the role of effective project documentation in executing successful
development projects, and share why I believe this is even more important in an Open software
development model than in traditional, closed-source projects.
The Traditional Software Project Model
The traditional project development model is usually the classic waterfall model of software engineering.
The project is conceived, funded, and staffed. At that point the conception phase moves into requirements
analysis, functional specifications are developed, and the architecture is determined. Next, the design
is specified, an implementation created, and the implementation is checked against the specifications of
the architecture. If everything checks out, the project is accepted by the sponsors and deployed.
At each stage of the traditional software project, new stakeholders are introduced, and these stakeholders have
progressively lower levels of focus. If the communication of the concept is not clear, the architects may come up with
a proposal which doesn't match the original concept. If the architecture is over- or under-constrained,
the implementation may be untenable or incomplete. If the validation specification does not match
the architecture the acceptance may not match the implemented product. Finally, if the end-user documentation
does not match the architecture and implementation, the end-user will find using the product to
be time consuming, frustrating, and costly. Anyone who has played the telephone game and observed how
garbled even a simple message can become can appreciate how even a small defect in the communication chain of a
complex project can lead to catastrophy in the final product!
The Open Software Development Project Model
The Open software development model differs from the traditional model in several imporant ways.
First, the concept and requirements typically come directly from the end-user(s), and are usually
accompanied by a rough implementation in the form of a prototype. At this point, the community
critiques the proposal and implementation. Community members attempt to reverse-engineer any
unspecified requirements, and suggest new concepts for inclusion. In the case of large software
projects, the development may shift back into the traditional model after this point. With smaller
projects, the process may be more iterative, with the traditional validation phase leading to revised
requirements, and the cycle repeating itself.
Second, in the Open model the stakeholders at each stage are different. In the case of the traditional model,
the needs of the business (the business case) ultimately drive the scope and requirements
of a project; the concept phase is nearly set in stone by the time the implementors start to engage.
In the Open development model, the focus is usually more on the desired outcome
of the end-users of the product. The requirements, and even the high level concepts
are more free to evolve over time as the project progresses.
The Role of Communication in Software Development
In my experience, most traditional software projects make it through the first few stages (concept
and requirements) with relatively few problems. By the time the implementors are brought on-board,
the functional specifications are baked, and they map back to the original concepts and requirements
fairly accurately. The big issues such as projected development and scope have been settled by the
project bosses. Due to the senior-level of involvement up to this point, the accuracy and quality
of specification documents is usually very good.
The really successful software projects become successful because they give the right level
of attention to clearly communicating the key concepts and requirements. Software engineering experts
have differing opinions on how much of the total effort should be given to a project's early phases, but
my gut feel is that it is well over 25%, and may be as much as 50%.
To make an Open project which is proposed in the form of a prototype successful, it is even more
important to keep in mind that the key concepts and requirements need to be thoroughly documented for
the community to agree on them. There are several reasons for this:
- Engineers on traditional development projects are usually geographically co-located, and utilize
high bandwidth forms of communication such as meetings and whiteboard sessions to reach decisions
and agreements. Open development projects are almost always geographically distributed. E-mail and
project web pages, which are lower-bandwidth forms of communication, are typically the dominant forms
of reaching decisions and agreements; this makes increased accuracy and completeness a necessity.
- Unlike in the traditional model where the stakeholders are the product bosses, Open projects have
the entire community (and the end users) as the stakeholders! If the community can't agree on the high
level objectives and functional requirements of a software project, the project is doomed from the start.
Documenting Concepts, Requirements, and Architecture
The concepts and requirements comprise the big picture view of the software project. These define aspects
such as the major capabilies of the product and the intended target market. The architecture defines the
input/output methods, interoperability features, and user interface.
The first nail in the road at the early phases of a project is that seemingly unimportant details
such as how the requirements were gathered are often not recorded. High-level tradeoffs made in mapping the
requirements into architecture are often not well documented (if at all). As we'll
see, this is a disaster waiting to happen down the road when the implementors hit a roadblock in the design.
A less obvious trap at this point is that the requirements and architecture are usually baked
by experienced software engineers with a good, albeit incomplete, idea of the requirements of the implementors.
On the other hand, implementors are usually less-experienced, and have a more localized focus -- meaning
that there is an inherent opportunity for information loss when translating the requirements into software
constructs. If an implementor doesn't understand how flexible a requirement is, or why the requirement is
there at all, the implementation may be inadequate or over-constrained.
The latter effect is magnified in the Open development model, where the implementors do not work
with the software as their day job; if they do, they often don't have the benefit of years of experience
networking with engineers inside a major software company. If an Open development project is going to be taken on
by less-experienced implementors, then clearly communicating the concepts, requirements, and architecture of the
software is of paramount importance.
Documenting the Design
To help avoid pitfalls during the implementation phase, I believe it is essential that the design team
thoroughly document the aspects of their implementation blueprints. The apsects which should be thoroughly
documented include any and all of:
- Design Trade-offs
- Design Decisions
Let's break down each of these and examine them independently.
We have all heard the cliche' about how assumptions are bad. However, reality dictates that no
matter how well the requirements are written and the architecture is specified, there will always
be room for interpretation. The idea behind documenting all assumptions regarding the concepts,
requirements, and architecture is that the folks who created them can review these assumptions
and provide clarifications when necessary. In some circumstances there may be holes in the
original specifications which result in assumptions about major requirements! If in doubt,
and it's open to interpretation -- document your assumptions about it! The time spent up front
will save you a lot of frustration down the road by preventing an implementation going to the
validation phase, only for the end-user to come back and say this doesn't work the way I expected it to!
There's also an advantage to catching holes in our assumptions early which is particuarly
helpful in the context of an Open development community. We're all human, and as such we have tendencies
to blame others rather than ourselves when things go wrong. Having members of a community come back after an
implementor has worked long and hard on a problem only to tell him/her that the hard work that person
has done is incorrect is a big blow to the implementor's pride. In the worst case, flame wars may insue,
and the contributor ends up leaving the community. A lot of frustration and wasted effort can be
eliminated if the community has the opportunity to identify misguided assumptions and correct them
One major difference I've noticed between experienced, high-caliber software engineers and
average software engineers is that experienced software engineers know how to leverage constraints
in order to optimize solutions to difficult problems.
Let me give a concrete example from a project I worked on a few years ago: a previous team worked
hard at solving a difficult kernel problem in the area of virtual memory management. The team
came up with an approach, worked for about a year, and had a working solution -- only to have their
approach nailed to the wall because of a design flaw which could result in data corruption.
when I took over the project it didn't take me long to decide that the systemic problem with their
approach revolved around the constraints. Specifically, the project team's approach failed because
they had designed around a constraint that a specific DDI call had to remain supported. Because the
DDI call in question was part of an obsolete framework, removing support for the call was the best
way to solve the problem, and it removed the potential hole where an obsolete or incompatible device
driver might cause kernel data corruption when the new feature was enabled.
In a way constraints are assumptions -- but they are assumptions that aren't related to the specifications.
Which functions a library supports, which platforms will best run an application, and which programming
languages to use may all be constraints of an implementation on which the requirements and architecture
are entirely mute.
Another example of leveraging constraints to your advantage in an implementation is
creating a dependency on a particular system library, rather than developing your own
Some folks look negatively on design-tradeoffs -- they refer to the practice of making trade offs in a
design as cutting corners. However, software is like any other engineering practice -- we need to
balance effort and complexity against functionality. Finding a happy middle ground is a constant battle.
Most high quality projects I've seen do a good job of articulating their design tradeoffs.
This is because there are two paths to detecting an incorrect design tradeoff: in the first case, the
tradeoff is documented, and someone detects the problem before it poses a major problem; in
the second case, the product blows up in the hands of the end-user! I don't know about you, but I
prefer constructive criticism over catastrophic failure any day!
I prefer to think of the design decisions as the politics of software design. This position may not
be popular, but I think it gets the point across well. If others don't understand the reasoning behind
your decisions, they will be more inclined to disagree with the outcome. If you do not have a strong case,
and others hold enough clout over you, they may convince others that their approach is right, and yours
The best defense against showdowns based on emotion and past experience (versus the facts!) is to clearly
articulate the reasons you decided something is best implemented a certain way. In many cases, the decision is
unimporant and ends up being arbitrary (for instance, how often to poll a descriptor). In other cases, the
decision is the result of a thorough analysis. In yet other cases, there may be mistakes that you made
others can find that show your decision is flawed. Regardless of the scenario, clear and concise documentation
of the reasoning behind a design, and not just the documentation of the final design, will lead to a better
Putting Project Communication into Practice
Of course there is reading about project communication, and there is doing it
To ensure your development project is more likely to be successful, I recommend doing the following.
Start out with a communication plan
The communication plan should be your first deliverable. This plan should lay out which e-mail aliases
will be used to communicate regarding various aspects of the project, where the web page is, and what content
it will host. The communication plan should also list the documentation deliverables of each phase of the
project, and who the consumers are. Get the members of the community to buy-in to your plan before continuing!
Document each project phase
As you progress through your project, consider documentation to be as important as any other deliverables,
and do not consider moving onto the next phase until all of the deliverables are complete. Without full
agreement by everyone that the project documentation is accurate and complete, you have no assurances
that any of what you are delivering is!
Refer to your documentation often
In addition to reaching agreement before moving forward, the best aspect of project documentation is that
it will still be around in a year when you have forgotten the important details (I think we did this because ...
Why did we do that again?). Use the project documentation you produce to guide you along.
The minimalist approach
Suppose that I am proposing a bugfix. As long as there is no impact of the design of the system, and no change in the user's experience, only a small design document is necessary. My design may be as little as a few paragraphs to accompany the diffs in an e-mail, explaining the root cause of the bug, how I arrived at the fix, and what the fix does differently to correct the abnormal behavior.
Use common sense! If a bugfix is a one-line change to correct a misspelling in a code comment, it's obvious to everyone that no design documentation is required -- the diffs speak for themselves. In anything more complex than the most trivial, mechanical change, there was some thought process involved. The idea is to get that thought process out into the open, where others can review not just the fix, but what you were thinking when you wrote it.
Suppose I'm implementing a new function call in a shared library. The end-users will be software developers. The stakeholders will be whoever requested the interface. I'd start by collecting the requirements, which probably are coming from one or more persons who proposed the interface. Once the requirements fleshed out and agreed upon, I'd draw up an interface specification. Since all library calls require a man page, I would do this in the form of a mock-up man page, and send it out for wider review, making sure there are several experienced software developers among the reviewers. Once the interface was baked, I would sit down and draft up an implementation proposal, listing all of the tradeoffs I could think of. Once I had that in place I would request a peer review of my proposed implementation, and after iterating on it I would continue onto the code.
As usual a one-size-fits-all approach isn't going to work, so you'll need to consider many factors -- size of the team, importance of the functionality, dependencies, schedule, etc. A macro project may range from an RFE (request for enhancement) to a full blown software package.
At a minimum, I would suggest breaking up the project into the following stages, each paired with a set of project documentation deliverables:
- List of key requirements and features
- Component block diagram
- Interface specifications
- Dependency and data flow diagrams
- Detailed specifications for each component
- Constraints, risks, assumptions, and trade-offs
- Code comments
- Big theory statements (to describe algorithms)
- Test plan
- End-user documentation
If you are putting together a good-sized project and you lack experience, I would suggest seeking outside help. The OpenSolaris community (general discussion list) is a good place to start.