By nwooler on Jan 08, 2009
These are the types of processes that quality open source projects do as a part of the project development process. Indira Thangasamy, produced a similar article on how they approach QA within the OpenSSO project. As companies evaluate other open source projects, especially in these challenging economic times where cost reduction provide stronger rational's to consider starting projects using open source software. The quality approach of communities becomes an important differntiator as companies use open source in production and customer facing systems.
Here is a quick overview of the test harness used on OpenDS:
We use open-source, Java platform-based test tools, such as the following, not only to demonstrate our support for open source but also to ensure that they are accessible to everyone:
- TestNG — For running unit tests, same as in OpenSSO
- Checkstyle — For checking code style and rules before commitment
- EMMA — For measuring code coverage
- Software Testing Automation Framework (STAF) and STAF eXecution engine (STAX) — For running functional and system tests
Here are a couple of other highlights:
- Unit Testing and Automation: "Testing starts in the programming phase with unit tests, which verify that the code works as intended and which must exist for all features. Today, we run 30,000 automated unit tests daily on different Java virtual machines. No code can be integrated without satisfying the precommit requirements."
- Code coverage — With open-source EMMA, we find out the number of code lines, blocks, methods, and classes that are exposed by the unit and functional tests. Part of that information pinpoints the amount of the code tested as a percentage of the total, defining if we've met the quality criteria. We also define which areas of the code are not tested, called coverage holes, and create new tests to fill them.
- Feature coverage — OpenDS delivers features that customers want, that is, customer requirements. Each feature is recorded as an issue in the Issue Tracker, a tool that monitors defects. This data tells us the state the features are in and their status: Ready for Test or Tested.
- Documentation coverage — To ensure that the documentation is reviewed according to the test plan, we adopt a two-phase documentation review process: a technical review of the content followed by a formal QA review. Like the product features, the documentation is divided into categories—books, chapters, and sections—that are recorded in the Issue tracker. Through this coverage, we measure the percentage of the documentation reviewed over time and identify the reviewers and review status.
- Defect rates — This is a traditional measure. The goal is to have no high-priority bugs open at release time. Our Bug Council constantly studies the defects and assesses the risks to customers. We also plot simple graph trends to gauge how well the project is converging.
Thanks to Gary and Marina for publishing this article and allowing the community to learn from your experience.