Monday Oct 29, 2012

JDK bug migration: components and subcomponents

One subtask of the JDK migration from the legacy bug tracking system to JIRA was reclassifying bugs from a three-level taxonomy in the legacy system, (product, category, subcategory), to a fundamentally two-level scheme in our customized JIRA instance, (component, subcomponent). In the JDK JIRA system, there is technically a third project-level classification, but by design a large majority of JDK-related bugs were migrated into a single "JDK" project. In the end, over 450 legacy subcategories were simplified into about 120 subcomponents in JIRA. The 120 subcomponents are distributed among 17 components. A rule of thumb used was that a subcategory had to have at least 50 bugs in it for it to be retained.

Below is a listing the component / subcomponent classification of the JDK JIRA project along with some notes and guidance on which OpenJDK email addresses cover different areas. Eventually, a separate incidents project to host new issues filed at will use a slightly simplified version of this scheme.

The preponderance of bugs and subcomponents for the JDK are in library-related areas, with components named foo-libs and subcomponents primarily named after packages. While there was an overall condensation of subcomponents in the migration, in some cases long-standing informal divisions in core libraries based on naming conventions in the description were promoted to formal subcomponents. For example, hundreds of bugs in the java.util subcomponent whose descriptions started with "(coll)" were moved into java.util:collections. Likewise, java.lang bugs starting with "(reflect)" and "(proxy)" were moved into java.lang:reflect.

  • client-libs (Predominantly discussed on 2d-dev and awt-dev and swing-dev.)
    • 2d
    • demo
    • java.awt
    • java.awt:i18n
    • java.beans (See beans-dev.)
    • javax.accessibility
    • javax.imageio
    • javax.sound (See sound-dev.)
    • javax.swing
  • core-libs (See core-libs-dev.)
    • java.lang
    • java.lang.invoke
    • java.lang:class_loading
    • java.lang:reflect
    • java.math
    • java.nio (Discussed on nio-dev.)
    • java.nio.charsets
    • java.rmi
    • java.sql
    • java.sql:bridge
    • java.text
    • java.util
    • java.util.concurrent
    • java.util.jar
    • java.util.logging
    • java.util.regex
    • java.util:collections
    • java.util:i18n
    • javax.annotation.processing
    • javax.lang.model
    • javax.naming (JNDI)
    • javax.script
    • javax.script:javascript
    • javax.sql
    • org.openjdk.jigsaw (See jigsaw-dev.)
  • security-libs (See security-dev.)
    • javax.crypto (JCE: includes SunJCE/MSCAPI/UCRYPTO/ECC)
    • javax.crypto:pkcs11 (JCE: PKCS11 only)
    • (JSSE, includes
    • javax.smartcardio
    • javax.xml.crypto
    • org.ietf.jgss
    • org.ietf.jgss:krb5
  • other-libs
    • corba
    • corba:idl
    • corba:orb
    • corba:rmi-iiop
    • javadb
    • other (When no other subcomponent is more appropriate; use judiciously.)

Most of the subcomponents in the xml component are related to jaxp.

  • xml
    • jax-ws
    • jaxb
    • javax.xml.parsers (JAXP)
    • (JAXP)
    • javax.xml.transform (JAXP)
    • javax.xml.validation (JAXP)
    • javax.xml.xpath (JAXP)
    • jaxp (JAXP)
    • org.w3c.dom (JAXP)
    • org.xml.sax (JAXP)

For OpenJDK, most JVM-related bugs are connected to the HotSpot Java virtual machine.

The full JDK bug database contains entries related to legacy virtual machines that predate HotSpot as well as retired APIs.

  • vm-legacy
    • jit (Sun Exact VM)
    • jit_symantec (Symantec VM, before Exact VM)
    • jvmdi (JVM Debug Interface )
    • jvmpi (JVM Profiler Interface )
    • runtime (Exact VM Runtime)

Notable command line tools in the $JDK/bin directory have corresponding subcomponents.

Some aspects of JDK infrastructure directly affect JDK Hg repositories, but other do not.

  • infrastructure
    • build (See build-dev and build-infra-dev.)
    • licensing (Covers updates to the third party readme, licenses, and similar files.)
    • release_eng (Release engineering)
    • staging (Staging of web pages related to JDK releases.)

The specification subcomponent encompasses the formal language and virtual machine specifications.

  • specification
    • language (The Java Language Specification)
    • vm (The Java Virtual Machine Specification)

The code for the deploy and install areas is not currently included in OpenJDK.

  • deploy
    • deployment_toolkit
    • plugin
    • webstart
  • install
    • auto_update
    • install
    • servicetags

In the JDK, there are a number of cross-cutting concerns whose organization is essentially orthogonal to other areas. Since these areas generally have dedicated teams working on them, it is easier to find bugs of interest if these bugs are grouped first by their cross-cutting component rather than by the affected technology.

  • docs
    • doclet
    • guides
    • hotspot
    • release_notes
    • tools
    • tutorial
  • embedded
    • build
    • hotspot
    • libraries
  • globalization
    • locale-data
    • translation
  • performance
    • hotspot
    • libraries

The list of subcomponents will no doubt grow over time, but my inclination is to resist that growth since the addition of each subcomponent makes the system as a whole more complicated and harder to use.

When the system gets closer to being externalized, I plan to post more blog entries describing recommended use of various custom fields in the JDK project.

Thursday Oct 25, 2012

JDK bug migration: now backed by JIRA

The JDK bug migration from a Sun legacy system to JIRA has reached another planned milestone: the data displayed on is now backed by JIRA rather than by the legacy system. Besides maintaining the URLs to old bugs, bugs filed since the migration to JIRA are now visible too.

The basic information presented about a bug is the same as before, but reformatted and using JIRA terminology:

  • Instead of a "category", a bug now has a "component / subcomponent" classification. As outlined previously, part of the migration effort was reclassifying bugs according to a new classification scheme; I'll write more about the new scheme in a subsequent blog post.

  • Instead of a list of JDK versions a bug is "reported against," there is a list of "affected versions." The names of the JDK versions have largely been regularized; code names like "tiger" and "mantis" have been replaced by the release numbers like "5.0" and "1.4.2".

  • Instead of "release fixed," there are now "Fixed Versions."

  • The legacy system had many fields that could hold a sequence of text entries, including "Description," "Workaround", and "Evaluation." JIRA instead only has two analogous fields labeled as "Description" and a unified stream of "Comments."

Nearly coincident with switching to JIRA, we also enabled an agent which automatically updates a JIRA issue in response to pushes into JDK-related Hg repositories. These comments include the changeset URL, the user making the push, and a time stamp. These comments are first added when a fix is pushed to a team integration repository and then added again when the fix is pushed into the master repository for a release.

We're still in early days of production usage of JIRA for JDK bug tracking, but the transition to production went smoothly and over 1,000 new issues have already been filed. Many other facets of the migration are still in the works, including hosting new incidents filed at in a tailored incidents project in JIRA.

Wednesday Sep 26, 2012

JDK bug migration milestone: JIRA now the system of record

I'm pleased to announce the OpenJDK bug database migration project has reached a significant milestone: the JDK has switched from the legacy Sun "bugtraq" system to a new internal JIRA instance as the system of record for our bug tracking. This completes the initial phase of the previously described plan of getting OpenJDK onto an externally visible and writable bug tracker. The identities contained in the current system include recognized OpenJDK contributors.

The bug migration effort to date has been sizable in multiple dimensions. There are around 140,000 distinct issues imported into the JDK project of the JIRA instance, nearly 165,000 if backport issues to track multiple-release information are included. Separately, the Code Tools OpenJDK project has its own JIRA project populated with several thousands existing bugs. Once the OpenJDK JIRA instance is externalized, approved OpenJDK projects will be able to request the creation of a JIRA project for issue tracking.

There are many differences in the schema used to model bugs between the legacy bug system and the schema for the new JIRA projects. We've favored simplifications to the existing system where possible and, after much discussion, we've settled on five main states for the OpenJDK JIRA projects:

  • New
  • Open
  • In progress
  • Resolved
  • Closed

The Open and In-progress states can have a substate Understanding field set to track whether the issues has its "Cause Known" or "Fix understood". In the closed state, a Verification field can indicate whether a fix has been verified, unverified, or if the fix has failed.

At the moment, there will be very little externally visible difference between JIRA for OpenJDK and the legacy system it replaces. One difference is that bug numbers for newly filed issues in the JIRA JDK project will be 8000000 and above. If you are working with JDK Hg repositories, update any local copies of jcheck to the latest version which recognizes this expanded bug range. (The bug numbers of existing issues have been preserved on the import into JIRA). Relatively soon, we plan for the pages published on to be generated from information in JIRA rather than in the legacy system. When this occurs, there will be some differences in the page display and the terminology used will be revised to reflect JIRA usage, such as referring to the "component/subcomponent" of an issue rather than its "category". The exact timing of this transition will be announced when it is known.

We don't currently have a firm timeline for externalization of the JIRA system. Updates will be provided as they become available. However, that is unlikely to happen before JavaOne next week!

Tuesday Jun 05, 2012

Moving monarchs and dragons: migrating the JDK bugs to JIRA

Among insects, monarch butterflies and dragonflies have the longest migrations; migrating JDK bugs involves a long journey as well! As previously announced by Mark back in March, we've been working according to a revised plan to transition the JDK bug management from Sun's legacy system to initially an Oracle-internal JIRA instance which is afterward made visible and usable externally. I've been busily working on this project for the last few months and the team has made good progress on many aspects of the effort:

  • JDK bugs will be imported into JIRA regardless of age; bugs will also be imported regardless of state, including closed bugs. Consequently, the JDK bug project will start pre-populated with over 100,000 existing bugs, some dating all the way back to 1994. This will allow a continuity of information and allow new issues to be linked to old ones.

  • Using a custom import process, the Sun bug numbers will be preserved in JIRA. For example, the Sun bug with bug number 4040458 will become "JDK-4040458" in JIRA. In JIRA the project name, "JDK" in our case, is part of the bug's identifier. Bugs created after the JIRA migration will be numbered starting at 8000000; bugs imported from the legacy system have numbers ranging between 1000000 and 79999999.

  • We're working with the team to try to maintain continuity of the ability to both read JDK bug information as well as to file new incidents. At least for now, the overall architecture of will be the same as it is today: it will be a gateway bridging to an Oracle-internal system, but the internal system will change to JIRA from the legacy database. Generally we are aiming to preserve the visibility of bugs currently viewable on; however, bugs in areas not related to the JDK will not be visible after the transition to JIRA. New incoming incidents will be sent to a separate JIRA project for initial triage before possibly being moved into the JDK project.

  • JDK bug management leans heavily on being able to track the state of bugs in multiple releases, especially to coordinate delivering synchronized security releases (known as CPUs, critital patch updates, in Oracle parlance). For a security release, it is common for half a dozen or more release trains to be affected (for example, JDK 5, JDK 6 update, OpenJDK 6, JDK 7 update, JDK 8, virtual releases for HotSpot express, etc.). We've determined we need to track at least the tuple of (release, responsible engineer/assignee for the release, status in the release) for the release trains a fix is going into. To do this in JIRA, we are creating a separate port/backport issue type along with a custom link type to allow the multiple release information to be easily grouped and presented together.

  • The Sun legacy system had a three-level classification scheme, product, category, and subcategory. Out of the box, JIRA only has a one-level classification, component. We've implemented a custom second-level classification, subcomponent. As part of the bug migration we've taken the opportunity to think about how bugs should be grouped under a two-level system and we'll the new system will be simpler and more regular. The main top-level components of the JDK product will include:

    • core-libs
    • client-libs
    • deploy
    • install
    • security-libs
    • other-libs
    • tools
    • hotspot
    For the libs areas, the primary name of the subcomponent will be the package of the API in question. In the core-libs component, there will be subcomponents like:
    • java.lang
    • java.lang.class_loading
    • java.math
    • java.util
    • java.util:i18n
    In the tools component, subcomponents will primarily correspond to command names in $JDK/bin like, jar, javac, and javap.

    The first several bulk imports of the JDK bugs into JIRA have gone well and we're continuing to refine the import to have greater fidelity to the current data, including by reconstructing information not brought over in a structured fashion during the previous large JDK bug system migration back in 2004.

    We don't currently have a firm timeline of when the new system will be usable externally, but as it becomes available, I'll share further information in follow-up blog posts.




« June 2016

No bookmarks in folder