Sunday Jun 24, 2012

JPRT: A Build & Test System


A while back I did a little blogging on a system called JPRT, the hardware used and a summary on my weblog. This is an update on the JPRT system.

JPRT ("JDK Putback Reliablity Testing", but ignore what the letters stand for, I change what they mean every day, just to annoy people :\^) is a build and test system for the JDK, or any source base that has been configured for JPRT. As I mentioned in the above blog, JPRT is a major modification to a system called PRT that the HotSpot VM development team has been using for many years, very successfully I might add. Keeping the source base always buildable and reliable is the first step in the 12 steps of dealing with your product quality... or was the 12 steps from Alcoholics Anonymous... oh well, anyway, it's the first of many steps. ;\^)

Internally when we make changes to any part of the JDK, there are certain procedures we are required to perform prior to any putback or commit of the changes. The procedures often vary from team to team, depending on many factors, such as whether native code is changed, or if the change could impact other areas of the JDK. But a common requirement is a verification that the source base with the changes (and merged with the very latest source base) will build on many of not all 8 platforms, and a full 'from scratch' build, not an incremental build, which can hide full build problems. The testing needed varies, depending on what has been changed.

Anyone that was worked on a project where multiple engineers or groups are submitting changes to a shared source base knows how disruptive a 'bad commit' can be on everyone. How many times have you heard:
"So And So made a bunch of changes and now I can't build!".
But multiply the number of platforms by 8, and make all the platforms old and antiquated OS versions with bizarre system setup requirements and you have a pretty complicated situation (see

We don't tolerate bad commits, but our enforcement is somewhat lacking, usually it's an 'after the fact' correction. Luckily the Source Code Management system we use (another antique called TeamWare) allows for a tree of repositories and 'bad commits' are usually isolated to a small team. Punishment to date has been pretty drastic, the Queen of Hearts in 'Alice in Wonderland' said 'Off With Their Heads', well trust me, you don't want to be the engineer doing a 'bad commit' to the JDK. With JPRT, hopefully this will become a thing of the past, not that we have had many 'bad commits' to the master source base, in general the teams doing the integrations know how important their jobs are and they rarely make 'bad commits'. So for these JDK integrators, maybe what JPRT does is keep them from chewing their finger nails at night. ;\^)

Over the years each of the teams have accumulated sets of machines they use for building, or they use some of the shared machines available to all of us. But the hunt for build machines is just part of the job, or has been. And although the issues with consistency of the build machines hasn't been a horrible problem, often you never know if the Solaris build machine you are using has all the right patches, or if the Linux machine has the right service pack, or if the Windows machine has it's latest updates. Hopefully the JPRT system can solve this problem. When we ship the binary JDK bits, it is SO very important that the build machines are correct, and we know how difficult it is to get them setup. Sure, if you need to debug a JDK problem that only shows up on Windows XP or Solaris 9, you'll still need to hunt down a machine, but not as a regular everyday occurance.

I'm a big fan of a regular nightly build and test system, constantly verifying that a source base builds and tests out. There are many examples of automated build/tests, some that trigger on any change to the source base, some that just run every night. Some provide a protection gateway to the 'golden' source base which only gets changes that the nightly process has verified are good. The JPRT (and PRT) system is meant to guard the source base before anything is sent to it, guarding all source bases from the evil developer, well maybe 'evil' isn't the right word, I haven't met many 'evil' developers, more like 'error prone' developers. ;\^) Humm, come to think about it, I may be one from time to time. :\^{ But the point is that by spreading the build up over a set of machines, and getting the turnaround down to under an hour, it becomes realistic to completely build on all platforms and test it, on every putback. We have the technology, we can build and rebuild and rebuild, and it will be better than it was before, ha ha... Anybody remember the Six Million Dollar Man? Man, I gotta get out more often.. Anyway, now the nightly build and test can become a 'fetch the latest JPRT build bits' and start extensive testing (the testing not done by JPRT, or the platforms not tested by JPRT).

Is it Open Source? No, not yet. Would you like to be? Let me know. Or is it more important that you have the ability to use such a system for JDK changes?

So enough blabbering on about this JPRT system, tell me what you think.
And let me know if you want to hear more about it or not.

Stay tuned for the next episode, same Bloody Bat time, same Bloody Bat channel. ;\^)


Tuesday Apr 17, 2012

JDK Build Musings

Here are some somewhat random musings on building the jdk and various build system observations. It might be observed that some of these may sound like whining, I can assure you that whining is not allowed in my blog, only constructive criticism, it's everyone else that is whining. :^) Apologies for the length.

Build and test of the JDK has multiple dimensions, and I cannot say that I have them all covered here, it is after all, just a blog, and I am not endowed with any supernatural guidance here.

  • Continuous Build & Smoke Test
    Every component, every integration area, every junction point or merge point should be constantly built and smoke tested. I use the term smoke test because completely testing a product can take many weeks, and many tests do not lead themselves to being fully and reliably automated. The job of complete testing belongs to the testing organization. Smoke tests should provide a reasonable assurance that a product is not brain dead and had no major flaws to prevent further testing later on down the road. These smoke tests should be solid reliable tests, and any failure of these tests should signify a flaw in the changes to the product and raise major red flags for the individuals or teams that integrated any recent changes. Over the last year or more we have been trying to identify these tests for the jdk and it's not an easy task.
    Everyone cuts corners for the sake of productivity, it's just important to cut those corners with eyes wide open. The ideal would be that every changeset was known to have passed the build and smoke test, the reality is far less than that, but we know where we want to be.
  • Build and Test Machines
    The hardware/machine resources for a build and test system is cheap, and a bargain if it keeps all developers shielded from bad changes, finding issues as early in the development process as possible. But it is also true that hardware/machine resources do not manage themselves, so there is also an expense to managing the systems, some of it can be automated but not everything. Virtual machines can provide benefits here, but they also introduce complications.
  • Continuous Integration
    Depending on who you talk to, this can mean a variety of things. If it includes building and smoke testing before the changes are integrated, this is a fantastic thing. If people consider this to mean that developers should continuously integrate changes without any verifications whatsoever that the changes work and don't cause regressions, that could be a disaster. Some so called 'Wild West' projects purposely want frequent integrations with little or no build and test verifications. Granted, for a small tight team, going 'Wild West' can work very well, but not when the lives of innocent civilians are at risk. Wild West projects must be contained, all members must agile, wear armor, and be willing to accept the consequences of arbitrary changes sending a projectile through your foot.
  • Multiple Platforms
    The JDK is not a pure Java project and builds must be done on a set of different platforms. When I first started working on the JDK, it became obvious to me that this creates a major cost to the project, or any project. Expecting all developers to have access to all types of machines, be experienced with all platforms, and to take the time to build and test manually on all of them is silly, you need some kind of build and test system to help them with this. Building on multiple platforms (OS releases or architectures) is hard to setup, regardless of the CI system used, this is a significant issue that is often underestimated. Typically the CI system wants to try and treat all systems the same and the fact of the matter is, they are not, and somewhere these differences have to be handled very carefully. Pure Linux projects, or pure Windows projects will quickly become tethered to that OS and the various tools on them. Sometimes that tethering is good, sometimes not.
  • Multiple Languages
    Again, the JDK is not a pure Java project, many build tools try and focus on one language or set of languages. Building a product that requires multiple languages, where the components are tightly integrated, is difficult. Pure Java projects and pure C/C++ projects have a long list of tools and build assists in creating the resulting binaries. Less so for things like the JDK, where not only do we have C/C++ code in the JVM, but C/C++ code in the various JNI libraries, and C/C++ code in JVM agents (very customized). The GNU make tool is great for native code, the Ant tool is great for small Java projects, but there aren't many that work well in all cases.
    Picking the right tools for the JDK build is not a simple selection.
  • Multiple Compilers
    Using different C/C++ compilers requires a developer to be well aware of the limitations of all the compilers, and to some degree if the Java code is also being compiled by different Java compilers, the same awareness is needed. This is one of the reasons that builds and tests on all platforms is so important and also why changing compilers, even just new versions of the same compiler can make people paranoid.
  • Partial Builds
    With the JDK we have a history of doing what we call partial builds. The hotspot team rarely builds the entire jdk, but instead just builds hotspot (because that is the only thing they changed) and then places their hotspot in a vetted jdk image that was built by the Release Engineering team at the last build promotion. Dito for the jdk teams that don't work on hotspot, they rarely build hotspot. This was and still is considered a developer optimization, but is really only possible because of the way the JVM interfaces to the rest of the jdk, it rarely changes. To some degree, successful partial builds can indicate that the changes have not created an interface issue and can be considered somewhat 'compatible'.
    These partial builds create issues when there are changes in both hotspot and the rest of the jdk, where both changes need to be integrated at the same time, or more likely, in a particular order, e.g. hotspot integrates a new extern interface, later the jdk team integrates a change that uses or requires that interface, ideally after the hotspot changes have been integrated into a promoted build so everyone's partial builds have a chance of working.
    The partial builds came about mostly because of build time, but also because of the time and space needed to hold all the sources of parts of the product you never really needed. I also think there is a comfort effect by a developer not having to even see the sources to everything he or she doesn't care about. I'm not convinced that the space and time of getting the sources is that significant anymore, although I'm sure I would get arguments on that. The build speed could also become less of an issue as the new build infrastructure speeds up building and makes incremental builds work properly. But stay tuned on this subject, partial builds are not going away, but it's clear that life would be less complicated without them.
  • Build Flavors
    Similar to many native code projects we can build a product, or a debug, or a fastdebug version. My term for these is build flavors. My goal in the past is to make sure that the build process stays the same, and it's just the flavor that changes. Just like ice cream. ;^) (fastdebug == -O -g + asserts).
  • Plug and Play
    Relates to build flavors, it has been my feeling that regardless of the build flavor, the API's should not change. This allows for someone to take an existing product build, replace a few libraries with their debug versions, and run tests that will run with the best performance possible, and be able to debug in the area of interest. This cannot happen if the debug or fastdebug versions have different APIs, like MSVCRTD.DLL and MSVCRT.DLL.
  • Mercurial
    Probably applies to Git or any distributed Source Code Management system too.
    Face it, DSCM's are different. They provide some extremely powerful abilities over single repository model SCM's, but they also create unique issues. The CI systems typically want to treat these SCM systems just like SVN or CVS, in my opinion that is a mistake. I don't have any golden answers here, but anyone that has or does work with a distributed SCM, will struggle with CI systems that treat Mercurial like subversion.
    The CI systems are not the only ones. Many tools seem to have this concept of a single repository that holds all the changes, when in reality with a DSCM, the changes can be anywhere, and may or may not become part of any master repository.
  • Nested Repositories
    Not many projects have cut up the sources like the OpenJDK. There were multiple reasons for it, but it often creates issues for tools that either don't understand the concept of nested repositories, or just cannot handle them. It is not clear at this time how this will change in the future, but I doubt they will go away. But it has been observed by many that the lack of bookkeeping with regards to the state of all the repositories can be an issue. The build promotion tags may not be enough to track how all the repository changeset states line up with built together.
  • Managing Build and Test Dependencies
    Some build and test dependencies are just packages or products installed on a system, I've often called those "system dependencies". But many are just tarballs or zip bundles that needs to be placed somewhere and referred to. In my opinion, this is a mess, we need better organization here. Yeah yeah, I know someone will suggest Maven or Ivy, but it may not be that easy.
    We will be trying to address this better in the future, no detailed plans yet, but we must fix this and fix it soon.
  • Resolved Bugs and Changesets
    Having a quick connection between a resolved bug and the actual changes that fixed it is so extremely helpful that you cannot be without this. The connection needs to be both ways too. It may be possible to do this completely in the DSCM (Mercurial hooks), but in any case it is really critical to have that easy path between changes and bug reports. And if the build and test system has any kind of archival capability, also to that job data.
  • Automated Testing
    Some tests cannot be automated, some tests should not be automated, some automated tests should never be run as smoke tests, some smoke tests should never have been used as smoke tests, some tests can seriously mess up automation and even the system being used, ... No matter what, automating testing is not easy. You cannot treat testing like building, it has unique differences that cannot be ignored. If you want the test runs to be of the most benefit to a developer, you cannot stop on the first failure, you need to find all the failures. That failure list may be the evidence that links the failures to the change causing the failures, e.g. only tests using -server fail, or only tests on X64 systems fail, etc. At the same time, it is critical to drive home the fact that the smoke tests "should never fail", it is a slippery slope to start allowing smoke tests to fail. Sometimes, you need hard and fast rules on test failures, the smoke tests are those. If accepting failing smoke tests is a policy, that same policy needs to exclude the failing smoke test for everyone else so that life can go on for everyone else.
    In an automated build and test system, you have to protect yourself from the tests polluting the environment or the system and impacting the testing that follows it. Redirection of the user.home and properties can help, or at least making sure these areas are consistent freom test run to test run. Creating a separate and unique DISPLAY for X11 systems can also protect your test system from being impacted by automated GUI tests that can change settings of the DISPLAY.
  • Distributed Builds
    Unless you can guarantee that all systems used are producing the exact same binary bits, in my opinion, distributed builds are unpredictable and therefore unreliable. A developer might be willing to accept this potential risk, but a build and test system cannot, unless it has extremely tight control over the systems in use. It has been my experience that parallel compilations (GNU make -j N) on systems with many CPUs is a much preferred and more reliable way to speed up builds.
    However, if there are logically separate builds or sub-builds that can be distributed to different systems, that makes a great deal of sense. Having the debug, fastdebug, and product builds done on separate machines is a big win. Cutting up the product build can create difficult logistics in terms of pulling it all together into the final build image.
  • Distributed Tests
    Having one large testbase, and one large testlist, and requiring one very long testrun that can take multiple hours is not ideal. Generally, you want to make the testbase easy to install anywhere, and create batches of tests, so that using multiple machines of the same OS/arch can allow for the tests to be run in a distributed way. Getting the same testing done in a fraction of the time of running one large batch. If the batches are too small, you spend more time on test setup than running the test. The goal should be to run the smoke tests as fast as possible in the most efficient way possible, and more systems to test with should translate into the tests getting done sooner. This also allows for new smoke test additions as the tests run faster and faster. Unlike distributed building, the testing is not creating the product bits, and even if the various machines used are slightly different, that comes closer to matching the real world anyway. Ideally you would want to test all possible real world configurations, but we all know how impractical that is.
  • Killing Builds and Tests
    At some point, you need to be able to kill off a build or test, probably many builds and many tests on many different systems. This can be easy on some systems, and hard with others. Using Virtual Machines or ghosting of disk images provides a chance of just system shutdowns and restarts with a pristine state, but that's not simple logic to get right for all systems.
  • Automated System Updates
    Having systems do automatic updates while builds and tests are running is insane. The key to a good build and test system is reliability, you cannot have that if the system you are using is in a constant state of flux. System updates must be contained and done on a schedule that prevents any disturbance to the build and tests going on in the systems. It is completely unacceptable to change the system during a build or test.
  • AV Software
    AV software can be extremely disturbing to the performance of a build and test system. It is important, but must be done in a way that preserves the stability and reliability of the build and test system. The dynamic AV scanning is a great invention, but has the potential to disturb build and test processes in very negative ways.

Hopefully this blabbering provided some insights into the world of JDK build and test.


Wednesday Apr 11, 2012

JDK8 New Build Infrastructure

I unintentionally posted this before I verified everything, so once I have verified it all works, I'll updated this post. But this is what should work...

Most Interesting Builder in the World:

"I don't always build the jdk, but when I do, I prefer The New JDK8 Build Infrastructure. Stay built, my friends."

So the new Build Infrastructure changes have been integrated into the jdk8/build forest along side the older Makefiles (newer in makefiles/ and older ones in make/). The default is still the older makefiles.

Instructions can be found in the Build-Infra Project User Guide. The Build-Infra project's goal is to create the fastest build possible and correct many of the build issues we have been carrying around for years. I cannot take credit for much of this work, and wish to recognize the people who do so much work on this (and will probably still do more), see the New Build Infrastructure Changeset for a list of these talented and hard working JDK engineers. A big "THANK YOU" from me.

Of course, every OS and system is different, and the focus has been on Linux X64 to start, Ubuntu 11.10 X64 in particular. So there are at least a base set of system packages you need. I decided to use Ubuntu 12.04, hot off the press, you should run the following after getting into a root permissions situation (e.g. have run "sudo bash"):

  1. apt-get install aptitude
  2. aptitude update
  3. apt-get purge openjdk-6*
  4. aptitude install mercurial openjdk-7-jdk rpm ssh expect tcsh csh ksh gawk g++ ccache build-essential lesstif2-dev

Then get the jdk8/build sources:

  1. hg clone jdk8-build
  2. cd jdk8-build
  3. sh ./

Then do your build:

  1. chmod a+x common/bin/*
  2. cd common/makefiles
  3. bash ../autoconf/configure
  4. make

We still have lots to do, but this is a tremendous start.


Friday Nov 18, 2011

OpenJDK B24 Available

The latest OpenJDK6 release B24 is now available, details are at


Wednesday Jul 06, 2011

OpenJDK6 B23 Available

On July 6, 2011 the source bundle for OpenJDK 6 b23 was published at The sources are also available as Mercurial repositories from, as follows:

  • hg clone jdk6
  • cd jdk6
  • sh ./

Major changes include the latest round of security fixes, Hotspot 20, new jaxp bundle, new jaxws bundle, Release Process document addition, scripts used to do the release in make/scripts, and many other changes that have accumulated over a longer period than should have been allowed. The author takes all responsibility for the tardiness of this release, my apologies. Special thanks to Andrew John Hughes who syncs in HotSpot releases and generally helps shepherd changes into OpenJDK6 from the IcedTea project.

See the detailed list of all the changes in OpenJDK6 B23 for more information.

I will be doing test results differently than Joe did. Basically, if you build and test as follows

  • cd jdk6
  • make
  • make test
All the jdk regression tests run with make test should just pass. Over time we will fix the tests that have been excluded, possibly add more tests, and exclude tests that fail to demonstrate stability (with a bug filed against the test). Ultimately this should allow any OpenJDK6 developer to build and test his change without resorting to herculean archival measures to determine which tests were impacted by their changes.

Additional thanks to Joe Darcy for his direction, David Katleman for his Release Engineering work, and Lana Steuck for her reviews on the Release Process document.


Wednesday Jun 29, 2011

OpenJDK Systems Down

We are experiencing some problems with building cooling and had an emergency shutdown of many machines in our lab, including the openjdk machines that handle the repositories and mailing lists.



Various blogs on JDK development procedures, including building, build infrastructure, testing, and source maintenance.


« July 2016