Sunday Jun 24, 2012

JPRT: A Build & Test System


A while back I did a little blogging on a system called JPRT, the hardware used and a summary on my weblog. This is an update on the JPRT system.

JPRT ("JDK Putback Reliablity Testing", but ignore what the letters stand for, I change what they mean every day, just to annoy people :\^) is a build and test system for the JDK, or any source base that has been configured for JPRT. As I mentioned in the above blog, JPRT is a major modification to a system called PRT that the HotSpot VM development team has been using for many years, very successfully I might add. Keeping the source base always buildable and reliable is the first step in the 12 steps of dealing with your product quality... or was the 12 steps from Alcoholics Anonymous... oh well, anyway, it's the first of many steps. ;\^)

Internally when we make changes to any part of the JDK, there are certain procedures we are required to perform prior to any putback or commit of the changes. The procedures often vary from team to team, depending on many factors, such as whether native code is changed, or if the change could impact other areas of the JDK. But a common requirement is a verification that the source base with the changes (and merged with the very latest source base) will build on many of not all 8 platforms, and a full 'from scratch' build, not an incremental build, which can hide full build problems. The testing needed varies, depending on what has been changed.

Anyone that was worked on a project where multiple engineers or groups are submitting changes to a shared source base knows how disruptive a 'bad commit' can be on everyone. How many times have you heard:
"So And So made a bunch of changes and now I can't build!".
But multiply the number of platforms by 8, and make all the platforms old and antiquated OS versions with bizarre system setup requirements and you have a pretty complicated situation (see

We don't tolerate bad commits, but our enforcement is somewhat lacking, usually it's an 'after the fact' correction. Luckily the Source Code Management system we use (another antique called TeamWare) allows for a tree of repositories and 'bad commits' are usually isolated to a small team. Punishment to date has been pretty drastic, the Queen of Hearts in 'Alice in Wonderland' said 'Off With Their Heads', well trust me, you don't want to be the engineer doing a 'bad commit' to the JDK. With JPRT, hopefully this will become a thing of the past, not that we have had many 'bad commits' to the master source base, in general the teams doing the integrations know how important their jobs are and they rarely make 'bad commits'. So for these JDK integrators, maybe what JPRT does is keep them from chewing their finger nails at night. ;\^)

Over the years each of the teams have accumulated sets of machines they use for building, or they use some of the shared machines available to all of us. But the hunt for build machines is just part of the job, or has been. And although the issues with consistency of the build machines hasn't been a horrible problem, often you never know if the Solaris build machine you are using has all the right patches, or if the Linux machine has the right service pack, or if the Windows machine has it's latest updates. Hopefully the JPRT system can solve this problem. When we ship the binary JDK bits, it is SO very important that the build machines are correct, and we know how difficult it is to get them setup. Sure, if you need to debug a JDK problem that only shows up on Windows XP or Solaris 9, you'll still need to hunt down a machine, but not as a regular everyday occurance.

I'm a big fan of a regular nightly build and test system, constantly verifying that a source base builds and tests out. There are many examples of automated build/tests, some that trigger on any change to the source base, some that just run every night. Some provide a protection gateway to the 'golden' source base which only gets changes that the nightly process has verified are good. The JPRT (and PRT) system is meant to guard the source base before anything is sent to it, guarding all source bases from the evil developer, well maybe 'evil' isn't the right word, I haven't met many 'evil' developers, more like 'error prone' developers. ;\^) Humm, come to think about it, I may be one from time to time. :\^{ But the point is that by spreading the build up over a set of machines, and getting the turnaround down to under an hour, it becomes realistic to completely build on all platforms and test it, on every putback. We have the technology, we can build and rebuild and rebuild, and it will be better than it was before, ha ha... Anybody remember the Six Million Dollar Man? Man, I gotta get out more often.. Anyway, now the nightly build and test can become a 'fetch the latest JPRT build bits' and start extensive testing (the testing not done by JPRT, or the platforms not tested by JPRT).

Is it Open Source? No, not yet. Would you like to be? Let me know. Or is it more important that you have the ability to use such a system for JDK changes?

So enough blabbering on about this JPRT system, tell me what you think.
And let me know if you want to hear more about it or not.

Stay tuned for the next episode, same Bloody Bat time, same Bloody Bat channel. ;\^)


Thursday Dec 10, 2009

Mercurial Forest: Pet Shell Trick of the Day

Now be careful with this, but here is a simple bash shell script that will do Mercurial hg commands on a forest pretty quickly. It assumes that the forests are no deeper than 3, e.g. \*/\*/\*/.hg. Every hg command is run in parallel shell processes, so very large forests might be dangerous, use at your own risk:


# Shell script for a fast parallel forest command

rm -f -r ${tmp}
mkdir -p ${tmp}

# Remove tmp area on A. B. Normal termination
trap 'rm -f -r ${tmp}' EXIT

# Only look in specific locations for possible forests (avoids long searches)
hgdirs=`ls -d ./.hg ./\*/.hg ./\*/\*/.hg ./\*/\*/\*/.hg 2>/dev/null`

# Derive repository names from the .hg directory locations
for i in ${hgdirs} ; do
  repos="${repos} `echo ${i} | sed -e 's@/.hg$@@'`"

# Echo out what repositories we will process
echo "# Repos: ${repos}"

# Run the supplied command on all repos in parallel, save output until end
for i in ${repos} ; do
  n=`expr ${n} '+' 1`
    cline="hg --repository ${i} $\*"
    echo "# ${cline}"
    eval "${cline}"
    echo "# exit code $?"
  ) > ${tmp}/repo.${n} ; cat ${tmp}/repo.${n} &

# Wait for all hg commands to complete

# Cleanup
rm -f -r ${tmp}

# Terminate with exit 0 all the time (hard to know when to say "failed")
exit 0

Run it like: hgforest status, or hgforest pull -u.

As always should any member of your IMF forest be caught or killed, the secretary will disavow all knowledge of your actions. This tape will self-destruct in five seconds. Good luck Jim.


Monday Jun 15, 2009

Warning Hunt: OpenJDK, NetBeans, Warnings, and FindBugs

Did you see Bill's FindBugs slides from JavaONE 2009? You should create some step by step directions on getting started with NetBeans, FindBugs and the OpenJDK. We need to get developers working on this.
Humm, Ok, I'll look into that.
Don't just "look into it", do it!
Ok ok already, I'll "do it".
And try and talk about how to fix warnings, and especially the FindBugs errors, maybe talk about some kind of best practices.
Yeah yeah, and I'll take care of the world peace problem too.
Smart ass, how did you ever get this job in the first place.
Everything else I wanted to do was illegal, creating software seemed like the most criminal thing I could do that wouldn't put be in jail.
Oh geez, why does this seem like a Dilbert comic? Get to work!
Aye aye Captain! Batton down the hatches, full speed ahead, we be looking for the White Whale...
"Drink, ye harpooners! drink and swear, ye men that man the deathful whaleboat's bow -- Death to Moby Dick!"
Oh for heavens sake, maybe you should look into a mental institution.

Ok folks here we go...

My sincere apologies to the Whales and Whale lovers out there, just remember Moby Dick is just a fictional story and I am in no way condoning the slaughter of those innocent and fantastic animals.

NetBeans, FindBugs, and Hunting Moby Dick

For a developer that wants to actually change the source code, the very best way to work with a tool like FindBugs is through an IDE.

  1. Get .

    Download and install NetBeans 6.5.1, I used the smallest "Java SE" version but any one of the "Java" versions will do. It installs quickly and easily. I would normally recommend the newer NetBeans 6.7, and I am trying 6.7 myself right now, but so far I have been unable to get the other steps below to work with 6.7, so I'd stick with 6.5.1 for now.

  2. Turbo charge it .

    Configure the NetBeans memory settings for the best performance, this is optional. See my NetBeans performance blog. Effectively we want to boost the memory sizes to improve the interactive experience. Of course, if you have less than 1Gb RAM, you may want to avoid this completely or adjust the -Xmx number down:

    Mac<1> mkdir -p ${HOME}/.netbeans/6.5/etc
    Mac<2> cat >> ${HOME}/.netbeans/6.5/etc/netbeans.conf
    netbeans_default_options="-J-Xms256m -J-Xmx768m -J-XX:PermSize=32m -J-XX:MaxPermSize=160m
    -J-Xverify:none -J-Dapple.laf.useScreenMenuBar=true -J-XX:+UseConcMarkSweepGC
    -J-XX:+CMSClassUnloadingEnabled -J-XX:+CMSPermGenSweepingEnabled"
    NOTE: Keep in mind that if you run FindBugs inside NetBeans as I am suggesting here, you may need to adjust the above -Xmx number even higher depending on what you are running FindBugs on (e.g. how many class files or the size of the jar file). Running FindBugs as a separate process allows you to avoid using NetBeans heap space, but you won't get the good integration with the IDE that I'm advertising here, and what I think you need to actively work on fixing the bugs. FindBugs does use up some heap space, just make sure you have enough.
  3. Get FindBugs .

    The NetBeans FindBugs plugin is a bit hard to find. You need to tell NetBeans where the UpdateCenter center is that has this plugin. The NetBeans plugin is not maintained by the FindBugs developers. To do this you need to start up NetBeans 6.5.1 and do the following:

    1. Select "Tools->Plugins".
    2. Select "Settings".
    3. Select "Add"
    4. Enter any name you want in the "Name:" field, and for the "URL:" field use:
      then select "OK".
    5. Select "Available Plugins", select the "SQE Java" plugin, select "Install", and go through the plugin install process.
    You should now see several new icons in NetBeans, specifically the icons for the SQE site , the FindBugs tool , and the PMD tool . My experience with FindBugs has been very positive and I trust this tool completely, rarely has it been wrong or given bad advice. On PMD, you will note that the PMD tool has the image of a gun in it's icon, for a good reason, be very careful with PMD, shooting yourself in the foot is pretty easy to do.
  4. Find a NetBeans project to play with. My first example is jmake, a small Java project available from Get yourself a clone with:
    hg clone ${HOME}/jmake
    Then use the NetBeans "Open Project..." to open up this directory ${HOME}/jmake as a NetBeans project. It should load up fine, build cleanly, and you can try using FindBugs to look at and even fix some of the bugs.
    Note: You need to be a developer member of the jmake project to push changes back.

    Or you can get sources, and either find a NetBeans project or create one. I chose to create one and picked the jdwpgen tool in the jdk/make/tools/ directory. So I did this:

    1. Cloned the OpenJDK sources (just a partial forest):
      hg clone ${HOME}/myjdk7
      cd ${HOME}/myjdk7
      hg clone
    2. Tell NetBeans to create a "New Project", pick "Java Project with Existing Sources", naming the project "jdwpgen", and I'll locate this throw away project at ${HOME}/jdwpgen, then select "Next".
    3. Then I tell it to use the source directory ${HOME}/myjdk7/jdk/make/tools/src/, and select "Next".
    4. Then I specify build/tools/jdwpgen/\*\* in the "Includes", and select "Finish" to complete the project definition. This process should work for any clean subset of Java sources in the OpenJDK, but I can't attest to that fact, yet.
    5. Try a build (should compile 50 files or so), run FindBugs, try and fix something.
      Note: You will need to be a member of the OpenJDK project and be able to create correct changesets to push changes back, this normally requires a BugID, and will require that certain changeset comment and whitespace rules are followed. (Yeah yeah, I know we are working on this, someday I'll be able to describe this better and it will all be open and completely logical ;\^). Anyone outside of Sun that has tried to push changes into OpenJDK will confirm how difficult this currently is, but we are working on it.

That's it, play with it, try fixing FindBugs errors or compiler warning errors. Aye, we didn't manage to kill Moby Dick, but we sure sunk in a few good harpoons.

Command Line Usage

The FindBugs and PMD tools can be run from the command line to create all kinds of reports. You will need to download and install FindBugs and/or PMD of course.

Some sample FindBugs command lines:

findbugs -maxHeap 512 -textui -effort:default -low -exitcode ${HOME}/jdwpgen/dist/jdwpgen.jar
findbugs -maxHeap 512 -textui -effort:default -low -exitcode -xml:withMessages -output ${HOME}/jdwpgen/build/findbugs.xml ${HOME}/jdwpgen/dist/jdwpgen.jar

Some sample PMD command lines: ${HOME}/myjdk7/jdk/make/tools/src text basic,imports,unusedcode ${HOME}/myjdk7/jdk/make/tools/src xml basic,imports,unusedcode > ${HOME}/jdwpgen/build/pmd_out.xml

I use the command line versions when I just want the data for statistic purposes or as part of a batch build process, which I'll talk about later.

Mac, NetBeans/Ant, and PATHs

FYI... anyone that works with Ant scripts (all NetBeans users) has found out that the best way to get the Ant <exec> to work well is to set PATH properly, but controlling PATH from a GUI launch is tricky. For the Mac see my previous post on NetBeans, Mac and PATHs, which explains the Mac tricks needed to set the PATH for NetBeans.

For Developers, Hudson IS The Answer...

Have you used Hudson? If you haven't, you should, I'll blog about it more very soon, but to get yourself started, do this:

  1. Download Hudson (hudson.war)
  2. In a separate shell window, run:
    java -jar hudson.war
  3. Go to http://localhost:8080/ and play with your very own Hudson, create a new job, browse the many plugins available,...

I'll talk about Hudson and how to connect it up to FindBugs, PMD, Warning Messages, etc. in a future blog...


Monday Apr 13, 2009

Source Repository Rules

A few commandments for dealing with Software repositories (source bases) and the build results of those repositories.

  1. There shalt not be binary files in the repository.
    Binary files (executables, native libraries, zip files, jar files, etc.) are NOT source and should not be managed in a source repository.
  2. Keep thy path names simple.
    Directory names and filenames in the repositories should never contain blanks or non-printing characters. Certain characters such as '$' should also be avoided.
  3. There shall be one newline convention.
    The contents of all source files should follow the standard unix conventions on newlines (no \^M's).
  4. Generated source files shall not be added as managed files.
    Source files generated during the build process should not be managed files in a repository.
  5. All output from the build shall be kept separated from the source.
    All files generated during the build should land in a well defined output only directory such as build/ or dist/. The src/ directory should never get written to during the build process.

Of course the commandments will change from time to time. Reminds me of this scene in Mel Brooks "History of the World Part II" movie, loved the ending "These 15 Commandments... oops... 10 Commandments!". ;\^)


Monday Feb 16, 2009

JavaFX Compiler Setup Files

So what happened to my Java and OpenJDK/JDK blogs for the past few months?

Yup, JavaFX bit me. I've been busy working on a special build project for JavaFX, including learning all about the Hudson continuous build system, Ant build script capabilities, and dealing with another forest of Mercurial repositories. Along the way I discovered some new build tricks which I will try and share in separate postings.

The OpenJFX Compiler Project is now sporting a new Mercurial repository (previously was in SubVersion), and a new setup repository that has taken up a great deal of my time. This OpenJFX Compiler project is the "open source" part of JavaFX right now.

The OpenJFX Compiler Setup Files is an open Mercurial repository that allows for building of the entire JavaFX product from a forest of repositories, not all open of course.

Now before you post a question asking why JavaFX isn't all open source, you would be asking the wrong person, I don't know and I have very little influence over this, see the open source statement here. Also see the OpenJFX Data Site for more information on what is visible in the OpenJFX Compiler project.

Some of the issues I tried to tackle with this new setup may relate to many other projects:

  • Dealing ant scripts on a very large project full of hundreds of ant scripts.
  • Dealing with ant and native compilation, including using GNU make, Mac xcodebuild, Visual Studio devenv, and the always painful Visual Studio vcvars32.bat environment variable setup on Windows.
  • Builds of a Mercurial forest, allowing maximum independence but sharing what makes sense to share.
  • Hudson setups and benefits, and the limitations of continuous build system with a distributed source code management system (Mercurial).
  • Ant tricks and limitations I found.
  • Source repository rules, what you should and should not put into your repository.

More on these topics later...


Monday Dec 08, 2008

OpenJDK6 Repositories!

I see it! It actually exists! Yes Virginia, There Is a Santa Claus!

Actually that's a whale we saw in Alaska this summer. No I'm not trying to insult Santa. ;\^)

Seriously, we have our first cut at generated OpenJDK6 repositories:

You can browse all 7 repositories:

Or you can clone them, or do a forest clone to get the entire forest:

hg fclone yourjdk6
(See the OpenJDK Developer Guide for more information on how to setup Mercurial and the forest extension).

A few important notes:

  • These should be treated as experimental and read-only, official ones should be next week
  • They should match the contents of the OpenJDK6 source bundles, except:
    • No control directory, these files are in the top repository now
    • Previously you had to 'cd control/make && gnumake', now just 'cd . && gnumake'
    • README-builds.html is in the top repository, it's movement has created a little confusion in the changesets, ultimately we will have one copy.
  • Contributed changes should be documented in the changeset comments, if the contribution information is missing please let me know
  • These repositories were created from the TeamWare workspaces and a set of patches and documentation on those patches, we may have to re-create them. If we re-create repositories again, the old ones will not be related to the new ones. So any changesets you create with your clones should be viewed as temporary until the final repositories are put in place.
  • The hotspot repository may be completely replaced when we upgrade to HS14, so when that happens you may need to re-clone the hotspot repository.

Please let me know if you see anything wrong with these repositories.

The target date for official repositories is next week, once it is official we can add more changesets to correct problems, but we can't go back and change the changesets already created.



Various blogs on JDK development procedures, including building, build infrastructure, testing, and source maintenance.


« April 2014