Monday Dec 07, 2009

Have I become a Fuzz Tester?

Everything was a little fuzzy in Livermore, California this morning:

A very rare snowfall for this area, we are at 500 feet above sea level, snow was falling as low as 200 feet early this morning. But that's just my "Fuzz" picture intro...

I read an article on Secure Software Testing and it talked about Fuzz Testing, and now I'm wondering if I've become a Fuzz Tester, and what exactly does that mean? :( Should I see a doctor? "Fuzz", what a funny word, back in the 70's that was one of the more polite slang words for the Police.

Years ago, when I worked on C compilers, one of the first things my fellow Fortran compiler developer did to test my C compiler was to feed it a Fortran source file as input. If my compiler core dumped instead of generating an error message, it would have failed the "garbage input" test. I consequently would cd into his home directory and run "f77 \*", but I could never get his damn Fortran compiler to crash, maybe it just liked eating garbage. ;\^) Anyway, it appears this kind of "garbage input" testing is a form of "Fuzz Testing", feed your app garbage input and see if you can get it to misbehave.

So back to my primary topic at hand, OpenJDK testing. Lately I've been trying to get to a point where running the jdk tests is predictable. I've done that by weeding out tests that are known failures or unpredictable, and running and re-running the same tests over and over and over, on different systems and with slightly different configurations (-server, -client, -d64).

After reading that Fuzz article, I've come to the conclusion that I've been doing a type of Fuzz Testing, by varying one of the inputs in random ways, the system itself. But that's a silly thought, that would make us all Fuzz Testers, because who runs tests with the exact same system state every time? Unless you saved a private virtual machine image and restarted the system every time, how could you ever be sure the system state was always the same? And even then, depending on how fast the various system services run, even a freshly started virtual machine could have a different state if you started a test 5 seconds later than the last time. And what about the network? There is no way to do that, well I'm sure someone will post a comment telling me how you can do that. And even if you could, what would be the point? If everything is exactly the same, of course it will do the same thing, right? So there is always Fuzz, and you always want some Fuzz, who really wants to be completely Fuzz-less? My brain is now getting Fuzzy. :\^(

When looking at the OpenJDK testing, I just want to be able to run a set of robust tests, tests that are immune to a little "system fuzz". In particular system fuzz created by the test itself or it's related tests, "self induced fuzz failures" seems to be a common problem. Sounds like some kind of disease, H1Fuzz1, keep that hand sanitizer ready. Is it reasonable to expect tests to be free of "self induced fuzz failures"? Or should testing guarantee a fuzz free environment and only run a test one at time?

I'm determined to avoid tests with H1Fuzz1, or find a cure. So I recently pushed some changes into the jdk7/tl forest jdk repository:

With more improvements on the way, and the same basic change is planned for OpenJDK6. This should make it easier to validate your jdk build by running only the jdk regression/unit tests in the repository that are free of H1Fuzz1. They also should run as quickly as possible. You will need a built jdk7 image and also the jtreg tool installed. To download and install the latest jtreg tool do the following:

export JT_HOME=`pwd`/jtreg

Build the complete jdk forest, or just the jdk repository:

cd jdk/make && gmake ALT_JDK_IMPORT_PATH=previously_built_jdk7_home all images

Then run all the H1Fuzz1-free tests:

cd jdk/test && gmake -k jdk_all [PRODUCT_HOME=jdk7_home]

There are various batches of tests, jdk_all runs all of them and if your machine can handle it, use gnumake -j 4 jdk_all to run up to 4 of the batches in parallel. Some batches are run with jtreg -samevm for faster results. The tests in the ProblemList.txt file are not run, and hopefully efforts are underway to reduce the size of this list by curing H1Fuzz1.

As to the accuracy of the ProblemList.txt file, it could be wrong, and I may have slandered some perfectly good tests by accusing them of having H1Fuzz1. My apologies in advance, let me know if any perfectly passing tests made the list and I will correct it. It is also very possible that I missed some tests, so if you run into tests that you suspect might have H1Fuzz1, we can add them to the list. On the other hand, curing H1Fuzz1 is a better answer. ;\^)

That's enough Fuzz Buzz on H1Fuzz1.


Wednesday Nov 25, 2009

Faster OpenJDK Build Tricks

Here are a few tips and tricks to get faster OpenJDK builds.

  • RAM

    RAM is cheap, if you don't have at least 2Gb RAM, go buy yourself some RAM for Xmas. ;\^)


    Use local disk if at all possible, the difference in build time is significant. This mostly applies to the repository or forest you are building (and where the build output is also landing). Also, to a lesser degree, frequently accessed items like the boot jdk (ALT_BOOTDIR). Local disk is your best choice, and if possible /tmp on some systems is even better.


    This make variable (or environment variable) is used by the jdk repository and should be set to the number of native compilations that will be run in parallel when building native libraries. It does not apply to Windows. This is a very limited use of the GNU make -j N option in the jdk repository, only addressing the native library building. A recommended setting would be the number of cpus you have or a little higher, but no more than 2 times the number of cpus. Setting this too high can tend to swamp a machine. If all the machines you use have at least 2 cpus, using a standard value of 4 is a reasonable setting. The default is 1.


    Similar to PARALLEL_COMPILE_JOBS, this one applies to the hotspot repository, however, hotspot uses the GNU make -j N option at a higher level in the Makefile structure. Since more makefile rules get impacted by this setting, there may be a higher chance of a build failure using HOTSPOT_BUILD_JOBS, although reports of problems have not been seen for a while. Also does not apply to Windows. A recommended setting would be the same as PARALLEL_COMPILE_JOBS. The default is 1.

  • NO_DOCS=true

    Skip the javadoc runs unless you really need them.

Hope someone finds this helpful.


Monday May 11, 2009

Fedora 9 and Mac VMware Fusion 2.0.4

UPDATE: (5/11/2009) Patches for openjdk6 were deleted from this blog, not needed any more if using the latest OpenJDK 6 sources.

Thought I would provide some notes on setting up a Fedora 9 VMware image on my Mac laptop using VMWare Fusion 2.0.4. I used the same steps for both Fedora 9 32bit (i386) and 64bit (x86_64), however I had some trouble with installing x86_64, even seemed to trigger a MacOS panic at one point after doing the yum update, not sure what that was all about, only happened once. This was strictly a local Fedora install, so I didn't need to deal with any of the networking issues of setting up a real physical machine.

I'll try and re-create the order of things as best I can:

  1. Create VMware Virtual Machine from Fedora 9 install iso image. I set it up to have a 20GB disk (You cannot change this disk size afterwards!). I'm using 768Mb RAM (512Mb caused slow builds) and during the install I asked for the "Software Development" packages.

  2. Update your system and make sure you have all you need. Logged in as root:

    yum install kernel kernel-headers kernel-devel
    yum install hg ksh tcsh csh cups cups-devel freetype freetype-devel lesstif-devel
    yum groupinstall "X Software Development" "Development Tools" "Java Development"
    yum update
    This will take a while. A reboot after you are all done would be a good idea.
  3. Install VMware tools. Once you extract out the VMware tools folder vmware-tools-distrib, once again logged in as root do the following:

    cd vmware-tools-distro
    The list of questions to answer is long and convoluted, mostly the default answer works fine, but in some cases it seems to think you are using a remote login and you have to say "yes" to continue the installation.
  4. Mouse problems: For some reason all my single clicks were being treated as double clicks, which drove me nuts. I found this posting which solved the problem, I use option 2 and edited the file /etc/X11/xorg.conf and added the following lines, logged in as root:

    Section "ServerFlags"
            Option      "AutoAddDevices" "false"
    A reboot of your virtual machine is necessary to fix this.
  5. The default limit on file descriptors is very low, to allow for a larger limit the following addition to the file /etc/security/limits.conf will increase that limit, again logged in as root:

    \* soft nofile 64000
    \* hard nofile 64000
    You need to logout and back in for these new limits to be available.
  6. Recently it was discovered that the upgraded kernel-headers package has trimmed down the files it delivers to /usr/include/linux/ (i.e. dirent.h) and although this doesn't impact OpenJDK building, it could impact builds of parts of the Sun JDK (plugin). So to avoid this missing include file problem, you have to do this last step because the above steps need the latest and matching kernel-headers files. To get the older kernel-headers package run:

    yum remove kernel-headers glibc-headers
    yum install kernel-headers-2.6.25 glibc-headers
    Bugs have been filed on the Sun JDK to see if we can break this dependency on the /usr/include/linux/ files.

That's the basic system setup. In addition I also setup my own home directory with the following so I can build the OpenJDK:

  1. Get webrev tool:

    mkdir -p ${HOME}/bin
    cd ${HOME}/bin
    chmod a+x webrev
  2. Get latest ant:

    mkdir -p ${HOME}/import/ant_home
    cd ${HOME}/import/ant_home
    tar -xzf apache\*.tar.gz
    mv apache-ant-1.7.1/\* .
  3. Get forest extension:

    mkdir -p ${HOME}/hgrepos
    cd ${HOME}/hgrepos
    hg clone hgforest
  4. Setup your ${HOME}/.hgrc file:

    cat > ${HOME}/.hgrc <<EOF
    username = ${USER}
    ssh = ssh -C
    groups = wheel
    clone = --pull
    fclone = --pull
    fetch = -m Merge
    ffetch = -m Merge
  5. Get OpenJDK7 sources (jdk7 build source forest):

    mkdir -p ${HOME}/hgrepos/jdk7
    cd ${HOME}/hgrepos/jdk7
    hg fclone jdk7-build
  6. Get OpenJDK6 sources (jdk6 master source forest):

    mkdir -p ${HOME}/hgrepos/jdk6
    cd ${HOME}/hgrepos/jdk6
    hg fclone jdk6-master

Now to see if I can build both OpenJDK7 and OpenJDK6:

# To get rid of a few sanity errors
export LANG

# My own private copy of ant
export ANT_HOME

# Use the JDK that is part of Fedora 9

# Add java and ant to the PATH
export PATH

# Go to the root of the jdk7 source forest
cd ${HOME}/hgrepos/jdk7/jdk7-build

# Build jdk7
#  Don't run javadoc, too slow, needs 1024Mb RAM minimum
make NO_DOCS=true

# Go to the root of the jdk6 source forest
cd ${HOME}/hgrepos/jdk6/jdk6-master

# Build jdk6
#  Don't run javadoc, too slow, needs 1024Mb RAM minimum
make NO_DOCS=true

SUCCESS! They both build.


Monday Dec 08, 2008

OpenJDK6 Repositories!

I see it! It actually exists! Yes Virginia, There Is a Santa Claus!

Actually that's a whale we saw in Alaska this summer. No I'm not trying to insult Santa. ;\^)

Seriously, we have our first cut at generated OpenJDK6 repositories:

You can browse all 7 repositories:

Or you can clone them, or do a forest clone to get the entire forest:

hg fclone yourjdk6
(See the OpenJDK Developer Guide for more information on how to setup Mercurial and the forest extension).

A few important notes:

  • These should be treated as experimental and read-only, official ones should be next week
  • They should match the contents of the OpenJDK6 source bundles, except:
    • No control directory, these files are in the top repository now
    • Previously you had to 'cd control/make && gnumake', now just 'cd . && gnumake'
    • README-builds.html is in the top repository, it's movement has created a little confusion in the changesets, ultimately we will have one copy.
  • Contributed changes should be documented in the changeset comments, if the contribution information is missing please let me know
  • These repositories were created from the TeamWare workspaces and a set of patches and documentation on those patches, we may have to re-create them. If we re-create repositories again, the old ones will not be related to the new ones. So any changesets you create with your clones should be viewed as temporary until the final repositories are put in place.
  • The hotspot repository may be completely replaced when we upgrade to HS14, so when that happens you may need to re-clone the hotspot repository.

Please let me know if you see anything wrong with these repositories.

The target date for official repositories is next week, once it is official we can add more changesets to correct problems, but we can't go back and change the changesets already created.



Various blogs on JDK development procedures, including building, build infrastructure, testing, and source maintenance.


« April 2014