Monday Dec 07, 2009

Have I become a Fuzz Tester?

Everything was a little fuzzy in Livermore, California this morning:

A very rare snowfall for this area, we are at 500 feet above sea level, snow was falling as low as 200 feet early this morning. But that's just my "Fuzz" picture intro...

I read an article on Secure Software Testing and it talked about Fuzz Testing, and now I'm wondering if I've become a Fuzz Tester, and what exactly does that mean? :( Should I see a doctor? "Fuzz", what a funny word, back in the 70's that was one of the more polite slang words for the Police.

Years ago, when I worked on C compilers, one of the first things my fellow Fortran compiler developer did to test my C compiler was to feed it a Fortran source file as input. If my compiler core dumped instead of generating an error message, it would have failed the "garbage input" test. I consequently would cd into his home directory and run "f77 \*", but I could never get his damn Fortran compiler to crash, maybe it just liked eating garbage. ;\^) Anyway, it appears this kind of "garbage input" testing is a form of "Fuzz Testing", feed your app garbage input and see if you can get it to misbehave.

So back to my primary topic at hand, OpenJDK testing. Lately I've been trying to get to a point where running the jdk tests is predictable. I've done that by weeding out tests that are known failures or unpredictable, and running and re-running the same tests over and over and over, on different systems and with slightly different configurations (-server, -client, -d64).

After reading that Fuzz article, I've come to the conclusion that I've been doing a type of Fuzz Testing, by varying one of the inputs in random ways, the system itself. But that's a silly thought, that would make us all Fuzz Testers, because who runs tests with the exact same system state every time? Unless you saved a private virtual machine image and restarted the system every time, how could you ever be sure the system state was always the same? And even then, depending on how fast the various system services run, even a freshly started virtual machine could have a different state if you started a test 5 seconds later than the last time. And what about the network? There is no way to do that, well I'm sure someone will post a comment telling me how you can do that. And even if you could, what would be the point? If everything is exactly the same, of course it will do the same thing, right? So there is always Fuzz, and you always want some Fuzz, who really wants to be completely Fuzz-less? My brain is now getting Fuzzy. :\^(

When looking at the OpenJDK testing, I just want to be able to run a set of robust tests, tests that are immune to a little "system fuzz". In particular system fuzz created by the test itself or it's related tests, "self induced fuzz failures" seems to be a common problem. Sounds like some kind of disease, H1Fuzz1, keep that hand sanitizer ready. Is it reasonable to expect tests to be free of "self induced fuzz failures"? Or should testing guarantee a fuzz free environment and only run a test one at time?

I'm determined to avoid tests with H1Fuzz1, or find a cure. So I recently pushed some changes into the jdk7/tl forest jdk repository:

http://hg.openjdk.java.net/jdk7/tl/jdk/rev/af9346401220

With more improvements on the way, and the same basic change is planned for OpenJDK6. This should make it easier to validate your jdk build by running only the jdk regression/unit tests in the repository that are free of H1Fuzz1. They also should run as quickly as possible. You will need a built jdk7 image and also the jtreg tool installed. To download and install the latest jtreg tool do the following:

wget http://www.java.net/download/openjdk/jtreg/promoted/b03/jtreg-4_0-bin-b03-31_mar_2009.zip
unzip jtreg-4_0-bin-b03-31_mar_2009.zip
export JT_HOME=`pwd`/jtreg

Build the complete jdk forest, or just the jdk repository:

gmake
-OR-
cd jdk/make && gmake ALT_JDK_IMPORT_PATH=previously_built_jdk7_home all images

Then run all the H1Fuzz1-free tests:

cd jdk/test && gmake -k jdk_all [PRODUCT_HOME=jdk7_home]

There are various batches of tests, jdk_all runs all of them and if your machine can handle it, use gnumake -j 4 jdk_all to run up to 4 of the batches in parallel. Some batches are run with jtreg -samevm for faster results. The tests in the ProblemList.txt file are not run, and hopefully efforts are underway to reduce the size of this list by curing H1Fuzz1.

As to the accuracy of the ProblemList.txt file, it could be wrong, and I may have slandered some perfectly good tests by accusing them of having H1Fuzz1. My apologies in advance, let me know if any perfectly passing tests made the list and I will correct it. It is also very possible that I missed some tests, so if you run into tests that you suspect might have H1Fuzz1, we can add them to the list. On the other hand, curing H1Fuzz1 is a better answer. ;\^)

That's enough Fuzz Buzz on H1Fuzz1.

-kto

Wednesday Nov 25, 2009

Faster OpenJDK Build Tricks

Here are a few tips and tricks to get faster OpenJDK builds.

  • RAM

    RAM is cheap, if you don't have at least 2Gb RAM, go buy yourself some RAM for Xmas. ;\^)

  • LOCAL DISK

    Use local disk if at all possible, the difference in build time is significant. This mostly applies to the repository or forest you are building (and where the build output is also landing). Also, to a lesser degree, frequently accessed items like the boot jdk (ALT_BOOTDIR). Local disk is your best choice, and if possible /tmp on some systems is even better.

  • PARALLEL_COMPILE_JOBS=N

    This make variable (or environment variable) is used by the jdk repository and should be set to the number of native compilations that will be run in parallel when building native libraries. It does not apply to Windows. This is a very limited use of the GNU make -j N option in the jdk repository, only addressing the native library building. A recommended setting would be the number of cpus you have or a little higher, but no more than 2 times the number of cpus. Setting this too high can tend to swamp a machine. If all the machines you use have at least 2 cpus, using a standard value of 4 is a reasonable setting. The default is 1.

  • HOTSPOT_BUILD_JOBS=N

    Similar to PARALLEL_COMPILE_JOBS, this one applies to the hotspot repository, however, hotspot uses the GNU make -j N option at a higher level in the Makefile structure. Since more makefile rules get impacted by this setting, there may be a higher chance of a build failure using HOTSPOT_BUILD_JOBS, although reports of problems have not been seen for a while. Also does not apply to Windows. A recommended setting would be the same as PARALLEL_COMPILE_JOBS. The default is 1.

  • NO_DOCS=true

    Skip the javadoc runs unless you really need them.

Hope someone finds this helpful.

-kto

Friday Aug 14, 2009

Warning Hunt: Hudson and Charts on FindBugs, PMD, Warnings, and Test Results

"Drink, ye harpooners! drink and swear, ye men that man the deathful whaleboat's bow -- Death to Moby Dick!"

Once again, my sincere apologies to the Whales and Whale lovers out there, just remember Moby Dick is just a fictional story and I am in no way condoning the slaughter of those innocent and fantastic animals.

Hudson and Trends

Using Hudson to watch or monitor trends in your projects is a fantastic tool, although my graphs in this example are not that interesting, you can see how this could help in keeping track of recent test, warnings, findbugs, and pmd trends:

This blog should demonstrate how you can reproduce what you see above.

To get Hudson and start it up, do this:

  1. Download Hudson (hudson.war)
  2. In a separate shell window, run:
    java -jar hudson.war

Congratulations, you now have Hudson running. Now go to http://localhost:8080/ to view your Hudson system. Initially you need to add some Hudson plugins, so you need to:

  1. Select Manage Hudson
  2. Select Manage Plugins
  3. Select Available
  4. Select the check boxes for the following plugins: Findbugs Plugin, Mercurial Plugin, and PMD Plugin
  5. Select Install at the bottom of the page.
  6. Select Restart Now

We need a small Java project to demonstrate the charting, and one that has been setup to create the necessary charting data. You will need Mercurial installed, so if hg version doesn't work, get one of the Mercurial binary packages installed first. I will use this small jmake project from kenai.com. So get yourself a mercurial clone repository with:

hg clone https://kenai.com/hg/jmake~mercurial ${HOME}/jmake
and verify you can build it from the command line first:
cd ${HOME}/jmake
ant import
ant
The ant import will download junit, findbugs, and pmd. Access to junit, findbugs, and pmd could be done in a variety of ways, this import mechanism is just happens to be the way this ant script is setup.

This jmake build.xml script is already setup to create the necessary xml files that Hudson will chart for junit, findbugs, and pmd. You may want to look at the build.xml file, specifically the lines at the end of the file for how this is done. Your build.xml needs to create the raw data (the xml files) for the charting to work. This build.xml file happens to be a NetBeans configured ant script so it can be used in NetBeans too.

Now let's create a Hudson job to monitor your jmake repository. Go to http://localhost:8080/ and:

  1. Select New Job
  2. Name the job, I'll use "jmake".
  3. Select Build a free-style software project.
  4. Select OK.

Next we need to configure the job. You should be sitting in the job configuration page now, this is what you need to do:

  1. Under Source Code Management, select Mercurial and enter the path ${HOME}/jmake.
  2. Under Build Triggers select Poll SCM and enter \*/5 \* \* \* \* so that the repository is checked every 5 minutes.
  3. Under Build select Add Build Step and Invoke Ant, add import as the ant target.
  4. Select Add Build Step again and Invoke Ant, leave the ant target blank.
  5. Select Publish JUnit test report result and provide the path build/test/results/TESTS-\*.xml
  6. Select Scan for compiler warnings.
  7. Select Publish PMD analysis report and provide the path build/pmd.xml.
  8. Select Publish Findbugs analysis report and provide the path build/findbugs.xml
  9. Finally hit the Save at the bottom of the job configuration page.

Lastly, you need to use the Build Now a few times to build up some build history before the charts show up.

This Hudson job will automatically run when changes are added to the repository.

Hope this helps. Using Hudson for this is a definite plus, everyone should invest some time in setting these kind of systems up.

-kto

Wednesday Dec 31, 2008

Where has JVMPI gone?

Have you seen this error before?


FATAL ERROR: JVMPI, an experimental interface, is no longer supported.
Please use the supported interface: the JVM Tool Interface (JVM TI).
For information on temporary workarounds contact: jvmpi_eol@sun.com

For a long time now, since we released JDK 1.5, we have been warning people that the VM profiling interface JVMPI is going away. Starting with the JDK 6 update 3 release (JDK6u3), it is gone for good.

If you really need JVMPI, your best bet is to use a JDK 1.5 or older release, and also find out about transitioning to JVM TI. More often than not, you have become dependent on a tool that uses JVMPI, in which case you should try and upgrade that tool to a version that uses JVM TI instead. But if you have written your own JVMPI code, see the JVMPI transition article at http://java.sun.com/developer/technicalArticles/Programming/jvmpitransition/ for help in transitioning to JVM TI.

NOTE: Getting this message indicates that JVMPI has been requested of the JVM. A request for JVMPI must be made prior to JVM initialization and regardless of whether JVMPI is eventually used at runtime, just the request for it will have a negative performance impact on your Java application. In most situations, JVMPI should never be requested unless some kind of performance work is being done and slower performance is considered acceptable. JVM TI does not have many of the JVMPI limitations.

A few references of interest:

-kto

Monday Dec 08, 2008

OpenJDK6 Repositories!

I see it! It actually exists! Yes Virginia, There Is a Santa Claus!

Actually that's a whale we saw in Alaska this summer. No I'm not trying to insult Santa. ;\^)

Seriously, we have our first cut at generated OpenJDK6 repositories:

http://hg.openjdk.java.net/jdk6/jdk6

You can browse all 7 repositories:

Or you can clone them, or do a forest clone to get the entire forest:

hg fclone http://hg.openjdk.java.net/jdk6/jdk6 yourjdk6
(See the OpenJDK Developer Guide for more information on how to setup Mercurial and the forest extension).

A few important notes:

  • These should be treated as experimental and read-only, official ones should be next week
  • They should match the contents of the OpenJDK6 source bundles, except:
    • No control directory, these files are in the top repository now
    • Previously you had to 'cd control/make && gnumake', now just 'cd . && gnumake'
    • README-builds.html is in the top repository, it's movement has created a little confusion in the changesets, ultimately we will have one copy.
  • Contributed changes should be documented in the changeset comments, if the contribution information is missing please let me know
  • These repositories were created from the TeamWare workspaces and a set of patches and documentation on those patches, we may have to re-create them. If we re-create repositories again, the old ones will not be related to the new ones. So any changesets you create with your clones should be viewed as temporary until the final repositories are put in place.
  • The hotspot repository may be completely replaced when we upgrade to HS14, so when that happens you may need to re-clone the hotspot repository.

Please let me know if you see anything wrong with these repositories.

The target date for official repositories is next week, once it is official we can add more changesets to correct problems, but we can't go back and change the changesets already created.

-kto

About

Various blogs on JDK development procedures, including building, build infrastructure, testing, and source maintenance.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today