Friday Aug 14, 2009

Warning Hunt: Hudson and Charts on FindBugs, PMD, Warnings, and Test Results

"Drink, ye harpooners! drink and swear, ye men that man the deathful whaleboat's bow -- Death to Moby Dick!"

Once again, my sincere apologies to the Whales and Whale lovers out there, just remember Moby Dick is just a fictional story and I am in no way condoning the slaughter of those innocent and fantastic animals.

Hudson and Trends

Using Hudson to watch or monitor trends in your projects is a fantastic tool, although my graphs in this example are not that interesting, you can see how this could help in keeping track of recent test, warnings, findbugs, and pmd trends:

This blog should demonstrate how you can reproduce what you see above.

To get Hudson and start it up, do this:

  1. Download Hudson (hudson.war)
  2. In a separate shell window, run:
    java -jar hudson.war

Congratulations, you now have Hudson running. Now go to http://localhost:8080/ to view your Hudson system. Initially you need to add some Hudson plugins, so you need to:

  1. Select Manage Hudson
  2. Select Manage Plugins
  3. Select Available
  4. Select the check boxes for the following plugins: Findbugs Plugin, Mercurial Plugin, and PMD Plugin
  5. Select Install at the bottom of the page.
  6. Select Restart Now

We need a small Java project to demonstrate the charting, and one that has been setup to create the necessary charting data. You will need Mercurial installed, so if hg version doesn't work, get one of the Mercurial binary packages installed first. I will use this small jmake project from kenai.com. So get yourself a mercurial clone repository with:

hg clone https://kenai.com/hg/jmake~mercurial ${HOME}/jmake
and verify you can build it from the command line first:
cd ${HOME}/jmake
ant import
ant
The ant import will download junit, findbugs, and pmd. Access to junit, findbugs, and pmd could be done in a variety of ways, this import mechanism is just happens to be the way this ant script is setup.

This jmake build.xml script is already setup to create the necessary xml files that Hudson will chart for junit, findbugs, and pmd. You may want to look at the build.xml file, specifically the lines at the end of the file for how this is done. Your build.xml needs to create the raw data (the xml files) for the charting to work. This build.xml file happens to be a NetBeans configured ant script so it can be used in NetBeans too.

Now let's create a Hudson job to monitor your jmake repository. Go to http://localhost:8080/ and:

  1. Select New Job
  2. Name the job, I'll use "jmake".
  3. Select Build a free-style software project.
  4. Select OK.

Next we need to configure the job. You should be sitting in the job configuration page now, this is what you need to do:

  1. Under Source Code Management, select Mercurial and enter the path ${HOME}/jmake.
  2. Under Build Triggers select Poll SCM and enter \*/5 \* \* \* \* so that the repository is checked every 5 minutes.
  3. Under Build select Add Build Step and Invoke Ant, add import as the ant target.
  4. Select Add Build Step again and Invoke Ant, leave the ant target blank.
  5. Select Publish JUnit test report result and provide the path build/test/results/TESTS-\*.xml
  6. Select Scan for compiler warnings.
  7. Select Publish PMD analysis report and provide the path build/pmd.xml.
  8. Select Publish Findbugs analysis report and provide the path build/findbugs.xml
  9. Finally hit the Save at the bottom of the job configuration page.

Lastly, you need to use the Build Now a few times to build up some build history before the charts show up.

This Hudson job will automatically run when changes are added to the repository.

Hope this helps. Using Hudson for this is a definite plus, everyone should invest some time in setting these kind of systems up.

-kto

Monday Jun 15, 2009

Warning Hunt: OpenJDK, NetBeans, Warnings, and FindBugs


Did you see Bill's FindBugs slides from JavaONE 2009? You should create some step by step directions on getting started with NetBeans, FindBugs and the OpenJDK. We need to get developers working on this.
Humm, Ok, I'll look into that.
Don't just "look into it", do it!
Ok ok already, I'll "do it".
And try and talk about how to fix warnings, and especially the FindBugs errors, maybe talk about some kind of best practices.
Yeah yeah, and I'll take care of the world peace problem too.
Smart ass, how did you ever get this job in the first place.
Everything else I wanted to do was illegal, creating software seemed like the most criminal thing I could do that wouldn't put be in jail.
Oh geez, why does this seem like a Dilbert comic? Get to work!
Aye aye Captain! Batton down the hatches, full speed ahead, we be looking for the White Whale...
"Drink, ye harpooners! drink and swear, ye men that man the deathful whaleboat's bow -- Death to Moby Dick!"
Oh for heavens sake, maybe you should look into a mental institution.

Ok folks here we go...

My sincere apologies to the Whales and Whale lovers out there, just remember Moby Dick is just a fictional story and I am in no way condoning the slaughter of those innocent and fantastic animals.

NetBeans, FindBugs, and Hunting Moby Dick

For a developer that wants to actually change the source code, the very best way to work with a tool like FindBugs is through an IDE.

  1. Get .

    Download and install NetBeans 6.5.1, I used the smallest "Java SE" version but any one of the "Java" versions will do. It installs quickly and easily. I would normally recommend the newer NetBeans 6.7, and I am trying 6.7 myself right now, but so far I have been unable to get the other steps below to work with 6.7, so I'd stick with 6.5.1 for now.

  2. Turbo charge it .

    Configure the NetBeans memory settings for the best performance, this is optional. See my NetBeans performance blog. Effectively we want to boost the memory sizes to improve the interactive experience. Of course, if you have less than 1Gb RAM, you may want to avoid this completely or adjust the -Xmx number down:

    
    Mac<1> mkdir -p ${HOME}/.netbeans/6.5/etc
    Mac<2> cat >> ${HOME}/.netbeans/6.5/etc/netbeans.conf
    netbeans_default_options="-J-Xms256m -J-Xmx768m -J-XX:PermSize=32m -J-XX:MaxPermSize=160m
    -J-Xverify:none -J-Dapple.laf.useScreenMenuBar=true -J-XX:+UseConcMarkSweepGC
    -J-XX:+CMSClassUnloadingEnabled -J-XX:+CMSPermGenSweepingEnabled"
    
    NOTE: Keep in mind that if you run FindBugs inside NetBeans as I am suggesting here, you may need to adjust the above -Xmx number even higher depending on what you are running FindBugs on (e.g. how many class files or the size of the jar file). Running FindBugs as a separate process allows you to avoid using NetBeans heap space, but you won't get the good integration with the IDE that I'm advertising here, and what I think you need to actively work on fixing the bugs. FindBugs does use up some heap space, just make sure you have enough.
  3. Get FindBugs .

    The NetBeans FindBugs plugin is a bit hard to find. You need to tell NetBeans where the UpdateCenter center is that has this plugin. The NetBeans plugin is not maintained by the FindBugs developers. To do this you need to start up NetBeans 6.5.1 and do the following:

    1. Select "Tools->Plugins".
    2. Select "Settings".
    3. Select "Add"
    4. Enter any name you want in the "Name:" field, and for the "URL:" field use:
      https://sqe.dev.java.net/updatecenters/sqe/updates.xml
      then select "OK".
    5. Select "Available Plugins", select the "SQE Java" plugin, select "Install", and go through the plugin install process.
    You should now see several new icons in NetBeans, specifically the icons for the SQE site , the FindBugs tool , and the PMD tool . My experience with FindBugs has been very positive and I trust this tool completely, rarely has it been wrong or given bad advice. On PMD, you will note that the PMD tool has the image of a gun in it's icon, for a good reason, be very careful with PMD, shooting yourself in the foot is pretty easy to do.
  4. Find a NetBeans project to play with. My first example is jmake, a small Java project available from kenai.com. Get yourself a clone with:
    hg clone https://kenai.com/hg/jmake~mercurial ${HOME}/jmake
    Then use the NetBeans "Open Project..." to open up this directory ${HOME}/jmake as a NetBeans project. It should load up fine, build cleanly, and you can try using FindBugs to look at and even fix some of the bugs.
    Note: You need to be a developer member of the jmake project to push changes back.

    Or you can get sources, and either find a NetBeans project or create one. I chose to create one and picked the jdwpgen tool in the jdk/make/tools/ directory. So I did this:

    1. Cloned the OpenJDK sources (just a partial forest):
      hg clone http://hg.openjdk.java.net/jdk7/jdk7 ${HOME}/myjdk7
      cd ${HOME}/myjdk7
      hg clone http://hg.openjdk.java.net/jdk7/jdk7/jdk
    2. Tell NetBeans to create a "New Project", pick "Java Project with Existing Sources", naming the project "jdwpgen", and I'll locate this throw away project at ${HOME}/jdwpgen, then select "Next".
    3. Then I tell it to use the source directory ${HOME}/myjdk7/jdk/make/tools/src/, and select "Next".
    4. Then I specify build/tools/jdwpgen/\*\* in the "Includes", and select "Finish" to complete the project definition. This process should work for any clean subset of Java sources in the OpenJDK, but I can't attest to that fact, yet.
    5. Try a build (should compile 50 files or so), run FindBugs, try and fix something.
      Note: You will need to be a member of the OpenJDK project and be able to create correct changesets to push changes back, this normally requires a BugID, and will require that certain changeset comment and whitespace rules are followed. (Yeah yeah, I know we are working on this, someday I'll be able to describe this better and it will all be open and completely logical ;\^). Anyone outside of Sun that has tried to push changes into OpenJDK will confirm how difficult this currently is, but we are working on it.

That's it, play with it, try fixing FindBugs errors or compiler warning errors. Aye, we didn't manage to kill Moby Dick, but we sure sunk in a few good harpoons.

Command Line Usage

The FindBugs and PMD tools can be run from the command line to create all kinds of reports. You will need to download and install FindBugs and/or PMD of course.

Some sample FindBugs command lines:

findbugs -maxHeap 512 -textui -effort:default -low -exitcode ${HOME}/jdwpgen/dist/jdwpgen.jar
findbugs -maxHeap 512 -textui -effort:default -low -exitcode -xml:withMessages -output ${HOME}/jdwpgen/build/findbugs.xml ${HOME}/jdwpgen/dist/jdwpgen.jar

Some sample PMD command lines:

pmd.sh ${HOME}/myjdk7/jdk/make/tools/src text basic,imports,unusedcode
pmd.sh ${HOME}/myjdk7/jdk/make/tools/src xml basic,imports,unusedcode > ${HOME}/jdwpgen/build/pmd_out.xml

I use the command line versions when I just want the data for statistic purposes or as part of a batch build process, which I'll talk about later.

Mac, NetBeans/Ant, and PATHs

FYI... anyone that works with Ant scripts (all NetBeans users) has found out that the best way to get the Ant <exec> to work well is to set PATH properly, but controlling PATH from a GUI launch is tricky. For the Mac see my previous post on NetBeans, Mac and PATHs, which explains the Mac tricks needed to set the PATH for NetBeans.

For Developers, Hudson IS The Answer...

Have you used Hudson? If you haven't, you should, I'll blog about it more very soon, but to get yourself started, do this:

  1. Download Hudson (hudson.war)
  2. In a separate shell window, run:
    java -jar hudson.war
  3. Go to http://localhost:8080/ and play with your very own Hudson, create a new job, browse the many plugins available,...

I'll talk about Hudson and how to connect it up to FindBugs, PMD, Warning Messages, etc. in a future blog...

-kto

Sunday Mar 02, 2008

NetBeans, Mercurial, Ant, Mac OS X, and getting the right PATH set

It took be some time, but I figured out how to change the environment variables that the launched Mac applications will get. Why? Because I wanted NetBeans to be running with a PATH environment variable setting that matched what Ant got when I used the build.xml file from the command line. When running the "<exec>" Ant task to use an executable, full paths or modifying the PATH is very platform specific, hard to maintain, and a huge pain. Having the PATH set properly in the environment is the best way.

Now I could launch "netbeans" from a command line and it would work, but here is the answer for setting the environment for applications launched:


<> cat ~/.MacOSX/environment.plist 
{
PATH = "/Users/ohair/ant/bin:/Users/ohair/findbugs/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/sbin:/sbin";
}

The directory ~/.MacOSX will need to be created. Apparently this file is read in at login time, so if you change it you will need to logout and log back in again for any change to make a difference. In my case I added the path to my Ant, my Findbugs, and /usr/local/bin which contains the Mercurial (hg) I want to have available in the PATH. I'm not sure NetBeans will actually use the version of Ant in the PATH, but that hasn't been an issue for me.

Why Does hg need to be in the PATH?

It was the need for /usr/local/bin (hg) in the path (/usr/local/bin) that got me started on all this because in my Ant file I wanted the build to automatically pull the version information out of the repository and make it available to the built product as a property setting. Effectively I need to run:


    # Get the last changeset
    hg tip --template '{node|short}\\n'
    # Get the latest tag with a Version string in it
    hg log -l 1 --template '{desc|firstline}\\n' -k "Version:"
    # Get the date of the last changeset
    hg tip --template '{date|shortdate}\\n'

I used Ant rules something like this:


    <target name="hgpropfile" description="Create property file">
        <exec executable="hg" outputproperty="hg.tip.id">
            <arg value="tip"/>
            <arg value="--template"/>
            <arg value="{node|short}\\n"/>
        </exec>
        <exec executable="hg" outputproperty="hg.last.tag.summary">
            <arg value="log"/>
            <arg value="-l"/>
            <arg value="1"/>
            <arg value="-k"/>
            <arg value="Version:"/>
            <arg value="--template"/>
            <arg value="{desc|firstline}\\n"/>
        </exec>
        <exec executable="hg" outputproperty="hg.tip.date">
            <arg value="tip"/>
            <arg value="--template"/>
            <arg value="{date|shortdate}\\n"/>
        </exec>
<!-- Indentation is critical here -->
<echo file="${dist.dir}/config-default.properties" append="false">
product.version=${hg.tip.date} ${hg.last.tag.summary} [${hg.tip.id}]
</echo>
    </target>

This property and it's value would need to be read in or made available at runtime, the actual Java code would just need to getProperty("product.version") to get the version string.

I used a tag to track the version code name, with the most recent tag containing "Version:" provides the product code name. The date and changeset id come from the tip, or most recent changeset.

Automating the creation of a new version code name tag can be done with a special Ant target used when needed. Effectively it needs to run:


    hg tag -f -m "Version: Name" TAG-YYYY-MM-DD
I just manually created a file with a few hundred code names and pick one based on the day of the year. This "AllVersions" file could look as simple as:
    Humor Risk (1921), previewed once and never released; thought to be lost
    The Cocoanuts (1929), released by Paramount Pictures
    Animal Crackers (1930), released by Paramount
    The House That Shadows Built (1931), released by Paramount (short subject)
    Monkey Business (1931), released by Paramount
    Horse Feathers (1932), released by Paramount
    Duck Soup (1933), released by Paramount
    A Night at the Opera (1935), released by MGM
    A Day at the Races (1937), released by MGM
    Room Service (1938), released by RKO Radio Pictures
    At the Circus (1939), released by MGM
    Go West (1940), released by MGM
    The Big Store (1941), released by MGM
    A Night in Casablanca (1946), released by United Artists
    Love Happy (1949), released by United Artists
    The Story of Mankind (1957), released by Warner Brothers
But ideally you would want enough code names in the list to avoid the name getting re-used too many times. The "hg tip --template '{node|short}\\n'" is your real version, these code names are just a way to help people quickly identify a version.

The Ant target looks something like:


    <target name="new_version" description="Create new version tag">
        <tstamp>
            <format property="date.year.day" pattern="D"/>
        </tstamp>
        <tstamp>
            <format property="date.ymd" pattern="yyyy-MM-dd"/>
        </tstamp>
        <property name="versions.file" value="AllVersions"/> 
        <exec executable="wc" outputproperty="version.count.temp" input="${versions.file}">
            <arg value="-l"/>
        </exec>
        <exec executable="sed" outputproperty="version.count" inputstring="${version.count.temp}">
           <arg value="-e"/>
           <arg value="s@\^[\\ ]\*\\([1-9][0-9]\*\\)$@\\1@"/>
        </exec>
        <exec executable="expr" outputproperty="version.selection">
            <arg value="${date.year.day}"/>
            <arg value="%"/>
            <arg value="${version.count}"/>
        </exec>
        <exec executable="sed" outputproperty="version.name" input="${versions.file}">
            <arg value="-e"/>
            <arg value="${version.selection},${version.selection}p"/>
            <arg value="-n"/>
        </exec>
        <exec executable="hg">
            <arg value="tag"/>
            <arg value="-f"/>
            <arg value="-m"/>
            <arg value="Version: ${version.name}"/>
            <arg value="TAG${date.ymd}"/>
        </exec>  
    </target>

Or in a Makefile you could do a similar thing:


DATE_YEAR_DAY:=$(shell date +%j)
DATE_YMD:=$(shell date +%Y-%m-%d)
VERSIONS_FILE=AllVersions
VERSION_COUNT:=$(shell cat $(VERSIONS_FILE) | wc -l | sed -e 's@\^[\\ ]\*\\([1-9][0-9]\*\\)$@\\1@')
VERSION_SELECTION:=$(shell expr $(DATE_YEAR_DAY) '%' $(VERSION_COUNT))
VERSION_NAME:=$(shell cat $(VERSIONS_FILE) | sed -n -e "$(VERSION_SELECTION),$(VERSION_SELECTION)p" -n)
new_version:
        hg tag -f -m "Version: $(VERSION_NAME)" TAG$(DATE_YMD)

Now creating tags like this may not be advised or necessary with all repositories, but the basic principle can work in many situations. For example, with the OpenJDK project, the Release Engineering people will create the major milestone tags, and using those you could effectively identify a JDK version with a name (e.g. JDK 7 Build 23), and an exact changeset id. The trick is to get the version information from the repository, into the build tool (make or ant), into the product installation or baked into the product executable, and then available at runtime by the product plus easily seen when looking at an installation of the product. One issue I see is dealing with the situation where you are building a plain source tree without the Mercurial data or the Mercurial tools, somehow when the plain source tree is created the version data would need to be left in the source bundle.

Hope this is of some use to people. I'm sure there might be a better way, so if anyone has any ideas please add your comments. Ultimately I'd like a product to be able to provide enough details to a user so that the original source tree could be made quickly available. Given the changeset id, the exact and complete source could be re-created with hg clone --rev, of course that gets more complicated with a forest, but still pretty simple.

-kto

Friday Feb 29, 2008

Why is NetBeans so slow? Here is a tip.

I use NetBeans 6 all the time on my Mac laptop and have learned how to configure it to run more effectively for me. NetBeans, like any Java application needs room to work, and the default memory settings for Java and NetBeans are usually not what I consider ideal. Of course how would they know what is ideal, you need to tell it. Granted my solution may not be ideal for others, but it's worth playing with. I'm using jdk5 update 13 on my Mac, but these same setting should apply to jdk6 also. And it's not just for The Mac or even NetBeans 6, this should work on any platform.


KellyMacBook<3> cat ~/.netbeans/6.0/etc/netbeans.conf
netbeans_default_options="-J-Xms256m -J-Xmx512m -J-XX:PermSize=32m -J-XX:MaxPermSize=160m
-J-Xverify:none -J-Dapple.laf.useScreenMenuBar=true -J-XX:+UseConcMarkSweepGC
-J-XX:+CMSClassUnloadingEnabled -J-XX:+CMSPermGenSweepingEnabled"

Any change to this file will require a reboot (there should be one line in the file above, I had to break it up to get the blog to display it in a readable way).

Also, turn on the View->Toolbars->Memory to keep track of the NetBeans memory usage, clicking on it will force a GC, and you can get a feeling for how much memory your NetBeans session needs, making adjustments to the maximums above. And my recommendation is that the maximum (-J-Xmx) setting should never be more than the RAM on the machine minus 512m, or system memory thrashing may occur.

Additional information can be found at the NetBeans FAQ on GC pauses and the NetBeans FAQ on configuration.

The above settings make a world of difference in the way NetBeans performs for me.

-kto

Tuesday May 08, 2007

NetBeans 6 Mercurial Village

NetBeans 6 and Mercurial

I just happened to have a picture of the Village where NetBeans 6 and it's new Mercurial plugin were created.

Ok, not really, I'm just kidding, this is actually Plimoth Plantation a re-creation of the original 1627 settlement, in Plymouth, Mass. But both are pretty cool, if you are ever in Massachusetts, make sure you put Plymouth on the list. It's quite educational.

As for NetBeans 6, it's much easier to access than Plimoth Village :\^) Just go to www.netbeans.org, download and install NetBeans 6 Preview (Milestone 9). Once you get it started, go to Tools->Plugins, find the Mercurial plugin and install it. The editor has changed quite a bit, but seems pretty solid so far. I have a small Java project called JPRT (maybe 150 source files, maybe 35,000 lines of Java code), that has been in Mercurial for quite some time now, and this NetBeans 6 and Mercurial plugin is just right for me. Very handy. I'm in the process of converting to NetBeans 6 now, should be interesting.

The complete OpenJDK was released today at JavaOne 2007 and is now available at openjdk.java.net. Along with the OpenJDK sources are some NetBeans 6 projects, which are documented at openjdk.java.net/groups/nb-projects. As of today, these projects haven't been tested with an OpenJDK Mercurial repository yet, we will be doing that in the next few months.

Granted we haven't released any OpenJDK Mercurial repositories just yet, but as soon as we catch our breath from doing the OpenJDK launch, we will start on the Mercurial transition.

By the way, a great deal of people have been working long hours to make this OpenJDK release at JavaOne 2007 happen. I've been involved in some of it myself, but it took a dedicated and talented team of people to make this all happen. It might seem like a trivial thing to open up some sources, but trust me, it wasn't trivial. My special thanks go out to the Release Engineering Team, who spent some long nights getting those OpenJDK source bundles ready.

-kto

About

Various blogs on JDK development procedures, including building, build infrastructure, testing, and source maintenance.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today