Wednesday Sep 28, 2011

Tip #9 - Advanced Debugger Attach

GlassFish has server-side commands, for instance create-instance, that in-turn call the client side.  It generally does this by calling asadmin's cousin:


with the command. 

But what if you need to debug the spawned client JVM?  Impossible?  No!  Trivial!

Simply do this:

edit nadmin (or nadmin.bat if you are using Windows) and add the java debugging args.


-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=1234

 Now simply run the server-side command and then leisurely attach a debugger to port 1234 (make sure you have a breakpoint waiting or you'll miss out). 

If you use NetBeans you can easily debug both server and client at the same time.

Don't forget to undo nadmin when you are done! 

Tuesday Sep 27, 2011

Tip #8 Watch Those RegEx calls -- they are S-L-O-W

I see this construct quite a bit:

String someString ....

This is probably a huge waste of time (probably because I didn't time it).  Not only does it do much much much more complex regular expression parsing, for no good reason, it is also much less readable then the "correct" way to do it:

String someString ....

Monday Sep 26, 2011

Tip #7 - Absolute Filenames in Java

I'm busy reading lots and lots of code lately because I need to implant DCOM support.  I just saw this:

  if (nodeDirFile.isAbsolute()) {
                return nodeDir;

Would this surprise you?

File f = new File("/foo/../foo/.././././././foo/././././././../foo/goo");

will return true.  That is an absolute path as far as Java IO is concerned.  I.e. if it BEGINS with something absolute -- then it is absolute.  Even if it's really really ugly.

What was probably desired is getCanonical().  There is one and only canonical filenames per file.  There are infinite "absolute" filenames per file.

Monday Aug 15, 2011

Tip #6 Windows Script to change directory QUICKLY

GlassFish's source tree, like any huge software has enormously long directory paths.  My goal is to try to avoid ever typing them in. Even with the shell auto-filling it still takes too much time, IMHO.

I created 2 scripts on my Windows systems:

1)  "cdn.bat"

cd \gf\main\nucleus\*%1*\*%2*

2) "cda.bat"

cd \gf\main\appserver\*%1*\*%2*


If I want to change my directory to,say flashlight-framework I just do this:

cdn fl fr

This command instantly moves me to cluster-cli

cdn clu cl

Friday Aug 12, 2011

Tip #5 Avoid RegEx When Unnecessary

I see this in GlassFish source code frequently:

string = string.replaceAll("\\\\", "/");

 Whoa!  FOUR backslashes?!?  Yes.  That's a regular expression.  What we see is FOUR but inside the string itself are TWO literal actual backslashes. The backslashes have to be doubled so that the RegEx parser knows that we are looking for a literal backslash.

What the author of this code really wants to do is something like this:

c:\a\b\c\d  ==>  c:/a/b/c/d

 This can easily be done without the more confusing regex notation and (I assume) the slower regex code like so:

string = string.replace('\\', '/');

Tuesday Aug 09, 2011

Tip #4: Stop and Think before you make that Class public

I run into this time-wasting situation fairly frequently:

I need to change the signature of a Java class, in GlassFish,  that has been declared public.  Uh-oh!  It's public so I can't just change it. 

So -- my first step is to grep through every single java file in GlassFish.  That means  7,772 files currently!

Now I have to wonder -- is the class being used from modules outside of GlassFish?  Uh-oh -- now I have to run massive automated tests to make sure I don't break something.

Ultimately you can never be totally sure that public class is not being used somewhere.  E.g. say the name of the class is FooGooHoo.  What if there is code doing a Class.forName("Foo" + "Goo" + "Hoo") ?  I'll never find it with a simple search. 

This is all expected stuff with the public keyword.  But what if the class is exclusively used inside of its own package?  Then I just  wasted a whole lot of my employer's money chasing down non-existent references. 

On the other hand if the class had its visibility set to pkg-private in the first place, all I had to do was to search inside that one package for uses.  Java guarantees no other package can see it.  And if other code somehow subverts that policy -- tough!!

Moral of the Story:

Only mark classes and methods as public when you are absolutely positively sure that it must be public.  When in doubt make it pkg-private and avoid protected too.

Monday Aug 08, 2011

Tip #3: Make Life Easier for Your Code Reviewer

There are probably a hundred applications that will package up code changes in various cool ways.  You could also write your own -- unfortunately it's one of those cases where the time you save using the tool will never approach the time you spent making the tool. 

I get code review requests fairly often and they are almost always like this:

 Below are the diffs for my change.  Please review

 And as promised the output of some diff tool is below in the email.  What if the context around the diffs isn't big enough?  What if I hate the diff tool formatting?  What if I want to compile and run the actual changes in my environment?

I can't do any of the above quickly and easily with the raw diffs embedded in the email.  Even copying and pasting them is usually a big pain because of the prepended plus signs.

 Here are my requirements for code review submitters (Please?!?)

  • Send out the actual changed source files
    • Make it very very easy to get those source file into the right directory.  Don't make me type in commands with endlessly long paths!
  • Send me the diffs as a file and have it appear directly in the email (maybe the change is easy and I don't need to waste time dealing with file attachments)

And here is the solution I use.  It is simple and old-fashioned and works great:

  1.  Create the following script (Windows version).  I use this script constantly when I'm working on source code.  That's why I named it "s.bat".  Usually I don't use the "dome.bat" file but I always create it just in case.  I sort the output to get all the modified files together.

    svn status | sort | tee dome.bat

  2. To submit the changes to a code reviewer, I want him or her to get the diffs file, and all changed files.  I collect the diffs in a file like so (make a script out of it!)
    svn diff > diffs.txt
  3. Here is the trick.  Edit the dome.bat file from step 1 above.  Delete the uninteresting lines and QUICKLY edit the Modified or Added lines to do this:
    jar cvfM mychanges.jar diffs.txt c:/some/gigantic/ugly/long/path/ c:/some/even/uglier/gigantic/long/path/
  4. Now I attach mychanges.jar to an email and also copy the diffs to the email message.  I tell the reviewer which directory to change to and how to unpack the source files.  E.g.
    "cd to core/kernel and run jar xvf mychanges.jar to overlay my changes in your installation"

It sounds like a big deal but it shouldn't take more than a couple minutes and it makes life more pleasant for the reviewer.  Note how the reviewer can do this:

jar cvfM src
jar xvf mychanges.jar
run his or her favorite diff tools
compile and build, look at the code in a debugger or whatever
when review is complete, jar xvf

Thursday Jul 28, 2011

Tip #2 Step Through Every Line of Code in a Debugger

This is so simple.  The place where I catch most of my bugs -- before the code is even checked-in.  


Step Through Every Line of Code You Write in a Debugger

 I'm frequently amazed at what I bump into during this pass-through.  I've found all sorts of embarrassing problems.  You never see them, though, because I caught them before checking in.

Nobody writes bug-free code.  But you can check-in bug-free (or nearly so) code.

Some people may counter with: "Oh ho!  I have no time to step through the code in a debugger!"  My answer is that the time it takes to  step through the code in a debugger is normally a tiny percentage of the time it takes to do all of these things:

  • Think up what to write
  • Write it
  • Fix compile errors etc.
  • Fix more obvious errors from simple tests
  • write unit tests
  • write dev tests
  • close an issue in Issue Tracker
  • check-in comments
  • Ask for a code review, get comments, respond. make changes

And I mean every line of code.  Modern java debuggers allow you to change values of variables, etc., to force your execution path into error handling code.

If you start do this zealously it may force you into picking up other good development habits too, that make it easier to step-through all of your code.

Dtrace of Clusters in GlassFish - A Quick Tutorial

This is the second blog in the series about DTrace and GlassFish 3.1.1  You should read the first blog before this one.

Continuing where I left off in the previous blog I would like to analyze the usage of MonApp in 2  different instances.  Specifically I did this:

  1. undeploy from DAS
    asadmin undeploy MonApp
  2. create a cluster
    asadmin create-cluster c1
  3. create two clustered instances
    asadmin create-local-instance --cluster c1 i1
    asadmin create-local-instance --cluster c1 i2
  4. deploy to the cluster
    asadmin deploy MonApp.war
  5. start the instances
    asadmin start-cluster c1
  6. OPTIONAL - turn the levels up to HIGH for all Monitoring Modules as in the previous blog.  Just add the args --target c1
  7. turn-on DTrace support
    asadmin enable-monitoring --target c1 --dtrace
  8. Load both off the URLs in a browser in 2 different tabs:
  9. Verify DTrace "sees" the MonApp Probes
    dtrace -l | grep foo
    39785 fooblog20527           samples                      ProbeServlet myProbe
    39841 fooblog20544           samples                      ProbeServlet myProbe
  10. Run list-instances to get the instance pids so you can tell the probes apart
    asadmin list-instances --long
    i1    localhost  24848  20527  c1        running
    i2    localhost  24849  20544  c1        running
  11. If you had a load balancer setup (I don't) you could start hammering the App and see which instances are servicing the calls.  What I do is start the dtrace script:
    dtrace -n 'fooblog*::: { printf("PID:%d APP:%s %s", pid, execname, copyinstr(arg0)); }'
  12. Now I sit in the browser and randomly reload the 2 pages for a while and look at DTrace output:
    dtrace: description 'fooblog*::: ' matched 2 probes
    CPU     ID                    FUNCTION:NAME
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #3
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #4
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #5
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #3
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #4
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #5
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #6
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #6
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #7
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #8
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #7
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #8
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #9
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #10
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #9


Introducing DTrace in GlassFish 3.1.1

This is the first in a series of blogs about Dtrace in GlassFish.  I will present concrete examples  in upcoming blogs.  This blog is an overview of this powerful new capability.

An entire book could be (and has been) written just to describe DTrace itself.   There are many many useful documents on the internet about DTrace and its usage.  At the end of this blog I've referenced a few good ones that I've read.

DTrace is a tool available currently on Solaris 10 and Mac OS X that will dynamically instrument already-running code  for the purposes of seeing what is going on inside software.  This includes the OS itself.  You can probe into kernel-mode in addition to user-mode. 

GlassFish automatically supports DTrace.  For every Probe that is registered in GlassFish, a corresponding DTrace probe is generated and made available to DTrace.  This includes any probes that you create yourself such as in your Applications.  See  [1] for a tutorial on this.

In order for the DTrace probes to be available you need to do 3 things from the GlassFish end:

  1. GlassFish must be running in JDK 7
  2. Dtrace must be enabled.  Simply run this command:
    asadmin enable-monitoring --dtrace
  3. You must have glassfish-dtrace.jar copied to the GlassFish modules directory.  Update center will set things up for you by doing the following
    1. You must use an official distribution:
      1. Open Source Distribution
      2. Oracle GlassFish Server Distribution
      3. Official Oracle GlassFish Server page is located here
    2. Run: <installdir>/bin/updatetool to bootstrap update clients.
    3. Run:  <installdir>/bin/pkg install glassfish-monitoring-dtrace@3.1.1 to install the DTrace package

Of course you also have to turn on the monitoring levels for what you are interested in.  To turn Monitoring up full-blast for everything run these commands (you can make one huge command out of it and stuff it into a script like I do):

asadmin enable-monitoring --target server --modules deployment=HIGH
asadmin enable-monitoring --target server --modules connector-connection-pool=HIGH
asadmin enable-monitoring --target server --modules connector-service=HIGH
asadmin enable-monitoring --target server --modules ejb-container=HIGH
asadmin enable-monitoring --target server --modules http-service=HIGH
asadmin enable-monitoring --target server --modules jdbc-connection-pool=HIGH
asadmin enable-monitoring --target server --modules jms-service=HIGH
asadmin enable-monitoring --target server --modules jvm=HIGH
asadmin enable-monitoring --target server --modules orb=HIGH
asadmin enable-monitoring --target server --modules thread-pool=HIGH
asadmin enable-monitoring --target server --modules transaction-service=HIGH
asadmin enable-monitoring --target server --modules web-container=HIGH
asadmin enable-monitoring --target server --modules jersey=HIGH
asadmin enable-monitoring --target server --modules jpa=HIGH
asadmin enable-monitoring --target server --modules security=HIGH
asadmin enable-monitoring --target server --modules web-services-container=HIGH

Obviously you have to be on an OS that has dtrace!

You can quickly and easily see that everything is registered correctly and available for running in DTrace scripts with this command:

dtrace -l | grep glassfish

This will show 151 or so built-in GlassFish probes.  Here is one line of the output:

39462 glassfish20383               web                        web-module webModuleStartedEvent
20383 is the Process ID of the GlassFish Server.  Whenever a web module is started the DTrace probe will be called with some arguments like the name of the web-module, etc.

The beauty of DTrace is that the DTrace probe inside of GlassFish is a no-op unless a DTrace script happens to be running.  In that case and in only that case the DTrace probe code gets dynamically instrumented to call into the OS's DTrace framework.  When the script ends, the probe gets un-instrumented.  There is almost no overhead unless a DTrace script is running.

There are limitless applications for this technology.  You can create on-the-fly scripts that attach to an already-running GlassFish server.  Everything can be enabled dynamically - the server never needs to be restarted for the magic to happen.  DTrace is so powerful it has its own programming language named "d".  It is beyond the scope of this blog to do more than chip at the surface of the DTrace engine.  

An example might be choosing a pair of bookend probes.  E.g. a start-transaction/end-transaction pair.  Or a HTTP service startRequest/endRequest pair.  You can easily have a DTrace script measure the time between the 2 calls and print out this information.  Since it is a programming language you can even save them into variables and crunch the data at the end of the script. 

Another example might be that you want to see every file that is opened - but only between a bookend pair (i.e. you are filtering the deluge of file openings to those that happen to occur during the time-slice of your bookend).

Create or download the app from [1].  Now add something more interesting to the webapp like, for instance, a form.  Now call a probe everytime the form is processed.  Pass a String the user enters in the form to the Probe.  That String will be available in real-time to a DTrace script!  This is a lot easier than it sounds.  I deployed the app (MonApp) from the blog[1] and then ran a super-simple DTrace script.  What the App does is call the probe method each time you refresh it.  wget is an easy way to do the refresh. It increments a counter and calls the probe with a string like so:  "Hello #XX".

I didn't even bother writing a script file - I just ran it from a command line:

dtrace -n 'fooblog*::: { printf("PID:%d APP:%s %s", pid, execname, copyinstr(arg0)); }'
dtrace: description 'fooblog*::: ' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #22
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #23
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #24
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #25
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #26
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #27

I got one output line every time I refreshed MonApp in a browser.  This is incredibly powerful!  A string was sent from the user in a browser to a Java WebApp which then sent it off to a native program, DTrace, that was looking for it with a sophisticated scripting language.  All of this can be done without ever restarting the GlassFish Server.  There are truly endless possibilities.  And once you kill the script -- all the instrumentation code is removed.

 In case you are wondering about fooblog*:::  -- fooblog is the main name I gave it.  The * is needed because the Monitoring Framework automatically appends the PID of the server (very handy if you are DTracing with multiple GlassFish servers simultaneously).  The ::: means match ANY probe that has the main name that starts with fooblog.  Whew.  Hard to explain in English, easy to explain with one command:

~/gf_other/value-add/monitoring/dtrace/dtrace-samples/dtrace-scripts>dtrace -l | grep fooblog
39554 fooblog20383           samples                      ProbeServlet myProbe

To be completely specific in the earlier command would be:

dtrace -n 'fooblog20383:samples:ProbeServlet:myProbe { ..... }

Dtrace uses the quadruple of names to make the namespace less unwieldy and easier to filter.


 [1] My blog about adding Monitoring probes to a WebApp

Tuesday Jul 19, 2011

Tip #-1 How Does GlassFish Pick the JVM to Run In?

 If you start a GlassFish Server by using asadmin, then you may be interested in precisely how the JVM that will run GlassFish Server is chosen.  Note that if you restart via the Admin Console (GUI) you are indirectly using asadmin to restart the server and the following rules apply as well.  If you start the Server some other way, like

java -jar glassfish.jar

then you are in charge of choosing the JVM yourself.

In order of precedence from high to low:

  1. The jdk location specified in domain.xml in the java-config  ("java-home")
  2. The contents of AS_JAVA in the asenv property file (in glassfish/config)
  3. Whatever the env. variable, JAVA_HOME, is pointing at.
  4. The java.home system property is used like so:
    1. Parent directory of java.home -- this is because sometimes the JRE's java is asadmin's JVM and the JDK may be in the directory above.
    2. Contents of java.home

In every case I guarantee a valid java.  E.g. you can set 1,2,3 to garbage and we'll still find a java to use.

Note that we never explicitly use java from the path -- if there is one.  It may be implicitly used though by using #4 above since asadmin itself is definitely running inside of a valid JVM.

p.s.  If you are wondering where the code is -- 90% of it is inside
10% is in

-- AsenvPropertyReader can't access domain.xml.  GFLauncher can.  GFLauncher's job is to see if (1) above applies.  If not he defers to AsenvPropertyReader.


Wednesday Jun 15, 2011

Tip #1 Finding Source Files In a Gazillion Modules

 Tip #1  June 15, 2011

 GlassFish has a LOT of modules (463 pom.xml files) and a lot of java files (7810).  Here is an easy way to instantly locate source files:

    1. From the source root (Windows) run
      dir *.java /s/b >\gffiles.txt
    2. Create a script with this in it
      grep -i %1 \gffiles.txt
    3. Run the script and you'll see where the java file lives.
    4. Update once in a while -- or just wait until you don't find a match.


d:\gf\v3>gf411 counts

d:\gf\v3>grep -i counts d:\gffiles.txt

Note:  I finally did this after dozens (scores?  hundreds?) of times manually searching for files.  Don't waste your time!

Friday Apr 01, 2011

Test Driven Development in a Large Team

I had a problem to work out.  I know of plenty of wrong-headed things that Monitoring does.  I want to clean up this behavior for GlassFish 3.2  But I want to use the tenets of Test Driven Development.  Namely -- tests FIRST, implement SECOND.  Normally that would mean - add some tests now.  The tests will fail every time until we get around to fixing the code.  Eventually the tests will all start passing as we add in all the fixes.  Overtime we see lots of failures dropping to zero failures.

But the problem is that we can't do this is a multiple-developer project with continuous automated runs of the enormous suites of tests.  We can't add tests that will fail (at least not for more than just one cycle).  And I don't want to maintain yet another Hudson job that runs a subset of tests just for monitoring.  Plus it just doesn't feel right.  The tests ought to pass ALL THE TIME.

An excellent solution was provided by Carla Mott.  Simple and elegant:

1) Create a test for what you want the new behavior to be.
2) Now set the results of that test to the converse of the actual results -- i.e. if they pass, report as a failure and vice versa

The tests will pass every time until you fix the issue.  Then it will fail every time until you fix the test. 

We can add lots of these tests for new features, now.  When we implement these fixes when we are under the steaming pressure head of deadlines and milestones we won't have to 'remember' to add Dev Tests.  They are already done and will break like popcorn popping as we complete tasks.  It's impossible to not have working devtests upon completion.

Wednesday Mar 23, 2011

CLI Logfile and DevTests

Recently a colleague added a test to the Admin Development Tests.  This is a huge suite of tests that we developers use as a sanity test that exercise the administration of a running GlassFish server.

here is the symptom of the problem -

 The test failed on her machine.
The test failed on my machine
The test succeeded on the Hudson build that runs it automatically -- it passed 100% of the time

I finally tracked this problem down today with the help of an invaluable tool for asadmin.  I always set this variable in my environment:


When this is set -- every asadmin command is written into the logfile.  It includes the full command with all arguments, the time-stamp and the return value.

I quickly figured out the problem by looking in this file on my platform, Windows, and on the Hudson platform, Linux.  Here it is:

Command inside the test itself:  asadmin list -m \*

Command inside cli.log

03/22/2011 21:55:27 EXIT: 1 asadmin --user admin --passwordfile D:\\gf\\v2\\appserv-tests/config/adminpassword.txt --host localhost --port 4848 --echo=true --terse=true list -m .svn apps build build.xml byron derby.log dome.bat foo foo2 foo2.bat mon.bat nbproject pom.xml q qqq resources seconddb src 

OOPS!!  The star is special in Windows calling the asadmin script -- even from Runtime.exec().  The star was replaced with the names of every file that happened to be sitting in the directory that the tests are running in.

 Solution:  In the code use "\\"\*\\"" for Windows and '\*' otherwise

Sunday Feb 27, 2011

Monitoring Data -- How To Open The Firehose

Are you interested in getting raw Monitoring Data without using JavaScript or Dtrace?  That's why we include the special command:

asadmin get -m

Looks can be deceiving.  It appears that the -m is just some command line option.  Not so.  That turns on a huge pile of code that plows through the run-time tree of monitoring data.  This data can be accessed from instances, clusters of instances and DAS -- in fact ALL the data from all of them can be combined and accessed all at the same time!

This command was available in V3.0  Unfortunately it was very slow if you wanted to collect everything (what I call Firehose Mode).  So what was done was a lot of regular expression parsing to narrow down the matching data in order to get things going faster.

I reworked this command for GlassFish 3.1 to support the clustering feature.  I started DAS (Domain Administration Server) and one instance.  I ran

asadmin get -m "\*"  // Firehose output for DAS and the instance

Then I waited.  And waited.  And waited and waited.  About 90 seconds later the server got an out-of-memory error.  I never saw any output at all.  Oops!

I discovered the performance bottle-neck.  It was unbelievable what the problem is.  If you are interested I've blogged about it here.

We spent a lot of time trying to get the get -m command to handle regular expressions well.  Below are some examples. 

My environment: I have a cluster, c1.  c1 has 2 instances, i1 and i2.  Both instances are running with monitoring-levels all set to HIGH.  DAS is also running.  All servers are running on the same machine so I don't muddy the water with network latency etc.

  • asadmin get -m c1 # this retrieves ALL monitoring data for every running instance in c1 (~14,000 lines of output)
  • asadmin get -m c1.\* # does exactly the same thing as above
  • asadmin get -m i2.web.servlet.activeservletsloadedcount-current  # Example drill-down.
    i2.web.servlet.activeservletsloadedcount-current = 6
  • asadmin get -m "\*" | wc -l # returned 20,000+ lines.  It took around 1 second on my laptop.  
  • asadmin get -m \*.web.servlet.activeservletsloadedcount-current
    server.web.servlet.activeservletsloadedcount-current = 2  #DAS
    i1.web.servlet.activeservletsloadedcount-current = 6  
    i2.web.servlet.activeservletsloadedcount-current = 6

The command is so incredibly fast you can simply collect everything in a file and then run text processing commands on that file and avoid calling the command multiple times:

asadmin get -m "\*" > output

grep hello output # just an example




« August 2016