Thursday Jul 28, 2011

Dtrace of Clusters in GlassFish - A Quick Tutorial

This is the second blog in the series about DTrace and GlassFish 3.1.1  You should read the first blog before this one.

Continuing where I left off in the previous blog I would like to analyze the usage of MonApp in 2  different instances.  Specifically I did this:

  1. undeploy from DAS
    asadmin undeploy MonApp
  2. create a cluster
    asadmin create-cluster c1
  3. create two clustered instances
    asadmin create-local-instance --cluster c1 i1
    asadmin create-local-instance --cluster c1 i2
  4. deploy to the cluster
    asadmin deploy MonApp.war
  5. start the instances
    asadmin start-cluster c1
  6. OPTIONAL - turn the levels up to HIGH for all Monitoring Modules as in the previous blog.  Just add the args --target c1
  7. turn-on DTrace support
    asadmin enable-monitoring --target c1 --dtrace
  8. Load both off the URLs in a browser in 2 different tabs:
    http://localhost:28080/MonApp
    http://localhost:28081/MonApp
  9. Verify DTrace "sees" the MonApp Probes
    dtrace -l | grep foo
    39785 fooblog20527           samples                      ProbeServlet myProbe
    39841 fooblog20544           samples                      ProbeServlet myProbe
  10. Run list-instances to get the instance pids so you can tell the probes apart
    asadmin list-instances --long
    NAME  HOST       PORT   PID    CLUSTER  STATE
    i1    localhost  24848  20527  c1        running
    i2    localhost  24849  20544  c1        running
  11. If you had a load balancer setup (I don't) you could start hammering the App and see which instances are servicing the calls.  What I do is start the dtrace script:
    dtrace -n 'fooblog*::: { printf("PID:%d APP:%s %s", pid, execname, copyinstr(arg0)); }'
  12. Now I sit in the browser and randomly reload the 2 pages for a while and look at DTrace output:
    dtrace: description 'fooblog*::: ' matched 2 probes
    CPU     ID                    FUNCTION:NAME
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #3
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #4
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #5
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #3
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #4
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #5
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #6
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #6
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #7
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #8
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #7
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #8
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #9
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #10
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #9

     

Introducing DTrace in GlassFish 3.1.1

This is the first in a series of blogs about Dtrace in GlassFish.  I will present concrete examples  in upcoming blogs.  This blog is an overview of this powerful new capability.

An entire book could be (and has been) written just to describe DTrace itself.   There are many many useful documents on the internet about DTrace and its usage.  At the end of this blog I've referenced a few good ones that I've read.

DTrace is a tool available currently on Solaris 10 and Mac OS X that will dynamically instrument already-running code  for the purposes of seeing what is going on inside software.  This includes the OS itself.  You can probe into kernel-mode in addition to user-mode. 

GlassFish automatically supports DTrace.  For every Probe that is registered in GlassFish, a corresponding DTrace probe is generated and made available to DTrace.  This includes any probes that you create yourself such as in your Applications.  See  [1] for a tutorial on this.

In order for the DTrace probes to be available you need to do 3 things from the GlassFish end:

  1. GlassFish must be running in JDK 7
  2. Dtrace must be enabled.  Simply run this command:
    asadmin enable-monitoring --dtrace
  3. You must have glassfish-dtrace.jar copied to the GlassFish modules directory.  Update center will set things up for you by doing the following
    1. You must use an official distribution:
      1. Open Source Distribution
      2. Oracle GlassFish Server Distribution
      3. Official Oracle GlassFish Server page is located here
    2. Run: <installdir>/bin/updatetool to bootstrap update clients.
    3. Run:  <installdir>/bin/pkg install glassfish-monitoring-dtrace@3.1.1 to install the DTrace package

Of course you also have to turn on the monitoring levels for what you are interested in.  To turn Monitoring up full-blast for everything run these commands (you can make one huge command out of it and stuff it into a script like I do):

asadmin enable-monitoring --target server --modules deployment=HIGH
asadmin enable-monitoring --target server --modules connector-connection-pool=HIGH
asadmin enable-monitoring --target server --modules connector-service=HIGH
asadmin enable-monitoring --target server --modules ejb-container=HIGH
asadmin enable-monitoring --target server --modules http-service=HIGH
asadmin enable-monitoring --target server --modules jdbc-connection-pool=HIGH
asadmin enable-monitoring --target server --modules jms-service=HIGH
asadmin enable-monitoring --target server --modules jvm=HIGH
asadmin enable-monitoring --target server --modules orb=HIGH
asadmin enable-monitoring --target server --modules thread-pool=HIGH
asadmin enable-monitoring --target server --modules transaction-service=HIGH
asadmin enable-monitoring --target server --modules web-container=HIGH
asadmin enable-monitoring --target server --modules jersey=HIGH
asadmin enable-monitoring --target server --modules jpa=HIGH
asadmin enable-monitoring --target server --modules security=HIGH
asadmin enable-monitoring --target server --modules web-services-container=HIGH

Obviously you have to be on an OS that has dtrace!

You can quickly and easily see that everything is registered correctly and available for running in DTrace scripts with this command:

dtrace -l | grep glassfish

This will show 151 or so built-in GlassFish probes.  Here is one line of the output:

39462 glassfish20383               web                        web-module webModuleStartedEvent
20383 is the Process ID of the GlassFish Server.  Whenever a web module is started the DTrace probe will be called with some arguments like the name of the web-module, etc.

The beauty of DTrace is that the DTrace probe inside of GlassFish is a no-op unless a DTrace script happens to be running.  In that case and in only that case the DTrace probe code gets dynamically instrumented to call into the OS's DTrace framework.  When the script ends, the probe gets un-instrumented.  There is almost no overhead unless a DTrace script is running.

There are limitless applications for this technology.  You can create on-the-fly scripts that attach to an already-running GlassFish server.  Everything can be enabled dynamically - the server never needs to be restarted for the magic to happen.  DTrace is so powerful it has its own programming language named "d".  It is beyond the scope of this blog to do more than chip at the surface of the DTrace engine.  

An example might be choosing a pair of bookend probes.  E.g. a start-transaction/end-transaction pair.  Or a HTTP service startRequest/endRequest pair.  You can easily have a DTrace script measure the time between the 2 calls and print out this information.  Since it is a programming language you can even save them into variables and crunch the data at the end of the script. 

Another example might be that you want to see every file that is opened - but only between a bookend pair (i.e. you are filtering the deluge of file openings to those that happen to occur during the time-slice of your bookend).

Create or download the app from [1].  Now add something more interesting to the webapp like, for instance, a form.  Now call a probe everytime the form is processed.  Pass a String the user enters in the form to the Probe.  That String will be available in real-time to a DTrace script!  This is a lot easier than it sounds.  I deployed the app (MonApp) from the blog[1] and then ran a super-simple DTrace script.  What the App does is call the probe method each time you refresh it.  wget is an easy way to do the refresh. It increments a counter and calls the probe with a string like so:  "Hello #XX".

I didn't even bother writing a script file - I just ran it from a command line:

dtrace -n 'fooblog*::: { printf("PID:%d APP:%s %s", pid, execname, copyinstr(arg0)); }'
dtrace: description 'fooblog*::: ' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #22
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #23
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #24
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #25
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #26
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #27

I got one output line every time I refreshed MonApp in a browser.  This is incredibly powerful!  A string was sent from the user in a browser to a Java WebApp which then sent it off to a native program, DTrace, that was looking for it with a sophisticated scripting language.  All of this can be done without ever restarting the GlassFish Server.  There are truly endless possibilities.  And once you kill the script -- all the instrumentation code is removed.

 In case you are wondering about fooblog*:::  -- fooblog is the main name I gave it.  The * is needed because the Monitoring Framework automatically appends the PID of the server (very handy if you are DTracing with multiple GlassFish servers simultaneously).  The ::: means match ANY probe that has the main name that starts with fooblog.  Whew.  Hard to explain in English, easy to explain with one command:

~/gf_other/value-add/monitoring/dtrace/dtrace-samples/dtrace-scripts>dtrace -l | grep fooblog
39554 fooblog20383           samples                      ProbeServlet myProbe

To be completely specific in the earlier command would be:

dtrace -n 'fooblog20383:samples:ProbeServlet:myProbe { ..... }

Dtrace uses the quadruple of names to make the namespace less unwieldy and easier to filter.


References


 [1] My blog about adding Monitoring probes to a WebApp



Friday Apr 01, 2011

Test Driven Development in a Large Team

I had a problem to work out.  I know of plenty of wrong-headed things that Monitoring does.  I want to clean up this behavior for GlassFish 3.2  But I want to use the tenets of Test Driven Development.  Namely -- tests FIRST, implement SECOND.  Normally that would mean - add some tests now.  The tests will fail every time until we get around to fixing the code.  Eventually the tests will all start passing as we add in all the fixes.  Overtime we see lots of failures dropping to zero failures.

But the problem is that we can't do this is a multiple-developer project with continuous automated runs of the enormous suites of tests.  We can't add tests that will fail (at least not for more than just one cycle).  And I don't want to maintain yet another Hudson job that runs a subset of tests just for monitoring.  Plus it just doesn't feel right.  The tests ought to pass ALL THE TIME.

An excellent solution was provided by Carla Mott.  Simple and elegant:

1) Create a test for what you want the new behavior to be.
2) Now set the results of that test to the converse of the actual results -- i.e. if they pass, report as a failure and vice versa

Result: 
The tests will pass every time until you fix the issue.  Then it will fail every time until you fix the test. 

We can add lots of these tests for new features, now.  When we implement these fixes when we are under the steaming pressure head of deadlines and milestones we won't have to 'remember' to add Dev Tests.  They are already done and will break like popcorn popping as we complete tasks.  It's impossible to not have working devtests upon completion.


Sunday Feb 27, 2011

GlassFish 3.1 Scripting of Monitoring Data for Clusters and Instances

Monitoring Scripting is a value-add feature and is available only in the Oracle GlassFish server which is available here.

GlassFish 3.1 introduces Clusters and Server Instances.  Monitoring now includes full support of these new features.  In this blog I'll demonstrate a simple application of monitoring-scripting in a clustered environment. 

First we need to setup a cluster and start everything:
  1. Start DAS (Domain Administration Server)
    asadmin start-domain
  2. Create a cluster with 2 instances -- they will all run on this same machine
    asadmin create-cluster c1
    asadmin create-local-instance --cluster c1 i1
    asadmin create-local-instance --cluster c1 i2
  3. Set the monitoring level to HIGH for the cluster.  By default it is OFF
    asadmin enable-monitoring --modules web-container=HIGH --target c1
  4. Start the cluster of 2 instances
    asadmin start-cluster c1
  5. Check and make sure everything is running OK
    asadmin list-domains
    domain1 running
    asadmin list-instances --long
        NAME  HOST       PORT   PID   CLUSTER  STATE
        i1    localhost  24848  2464  c1        running
        i2    localhost  24849  5188  c1        running
  6. Deploy a simple app to the cluster
    asadmin deploy --target c1 hello.war
  7. "Hit" the Hello app on both instances
    wget http://localhost:28080/hello
    wget http://localhost:28081/hello

Now we have 3 servers running: the DAS and 2 clustered instances.  The web-container is sending events when interesting things happen.  Notice that we really have not done anything special for monitoring except the one enable command.  This looks complicated -- but heck you now have a running cluster with a deployed app etc.  with just a few quick commands.  Not bad! 

We want to monitor our Hello app.  I'm especially interested in seeing if, say, my Load Balancer is distributing client calls to Hello equally.  The first step is to get a list of all the probes.  I won't show the list here -- there are 136 probes.  Just run the command and you can see all of them.  I'm interested in the event that fires just before a servlet is run.  So I run this command:

 asadmin list-probes | grep servlet | grep before

glassfish:web:servlet:beforeServiceEvent (java.lang.String servletName, java.lang.String appName, java.lang.String hostName)

This describes one probe.  Everything you need to listen to this probe's event is in this one line.  So now we create a simple JavaScript program.  I'll just enter it here, JavaScript is documented in the official Oracle GlassFish Server and elsewhere.  It's pretty obvious what's going on by reading the program's source code.



function requestStartEvent(servletName,appName, hostName) {
        client.print( 'requestStartEvent called on App: ' + appName +
                      ', Servlet: ' + servletName + '\\n');
}

params = java.lang.reflect.Array.newInstance(java.lang.String, 3);
params[0]="servletName";
params[1]="appName";
params[2]="hostName";

scriptContainer.registerListener('glassfish:web:servlet:beforeServiceEvent', params, 'requestStartEvent');


I saved the script as servlet.js

 Now I run this script against the cluster and I start hitting hello in each instance (28080, 28081) sort of randomly just to get some output.   We see output every time Hello (or any servlet) is accessed.  Notice how it shows which server is affected:

D:\\gf_other\\value-add\\monitoring\\scripting\\samples>asadmin run-script --target c1 servlet.js
Listening to i1 on host localhost and port 24848
Listening to i2 on host localhost and port 24849

[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: default
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: default
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp


The beauty of this is that once you have the clusters and everything setup and running you can run scripts on all sorts of probes very quickly.  You don't even have to compile.  Simply edit the script and call run-script

For a nice demonstration watch the video that Tom Mueller created here.





Huge Performance Improvement

I was working on improving the performance and memory consumption in Monitoring and I bumped into one problem that caused gigantic performance degradation.  I'm talking about a 100 fold speed-up for large (5000-20000) data sets.

 The code was adding Strings into a data structure.  The way it was generating the Strings was like this:

....

String s;  // we are actually all done now.  "s" is fine

s = MessageFormat.format("{0}", s);

 I simply got rid of that line and everything was fixed!

Moral:  Be careful with MessageFormat






Monitoring Data -- How To Open The Firehose

Are you interested in getting raw Monitoring Data without using JavaScript or Dtrace?  That's why we include the special command:

asadmin get -m

Looks can be deceiving.  It appears that the -m is just some command line option.  Not so.  That turns on a huge pile of code that plows through the run-time tree of monitoring data.  This data can be accessed from instances, clusters of instances and DAS -- in fact ALL the data from all of them can be combined and accessed all at the same time!

This command was available in V3.0  Unfortunately it was very slow if you wanted to collect everything (what I call Firehose Mode).  So what was done was a lot of regular expression parsing to narrow down the matching data in order to get things going faster.

I reworked this command for GlassFish 3.1 to support the clustering feature.  I started DAS (Domain Administration Server) and one instance.  I ran

asadmin get -m "\*"  // Firehose output for DAS and the instance

Then I waited.  And waited.  And waited and waited.  About 90 seconds later the server got an out-of-memory error.  I never saw any output at all.  Oops!

I discovered the performance bottle-neck.  It was unbelievable what the problem is.  If you are interested I've blogged about it here.

We spent a lot of time trying to get the get -m command to handle regular expressions well.  Below are some examples. 

My environment: I have a cluster, c1.  c1 has 2 instances, i1 and i2.  Both instances are running with monitoring-levels all set to HIGH.  DAS is also running.  All servers are running on the same machine so I don't muddy the water with network latency etc.

  • asadmin get -m c1 # this retrieves ALL monitoring data for every running instance in c1 (~14,000 lines of output)
  • asadmin get -m c1.\* # does exactly the same thing as above
  • asadmin get -m i2.web.servlet.activeservletsloadedcount-current  # Example drill-down.
    i2.web.servlet.activeservletsloadedcount-current = 6
  • asadmin get -m "\*" | wc -l # returned 20,000+ lines.  It took around 1 second on my laptop.  
  • asadmin get -m \*.web.servlet.activeservletsloadedcount-current
    server.web.servlet.activeservletsloadedcount-current = 2  #DAS
    i1.web.servlet.activeservletsloadedcount-current = 6  
    i2.web.servlet.activeservletsloadedcount-current = 6

The command is so incredibly fast you can simply collect everything in a file and then run text processing commands on that file and avoid calling the command multiple times:

asadmin get -m "\*" > output

grep hello output # just an example



A Taste of Monitoring in GlassFish 3.1

Ok, so you've installed the Oracle GlassFish Server that you got from here.  You've created a cluster with a couple of instances.  The instances seem to be getting a bunch of traffic.  At least there are a lot of flashing lights on the ethernet cards and disk light is getting hammered.  But how can you find out what's really going on?  

That's the whole point of Monitoring.  We have provided a rich platform for getting Monitoring information in GlassFish.  You can ask for a trickle by zeroing-in on exactly the scraps of data you are interested in (e.g. please tell me every time the web-container creates a new thread) or turning the fire hose on and getting every monitored-event as it happens.

How do you want to see the data?

  •  In a browser? -- no problem, use either the Admin GUI or REST.
  • You are running in Solaris 10+ and you want to use DTrace?  We support it -- all monitorable events can be seen via native DTrace.
  • You want all of your Applications to get added into this cool Monitoring framework so you can access the data just like the built-in GlassFish Monitoring data?  No sweat, we have provided an easy way to add Monitoring probes to your applications.
  • Not good enough?  Too generic?  You want to be able to collect the data dynamically and structure it some other way using a JavaScript program?  Again, no problem.  You can run a JavaScript program, access as much or as little data as you want, dynamically,  and go crazy massaging the data any way you like.


There are a lot of facets to Monitoring in GlassFish 3.1  We will present many blogs with the details to get you going.  As those blogs are completed I'll circle back here and add the links.  In the meantime -- here is a very simple example of a "What-If" question and how to get the answers quickly.

How many classes have been loaded by the Domain Administration Server's JVM and what happens with the count when an app is deployed and then accessed?

asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 3887
asadmin deploy hello.war
Application deployed with name hello.
asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 5979
wget http://localhost:8080/hello
asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 6816


Wow!  That's a lot of classes.  Probably the jump from 3887 to 5979 was loading the web-container (this server had no apps deployed yet) and the next big pile of classes were jsp and servlet classes.  Let's do a sanity check by deploying another simple app:

asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 6819
asadmin deploy Math.war
Application deployed with name Math.
asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 6902

wget http://localhost:8080/Math
asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 6906

That's semi-interesting.  Deploying a simple web module (with a servlet and jsp)  results in 83 classes getting loaded.  Running the app results in 4 classes being loaded.


Monitoring Blogfest

Running Monitoring JavaScript Programs on Clusters and Instances


Wednesday Dec 02, 2009

Mort Learns How To Use Monitoring in a WebApp

 This blog is a follow-up of my earlier monitoring blog, and I recommend reading that one first.

This time we will setup probes inside of a user application.  We will not be in the internal GlassFish environment.  We need to use the Monitoring tools available from GlassFish OSGi.  But -- we will be in a plain vanilla web module, not an OSGi module.  Oh no!  How can we access the Habitat inside GlassFish?  JavaEE 6 comes to the rescue with the @Resource annotation!

    @Resource
    private ProbeProviderFactory probeProviderFactory;

    @Resource
    private ProbeClientMediator listenerRegistrar;

These 2 variables are what we need to register a probe and hook up the probe with a listener.

The web app consists of the usual boilerplate song and dance needed for any web app.  The business end of this web app is in two classes:

  1. The Servlet class -- ProbeServlet.java
  2. The Probe Listener class -- MyProbeListener.java

The servlet class has this method:

@Probe(name="myProbe")
public void myProbeMethod(String s) {
    System.out.println("inside myProbeMethod "         
        + "called with this arg: " + s);
    } 

and it has this annotation at the class level:

@ProbeProvider( moduleProviderName="fooblog", moduleName="samples", probeProviderName="ProbeServlet")

That's all you need to do to turn this servlet class into an official GlassFish V3 Probe!

To get the Probe registered with GlassFish you need to add one line of code that needs to run before you use the probe (I added it to the init method):

probeProviderFactory.getProbeProvider(getClass());

I made the servlet painfully simple.  Whenever you load or reload the servlet it calls myProbeMethod().  Of course all of this would be pointless without a listener class that is hooked-up to the probe. 

 The init code in the servlet hooks up the listener with this one line of code:

listenerRegistrar.registerListener(
    new MyProbeListener());

But wait!  How in the world is that listener class going to be connected to the probe class?!?  This is where the magic of annotation processing comes in.  The listener class is super-simple.  Here it is in it's entirety:

public class MyProbeListener {
  @ProbeListener("fooblog:samples:ProbeServlet:myProbe")
  public void probe1(String s) {
    System.out.println("PROBE LISTENER HERE!!!!");
  }
}

Note how the strings in the ProbeProvider annotation are used in the listener to identify the correct probe.

This is a very simple example but you can see how you can create listeners in their own separate apps and have probes in many apps.  To see the output simply tail the server log.  When you write to System.out from a Listener - it will appear in the server log.

The app is available in open source as a maven web-app.  It can be checked out using this URL:

https://svn.java.net/svn/glassfish~svn/trunk/v2/appserv-tests/devtests/admin/cli/apps/MonApp

Note:  If you make changes to the Listener class you should restart the server after redeploying.


Sample output from loading the app in a browser:

....|PROBE LISTENER HERE.  Called with this arg: Hello #1|#]
....|inside myProbeMethod called with this arg: Hello #1|#]

Wednesday Sep 16, 2009

Mort Learns Monitoring and Admin Commands for GlassFish

  • Monitoring is a powerful feature in GlassFish V3.
  • Admin Commands are a powerful feature in GlassFish V3.  

In this blog I'll show you how to create an Admin Command that exercises the Monitoring Facilities.

Glossary:

  • ProbeProvider - A class or interface that has methods (Probes) that can be called at runtime.
  • Probe - A method that can be called at runtime that in turn will call any Listeners that are registered
  • Listener - A method that is registered to receive "events" coming from a Probe
  • Admin Command - An easy to call command running on GlassFish V3.  You can access it via REST, HTTP or asadmin.  You could easily do this whole tutorial inside a servlet instead.  I chose to use an Admin Command so you can see how to create and use commands.

Background

In this tutorial we will work with a ProbeProvider interface.  We write an interface with the methods (Probes) that we wish to call at runtime.  We never create an implementation of this interface.  Instead we hand it off to GlassFish.  GlassFish will magically create an implementation class for the interface and return an instance of that implementation.  Under the hood, GlassFish has automagically inserted code into your methods that will call any Listeners that you subsequently register.

Using an interface is a fairly unusual case.  The much fancier and more usual case is that you give GlassFish an instance of any POJO that has the proper annotations.  GlassFish will then rewrite the actual bytecode of the class and add calls to the Listeners!

Admin Command

  1. The command should setup the Probes and Listeners correctly the first time the command is run
  2. Each subsequent call should fire the appropriate probe

 The complete class is here

You provide LCPP and LCListener.  LCPP is the Probe Provider interface and LCListener is the Listener class.  The 2 lines that do all the registering and magic are here:

probe = probeProviderFactory.getProbeProvider(LCPP.class);
listenerRegistrar.registerListener(new LCListener());

That's all!  You now have a probe object that you can call methods on that will automatically call your listener class.

Probe Provider Interface

The interface is here.  As you can see it is mostly  a bunch of annotations and 2 methods:

void ping();
void ping2(@ProbeParam("message") String s);

Note that they must return void   You should make all parameters primitives or String to avoid trouble.

Probe Listener

 The source is here.  Now you can see how the Listener gets hooked up with the Probe.  The Listener needs the values from the ProbeProvider annotation in the Probe Provider class.  It's a lot easier to see how to do it by looking at the source then for me to explain it.

How to Test

  1. Take the files and drop them into a GlassFish extension project.  If you have GlassFish source just do what I did -- drop them into the core/kernel module.  Or you can just use the files as a guide and, say, create a Servlet instead of using an Admin Command.
  2. build and deploy the jar
  3. start the server
  4. tail the server log (even better -- run  asadmin start-domain --verbose)
  5. call asadmin ping <optional-string-here>  
  6. Look for the listener messages in the log
  7. goto 5



About

ByronNevins

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today