Thursday Jul 28, 2011

Dtrace of Clusters in GlassFish - A Quick Tutorial

This is the second blog in the series about DTrace and GlassFish 3.1.1  You should read the first blog before this one.

Continuing where I left off in the previous blog I would like to analyze the usage of MonApp in 2  different instances.  Specifically I did this:

  1. undeploy from DAS
    asadmin undeploy MonApp
  2. create a cluster
    asadmin create-cluster c1
  3. create two clustered instances
    asadmin create-local-instance --cluster c1 i1
    asadmin create-local-instance --cluster c1 i2
  4. deploy to the cluster
    asadmin deploy MonApp.war
  5. start the instances
    asadmin start-cluster c1
  6. OPTIONAL - turn the levels up to HIGH for all Monitoring Modules as in the previous blog.  Just add the args --target c1
  7. turn-on DTrace support
    asadmin enable-monitoring --target c1 --dtrace
  8. Load both off the URLs in a browser in 2 different tabs:
    http://localhost:28080/MonApp
    http://localhost:28081/MonApp
  9. Verify DTrace "sees" the MonApp Probes
    dtrace -l | grep foo
    39785 fooblog20527           samples                      ProbeServlet myProbe
    39841 fooblog20544           samples                      ProbeServlet myProbe
  10. Run list-instances to get the instance pids so you can tell the probes apart
    asadmin list-instances --long
    NAME  HOST       PORT   PID    CLUSTER  STATE
    i1    localhost  24848  20527  c1        running
    i2    localhost  24849  20544  c1        running
  11. If you had a load balancer setup (I don't) you could start hammering the App and see which instances are servicing the calls.  What I do is start the dtrace script:
    dtrace -n 'fooblog*::: { printf("PID:%d APP:%s %s", pid, execname, copyinstr(arg0)); }'
  12. Now I sit in the browser and randomly reload the 2 pages for a while and look at DTrace output:
    dtrace: description 'fooblog*::: ' matched 2 probes
    CPU     ID                    FUNCTION:NAME
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #3
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #4
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #5
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #3
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #4
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #5
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #6
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #6
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #7
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #8
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #7
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #8
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #9
      0  39841             ProbeServlet:myProbe PID:20544 APP:java Hello #10
      0  39785             ProbeServlet:myProbe PID:20527 APP:java Hello #9

     

Introducing DTrace in GlassFish 3.1.1

This is the first in a series of blogs about Dtrace in GlassFish.  I will present concrete examples  in upcoming blogs.  This blog is an overview of this powerful new capability.

An entire book could be (and has been) written just to describe DTrace itself.   There are many many useful documents on the internet about DTrace and its usage.  At the end of this blog I've referenced a few good ones that I've read.

DTrace is a tool available currently on Solaris 10 and Mac OS X that will dynamically instrument already-running code  for the purposes of seeing what is going on inside software.  This includes the OS itself.  You can probe into kernel-mode in addition to user-mode. 

GlassFish automatically supports DTrace.  For every Probe that is registered in GlassFish, a corresponding DTrace probe is generated and made available to DTrace.  This includes any probes that you create yourself such as in your Applications.  See  [1] for a tutorial on this.

In order for the DTrace probes to be available you need to do 3 things from the GlassFish end:

  1. GlassFish must be running in JDK 7
  2. Dtrace must be enabled.  Simply run this command:
    asadmin enable-monitoring --dtrace
  3. You must have glassfish-dtrace.jar copied to the GlassFish modules directory.  Update center will set things up for you by doing the following
    1. You must use an official distribution:
      1. Open Source Distribution
      2. Oracle GlassFish Server Distribution
      3. Official Oracle GlassFish Server page is located here
    2. Run: <installdir>/bin/updatetool to bootstrap update clients.
    3. Run:  <installdir>/bin/pkg install glassfish-monitoring-dtrace@3.1.1 to install the DTrace package

Of course you also have to turn on the monitoring levels for what you are interested in.  To turn Monitoring up full-blast for everything run these commands (you can make one huge command out of it and stuff it into a script like I do):

asadmin enable-monitoring --target server --modules deployment=HIGH
asadmin enable-monitoring --target server --modules connector-connection-pool=HIGH
asadmin enable-monitoring --target server --modules connector-service=HIGH
asadmin enable-monitoring --target server --modules ejb-container=HIGH
asadmin enable-monitoring --target server --modules http-service=HIGH
asadmin enable-monitoring --target server --modules jdbc-connection-pool=HIGH
asadmin enable-monitoring --target server --modules jms-service=HIGH
asadmin enable-monitoring --target server --modules jvm=HIGH
asadmin enable-monitoring --target server --modules orb=HIGH
asadmin enable-monitoring --target server --modules thread-pool=HIGH
asadmin enable-monitoring --target server --modules transaction-service=HIGH
asadmin enable-monitoring --target server --modules web-container=HIGH
asadmin enable-monitoring --target server --modules jersey=HIGH
asadmin enable-monitoring --target server --modules jpa=HIGH
asadmin enable-monitoring --target server --modules security=HIGH
asadmin enable-monitoring --target server --modules web-services-container=HIGH

Obviously you have to be on an OS that has dtrace!

You can quickly and easily see that everything is registered correctly and available for running in DTrace scripts with this command:

dtrace -l | grep glassfish

This will show 151 or so built-in GlassFish probes.  Here is one line of the output:

39462 glassfish20383               web                        web-module webModuleStartedEvent
20383 is the Process ID of the GlassFish Server.  Whenever a web module is started the DTrace probe will be called with some arguments like the name of the web-module, etc.

The beauty of DTrace is that the DTrace probe inside of GlassFish is a no-op unless a DTrace script happens to be running.  In that case and in only that case the DTrace probe code gets dynamically instrumented to call into the OS's DTrace framework.  When the script ends, the probe gets un-instrumented.  There is almost no overhead unless a DTrace script is running.

There are limitless applications for this technology.  You can create on-the-fly scripts that attach to an already-running GlassFish server.  Everything can be enabled dynamically - the server never needs to be restarted for the magic to happen.  DTrace is so powerful it has its own programming language named "d".  It is beyond the scope of this blog to do more than chip at the surface of the DTrace engine.  

An example might be choosing a pair of bookend probes.  E.g. a start-transaction/end-transaction pair.  Or a HTTP service startRequest/endRequest pair.  You can easily have a DTrace script measure the time between the 2 calls and print out this information.  Since it is a programming language you can even save them into variables and crunch the data at the end of the script. 

Another example might be that you want to see every file that is opened - but only between a bookend pair (i.e. you are filtering the deluge of file openings to those that happen to occur during the time-slice of your bookend).

Create or download the app from [1].  Now add something more interesting to the webapp like, for instance, a form.  Now call a probe everytime the form is processed.  Pass a String the user enters in the form to the Probe.  That String will be available in real-time to a DTrace script!  This is a lot easier than it sounds.  I deployed the app (MonApp) from the blog[1] and then ran a super-simple DTrace script.  What the App does is call the probe method each time you refresh it.  wget is an easy way to do the refresh. It increments a counter and calls the probe with a string like so:  "Hello #XX".

I didn't even bother writing a script file - I just ran it from a command line:

dtrace -n 'fooblog*::: { printf("PID:%d APP:%s %s", pid, execname, copyinstr(arg0)); }'
dtrace: description 'fooblog*::: ' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #22
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #23
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #24
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #25
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #26
  0  39554             ProbeServlet:myProbe PID:20383 APP:java Hello #27

I got one output line every time I refreshed MonApp in a browser.  This is incredibly powerful!  A string was sent from the user in a browser to a Java WebApp which then sent it off to a native program, DTrace, that was looking for it with a sophisticated scripting language.  All of this can be done without ever restarting the GlassFish Server.  There are truly endless possibilities.  And once you kill the script -- all the instrumentation code is removed.

 In case you are wondering about fooblog*:::  -- fooblog is the main name I gave it.  The * is needed because the Monitoring Framework automatically appends the PID of the server (very handy if you are DTracing with multiple GlassFish servers simultaneously).  The ::: means match ANY probe that has the main name that starts with fooblog.  Whew.  Hard to explain in English, easy to explain with one command:

~/gf_other/value-add/monitoring/dtrace/dtrace-samples/dtrace-scripts>dtrace -l | grep fooblog
39554 fooblog20383           samples                      ProbeServlet myProbe

To be completely specific in the earlier command would be:

dtrace -n 'fooblog20383:samples:ProbeServlet:myProbe { ..... }

Dtrace uses the quadruple of names to make the namespace less unwieldy and easier to filter.


References


 [1] My blog about adding Monitoring probes to a WebApp



About

ByronNevins

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today