Monitoring in GlassFish v3

Monitoring in GlassFish v3 - Blog
  1. Introduction
    1. Probes
    2. Probe Provider
    3. Probe Listener & StatsProviders
  2. DTrace\*
  3. Scripting Client\*
  4. eXtending Probes using XML (XP-XML)
  5. Other Clients
    1. JMX/AMX
    2. REST
    3. Admin Console
    4. asadmin list-probes\*
    5. Help for probe\*
    6. asadmin get/list
    7. asadmin monitor
  6. Architecture
  7. References

\*- The features denoted with '\*' are the value-add features available only for the paid customers.


1. Introduction

Monitoring in GlassFish v3 has taken a huge leap when compared to the earlier versions. We provide a way to dynamically and non-intrusively generate monitoring events from any of the glassfish runtime classes, the ability to listen to these events, collect the statistics and expose these statistics through various standard clients.

Salient features of Monitoring Infrastructure are
  • Container(s) / sub-system only emit events/probes
    • Example: onWebRequest(...), onWebResponse(...)
    • Interesting values are passed as paramters
    • Containers do NOT maintain any stats
    • We use the term "providers"
    • If there are no listeners registered for this, then there is no overhead
  • Listeners receive the event that are emitted by providers
    • Listeners can examine the parameters passed
    • Listeners maintain the stats
  • The Monitoring framework connects “providers” and “listeners”
  • Supports various clients, like asadmin, JMX/AMX, REST, Admin GUI, Client-Scripting\* (JavaScript)
  • Monitoring events are exposed as probes in the DTrace\* world
  • Allows the Turning on/off, (configuring) the monitoring for components
\*- Available as a value-add feature, made available only to the paid customers.

Before we dwelve into the details, let us understand the terminology used in this document.

i. Probes

Probe points are the monitoring events, emitted when glassfish goes through that code path. The Probes are defined as part of ProbeProvider definition. The Providers and its probes are discovered and registered automatically by the Monitoring infrastructure and thus exposed to the DTrace, Client-Scripting and Probe Listener clients.

ii. Probe Providers

You will use the provider API to expose the probes for your subsystem. A provider is either a Class or  an XML file, defined by the contianer/module. It defines the set of events that are emitted by the provider. Providers are discovered and registered automatically, as and when the modules are loaded into the glassfish.

The following are the two ways on how the ProbeProviders are defined.

Probe Provider as a Class

To define the Probes you could choose to write a simple ProbeProvider class (doesn't have to confirm to any interfaces) and annotate the class and methods with appropriate constructs to qualify the methods in the class as Probes. Once you define the probes, you would instantiate this class in the container code (or wherever you would like to fire this from) and just give a call to the probe method. See the below example.

Defining Provider as a Class

@ProbeProvider(moduleProviderName="glassfish", moduleName="web", probeProviderName="jsp")
Class JspProbeProvider {
...
    @Probe(name="jspLoadedEvent")
    public void jspLoadedEvent(
        @ProbeParam("jsp")
        String jsp,
        @ProbeParam("hostName")
        String hostName) { }
....
}

Emitting the probe

Class JspProbeEmitterImpl.java {

...
public JspProbeEmitterImpl() {
           JspProbeProvider jspProvider = new JspProbeProvider();
                ......
          }
          public jspLoaded (String jsp, String vsId) {
               jspProvider.jspLoadedEvent(jsp, vsId);
          }
...
}

MANIFEST Entry

probe-provider-class-names:  org.glassfish.web.admin.monitor.RequestProbeProvider,org.glassfish.web.admin.monitor.JspProbeProvider


Probe Provider as an XML

This will allow the user to define probes in a non-intrusive way. Just feed the XML file to the Monitoring infrastructure and make the probes active. This will allow the users to conviniently define probes based on existing code which we cannot change. The XML will be bundled as part of the module and you need to put an entry in the manifest file for the Monitoring infrastructure to discover it. See below for an example of ProbeProvider definition, Usage and Manifest entry.

Defining the Provider XML

<probe-provider moduleProviderName="glassfish" moduleName="web" probeProviderName="jsp" >
    <probe name="jspLoadedEvent"                                                         
       <class>com.sun.enterprise.web.jsp.JspProbeEmitterImpl </class>
       <method>jspLoaded</method>
       <signature>void (String,String)</signature>
       <probe-param type="String" name="jsp"/>
       <probe-param type="String" name="hostName"/>
       <return-param type="void" />
       <location>RETURN/ENTRY/EXCEPTION</location>
    </probe>
</probe-provider>

The Container class's method being called

        Class JspProbeEmitterImpl.java{
        ...
      public jspLoaded (String jsp, String vsId) {
            ....
      }
       }
MANIFEST Entry

probe-provider-class-names:  org.glassfish.web.admin.monitor.RequestProbeProvider,org.glassfish.web.admin.monitor.JspProbeProvider


iii. Probe Listener

A Probe listener (part of the functionality of StatsProvider Object) is basically an object that receives callbacks from the providers. A method in the listener can be "marked" as callback method for a particular event from a specific provider.

Stats Providers

StatsProvider are the objects which gather and expose the monitoring data of various components. They do the following functionality
  • Allows configuration of a component for Monitoring i.e, Enable/Disable
  • Gathers statistics
    • Defines the Statistics Objects
    • May Listen (Via. Probe Listeners) to Probe events emitted by Provider
    • May collect statistics based on the probe events or however way it wants (for ex. using some specific container events)
    • Also may collect statistics from non-Probe Provider, Ex. JVM statistics
  • The statistics objects will be publised to the Monitoring Registry
  • They are the target objects for AMX/JMX and REST clients
Writing a Probe Listener and StatsProvider

@ManagedObject
class JspStatsProvider() {

    private CountStatistic activeJspsLoadedCount;
    private CountStatistic totalJspsLoadedCount;
....

    @ManagedAttribute("activejspsloadedcount")

    public CountStatistic  getActiveJspsLoadedCount(){
         return activeJspsLoadedCount;
    }

    @ManagedAttribute("totaljspsloadedcount")
    public CountStatistic  getTotalJspsLoadedCount(){
         return totalJspsLoadedCount;
    }

   @ProbeListener("glassfish:web:jsp:jspLoadedEvent")
    public void jspLoadedEvent(
        @ProbeParam("jsp")
        String jsp,

        @ProbeParam("hostName")
        String hostName) {

        activeJspsLoadedCount.increment();
        totalJspsLoadedCount.increment();
    }
....
}

Registering a StatsProvider Object

StatsProviderManager.register("jsp-container", PluginPoint.SERVER, "applications/app1/jsp1", new JspStatsProvider());


In the above example the StatsProvider object will be registered under the “server” node in the Monitoring registry. Third parameter gives the intermediary sub-tree (followed after the 'server' node) in the Monitoring registry under which the Statistics objects (defined in the StatsProvider object) would be created as leaf nodes. The fourth parameter is the StatsProvider object itself.

>asadmin get server.applications.app1.jsp1.\*   //will return these stats objects

2. DTrace

When the Probe Provider is registered with the monitoring infrastructure (Note that StatsProvider objects have nothing to do with this), the probes from glassfish components are automatically exposed to the amazing world of DTrace (for Solaris) so the DTrace clients can write their (D)scripts to listen to these probes. Note that as a provider, you wont have to do anything for DTrace to work.

3. Scripting Client\*

This is a Value-add feature and available only to the paying customers. We expose all our probes to a scripting client that would allow GF user to enable/disable the probes and get the data they want. This is similar to DTrace philosophy but will work on any OS for GlassFish runtime. Monitoring infrastructure exposes a Scripting Container, also the asadmin and GUI clients for deploying and running the scripts written in JavaScript and JRuby (next release) and opening up a Comet channel to display the data as and when the probe is fired (a Push model, does it asynchronously). Note that the script is something similar to a StatsProvider object in this case, the difference being, the user could write adhoc script (no compilation is necessary) and just run it on the fly to see the data instantly.

Javascript example

var njspLoaded=0;

function jspLoaded(hostName) {
    njspLoaded = njspLoaded + 1;
    client.print( '\\n js> jsp loaded event called on ' +
            'host = ' + hostName +
            //', app = ' + appName +
            ' and count = ' + njspLoaded);
}

params = java.lang.reflect.Array.newInstance(java.lang.String, 1);
params[0]="hostName";

scriptContainer.registerListener('glassfish:web:jsp:jspLoadedEvent', params, 'jspLoaded');


$> asadmin run-script <script-name>

js> jsp loaded event called on host = localhost , app = helloApp and count = 1
js> jsp loaded event called on host = localhost , app = helloApp and count = 2
js> jsp loaded event called on host = localhost , app = helloApp and count = 3
js> jsp loaded event called on host = localhost , app = helloApp and count = 2
\^C


Apart from the adhoc scripting support, we will also provide pre-canned script capability, which user would be able to invoke directly. The output from the scripting container is name=value pairs with the value coming from the Object.toString(). Following is the example

asadmin run-script --probe "glassfish:web:request:requestStart" --paramnames "moduleName,hostName"

The module owner doesn't have to do anything to get this functionality. Its available by default, when the container (or module) comes up with its probe provider.

4. eXtending Probes using XML (XP-XML)

This would be a great feature considering the amount of work required by the user to expose any Class method in the glassfish runtime as a Probe. We give this ability using the XP-XML mechanism. Just write an XML with the probe provider information, drop it into your $INSTALL/lib/monitor directory (note that the extension should be .xml) and we discover it automatically and consume it to expose the probes you desire. Once the probes are exposed, you will have huge set of tools to use those probes ranging from writing a StatsProvider thus exposing it through JMX/REST/asadmin clients or writing D-Scripts to analyze the glassfish system or by writing an adhoc javascript to listen to these probes.

XP-XML - sample xml

<?xml version="1.0" encoding="UTF-8"?>
<probe-providers module="org.glassfish.web.glue">

<!--  glassfish:web:jsp-new:\* -->

<probe-provider moduleProviderName="glassfish" moduleName="web" probeProviderName="jsp-new" class="com.sun.enterprise.web.jsp.JspProbeEmitterImpl">

<!--  glassfish:web:jsp-new:loaded -->
   <probe name="loaded">
      <method>jspLoadedEvent</method>
      <probe-param type="javax.servlet.Servlet" name="servlet" />
      <return-param type="void" />
   </probe>

</probe-provider>

</probe-providers>


5. Clients

We support various clients out of the box. You just need to follow the instructions to write the ProbeProviders and ProbeListeners and register with Monitoring Infrastructure, all the stats are made available with the following clients.

i. JMX/AMX

We use Gmbal infrastructure to expose the Statistics objects as JMX/AMX. As part of the MonitoringFramework.register API, we are exposing these Statistics objects as JMX/AMX automatically.

Monitoring with JConsole

ii. REST

We use the REST infrastructure to expose the Statistics objects as REST API. As part of the MonitoringFramework.register API we notify the REST infrastructure to expose these Statistics objects as REST automatically. Please see REST blog (by Rajeshwar) which explains clearly the various REST features for Monitoring.

REST root

REST server

REST jvmdetails

iii. Admin Console

Most of the statistics that you will see in the asadmin, will be exposed using the Admin Console.

Monitoring in Admin Cosole

iv. asadmin list-probes\*

GlassFish related probes can be either viewed with dtrace -l 'glassfish\*' command or using the asadmin list-probes command. This command will list all the probes in the GlassFish runtime.

List-Probes command output

v. Help for a probes

You can additionally view what each probe is all about, by doing

$> asadmin --help list-probes glassfish:web:jsp:jspLoaded

This will show the manpage for the specified probe.

vi. asadmin get/list

The asadmin get and list commands (with --monitor=true option to differentiate from the config get/list commands) will automatically expose the monitoring data, once you publish your statistic objects into the monitoring registry. It would look something like below

asadmin


vii. asadmin monitor

To be able to plug in your own monitoring type and view its related output, you should be able to use the pluggability feature of the CLI monitor command. Please follow the link for an explanation on how this works.

This command displays commonly monitored statistics for Enterprise Server components and services. Monitoring output is displayed continuously in a tabular format; the --interval option can be used to display output at a particular interval (default 30 seconds).

$> asadmin monitor --type=webmodule server
asc  ast   rst   st    ajlc  mjlc tjlc   aslc    mslc      tslc
0     0     0     1     0     0      0     8        8          8

$> asadmin monitor --type=jvm server
                               JVM Monitoring
UpTime(ms)                          Heap and NonHeap Memory(bytes)
current                   min             max             low       high      count
11623062             8585216    167313408   0          0          38531072

6. Architecture

To get a bigger picture of overall v3 monitoring, look at the following architecture diagram.

architecture

7. References

Making Module Monitorable
Guidelines for writing ProbeProviders and StatsProviders
Writing Probes - Presentation
Monitoring Functional Specification


Comments:

Very well written article and of course, great technology,

Ludo

Posted by Ludo on August 27, 2009 at 03:00 AM PDT #

This is awesome! Great work.

Posted by Axl Mattheus on August 28, 2009 at 04:31 PM PDT #

Thanks for the interesting blog!
I had problems trying the DTrace probes. Glassfish v3 b62 did not show DTrace as a component in the monitoring service section on the web admin console. Using dtrace -l 'glassfish\*' as you suggest above gives all probes on the system, but none contains glassfish in its name, so the syntax seem to be wrong. Trying "asadmin list-probes" returns:
CLI001 Invalid Command: list-probes
Command list-probes failed.
I had updated GF v3 through http://pkg.glassfish.org/stable/. Installing the add-on called management extensions didn't help either. Then I tried Glassfish v3 from java.sun.com and got it updated to Glassfish v3 b57 through http://pkg.glassfish.org/stable/. This time DTrace appeared as a component in the monitoring service section on the web admin console. However even after enabling it, I couldn't see any glassfish related probes and I observed the same as above for Glassfish v3 b62. So I have following questions:
1. what component is needed to make DTrace show in the web console
2. what component is needed so that 'asadmin list-probes' work
3. will the components from 1.+2. suffice for the glassfish probes for DTrace to work.
Thanks and best regards
Peter

Posted by Peter Doschkinow on September 13, 2009 at 05:41 PM PDT #

Peter,

As I have mentioned in the blog, the DTrace feature is available only to the paying customers. You can choose to write your own StatsProviders which would be listening to the Probes that are being fired by the Probe providers and could collect and eventually expose the stats (with our API) through asadmin/REST and MBeans.

-Prashanth

Posted by Prashanth Abbagani on September 23, 2009 at 01:27 AM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

bloggerprashanth

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today