Sunday Feb 27, 2011

A Taste of Monitoring in GlassFish 3.1

Ok, so you've installed the Oracle GlassFish Server that you got from here.  You've created a cluster with a couple of instances.  The instances seem to be getting a bunch of traffic.  At least there are a lot of flashing lights on the ethernet cards and disk light is getting hammered.  But how can you find out what's really going on?  

That's the whole point of Monitoring.  We have provided a rich platform for getting Monitoring information in GlassFish.  You can ask for a trickle by zeroing-in on exactly the scraps of data you are interested in (e.g. please tell me every time the web-container creates a new thread) or turning the fire hose on and getting every monitored-event as it happens.

How do you want to see the data?

  •  In a browser? -- no problem, use either the Admin GUI or REST.
  • You are running in Solaris 10+ and you want to use DTrace?  We support it -- all monitorable events can be seen via native DTrace.
  • You want all of your Applications to get added into this cool Monitoring framework so you can access the data just like the built-in GlassFish Monitoring data?  No sweat, we have provided an easy way to add Monitoring probes to your applications.
  • Not good enough?  Too generic?  You want to be able to collect the data dynamically and structure it some other way using a JavaScript program?  Again, no problem.  You can run a JavaScript program, access as much or as little data as you want, dynamically,  and go crazy massaging the data any way you like.

There are a lot of facets to Monitoring in GlassFish 3.1  We will present many blogs with the details to get you going.  As those blogs are completed I'll circle back here and add the links.  In the meantime -- here is a very simple example of a "What-If" question and how to get the answers quickly.

How many classes have been loaded by the Domain Administration Server's JVM and what happens with the count when an app is deployed and then accessed?

asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 3887
asadmin deploy hello.war
Application deployed with name hello.
asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 5979
wget http://localhost:8080/hello
asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 6816

Wow!  That's a lot of classes.  Probably the jump from 3887 to 5979 was loading the web-container (this server had no apps deployed yet) and the next big pile of classes were jsp and servlet classes.  Let's do a sanity check by deploying another simple app:

asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 6819
asadmin deploy Math.war
Application deployed with name Math.
asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 6902

wget http://localhost:8080/Math
asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 6906

That's semi-interesting.  Deploying a simple web module (with a servlet and jsp)  results in 83 classes getting loaded.  Running the app results in 4 classes being loaded.

Monitoring Blogfest

Running Monitoring JavaScript Programs on Clusters and Instances

Huge Performance Improvement

I was working on improving the performance and memory consumption in Monitoring and I bumped into one problem that caused gigantic performance degradation.  I'm talking about a 100 fold speed-up for large (5000-20000) data sets.

 The code was adding Strings into a data structure.  The way it was generating the Strings was like this:


String s;  // we are actually all done now.  "s" is fine

s = MessageFormat.format("{0}", s);

 I simply got rid of that line and everything was fixed!

Moral:  Be careful with MessageFormat

Automatic Starting Implementation Details for GlassFish 3.1

Process Nuts and Bolts

When you start a GlassFish 3.1 Server Service notice that there are 2 JVM processes running.  The 2 process are these:

  1. The GlassFish Server
  2. A JVM running asadmin start-[local-instance|domain] --verbose

Why?  We want to leverage the start command.  Asadmin does a great deal of work for you when you start a server.  The end-product of that work is mainly (but not only) the generation of a big complex JVM invocation command to start the server with.  I could have implemented the Services to call asadmin and have asadmin exit.  But then the Platform would see that asadmin exited and would report the service as "stopped".  It would not know in the future when the server really stops. 

When you start a server with the --verbose flag then the asadmin process waits for the server to die before exiting.  When the server stops -- asadmin stops and the Platform will update the state to "stopped".

Note that restart-[domain|instance] will still work perfectly.  In that case the server process dies but the asadmin process does not - not even for a moment.  The platform will not "see" anything happening.


I discovered that there seems to be a special way to create services for every different flavor of Linux.  And they are all different from each other.  Basically all of these different ways end up with writing out special files into a special area.  This one common denominator is to use the ancient UNIX way of setting up services which is, not surprisingly, editing lots of configuration files in a special area.  Even the exact details on the special area is different among the different flavors.  The code is flexible and will handle all known Linux flavors. The script can be found in /etc/init.d and links to it in the runlevel directories:  /etc/init.d/rc?.d

GlassFish 3.1 Scripting of Monitoring Data for Clusters and Instances

Monitoring Scripting is a value-add feature and is available only in the Oracle GlassFish server which is available here.

GlassFish 3.1 introduces Clusters and Server Instances.  Monitoring now includes full support of these new features.  In this blog I'll demonstrate a simple application of monitoring-scripting in a clustered environment. 

First we need to setup a cluster and start everything:
  1. Start DAS (Domain Administration Server)
    asadmin start-domain
  2. Create a cluster with 2 instances -- they will all run on this same machine
    asadmin create-cluster c1
    asadmin create-local-instance --cluster c1 i1
    asadmin create-local-instance --cluster c1 i2
  3. Set the monitoring level to HIGH for the cluster.  By default it is OFF
    asadmin enable-monitoring --modules web-container=HIGH --target c1
  4. Start the cluster of 2 instances
    asadmin start-cluster c1
  5. Check and make sure everything is running OK
    asadmin list-domains
    domain1 running
    asadmin list-instances --long
        i1    localhost  24848  2464  c1        running
        i2    localhost  24849  5188  c1        running
  6. Deploy a simple app to the cluster
    asadmin deploy --target c1 hello.war
  7. "Hit" the Hello app on both instances
    wget http://localhost:28080/hello
    wget http://localhost:28081/hello

Now we have 3 servers running: the DAS and 2 clustered instances.  The web-container is sending events when interesting things happen.  Notice that we really have not done anything special for monitoring except the one enable command.  This looks complicated -- but heck you now have a running cluster with a deployed app etc.  with just a few quick commands.  Not bad! 

We want to monitor our Hello app.  I'm especially interested in seeing if, say, my Load Balancer is distributing client calls to Hello equally.  The first step is to get a list of all the probes.  I won't show the list here -- there are 136 probes.  Just run the command and you can see all of them.  I'm interested in the event that fires just before a servlet is run.  So I run this command:

 asadmin list-probes | grep servlet | grep before

glassfish:web:servlet:beforeServiceEvent (java.lang.String servletName, java.lang.String appName, java.lang.String hostName)

This describes one probe.  Everything you need to listen to this probe's event is in this one line.  So now we create a simple JavaScript program.  I'll just enter it here, JavaScript is documented in the official Oracle GlassFish Server and elsewhere.  It's pretty obvious what's going on by reading the program's source code.

function requestStartEvent(servletName,appName, hostName) {
        client.print( 'requestStartEvent called on App: ' + appName +
                      ', Servlet: ' + servletName + '\\n');

params = java.lang.reflect.Array.newInstance(java.lang.String, 3);

scriptContainer.registerListener('glassfish:web:servlet:beforeServiceEvent', params, 'requestStartEvent');

I saved the script as servlet.js

 Now I run this script against the cluster and I start hitting hello in each instance (28080, 28081) sort of randomly just to get some output.   We see output every time Hello (or any servlet) is accessed.  Notice how it shows which server is affected:

D:\\gf_other\\value-add\\monitoring\\scripting\\samples>asadmin run-script --target c1 servlet.js
Listening to i1 on host localhost and port 24848
Listening to i2 on host localhost and port 24849

[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: default
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: default
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp

The beauty of this is that once you have the clusters and everything setup and running you can run scripts on all sorts of probes very quickly.  You don't even have to compile.  Simply edit the script and call run-script

For a nice demonstration watch the video that Tom Mueller created here.

Saturday Feb 26, 2011

Folks, Individuals, Resources, Head-Counts - Lend Me Your Ears

Okay FOLKS, we are developing a new product and we need our RESOURCES to start work on it right away.   Naturally INDIVIDUALS can get involved. But no PEOPLE are allowed!!

 Why in modern-day English do we bend over backwards to avoid using the words "person" and "people".  They seem to fit the bill just fine to me.  At meetings at work I hear things like:  "We don't have enough RESOURCES to get that done!"  Water?  Oil?  Mining rights?  Computers?  Robots?  No!  He meant "We don't have enough PEOPLE to get that done!".

Folks is interesting because it is a German word (Volk und Volken).  I guess that it's OK to disguise that you're really saying person or people by using a foreign word that means people.

Police/Government speak loves to use the term "Individual".  "Some INDIVIDUALS were noticed hanging around the area".  Hmm -- Individual \*\*what\*\*?  Individual lions?  rocks?  resources?  Isn't INDIVIDUAL meant to differentiate from GROUP.  Now we have a group of INDIVIDUALS.  Aiiieeee!

I guess the answer to the question is that we like to dehumanize our speech.  Seems sad and imprecise to me.

Tuesday Jul 20, 2010

Local Strings in GlassFish - A Proposal

There are 2 heavily used classes that help with obtaining localized strings at runtime.  The strings themselves are maintained in one properties file per package.  Usually what happens is that every class in the package will create a static string helper class instantiation.  Since they are static they will last forever once the class is loaded.  If the package has a lot of classes then there will be a lot of these instances.  But the instances are all really the same thing.  They all point at the same properties file in the package. 

My proposal is to have an extremely light-weight helper class, in every package that uses localized strings.  This one class has only 2 simple static methods that return strings from that package's properties file.  The class is copied around to every package and, literally, only one line needs to be changed -- the package declaration.

  • Classes in the package need not import anything -- it is already in the package.
  • Everything in the class is package-private -- no public or protected.
  • The class is naturally thread-safe
  • Classes simply use it like so:
    • String msg = Strings.get("labelHere", "arg1", "arg2");

I have 2 goals with this proposal

  • Make it easier to localize strings in GlassFish code.  I don't know how to make it any easier!
  • Avoid Code Duplication

Example Implementation.  You literally copy, paste and change the package name.  This implementation  uses LocalStringsImpl as the workhorse.  You could change things around  bit to use LocalStringManagerImpl instead.

package com.sun.enterprise.util.cluster;

import com.sun.enterprise.universal.i18n.LocalStringsImpl;

 \* Strings -- Get your Strings here.
 \* One file with Strings
 \* So one class for messing with them!
 \* Nothing in here is public or protected.  Only for use by this one java package.
 \* @author Byron Nevins

final class Strings {
    private Strings() {
        // no instances allowed!

    final static String get(String indexString) {
        return strings.get(indexString);

    final static String get(String indexString, Object... objects) {
        return strings.get(indexString, objects);

    final private static LocalStringsImpl strings = new LocalStringsImpl(Strings.class);

Friday Jul 16, 2010

GlassFish V3.1 Offline Configuration

Clustering support is available now in pre-release GlassFish 3.1  In this blog I will show you how to perform Offline Configuration in GlassFish 3.1

I've blogged about  why and how to do Offline Configuration here.

One big difference between GlassFish V2 and GlassFish 3.1 clustering is that node agents are not provided and used in GlassFish 3.1.  Instead ssh is utilized.  Using ssh in GlassFish 3.1 is the subject of a different future blog.  In this blog I'll demonstrate how to setup a distributed GlassFish 3.1 system without node-agents or new ssh features.

The Desired Environment

  • DAS running on improvident
  • One cluster
  • The cluster has one instance
  • DAS and the instance run on different machines


  • The names are just the names of the machines I happened to use.
  • This is as simple a distributed system as possible for demonstration purposes. 
  1. machine1 = samskritam
  2. machine2 = improvident

  • install the same  GlassFish 3.1installation files on each machine. 
  • DAS (domain1) is used on samskritam. 
  • The other machine never runs DAS.
  • improvident has this set in its environment -- This makes it easy to run asadmin commands.
    • AS_ADMIN_HOST=samskritam
  • In the Procedure section I'm using these conventions:
    1. I put the name of the machine where the command is running at the beginning of the line - you wouldn't type that in.
    2. Comments are in italics
    3. Output from asadmin is bold


  1. [samskritam] asadmin start-domain
  2. [samskritam] asadmin create-cluster mycluster
  3. [samskritam] asadmin create-node-config --nodehost improvident imp
  4. [samskritam] asadmin create-instance --cluster mycluster    --node imp mycluster_i1
  5. [improvident] asadmin create-local-instance --node imp mycluster_i1
  6. [improvident] asadmin start-local-instance mycluster_i1


On Samskritam:

samskritam% asadmin start-domain
Waiting for the server to start ....
Successfully started the domain : domain1
domain location: /home/kedar/glassfishv3/glassfish/domains/domain1
Log File: /home/kedar/glassfishv3/glassfish/domains/domain1/logs/server.log
Admin Port: 4848
samskritam% asadmin create-cluster mycluster
samskritam% asadmin create-node-config --nodehost improvident imp
samskritam% asadmin create-instance --cluster mycluster    --node imp mycluster_i1
remote failure:
Successfully created instance mycluster_i1 in the DAS configuration, but failed to create the instance files. Please run:

asadmin --host samskritam.SFBay.Sun.COM --port 4848 create-local-instance --node imp mycluster_i1

on improvident to complete the transaction.


On improvident:

~>asadmin create-local-instance --node imp mycluster_i1
Attempting to rendezvous with DAS on host samskritam port 4848
Uptime: 1 minutes, 31 seconds

The instance has rendezvoused with the DAS and will be using host samskritam port 4848 for future communication.

~>asadmin start-local-instance mycluster_i1
Waiting for the server to start ......................................
Successfully started the instance: mycluster_i1
instancelocation: /export/home/bnlocal/glassfishv3/glassfish/nodeagents/improvident/mycluster_i1
Log File: /export/home/bnlocal/glassfishv3/glassfish/nodeagents/improvident/mycluster_i1/logs/server.log
Admin Port: 24848
~>asadmin list-clusters
mycluster running

~>asadmin list-instances --verbose
Instance Name   Host Name     Admin Port   Cluster     Current State
mycluster_i1    improvident      24848      mycluster  Uptime: 15 minutes, 11 seconds

Thursday Mar 04, 2010

Run GlassFish V3 as a non-root Service on Linux Ubuntu/Debian


I already showed you how to setup GlassFish V3 as a Service on Linux here.  In that blog I used root as the GlassFish "owner" for simplicity.  In this blog we will do the same thing except with a non-root user.  If you already have a user and/or you have V3 installed you can skip some of the earlier steps.  Once again my main motivation is to get these procedures worked out carefully so that I can implement it automatically.  Currently we support Windows and Solaris 10.  For each step I'll also show you how to undo the change in case you want to rollback everything.

1. Create a user

I decided to create my own GlassFish user that exists solely to run GlassFish.  You could use your own existing account as well.  I also created a new group for use only by this user.  I.e. I don't want this user to have any Linux admin privileges.

sudo groupadd glassfish
sudo useradd -s /bin/bash -d /home/glassfish -m -g glassfish glassfish 


sudo userdel glassfish
sudo groupdel glassfish

2. Login as the new user in a terminal window and set a password (if desired)

sudo passwd glassfish
sudo -i -u glassfish

3. Install GlassFish V3 using user glassfish

cd ~


rm -rf ~/glassfishv3


4. Exit the shell from step 2


5. Copy the script to /etc/init.d  and configure it as a Service

sudo cp the-script /etc/init.d/glassfish
sudo update-rc.d glassfish defaults


sudo rm /etc/init.d/glassfish
sudo update-rc.d glassfish remove

All Done

You can start, stop or restart GlassFish V3 like so.  Note that you must do it with an account that has admin privileges.  The user "glassfish" ironically can not do this even though the server will run as "his" process.

sudo /etc/init.d/glassfish start|stop|restart

How to Allow GlassFish V3 to use ports less than 1024

 This technique configures the firewall to send all port 80 traffic to port 8080.  You can do the same thing for any other ports you need that are less than 1024.  I don't know how to undo it.

iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080
iptables -t nat -A PREROUTING -p udp -m udp --dport 80 -j REDIRECT --to-ports 8080

Service Script

# Platform Services for GlassFish

SU="su --login  $GF_USER --command  "

case "$1" in

    $SU "$ASADMIN start-domain > /dev/null 2>&1 &"



    $SU "$ASADMIN stop-domain > /dev/null 2>&1 &"



    $SU "$ASADMIN restart-domain > /dev/null 2>&1 &"


    echo "usage: $0 (start|stop|restart|help)"


Wednesday Feb 24, 2010

How to Run GlassFish V3 as a Service on Linux Ubuntu/Debian

We already support running GlassFish V3 as a service on Solaris 10 and Windows platforms (see my blog).  I have been investigating how to provide support for automatically starting GlassFish V3 as a service on Linux.  Of course before I can hope to do that -- I must be able to set it up manually.  In this blog I will take you through the manual steps needed to run GlassFish V3 as a service on Linux.

This procedure was worked out on my Linux system which happens to be Ubuntu.  Other flavors of Linux may have slightly different procedures.

One decision you need to make right up front is what Linux user should "own" GlassFish V3.  Typically root is used as the owner.  If you choose root as the user you get the advantage that you can use ports < 1024 without complex configuration changes to the system.  For this blog I used root.

 Here are the steps  -- the file named "glassfish" in step 4 is the simple init script which appears at the end of this blog.

  1. Install JDK 6 if needed
  2. Have root install GlassFish like so:
    1. cd /opt
    2. wget
    3. unzip
    4. rm
  3. cd /etc/init.d
  4. cp glassfish .
  5. update-rc.d glassfish defaults
  6. OPTIONAL /etc/init.d/glassfish start
  7. OPTIONAL Make sure GlassFish is running OK
  8. reboot -- you are done!

To start, stop, restart GlassFish simply run these commands:

sudo /etc/init.d/glassfish start
sudo /etc/init.d/glassfish stop
sudo /etc/init.d/glassfish restart


# glassfish init script for Linux
# Simplest possible case -- no password file, one default domain
# it would be simple to add such options


case "$1" in
    $GLASSFISH_HOME/bin/asadmin start-domain >/dev/null
    $GLASSFISH_HOME/bin/asadmin stop-domain >/dev/null
    $GLASSFISH_HOME/bin/asadmin restart-domain >/dev/null
    echo "usage: $0 (start|stop|restart|help)"

Monday Feb 15, 2010

GlassFish v2 Offline Configuration

Review of Clustering Features in GlassFish v2

A typical GlassFish v2 enterprise installation consists of one administration domain controlled by the Domain Administration Server (DAS) which handles all of the administration duties of the domain.  The  domain provides a common  authentication and administration point for a collection of Java EE server instances.

Each JavaEE server instance gets it's configuration from DAS and has its lifecycle controlled by a Node Agent.  A Node Agent is a very light-weight process that DAS interacts with.  Node Agents and remote Java EE server instances synchronize with DAS during startup to get the latest configuration from the central repository managed by DAS.

GlassFish v2 supports the concept of cluster where multiple Java EE server instances can be grouped for horizontal scalability. Each Java EE server instance in a cluster shares the same set of applications and configuration information.

Typical Configuration

Say you have a running DAS (Domain Administration Server) and you wish to add a cluster, a few instances and a Node Agent running on a separate machine.  Here is how you would do an online configuration:

  1. Login to the new Node Agent machine
  2. asadmin create-node-agent NA-Name
  3. asadmin create-cluster
  4. asadmin create-instance (probably need to run this command a few times)
  5. asadmin start-node-agent

 You can see how if you have a lot of machines this can become a headache.  Note that create-instance has many required parameters that I'm not showing here.

Offline Configuration

Offline Configuration lets you do all of the configuration right from the DAS machine.  On each Node Machine you run two very simple commands:

  • asadmin create-node-agent NA-Name
  • asadmin start-node-agent NA-Name

There are several good reasons for developing scripts for offline configuration of complex GlassFish installations:

  • You can develop scripts that will exactly recreate your system from scratch.  I'm talking about deploying your applications, setting complicated jvm-options, configuring database connections, setting up multiple clusters and instances -- the works!  This is very powerful.  I have a personal website that has been running v2 for 4 years or so.  I can recreate the entire site anytime I like on any new machine that has v2 installed.  It has a ton of applications, a database, etc.
  • You need the complicated configuration information to be used on just one machine, DAS, in a multiple machine installation.
  • You can replace a Node easily at any time
  • No restarts are needed during the creation of the instance, cluster and node-agent configuration since it is all being done offline.  Once ALL of the configuration is complete everything is started at once and no restarts will be needed.  This is a key difference with online configuration.
  • Faster to automate a multi-node GlassFish cluster setup with desired configuration.
  • It works!

Example Offline Configuration Session

Generally one would want to edit a script and run the script.  Here is a an example session to show you how to do it from a command line:

(For clarity the credentials such as --user and --passwordfile are left out of the commands)

  • asadmin start-domain
  • asadmin create-cluster c1
  • asadmin create-node-agent-config na1
  • asadmin create-node-agent-config na2
  • asadmin create-instance --cluster c1 --node-agent na1 i1_1
  • asadmin create-instance --cluster c1 --node-agent na2 i2_1
  • asadmin create-instance --cluster c1 --node-agent na2 i2_2

We didn't start anything.  All of the commands changed the configuration in the domain.xml.  Namely it created 1 new cluster with 2 node agents and a total of 3 instances.

To get everything running we do this:

  • For each machine (in this case 2 separate remote machines with v2 binaries installed)
    • asadmin create-node-agent na1 or na2
    • asadmin start-node-agent na1 or na2
    • note: the Node Agent name must match exactly the name in create-node-agent-config above.  Otherwise you will get a new Node Agent created!

Example Script

Kedar Mhaswade created  the attached example shell script.  The script demonstrates how you can automate building a four-machine Enterprise GlassFish v2 Installation.


Saturday Feb 13, 2010

How to make GlassFish 3.1 Platform Services Survive Logoff

Refer to my other blog for general details about setting up Platform Services

A running JVM does not survive the user logging off unless you wave a magic wand over the JVM startup parameters.  It is very simple.  If you give the -Xrs option then the JVM will completely ignore logoff events (as well as others).  The solution in V3 is very simple but first let me give you a little background.  Skip to the end of  this blog to see the solution if you are in a hurry.

 In V2 all you needed to do was add -Xrs as a jvm-option to domain.xml and you were all set.  If you tried this on V3 you'll have noticed that it does not work.  The reason is that V3 running as a service is really two separate JVM's.  There is an asadmin "watchdog" JVM and the V3 JVM.  This arrangement is exactly like running asadmin start-domain --verbose at a command line.  These 2 JVMs go through life together.  If one dies then the other is automatically killed.  That's the problem.  You can set the -Xrs flag in domain.xml but if the asadmin JVM does not also have it set, then logging off will kill the asadmin JVM which, in turn, will automatically take the V3 JVM down with it. 

The solution is simple: set -Xrs on both JVMs.  I verified that V3 running as a service will only survive a logoff if both JVM's have it set.  I tested all 4 combinations:

 domain.xml   -Xrs ?

 asadmin   -Xrs?

 Survives Logoff event?













My recommendation is to set -Xrs for asadmin once and then forget about it.  You can always adjust the behavior exclusively from domain.xml.


  1. asadmin create-jvm-options -Xrs

  2. edit install-root/bin/asadmin[.bat] and add the -Xrs option to the java command

To turn off the behavior run this command:

asadmin delete-jvm-options -Xrs

Wednesday Dec 02, 2009

Mort Learns How To Use Monitoring in a WebApp

 This blog is a follow-up of my earlier monitoring blog, and I recommend reading that one first.

This time we will setup probes inside of a user application.  We will not be in the internal GlassFish environment.  We need to use the Monitoring tools available from GlassFish OSGi.  But -- we will be in a plain vanilla web module, not an OSGi module.  Oh no!  How can we access the Habitat inside GlassFish?  JavaEE 6 comes to the rescue with the @Resource annotation!

    private ProbeProviderFactory probeProviderFactory;

    private ProbeClientMediator listenerRegistrar;

These 2 variables are what we need to register a probe and hook up the probe with a listener.

The web app consists of the usual boilerplate song and dance needed for any web app.  The business end of this web app is in two classes:

  1. The Servlet class --
  2. The Probe Listener class --

The servlet class has this method:

public void myProbeMethod(String s) {
    System.out.println("inside myProbeMethod "         
        + "called with this arg: " + s);

and it has this annotation at the class level:

@ProbeProvider( moduleProviderName="fooblog", moduleName="samples", probeProviderName="ProbeServlet")

That's all you need to do to turn this servlet class into an official GlassFish V3 Probe!

To get the Probe registered with GlassFish you need to add one line of code that needs to run before you use the probe (I added it to the init method):


I made the servlet painfully simple.  Whenever you load or reload the servlet it calls myProbeMethod().  Of course all of this would be pointless without a listener class that is hooked-up to the probe. 

 The init code in the servlet hooks up the listener with this one line of code:

    new MyProbeListener());

But wait!  How in the world is that listener class going to be connected to the probe class?!?  This is where the magic of annotation processing comes in.  The listener class is super-simple.  Here it is in it's entirety:

public class MyProbeListener {
  public void probe1(String s) {
    System.out.println("PROBE LISTENER HERE!!!!");

Note how the strings in the ProbeProvider annotation are used in the listener to identify the correct probe.

This is a very simple example but you can see how you can create listeners in their own separate apps and have probes in many apps.  To see the output simply tail the server log.  When you write to System.out from a Listener - it will appear in the server log.

The app is available in open source as a maven web-app.  It can be checked out using this URL:

Note:  If you make changes to the Listener class you should restart the server after redeploying.

Sample output from loading the app in a browser:

....|PROBE LISTENER HERE.  Called with this arg: Hello #1|#]
....|inside myProbeMethod called with this arg: Hello #1|#]

Wednesday Sep 16, 2009

Mort Learns Monitoring and Admin Commands for GlassFish

  • Monitoring is a powerful feature in GlassFish V3.
  • Admin Commands are a powerful feature in GlassFish V3.  

In this blog I'll show you how to create an Admin Command that exercises the Monitoring Facilities.


  • ProbeProvider - A class or interface that has methods (Probes) that can be called at runtime.
  • Probe - A method that can be called at runtime that in turn will call any Listeners that are registered
  • Listener - A method that is registered to receive "events" coming from a Probe
  • Admin Command - An easy to call command running on GlassFish V3.  You can access it via REST, HTTP or asadmin.  You could easily do this whole tutorial inside a servlet instead.  I chose to use an Admin Command so you can see how to create and use commands.


In this tutorial we will work with a ProbeProvider interface.  We write an interface with the methods (Probes) that we wish to call at runtime.  We never create an implementation of this interface.  Instead we hand it off to GlassFish.  GlassFish will magically create an implementation class for the interface and return an instance of that implementation.  Under the hood, GlassFish has automagically inserted code into your methods that will call any Listeners that you subsequently register.

Using an interface is a fairly unusual case.  The much fancier and more usual case is that you give GlassFish an instance of any POJO that has the proper annotations.  GlassFish will then rewrite the actual bytecode of the class and add calls to the Listeners!

Admin Command

  1. The command should setup the Probes and Listeners correctly the first time the command is run
  2. Each subsequent call should fire the appropriate probe

 The complete class is here

You provide LCPP and LCListener.  LCPP is the Probe Provider interface and LCListener is the Listener class.  The 2 lines that do all the registering and magic are here:

probe = probeProviderFactory.getProbeProvider(LCPP.class);
listenerRegistrar.registerListener(new LCListener());

That's all!  You now have a probe object that you can call methods on that will automatically call your listener class.

Probe Provider Interface

The interface is here.  As you can see it is mostly  a bunch of annotations and 2 methods:

void ping();
void ping2(@ProbeParam("message") String s);

Note that they must return void   You should make all parameters primitives or String to avoid trouble.

Probe Listener

 The source is here.  Now you can see how the Listener gets hooked up with the Probe.  The Listener needs the values from the ProbeProvider annotation in the Probe Provider class.  It's a lot easier to see how to do it by looking at the source then for me to explain it.

How to Test

  1. Take the files and drop them into a GlassFish extension project.  If you have GlassFish source just do what I did -- drop them into the core/kernel module.  Or you can just use the files as a guide and, say, create a Servlet instead of using an Admin Command.
  2. build and deploy the jar
  3. start the server
  4. tail the server log (even better -- run  asadmin start-domain --verbose)
  5. call asadmin ping <optional-string-here>  
  6. Look for the listener messages in the log
  7. goto 5

Monday Jul 27, 2009

How to Configure Domain.xml to use the Environment

I've been debugging GlassFish today.  I need to set the debugger to suspend=y or suspend=n depending on what I am doing.  Instead of constantly editing domain.xml I simply do this:

<java-config debug-options="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=${SUSPEND},address=9009" system-classpath="" classpath-suffix="">

And then I do this in the OS:

 Windows:  set SUSPEND=y

UNIX: export SUSPEND=n


Environmental variables are automatically searched when a token appears in domain.xml.  It also searches System Properties and other properties inside domain.xml.

Sunday May 31, 2009

Platform Services Available in V3

Platform Services have just been rolled out for the first 2 platforms:

  • Solaris  Management Facility (SMF)
  • Windows

In the simplest case where you have one default domain and you want to run it as a service you run this command:

asadmin create-service

Here are all of the options:

[--passwordfile password-file] [--name service-name] [--serviceproperties (name=value)+] [--dry-run=false] [--domaindir domain_parent_directory] [domain_name]

 Once you have created the service it can be controlled in various ways by operating system and native tools. 


We use Windows Service Wrapper as the tool for installing the service.  It can be used to start stop and uninstall the service.  It can be found in the bin subdirectory of the domain directory.  In the default case the name of the executable will be domain1Service.exe  The directory also contains the configuration file, domain1Service.xml.  Below are the available commands:

  • domain1Service start
  • domain1Service stop
  • domain1Service uninstall
  • domain1Service install
You can also manage the GlassFish service with Windows tools:  Services snap-in and the net command line program.

Solaris  Management Facility (SMF)

You create the service exactly the same as WIndows.  One big difference for most users is that on Windows most users have administration privileges.  I.e. everyone is super-user.  Not so on Solaris.  In order to create a service you must have SMF privileges.  How to set that up is beyond the scope of this blog, but root definitely has these privileges by default 

 Important: If root creates a service then only root can start that domain from then on.  The best way to set this up is to install and/or create the domain as root and then create the service as root.

Once you have created the service you need to start it yourself.  Here are the most typical Solaris commands of interest:

  • /usr/bin/svcs  -a | grep domain1  // status
  • /usr/sbin/svcadm enable domain1 // start
  • /usr/sbin/svcadm disable domain1 // stop
  • /usr/sbin/svccfg delete domain1 // uninstall

Future Directions

If there is interest, demand and time we will add support for controlling the service from CLI:

  • start
  • stop
  • uninstall
  • restart




« April 2014