Sunday Feb 27, 2011

Automatic Starting of Servers in GlassFish 3.1

Note: At the end of this blog are links to older blogs from V3.0 about this feature.  You may want to look them over as well.

The way we offer the ability to automatically start GlassFish servers is via "services" on the platform.  "Services" are an ancient technology, available on all of our supported platforms (and I would imagine is available on every serious OS).  Services allow applications to run automatically when a computer boots up.  Nobody needs to login -- the service just starts automatically.  A simple example of this is the FTP server.  When you boot up your computer you can access FTP from any other computer without touching the FTP computer.  This is very important in the event of a power failure.  On Windows you can setup automatic security updates and Windows will automatically reboot your computer.  If you have a Subversion server that is not configured as a service and you are halfway around the world depending on it -- you will become a Services Specialist in the near future when you return home!

GlassFish 3.1 Platform Services now supports all Linux versions, Windows and Solaris10/SMF

Services can be quite complex.  For instance you might want to have it do this:

  1. Automatically start a GlassFish server upon booting
  2. If the server crashes (no way!) restart it
    What if it crashes every time?  The machine will be in trouble with an infinite loop so:
  3. Try to restart 3 times and then give up.

Setting up this level of services-granularity is very difficult and expensive to implement perfectly.  For one - we are supporting several platforms and they all do these things completely differently. The hard part is setting up services at all to do the basic things.  Once that is complete you can easily use the platform's native tools for adjusting things just the way you like.  What we support is the classic services model -- which is to start the service upon booting, unattended.

I worked hard to make it easy to setup services.  Here's how you would create them for an installation that has one Domain:

asadmin create-service

If you want to make sure it would work but you aren't yet ready to pull the trigger, or you just want to see what it would do -- then run this command:

asadmin create-service --dry-run

If your domain is named, say. domain1 then the service's name will be domain1.  Simple.

 After the service is created you are greeted with a platform-specific message giving you the details on how to start the service.  We do NOT automatically start it.  I personally recommend rebooting immediately -- and it will start.

Windows is particularly easy and flexible to work with.  You can manage the GlassFish service(s) by:

  1. GUI -- right-click on Computer/Manage/Services
  2. sc.exe  very handy tool for querying/starting/stopping/restarting etc.
  3. In the bin directory of the domain or instance run the your-service-nameService.exe application that we put in there. 

How to setup automatic starting for an instance?

Easy!  Simply give the name of the instance as the final argument to create-service.  The following command will create a service for myinstance:

asadmin create-service myinstance

There is an undocumented, unsupported command that will delete a service for you.  It is very very easy to delete a service on SMF and Windows.  On Linux it can be tedious.  I've personally tested the command successfully but it has not been through the merciless testing of QA at Oracle.  Use it at your own risk.

asadmin _delete-service 

Important blog for making sure GlassFish doesn't stop when you logout.

I have another blog with implementation details if you are interested.

Some older blogs that pertain to V3.0:

V3.0 Platform Services for Windows and SMF

DIY Cookbook for Configuring Linux Services

How to Run on Linux as a Non-Root User

Huge Performance Improvement

I was working on improving the performance and memory consumption in Monitoring and I bumped into one problem that caused gigantic performance degradation.  I'm talking about a 100 fold speed-up for large (5000-20000) data sets.

 The code was adding Strings into a data structure.  The way it was generating the Strings was like this:


String s;  // we are actually all done now.  "s" is fine

s = MessageFormat.format("{0}", s);

 I simply got rid of that line and everything was fixed!

Moral:  Be careful with MessageFormat

GlassFish 3.1 Scripting of Monitoring Data for Clusters and Instances

Monitoring Scripting is a value-add feature and is available only in the Oracle GlassFish server which is available here.

GlassFish 3.1 introduces Clusters and Server Instances.  Monitoring now includes full support of these new features.  In this blog I'll demonstrate a simple application of monitoring-scripting in a clustered environment. 

First we need to setup a cluster and start everything:
  1. Start DAS (Domain Administration Server)
    asadmin start-domain
  2. Create a cluster with 2 instances -- they will all run on this same machine
    asadmin create-cluster c1
    asadmin create-local-instance --cluster c1 i1
    asadmin create-local-instance --cluster c1 i2
  3. Set the monitoring level to HIGH for the cluster.  By default it is OFF
    asadmin enable-monitoring --modules web-container=HIGH --target c1
  4. Start the cluster of 2 instances
    asadmin start-cluster c1
  5. Check and make sure everything is running OK
    asadmin list-domains
    domain1 running
    asadmin list-instances --long
        i1    localhost  24848  2464  c1        running
        i2    localhost  24849  5188  c1        running
  6. Deploy a simple app to the cluster
    asadmin deploy --target c1 hello.war
  7. "Hit" the Hello app on both instances
    wget http://localhost:28080/hello
    wget http://localhost:28081/hello

Now we have 3 servers running: the DAS and 2 clustered instances.  The web-container is sending events when interesting things happen.  Notice that we really have not done anything special for monitoring except the one enable command.  This looks complicated -- but heck you now have a running cluster with a deployed app etc.  with just a few quick commands.  Not bad! 

We want to monitor our Hello app.  I'm especially interested in seeing if, say, my Load Balancer is distributing client calls to Hello equally.  The first step is to get a list of all the probes.  I won't show the list here -- there are 136 probes.  Just run the command and you can see all of them.  I'm interested in the event that fires just before a servlet is run.  So I run this command:

 asadmin list-probes | grep servlet | grep before

glassfish:web:servlet:beforeServiceEvent (java.lang.String servletName, java.lang.String appName, java.lang.String hostName)

This describes one probe.  Everything you need to listen to this probe's event is in this one line.  So now we create a simple JavaScript program.  I'll just enter it here, JavaScript is documented in the official Oracle GlassFish Server and elsewhere.  It's pretty obvious what's going on by reading the program's source code.

function requestStartEvent(servletName,appName, hostName) {
        client.print( 'requestStartEvent called on App: ' + appName +
                      ', Servlet: ' + servletName + '\\n');

params = java.lang.reflect.Array.newInstance(java.lang.String, 3);

scriptContainer.registerListener('glassfish:web:servlet:beforeServiceEvent', params, 'requestStartEvent');

I saved the script as servlet.js

 Now I run this script against the cluster and I start hitting hello in each instance (28080, 28081) sort of randomly just to get some output.   We see output every time Hello (or any servlet) is accessed.  Notice how it shows which server is affected:

D:\\gf_other\\value-add\\monitoring\\scripting\\samples>asadmin run-script --target c1 servlet.js
Listening to i1 on host localhost and port 24848
Listening to i2 on host localhost and port 24849

[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: default
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: default
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp
[i2] requestStartEvent called on App: hello, Servlet: jsp
[i1] requestStartEvent called on App: hello, Servlet: jsp

The beauty of this is that once you have the clusters and everything setup and running you can run scripts on all sorts of probes very quickly.  You don't even have to compile.  Simply edit the script and call run-script

For a nice demonstration watch the video that Tom Mueller created here.

Monitoring Data -- How To Open The Firehose

Are you interested in getting raw Monitoring Data without using JavaScript or Dtrace?  That's why we include the special command:

asadmin get -m

Looks can be deceiving.  It appears that the -m is just some command line option.  Not so.  That turns on a huge pile of code that plows through the run-time tree of monitoring data.  This data can be accessed from instances, clusters of instances and DAS -- in fact ALL the data from all of them can be combined and accessed all at the same time!

This command was available in V3.0  Unfortunately it was very slow if you wanted to collect everything (what I call Firehose Mode).  So what was done was a lot of regular expression parsing to narrow down the matching data in order to get things going faster.

I reworked this command for GlassFish 3.1 to support the clustering feature.  I started DAS (Domain Administration Server) and one instance.  I ran

asadmin get -m "\*"  // Firehose output for DAS and the instance

Then I waited.  And waited.  And waited and waited.  About 90 seconds later the server got an out-of-memory error.  I never saw any output at all.  Oops!

I discovered the performance bottle-neck.  It was unbelievable what the problem is.  If you are interested I've blogged about it here.

We spent a lot of time trying to get the get -m command to handle regular expressions well.  Below are some examples. 

My environment: I have a cluster, c1.  c1 has 2 instances, i1 and i2.  Both instances are running with monitoring-levels all set to HIGH.  DAS is also running.  All servers are running on the same machine so I don't muddy the water with network latency etc.

  • asadmin get -m c1 # this retrieves ALL monitoring data for every running instance in c1 (~14,000 lines of output)
  • asadmin get -m c1.\* # does exactly the same thing as above
  • asadmin get -m i2.web.servlet.activeservletsloadedcount-current  # Example drill-down.
    i2.web.servlet.activeservletsloadedcount-current = 6
  • asadmin get -m "\*" | wc -l # returned 20,000+ lines.  It took around 1 second on my laptop.  
  • asadmin get -m \*.web.servlet.activeservletsloadedcount-current
    server.web.servlet.activeservletsloadedcount-current = 2  #DAS
    i1.web.servlet.activeservletsloadedcount-current = 6  
    i2.web.servlet.activeservletsloadedcount-current = 6

The command is so incredibly fast you can simply collect everything in a file and then run text processing commands on that file and avoid calling the command multiple times:

asadmin get -m "\*" > output

grep hello output # just an example

A Taste of Monitoring in GlassFish 3.1

Ok, so you've installed the Oracle GlassFish Server that you got from here.  You've created a cluster with a couple of instances.  The instances seem to be getting a bunch of traffic.  At least there are a lot of flashing lights on the ethernet cards and disk light is getting hammered.  But how can you find out what's really going on?  

That's the whole point of Monitoring.  We have provided a rich platform for getting Monitoring information in GlassFish.  You can ask for a trickle by zeroing-in on exactly the scraps of data you are interested in (e.g. please tell me every time the web-container creates a new thread) or turning the fire hose on and getting every monitored-event as it happens.

How do you want to see the data?

  •  In a browser? -- no problem, use either the Admin GUI or REST.
  • You are running in Solaris 10+ and you want to use DTrace?  We support it -- all monitorable events can be seen via native DTrace.
  • You want all of your Applications to get added into this cool Monitoring framework so you can access the data just like the built-in GlassFish Monitoring data?  No sweat, we have provided an easy way to add Monitoring probes to your applications.
  • Not good enough?  Too generic?  You want to be able to collect the data dynamically and structure it some other way using a JavaScript program?  Again, no problem.  You can run a JavaScript program, access as much or as little data as you want, dynamically,  and go crazy massaging the data any way you like.

There are a lot of facets to Monitoring in GlassFish 3.1  We will present many blogs with the details to get you going.  As those blogs are completed I'll circle back here and add the links.  In the meantime -- here is a very simple example of a "What-If" question and how to get the answers quickly.

How many classes have been loaded by the Domain Administration Server's JVM and what happens with the count when an app is deployed and then accessed?

asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 3887
asadmin deploy hello.war
Application deployed with name hello.
asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 5979
wget http://localhost:8080/hello
asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 6816

Wow!  That's a lot of classes.  Probably the jump from 3887 to 5979 was loading the web-container (this server had no apps deployed yet) and the next big pile of classes were jsp and servlet classes.  Let's do a sanity check by deploying another simple app:

asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 6819
asadmin deploy Math.war
Application deployed with name Math.
asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 6902

wget http://localhost:8080/Math
asadmin get -m server.jvm.class-loading-system.totalloadedclass-count-count
server.jvm.class-loading-system.totalloadedclass-count-count = 6906

That's semi-interesting.  Deploying a simple web module (with a servlet and jsp)  results in 83 classes getting loaded.  Running the app results in 4 classes being loaded.

Monitoring Blogfest

Running Monitoring JavaScript Programs on Clusters and Instances

Automatic Starting Implementation Details for GlassFish 3.1

Process Nuts and Bolts

When you start a GlassFish 3.1 Server Service notice that there are 2 JVM processes running.  The 2 process are these:

  1. The GlassFish Server
  2. A JVM running asadmin start-[local-instance|domain] --verbose

Why?  We want to leverage the start command.  Asadmin does a great deal of work for you when you start a server.  The end-product of that work is mainly (but not only) the generation of a big complex JVM invocation command to start the server with.  I could have implemented the Services to call asadmin and have asadmin exit.  But then the Platform would see that asadmin exited and would report the service as "stopped".  It would not know in the future when the server really stops. 

When you start a server with the --verbose flag then the asadmin process waits for the server to die before exiting.  When the server stops -- asadmin stops and the Platform will update the state to "stopped".

Note that restart-[domain|instance] will still work perfectly.  In that case the server process dies but the asadmin process does not - not even for a moment.  The platform will not "see" anything happening.


I discovered that there seems to be a special way to create services for every different flavor of Linux.  And they are all different from each other.  Basically all of these different ways end up with writing out special files into a special area.  This one common denominator is to use the ancient UNIX way of setting up services which is, not surprisingly, editing lots of configuration files in a special area.  Even the exact details on the special area is different among the different flavors.  The code is flexible and will handle all known Linux flavors. The script can be found in /etc/init.d and links to it in the runlevel directories:  /etc/init.d/rc?.d

Saturday Feb 13, 2010

How to make GlassFish 3.1 Platform Services Survive Logoff

Refer to my other blog for general details about setting up Platform Services

A running JVM does not survive the user logging off unless you wave a magic wand over the JVM startup parameters.  It is very simple.  If you give the -Xrs option then the JVM will completely ignore logoff events (as well as others).  The solution in V3 is very simple but first let me give you a little background.  Skip to the end of  this blog to see the solution if you are in a hurry.

 In V2 all you needed to do was add -Xrs as a jvm-option to domain.xml and you were all set.  If you tried this on V3 you'll have noticed that it does not work.  The reason is that V3 running as a service is really two separate JVM's.  There is an asadmin "watchdog" JVM and the V3 JVM.  This arrangement is exactly like running asadmin start-domain --verbose at a command line.  These 2 JVMs go through life together.  If one dies then the other is automatically killed.  That's the problem.  You can set the -Xrs flag in domain.xml but if the asadmin JVM does not also have it set, then logging off will kill the asadmin JVM which, in turn, will automatically take the V3 JVM down with it. 

The solution is simple: set -Xrs on both JVMs.  I verified that V3 running as a service will only survive a logoff if both JVM's have it set.  I tested all 4 combinations:

 domain.xml   -Xrs ?

 asadmin   -Xrs?

 Survives Logoff event?













My recommendation is to set -Xrs for asadmin once and then forget about it.  You can always adjust the behavior exclusively from domain.xml.


  1. asadmin create-jvm-options -Xrs

  2. edit install-root/bin/asadmin[.bat] and add the -Xrs option to the java command

To turn off the behavior run this command:

asadmin delete-jvm-options -Xrs

Tuesday Apr 24, 2007

Tip #-3 How to Debug Clustered AppServer Instances

There used to be a big problem in trying to attach a debugger to a clustered server instance in Glassfish.  As of today (build 45), it is a snap to do it!

The problem was that a cluster config could only specify one debugging port.  This would work if you started one and only one server instance.  As soon as you tried to start a second instance it would try to use the same jdwp port and it would fail to launch.  It was (almost) impossible to debug 2 instances on the same machine at the same time.

The solution is to defer the actual port number selection.  I.e. the instance needs to choose its own port.  Each instance specifies a different port and, voila, problem solved.

Here is how to do it:

1. Enable debugging support for the cluster
We are using a cluster named c1.  First make sure that debugging is enabled for the cluster:

asadmin set

2. Set a variable debugging port

The syntax is ugly.  Don't bother memorizing just do a get to see it, edit it and then do a set on the result like so: 

asadmin get = -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9009
asadmin set"-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=${DEBUG_PORT}"
Don't forget the double quotes, and no spaces around the equals sign!

 Another Tip:  asadmin get "\*" > afile is handy.  You can then grep for 'debug' in the file.  I use this technique all the time.  What human being wants to memorize this sort of crushing detail?

3. Set the instances' debugging ports

 In this case we have 2 instances, i1 and i2.  We will make the debugging ports 11111 and 22222 respectively:

GlassFish v2.x:

asadmin set i1.system-property.DEBUG_PORT=11111
i1.system-property.DEBUG_PORT = 11111
asadmin set i2.system-property.DEBUG_PORT=22222
i2.system-property.DEBUG_PORT = 22222

The above does NOT work in GlassFish 3.X.  Instead use the following commands:

asadmin create-system-properties --target i1 DEBUG_PORT=11111
asadmin create-system-properties --target i2 DEBUG_PORT=22222

That's it.  You're ready to debug multiple instances.




« September 2016