Under the hood of GlassFish v2 Clustering

GlassFish V2 is a production-ready Java EE 5 application server currently in beta. There was a lot of interest about GlassFish this year during JavaOne. I had a chance to talk to many customers interested in GlassFish. Some were already using Sun's distribution and dealing with very large deployments. In this blog, I will share few tricks and tips that may come in handy while dealing with GlassFish V2.

First, a quick re-cap. GlassFish V2 has support for profiles (OnePager, blog - [1], [2]) where the server installation is configured for different usage. Three profiles are available by default: developer, cluster, and enterprise. Developers would typically work with a single Java EE server instance and use the developer profile. Cluster and enterprise profiles are targeted for deployments.

A typical GlassFish V2 installation [in cluster or enterprise profile] consists of one or more administration domain. A domain provides a common authentication and administration point for a collection of Java EE server instances. Domain Administration Server (DAS) handles all the administrative operations in a domain.

Each remote machine [in cluster or enterprise profile] has a light weight process called Node Agent. DAS interacts with the Node Agent to manage the life-cycle operations (create, delete, start, stop) in a remote machine. Node Agents and remote Java EE server instances synchronizes with DAS during startup to get the latest configuration from central repository managed by DAS.

GlassFish V2 supports the concept of cluster where multiple Java EE server instances can be grouped for horizontal scalability. GlassFish V2 also supports in-memory replication feature where state of a deployed application is replicated to a peer in the cluster and thus enables the application to be highly available. Each Java EE server instance in a cluster shares the same set of applications and configuration information.

To summarize, GlassFish V2 offers enterprise features such as clustering (screencast, blog), high availability (screencast), and load balancing (GF page, blog) to support highly scalable, volume enterprise deployments for service-oriented architecture (SOA) and Web 2.0 applications. Refer to GlassFish V2 architecture wiki and JavaOne session TS-4436: Technical Overview of GlassFish v2 (abstract, blog) for more details.

Q: Does GlassFish V2 support hot deployment to a cluster?

Yes, GlassFish V2 offers multi-phase deployment to cluster and minimizes deployment related failures to distributed machines. Use the --target option in asadmin deploy command (example, %asadmin deploy --target cluster1 foo.war) or the Administration Console. Deployment is dynamic; i.e., deployed application bits are transferred automatically to the remote server instances. You don't need to restart the cluster after deploying an application. V2 also offers other deployment features such as auto-deploy, directory deploy, ant integration (blog), NetBeans Integration, etc. Refer to deployment documentation for more details.

Q: Is it possible to deploy the same application to multiple clusters?

Yes. Use asadmin create-application-ref (example, %asadmin create-application-ref --target cluster2 myApp ) and asadmin delete-application-ref commands with --target option in CLI or manage target for the deployed application in Administration Console.

Q: Is there a way to start a remote server instance when DAS is not running?

Yes, you may use the startserv/stopserv scripts under the bin directory (blog) or re-start the node agent. asadmin start-node-agent command will automatically start the remote server instances without synchronizing with DAS.

Q: How do I force a full synchronization of a remote server instance?

A hidden file named .com_sun_appserv_timestamp keeps track of the last successful synchronization time of DAS. If you remove these files manually (not an official interface) under the remote server instance(s), a full synchronization will happen during the next server re-start. If synchronization fails for any reason, these files are automatically removed to force a full synchronization. In GlassFish V2, significant improvement was made in synchronization performance (~40% - your mileage may vary).

The configuration under remote instances are treated as cache (for example, all files under nodeagents/na1/server1). In extreme cases, if you remove all files of a remote server instance and restart the node agent, the remote server instance (for example, server1) will be recreated and all necessary files will be synchronized. Refer to synchronization documentation for more details.

Q: Is there a way to backup/clone DAS so that it is not a single point of failure for management?

DAS has no role in application runtime functionality. So, when DAS is un-available, the cluster will continue to run. You may use %asadmin backup-domain command to create a backup of DAS. After that, please backup the zip from backup-domain command in a highly available storage. When /if DAS's machine crashes, you may use %asadmin restore-domain command with the previously backed-up zip to restore the domain in another machine. During DAS's startup, it will ping all the node agents to notify the new co-ordinates of DAS. Solaris [Sun] Cluster has support for Sun Java System Application Server. You may use it to provide automatic failover. Refer to this documentation for more details.

Q: Is there a way to specify machine specific JVM options for a server instance in a cluster?

Yes. You may use tokens (${token}) in the config element for a cluster and define the token property with different values for different server instance. Refer to administration reference guide for more details. Here is a quick example where we are setting max heap size for a server instance that is part of a cluster.

        <domain ...> <applications>...</applications> <resources>...</resources> <configs> <config ... name="cluster1-config"> ... <java-config ...> <jvm-options>${MAX_HEAP_SIZE}</jvm-options> </java-config> </config> </configs> <servers> <server ... config-ref="cluster1-config" name="server1"> <system-property name="MAX_HEAP_SIZE" value="-Xmx512m"/> </server> </servers> <clusters>...</clusters> <node-agents>...</node-agents> <lb-configs>...</lb-configs> <load-balancers>...</load-balancers> </domain>

Q: Is it possible to specify JNDI name at cluster scope? For example, is it possible to use two different connection pools for an application that is deployed to two clusters?

Yes. You may use the same tokenization strategy described above. Create a token for JDBC resource pool name and then specify the token property with desired (different) values for the two clusters. When application code looks up the JNDI name, it will end up connecting to two different database pools for the two different clusters. For example,

        <jdbc-resource ... jndi-name="jdbc/db" pool-name="${POOL_NAME}"> <cluster ... name="cluster1"> <system-property name="POOL_NAME" value="DerbyPool"/> </cluster> <cluster ... name="cluster2"> <system-property name="POOL_NAME" value="MySQLPool"/> </cluster>

Q: Say, I am dealing with a large deployment involving many clusters with multiple server instances (> 30). Is there a way to optimize DAS's foot-print?

During the cluster creation, instead of cloning the config for each cluster, use a reference to an existing config (example, cluster1-config). If you need to customize a value, use tokens and define them in each cluster. This will reduce the overall memory foot-print. Also, avoid deploying the applications to DAS (on Administration Console, target named server) unnecessarily. You may consider creating new domains (%asadmin create-domain) when appropriate.

Q: Is there a way share a library across a cluster?

Yes. If you copy the library jar under the cluster's config/lib directory (example, domains/domain1/config/cluster1-config/lib), the library will be available cluster wide. You will need to update the classpath-prefix or classpath-suffix. Refer to Sivakumar's blog for more classloader related issues. According to Siva's blog, library jars can be shared at domain and application scope also in GlassFish V2.

      • domains/domain1/lib - domain wide scope, common classloader add the jars automatically
      • domains/domain1/config/cluster1-config/lib - config wide, update classpath-prefix or classpath-suffix
      • domains/domain1/lib/applibs - application scope, added to application class loader automatically
      • domains/domain1/config/cluster1-config/lib/ext - adds to java.ext.dirs automatically

Q: Do you need special packaging of applications to use them in cluster?

No. Application code level changes are not needed. GlassFish V2 offers in-memory replication for high-availability with almost zero configuration also.

Q: Is it possible to install different versions on the same machine?

For GlassFish, this is not an issue. For SJS AS 9.1, you may also use the filebased installer. For native package based installation, SJS AS 8.x will be upgraded to SJS AS 9.1 on the same machine.

Q: Do you have support for monitoring?

GlassFish V2 offers a wide range of monitoring statistics. Statistics on server log, server runtime, deployed applications, resources and transactions are available. Refer to the following blogs ([1], [2]) for more details. Call Flow (tech tip) allows you to find out how a particular application response is being processed across different containers of the Application Server. Web services monitoring is also available (article) along with new improvements in CLI (blog) for mornitoring.

Q: Do you have any support for large data center deployments?

Yes, you may use N1 Service Provisioning System with SJS AS. We are working on a new plugin for SJS AS 9.1 (Sun's distribution of GlassFish V2) also.


Will the SJS AS 9.1 plugin for N1 SPS be as powerful as the plugin for BEA Weblogic? (possibility to create jdbc connections pools, war ,ear file deployment, all via N1 sps)

Posted by Thorleif Wiik on June 05, 2007 at 07:17 AM PDT #

Yes, SJS AS 9.1 plugin for N1 SPS will have support for installation, cluster creation, resource management, application deployment, HADB installation, etc. A big financial customer is currently using the plugin in production with SJS AS 8.x.

Posted by Nazrul on June 05, 2007 at 09:15 AM PDT #

Hi Nazrul. Great blog! One minor suggestion - it may be worth clarifying that the Architecture you are describing is for domains that use the Cluster profiles and that in Development profiles there is only a DAS involved, even in GFv2. - eduard/o

Posted by eduardo pelegri-llopart on June 05, 2007 at 02:03 PM PDT #

Hi Eduardo: I added some clarification about the profile support introduced GF V2. Thanks for the suggestion.

Posted by Nazrul on June 05, 2007 at 03:19 PM PDT #

Any (beta) release dates for the SJS AS 9.1 plugin for N1 SPS ?

Posted by Thorleif Wiik on June 07, 2007 at 03:32 AM PDT #

Yes, we are planning to do enterprise beta (beta 3) for SJS AS 9.1 in July 2007. If your company is interested to participate in this, please feel free to contact me.

Posted by Nazrul on June 07, 2007 at 04:59 PM PDT #

JDBC Update: Refer to these great blogs ([1], [2], [3], [4]) for JDBC connection pool related issues.

Posted by Nazrul on June 13, 2007 at 03:44 PM PDT #

Its a fine article which specifying almost functioning with clustering in glassfish

Posted by rajesh on October 10, 2008 at 10:10 PM PDT #

i am going to make project on web usage mining
i need an algorithm on clustering of weblogs

Posted by rahul on January 06, 2009 at 09:09 PM PST #

thank you for all those informations, it was very useful!

I was wondering, is it possible to use two different connection pools for an application that is deployed to ONE clusters?

Say, i have two clusters cloud, cloud1 and cloud2, with a load balancer in front of cloud1. Instances of cloud1 communicate with instances of cloud2 (via JNDI), but i only could make the instances of cloud1 communicate with the same instance of cloud2 (unique JNDI in the ear application).

So I'm looking for a workaround without having to place a load balancer between the clouds...

Any advise would be appreciated :)

Posted by Elyass on December 14, 2009 at 06:43 PM PST #

Hi Elyass,

No, you may not use two different connection pools for an application in the same cluster.


Posted by Nazrul on February 04, 2010 at 07:54 AM PST #

Your post is very helpful! I have a problem to setup the cluster, I wonder if you can give me some pointers. I successfully created the node agent on a different host, I then try to create a instance with that node agent, and when I start the instance, it says: " The Appserver instance test1234 does not exit. This could be caused by an error in a previous Node Agent Synchronization cycle. Please check the Node Agents log for the specific error." And I go to the remote host to look at the server.log for that node agent, it says something like TCP connection is locked, but then says nodeagent is starting the instance:
RMI TCP Connection(13)-;|NA-DEBUG: Got Lock: test1343|#]"
RMI TCP Connection(13)-;test1343;|NAGT0018:The nodeagent is starting the server instance test1343|#]"

Any pointer on how to solve it really appreciated.

Posted by Michelle Xue on April 23, 2010 at 09:01 AM PDT #


The following resources may help:


Posted by Nazrul on April 29, 2010 at 12:32 PM PDT #

I think you may be right in what you have said in this article. Whoever there are many people will not agree with your opinion.I loved your writing, please keep putting out more content like this!

Posted by Maxolip on July 16, 2010 at 11:17 PM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed

Ramblings about GlassFish


« February 2017