Wednesday Feb 24, 2010

The Art of Leaving ...

I hope I don't let it sound too emotional, but I don't want to let it feel untrue as well. Yes, after working for Sun Microsystems for more than nine wonderful years and Oracle Inc. for a few days, I have decided to leave. This is my last blog-post and today is my last day.

Valedictory function speeches, parting conversations etc. have always been a challenge for me. This blog-post is no different. There is one Sanskrit verse though that captures how I feel very succinctly and it is as follows

      गुणवद्वस्तु संसर्गाद् याति स्वल्पोपि गौरवम् |
      पुष्पमालानुषङ्गेण सूत्रं शिरसि धार्यते || 
This blog-post states the meaning correctly. In other words,
Any small, ordinary thing gets crowning glory if it associates itself with a great entity, just like a common thread gets honoured because of its association, vicinity with the garland it holds!
My tenure at Sun Microsystems has been a testimony of the above. It's this place that made my stay a memorable one. There are things that I could have done differently and I regret not having done so, but the Sunny atmosphere here is simply unforgettable. I mean where else does a programmer get an opportunity to have a Bill-Joy- Number of 2? Where does a programmer get an opportunity to see and experience amazing and wacky things being built by remarkable personalities? Of course, there are places elsewhere and this is a rhetoric, but that does not curtail my excitement one bit. Thank you, Sun Microsystems Inc. Individual colleagues, friends and well-wishers who influenced me are just too many to list here.

The culmination of my career at Sun was being a part of GlassFish team where I could contribute building one of the most loved open source enterprise products. I feel fortunate to be part of this team and since it's open source, I can contribute to it wherever I go. You can always reach me at and I will be hanging around in the GlassFish users'/developers' list.

Sunday Oct 18, 2009

GlassFish v3: Elements of Administrative Security ...

Takes you on the tour of administrative security of GlassFish v3.[Read More]

Monday Apr 28, 2008

How GlassFish DAS communicates with Node Agents and Instances ...

Question: Explain the communication details between Domain Admin Server, node-agents and server instances in Sun's Application Server 8.x and 9.x (GlassFish V2).


DAS: Domain Admin Server (One per domain) -- The process that controls the management of the entire domain.

NA: Node Agent -- Generally, one per box or Solaris container -- The process that controls the life cycle of server instances.

SI: Server Instance --  The real Java EE instances that run user applications in an enterprise.


1. Background: The domain.xml controls the configuration. At every node-agent, there are also a few configuration files that are consulted by every NA. See NA section at for details. Following are the points in time when the communication (for administration/management purpose) happens:

  • DAS communicates with each NA: Only when DAS needs to know NA's running status.
  • DAS communicates with each SI: When DAS needs to know SI's running status and when it needs to cascade the SI MBeans into the DAS's MBenServer.
  • NA communicates with DAS: During initial rendezvous (which may happen during creation of NA), synchronization of the NA itself and synchronization of each SI that NA is responsible for.
  • SI communicates with the DAS: Never, explicitly.

Thus, the communication is mainly driven by DAS. When the domain is created, the administration is configured to use an authentication realm named admin-realm. This realm points to what's called a FileRealm which is nothing but the implementation of a security realm implementation that uses admin-keyfile. If you see the domain's configuration, you'll find this file in config folder of that domain.

The communication happens over two channels. One is the HTTP channel and the other is RMI channel. For this purpose, there is a SynchronizationServlet and a System JMX Connector (standard in JDK 5) that is provided. Every DAS and SI, including the NA start a JMX RMI ConnectorServer that can be optionally configured to use transport layer security.

Every NA communicates with DAS multiple times, but the key points are of initial hand-shake and synchronization. The initial hand-shake is when NA makes DAS aware of its own existence and DAS correspondingly responds if it has the correct credentials. When the DAS is configured to have secure access (this is the default in enterprise profile domain), both the HTTP and JMX/RMI channels use Transport Layer Security with SSL/v3.  Note that during the initial hand-shake, the DAS knows about NA's existence alone. DAS does not release the contents of the domain's repository during this phase. This happens over HTTP channel since creation of node-agent takes the DAS's admin-port (default: 4848) as an option.

After an NA is created, the most natural step is to start that NA. This is done by executing the asadmin start-node-agent command. Since this is the first-time startup of the NA, NA syncs up with the DAS. Note that startup of NA requires the correct credentials (admin user name and admin password) to be supplied. The DAS compares them against its own admin-keyfile and the communication succeeds only when this succeeds. The NA startup also requires the master password to be provided on the command line because in order to start, the NA has to be able to unlock the security store (e.g. keystore.jks) that it synced from the DAS. Note that master password is never put on the wire! It has to be provided at the time of both DAS startup and every NA startup. For advanced use cases, there is an unattended boot scenario that is handled by using the option --savemasterpassword which should be used with care.

The reason NA needs the master password is also to pass it on to the SI's it starts (as part of start-instance or start-cluster) so that these instances are able to unlock the security store to get the primary keys and certificates. 

The NA always communicates with the DAS over JMX/RMI channel. Thus NA opens an RMI connection to the DAS where DAS is listening for RMI/JMX Connections. This is where the RMI Registry in DAS (default port 8686) comes into picture.

When the domain is created, it uses the self-signed certificate aliased s1as which is used for internal communication. This certificate is created anew every time a domain is created. The master password of a domain is what locks the server's keystore. In enterprise profile domain, NSS is used to manage the secure store, whereas in cluster profile domain, JKS manages the secure store. The semantics of the master password are unchanged in both the cases.

The Server Instances are synced with the DAS as part of either:

  1. start-instance, or
  2. start-cluster, or
  3. start-node-agent --syncinstances procedure.

For this synchronization, they use the HTTP layer and communicate with the SynchronizationServlet that's listening for sync requests. This servlet is (of course) running in the DAS.

The server instances get the admin credentials from the node-agent process in a secure manner (using stdin). This also evident when you try to use the startserv script that's located in instance's bin folder.

The process of DAS communicating with the NA and SI's is identical in that it communicates with them over RMI/JMX in the other direction.

2. Transport Layer Security:

This is achieved when we enable the security-enabled flag on the admin-listener and jmx-connector named system on the DAS and server instances. Note that  admin-listener (HTTP/S) is started only in the DAS. There is no admin-listener in server instances.

It's of course possible to use another CA-signed certificate for this purpose. It needs additional configuration after importing those certs in the store.

3. Authentication and Credentials:

Please see:


Thursday Nov 22, 2007

GlassFish Podcast about GlassFish Deployment ...

Check out my GlassFish Podcast on this subject ...


Thursday Oct 18, 2007

Upgrading from GlassFish V1 to GlassFish V2 and an Interesting Deployment ...

I came across this interesting use case while discussing GlassFish V1-> V2 upgrade.

As you might know, we have modified the process launcher a great deal for V2. This contributed to improved startup performance. In the process, following happened:

  1. asadmin start-domain does not invoke the domain-folder/bin/startserv script anymore.

A user was using V1 in an interesting way. The assumption was asadmin start-domain will always invoke the startserv script.

Here is how:

Consider that I have a heterogeneous environment, with various libraries being used by my application. We have a complicated build environment and we finally make sure that all the libraries needed by the application are saved at a particular location say /libraries and is captured in an environment variable AS_LIBRARIES as:

export AS_LIBRARIES=/libraries/foo.jar:/libraries/bar.jar ...

Then we modified the startserv script as: -Dmylibraries=$AS_LIBRARIES ...

and finally we modified domain.xml as:

<java-config classpath-suffix="${mylibraries}" ....


Now, this was clearly a hack and worked only because of peculiar way in which GlassFish V1 treats the system properties. This is clearly an area in GlassFish that needs improvement because of its over-dependence on (evil) System Properties. Here, however, the user had employed this to his advantage! In short, what he has done was:

Invented a way to pass an environment variable on the system whose value at a given point in time would be picked up by the domain dynamically, at the startup. That way, the user would just modify the value of the environment variable and restart the server to pick up the changes in the variable!

The ordeal is expressed here.

I liked the way they used it and I thought of helping him. Of course, there are better ways of doing this, but they had been having this for some time and apparently, this was the thing holding them back from upgrading to GlassFish V2.

I remembered suddenly, the GlassFish had an attribute named env-classpath-ignored which is set to true by default. This means, that whoever (user on Operating System) starts the application server, if s/he has an environment variable named CLASSPATH it is always ignored by the server by default. This works well in almost all cases, since CLASSPATH is being forgotten by the Java community.

I thought of suggesting this to him and instead of setting AS_LIBRARIES, if CLASSPATH is set as the environment variable in this user's environment, the problem would be solved.

The way the launcher deals with this is:

  • Look at the env-classpath-ignored and it it is set to false (non-default case), append its value to the -cp argument to the JVM.
  • Effectively place the jars pointed to by CLASSPATH into the application server JVM's System ClassLoader.

I thought the problem is solved!

But no, because of various class-loaders in application server (each for a specific purpose), this was not working correctly as some classes were loaded too early by the System ClassLoader. The correct solution was to:

  • Look at env-classpath-ignored and if it is false,
  • Get the value of CLASSPATH and set it as -Dcom.sun.aas.ClassPathSuffix along with classpath-suffix in domain.xml.

It worked perfectly and user tested it (on his laptop, in Mumbai).

Lessons learned:
- Don't rely on private interfaces of a product. Hacks are OK, but look at the consequences.
- Consider using ${path.separator} instead of ':' or ';' if you want GlassFish to virtualize the 
operating platform for you. So, while setting path-like entries in domain.xml, always use
${path.separator}. GlassFish is designed to do the right thing.

Thanks to both Byron Nevins and Sivakumar Thyagarajan for their help in this regard.

Sunday Oct 07, 2007

GlassFish Clustering Article on ...

Check out the clustering article on that I co-authored.

Sunday Sep 30, 2007

Yahoo! Pipes and Global GlassFish Aquarium ...

This post is based on two facts:

  1. It (Web 2.0?) has definitely started to change how we look at the Web. It is a collaboration of various technologies that is ultimately going to make the Internet a better place for everyone.
  2. Translation is probably still a topic of active research.

I am trying to suggest an approach to translate TheAquarium into various languages so that developers around the globe can experience it and get benefited by GlassFish technologies.

Yahoo! Pipes is an interesting technology but I am not going to delve into it because there are various resources that help you get started with it. When I looked into it, the first thought that came to my mind was to use it for GlassFish and then I saw the translation module. It sort of guided me into defining the pipes as shown in attached graphic.

TheAquarium Feeds in French 

This shows the pipe that takes the English feed from feedburner (which is rather weird) and then translates the titles into French and creates the list.

Here is the English-French TheAquarium Pipe I published.

My French friends are going to be happy with me.

I bet this technique can be exploited to translate the entire posts so that Global Developers are attracted to TheAquarium. We might not need the Language Buttons any more (although they serve a slightly different purpose).

Isn't the Web (ever) burgeoning?

Monday Sep 17, 2007

Five Reasons to Upgrade to GlassFish V2 ...

GlassFish V2 is now released. 

Download it here. GlassFish V2 

There have been quite a few firsts and positives about this release. Deployers, administrators and developers have all got reasons to pick GlassFish V2. Here are my top five reasons (not in any specific order) to pick GlassFish V2 as your platform:

  • Fully Open Sourced Product: GlassFish is 100% Open Source. This is an attractive option for the industry deployments and developers because of the obvious advantages. The user and developer community for GlassFish is thriving and the discussion forums have been invaluable to various users. There has been contributions from several users to the FAQ's based on their experiences. Over 3500 issues have been filed by the community at large in a year and quite a few of them have been fixed for V2. An excellent knowledge base is being developed at the Wiki. Users and developers have been contributing to documentation and its reviews. A search of specific terms on discussion forums is likely to direct you in the right direction if you are looking to solve a particular problem. The open source momentum is driving this effort fabulously.
Another interesting fallout of this is that GlassFish V2 has been successful in integrating various technologies so that deployers can pick and choose. These include SOA integration in terms of Open ESB and Web Services Interoperability Technologies.
  • Ease of Use for Developers and Deployers: GlassFish V1 was the reference implementation for Java EE 5 more than a year ago and that won the mindshare of several developers. The success of EJB 3.0 and JPA is evident. Developers like the ease with which they can develop the enterprise applications. The industry-leading IDE for Java EE 5 in NetBeans makes the development even easier. All of these advantages have been retained in GlassFish V2 as it is fully compliant with Java EE 5. With its introduction of usage profiles, it has been possible to get rid of the Product Editions, which were too course-grained and difficult to upgrade. GlassFish V2 adds the ability to manage the software for enterprise with ease. The attractive features for administrators and deployers include:
    • A Web-based Administrative Console with uniform look and feel.
    • An exhaustive, easy to use and uniform command line interface (asadmin).
    • An ability to add clustering support to an existing server, retaining your behavior (in other words, an easy upgrade to clustering).
    • Ability to monitor the servers in the cluster and the admin server itself using any tool of choice. The tools for monitoring include the asadmin, admin-console, JConsole and any JMX-enabled console (like HP OpenView) etc. The asadmin program has added a new command called "monitor" that gives you a more intuitive output, similar to several Unixy commands like iostat, vmstat etc.
    • Ability to run heterogeneous domains through use of profiles. Three profiles of interest are bundled with any distribution and they can be made to work if required components are just plugged in. This was not possible in the past with Sun's application server offering. The introduction of profiles makes it possible to have single download that gives you an ability to create administrative domains (protected by distinct security credentials) with specific behavior.
These administrative advances in GlassFish V2 make other offerings without any cluster management support look old-fashioned.
  • World-record Web-tier Performance: GlassFish V2 posted the best ever Specjappserver2004 performance numbers. The Grizzly front-end adds performance and features that are poised to make your deployment blazingly fast. Grizzly brings in the Comet Support and Port Unification. The additional Web Container enhancements are described by Jan Luehe.
  • Clustering and Availability: This is one of the distinguishing features of GlassFish V2. Clustering of the application server instances as far as the conversational state of Stateful Session Beans and HTTP Session is concerned would not have been easier to manage. Based on robust architecture of JXTA, this solution is close to zero-configuration and delivers reasonable availability. In case your business demands warrant a more available solution, you can always use the Highly Available Database (this is available as a binary download from The good thing about this is, it is pretty easy to interchange the in-memory session state replication and HADB solution, for well-written user applications.
  • Pluggability into Sun's Software Stack and Support Benefits: You can reap the benefits of alignment of Sun's server product offerings. Along with the GlassFish V2 release, Sun has also released Sun Java System Application Server and Application Server/Platform Suites. These integrate the Identity Management, BPEL Engine and so on. Sun also delivers the same set of GlassFish V2 bits into Java Enterprise System -- an integrated stack involving industry-leading directory server, identity manager, portal server and registry. The integration has been smoother than ever before.
Sun's support is more than equipped to guide you with right advice on deployment on Solaris, Linux and Windows platforms. GlassFish V2 also supports Mac OS X. If you have an adoption story, please let us know at stories portal.

Tuesday Apr 17, 2007

A recipe to create clusters with GlassFish V2 ...

I have been seeing several questions about clustering on GlassFish Users alias. Here is a simple recipe, very crude, but the one that works.

Questions? Shoot.

  • Domain Spans machines, usually. One of the machines is designated as DAS.
  • The DAS is the admin server and it manages the servers in the same domain. The servers can be clustered if administrator wants them so. The administrator has benefits of management when instances are clustered. Now, on the other machine, all you need is "bits".
  • Consider the scenario: You have "m1" and "m2" as two machines.
  • Let's say you want to create a cluster "c1" with instances "s1" and "s2" with node-agents "na1" on "m1" and "na2" on "m2".
  • Your DAS is on "m1".
  • Let's call the domain where all this "logically" resides as "domain1". You do the following:
    • 1- Download the glassfish.jar on both "m1" and "m2".
    • 2- On "m1", run "ant -f setup-cluster.xml". This creates the domain, domain1 that is capable of handling clusters.
    • 3- On "m1", you start domain, domain1.
    • 4- On "m1", /asadmin create-node-agent na1
    • 5- On "m2", run ant -f setup-cluster.xml on glassfish.jar you downloaded (after of course, java -jar ...) This creates a domain on that machine as well, but you don't need it. I will file an RFE in this regard.
    • 6- On "m2" you create a node-agent \*pointing to domain1 on "m1"\* as: /asadmin create-node-agent --host m1 --user admin-user na2
    • 7- On "m1", /asadmin start-node-agent na1 On "m2", /asadmin start-node-agent na2 Now, your topology is created and ready to go. You have a domain domain1, this spans "m1" and "m2". Node-agent "na1" is on "m1". It will control instances on "m1". Node-agent "na2" is on "m2". It will control instances on "m2". From this point on, forget you had machine "m2". The entire configuration can be done just from machine "m1". This is the beauty of the process, IMO.
    • 8- On "m1", just try: /asadmin create-cluster c1
    • 9- On "m1", /asadmin create-instance --cluster c1 --nodeagent na1 s1 This puts "s1" in cluster "c1". Note that this instance is controlled by Node Agent "na1". This instance uses "physical resources" from "m1".
    • 10- On "m1", (not "m2"!) /asadmin create-instance --cluster c1 --nodeagent na2 s2 11- On "m1", /asadmin start-cluster "c1". This is where it culminates. All you do is "start cluster". It starts instance "s1" on "m1" and "s2" on "m2". The instances have homogeneous configuration.

Thursday Apr 05, 2007

Sun Application Server Builds and Clustering ...

I think there is some confusion in this regard. I think I must clarify certain things here. This post is about which distributions of Sun's application server have clustering support "out-of-the-box".

Simple answer: ALL .

Long Answer:

Following distributions of application server all have clustering support built into them:

The reason this is possible is because the core application server code base is exactly the same in all the distributions. It is the various other 3PP (third party products) that differ. For example, the SDK bundle (it has different official name, check out for details) has other software like access manager and portal server support, whereas the EE bundle has HADB to store the state and has NSS and Load-balancer plugin among other things.

The features that enable this (amazing, IMO) feat are:

  • In-memory replication of conversational state.
  • Usage Profiles.
  • IIOP failover.
So, next time you need clustering support, don't look for EE, PE, SDK and GlassFish builds do the job. I will post more in this regard soon ...

Welcome to my blog where mostly my work related thoughts are expressed.


« July 2016