Wednesday Jul 09, 2008

Javadoc: An Observation

Disclaimer: I don't think this is not obvious (at least to some) and it (this observation) might have been made by many. I just reached it independently.

Javadoc is a great tool. It helps understand the basic hierarchy of an API and its public implementation. It's no replacement for looking into the source code, however. Sometimes, it might be misleading to draw conclusions based on what normal Javadoc shows, without looking into the code. Here is an example:

java.util.LinkedList illustrationThe java.util.LinkedList  inherits iterator() from java.util.AbstractSequentialList, which has a concrete implementation for iterator(). This method calls listIterator() which is inherited from java.util.AbstractList.  This method in turn calls listIterator(int index) which has concrete implementation in java.util.LinkedList  itself.

Thus, calling java.util.LinkedList.iterator()  will result in calling listIterator(int) on the same instance of LinkedList.

Did LinkedList really inherit iterator() from AbstractSequentialList in that case?

It of couse inherited the Java method named iterator(), but it does not inherit (and does override) the behavior for iterator().

Conclusion: Read source code, and open source software is beneficial.

Monday Apr 28, 2008

How GlassFish DAS communicates with Node Agents and Instances ...

Question: Explain the communication details between Domain Admin Server, node-agents and server instances in Sun's Application Server 8.x and 9.x (GlassFish V2).


DAS: Domain Admin Server (One per domain) -- The process that controls the management of the entire domain.

NA: Node Agent -- Generally, one per box or Solaris container -- The process that controls the life cycle of server instances.

SI: Server Instance --  The real Java EE instances that run user applications in an enterprise.


1. Background: The domain.xml controls the configuration. At every node-agent, there are also a few configuration files that are consulted by every NA. See NA section at for details. Following are the points in time when the communication (for administration/management purpose) happens:

  • DAS communicates with each NA: Only when DAS needs to know NA's running status.
  • DAS communicates with each SI: When DAS needs to know SI's running status and when it needs to cascade the SI MBeans into the DAS's MBenServer.
  • NA communicates with DAS: During initial rendezvous (which may happen during creation of NA), synchronization of the NA itself and synchronization of each SI that NA is responsible for.
  • SI communicates with the DAS: Never, explicitly.

Thus, the communication is mainly driven by DAS. When the domain is created, the administration is configured to use an authentication realm named admin-realm. This realm points to what's called a FileRealm which is nothing but the implementation of a security realm implementation that uses admin-keyfile. If you see the domain's configuration, you'll find this file in config folder of that domain.

The communication happens over two channels. One is the HTTP channel and the other is RMI channel. For this purpose, there is a SynchronizationServlet and a System JMX Connector (standard in JDK 5) that is provided. Every DAS and SI, including the NA start a JMX RMI ConnectorServer that can be optionally configured to use transport layer security.

Every NA communicates with DAS multiple times, but the key points are of initial hand-shake and synchronization. The initial hand-shake is when NA makes DAS aware of its own existence and DAS correspondingly responds if it has the correct credentials. When the DAS is configured to have secure access (this is the default in enterprise profile domain), both the HTTP and JMX/RMI channels use Transport Layer Security with SSL/v3.  Note that during the initial hand-shake, the DAS knows about NA's existence alone. DAS does not release the contents of the domain's repository during this phase. This happens over HTTP channel since creation of node-agent takes the DAS's admin-port (default: 4848) as an option.

After an NA is created, the most natural step is to start that NA. This is done by executing the asadmin start-node-agent command. Since this is the first-time startup of the NA, NA syncs up with the DAS. Note that startup of NA requires the correct credentials (admin user name and admin password) to be supplied. The DAS compares them against its own admin-keyfile and the communication succeeds only when this succeeds. The NA startup also requires the master password to be provided on the command line because in order to start, the NA has to be able to unlock the security store (e.g. keystore.jks) that it synced from the DAS. Note that master password is never put on the wire! It has to be provided at the time of both DAS startup and every NA startup. For advanced use cases, there is an unattended boot scenario that is handled by using the option --savemasterpassword which should be used with care.

The reason NA needs the master password is also to pass it on to the SI's it starts (as part of start-instance or start-cluster) so that these instances are able to unlock the security store to get the primary keys and certificates. 

The NA always communicates with the DAS over JMX/RMI channel. Thus NA opens an RMI connection to the DAS where DAS is listening for RMI/JMX Connections. This is where the RMI Registry in DAS (default port 8686) comes into picture.

When the domain is created, it uses the self-signed certificate aliased s1as which is used for internal communication. This certificate is created anew every time a domain is created. The master password of a domain is what locks the server's keystore. In enterprise profile domain, NSS is used to manage the secure store, whereas in cluster profile domain, JKS manages the secure store. The semantics of the master password are unchanged in both the cases.

The Server Instances are synced with the DAS as part of either:

  1. start-instance, or
  2. start-cluster, or
  3. start-node-agent --syncinstances procedure.

For this synchronization, they use the HTTP layer and communicate with the SynchronizationServlet that's listening for sync requests. This servlet is (of course) running in the DAS.

The server instances get the admin credentials from the node-agent process in a secure manner (using stdin). This also evident when you try to use the startserv script that's located in instance's bin folder.

The process of DAS communicating with the NA and SI's is identical in that it communicates with them over RMI/JMX in the other direction.

2. Transport Layer Security:

This is achieved when we enable the security-enabled flag on the admin-listener and jmx-connector named system on the DAS and server instances. Note that  admin-listener (HTTP/S) is started only in the DAS. There is no admin-listener in server instances.

It's of course possible to use another CA-signed certificate for this purpose. It needs additional configuration after importing those certs in the store.

3. Authentication and Credentials:

Please see:


Monday Dec 17, 2007

A strong case for <a seref="search string">?

This is of course a wacky idea and the one which search engines might/should jump on :). I checked the working draft for HTML 5 and could not find it there.

The idea is about search engines aiding all the HTML documents that are about a certain topic and want to intersperse hyper-links to certain terms that appear elsewhere on the web. Of course, I, as an author of a document that references such concepts or terms, don't want to waste time on finding the exact web-page that demonstrates it. The intent of such a link is to provide more information about that term anyway. Then, why not have a facility like:

  • <a seref="Effective Java">Effective Java</a>
which will bring the page from your favorite search engine with "Effective Java" as your query string.

This is useful for the author because when the author does not really want to be specific to point to a known document, s/he can always make use of search engines which are supposed to provide more up-to-date information on a topic of general interest. More importantly, it examines if search relevance that search engines tout about is any good!

I told ya -- it is wacky :), but I just implemented this in my previous post.

Default Singleton Boilerplate code in IDEA ...

While I was getting used to IDEA IntelliJ IDE, one "feature" caught my attention. They have a way to create a "Singleton" by way of a pop-up menu. Here is the code that generates the class for this controversial and damned pattern (now that Google has a Singleton Detector :-) ).
public class Single {
    private static Single ourInstance = new Single();

    public static Single getInstance() {
        return ourInstance;

    private Single() {
I see at least two problems with this "default" code:
  1. The class should have been declared final.
  2. The variable ourInstance should have been declared final.
  3. Motivation:

    Quality criterion for boilerplate code is same as that for real code (in these days of fast search engines :) ). Also, I am mildly surprised that it takes a lot of time for me to:
    • Create a Java Interface
    • Create a Class that implements that Interface with stubbed implementation for interface methods .
    This is true with both NetBeans 6.0 (which means it is true with any other IDE because NetBeans is the only IDE you need, these days :) ) and IDEA 7.0.

Thursday Nov 22, 2007

GlassFish Podcast about GlassFish Deployment ...

Check out my GlassFish Podcast on this subject ...


Thursday Nov 08, 2007

Gmail against Firebug ...

One of the things I like about Firefox is Firebug. I wonder why Gmail has started showing this ugly red flag recently ...



I think it is ugly because it does not make sense to have it as a red flag. People who run Firebug know that there is an overhead involved with it. 

But it's the G-muscle, what can you do?

Friday Nov 02, 2007

13949712720901ForOSX ...

Java 6 is what Leopard misses ...

Thursday Oct 18, 2007

Upgrading from GlassFish V1 to GlassFish V2 and an Interesting Deployment ...

I came across this interesting use case while discussing GlassFish V1-> V2 upgrade.

As you might know, we have modified the process launcher a great deal for V2. This contributed to improved startup performance. In the process, following happened:

  1. asadmin start-domain does not invoke the domain-folder/bin/startserv script anymore.

A user was using V1 in an interesting way. The assumption was asadmin start-domain will always invoke the startserv script.

Here is how:

Consider that I have a heterogeneous environment, with various libraries being used by my application. We have a complicated build environment and we finally make sure that all the libraries needed by the application are saved at a particular location say /libraries and is captured in an environment variable AS_LIBRARIES as:

export AS_LIBRARIES=/libraries/foo.jar:/libraries/bar.jar ...

Then we modified the startserv script as: -Dmylibraries=$AS_LIBRARIES ...

and finally we modified domain.xml as:

<java-config classpath-suffix="${mylibraries}" ....


Now, this was clearly a hack and worked only because of peculiar way in which GlassFish V1 treats the system properties. This is clearly an area in GlassFish that needs improvement because of its over-dependence on (evil) System Properties. Here, however, the user had employed this to his advantage! In short, what he has done was:

Invented a way to pass an environment variable on the system whose value at a given point in time would be picked up by the domain dynamically, at the startup. That way, the user would just modify the value of the environment variable and restart the server to pick up the changes in the variable!

The ordeal is expressed here.

I liked the way they used it and I thought of helping him. Of course, there are better ways of doing this, but they had been having this for some time and apparently, this was the thing holding them back from upgrading to GlassFish V2.

I remembered suddenly, the GlassFish had an attribute named env-classpath-ignored which is set to true by default. This means, that whoever (user on Operating System) starts the application server, if s/he has an environment variable named CLASSPATH it is always ignored by the server by default. This works well in almost all cases, since CLASSPATH is being forgotten by the Java community.

I thought of suggesting this to him and instead of setting AS_LIBRARIES, if CLASSPATH is set as the environment variable in this user's environment, the problem would be solved.

The way the launcher deals with this is:

  • Look at the env-classpath-ignored and it it is set to false (non-default case), append its value to the -cp argument to the JVM.
  • Effectively place the jars pointed to by CLASSPATH into the application server JVM's System ClassLoader.

I thought the problem is solved!

But no, because of various class-loaders in application server (each for a specific purpose), this was not working correctly as some classes were loaded too early by the System ClassLoader. The correct solution was to:

  • Look at env-classpath-ignored and if it is false,
  • Get the value of CLASSPATH and set it as -Dcom.sun.aas.ClassPathSuffix along with classpath-suffix in domain.xml.

It worked perfectly and user tested it (on his laptop, in Mumbai).

Lessons learned:
- Don't rely on private interfaces of a product. Hacks are OK, but look at the consequences.
- Consider using ${path.separator} instead of ':' or ';' if you want GlassFish to virtualize the 
operating platform for you. So, while setting path-like entries in domain.xml, always use
${path.separator}. GlassFish is designed to do the right thing.

Thanks to both Byron Nevins and Sivakumar Thyagarajan for their help in this regard.

Sunday Oct 07, 2007

GlassFish Clustering Article on ...

Check out the clustering article on that I co-authored.

Sunday Sep 30, 2007

Yahoo! Pipes and Global GlassFish Aquarium ...

This post is based on two facts:

  1. It (Web 2.0?) has definitely started to change how we look at the Web. It is a collaboration of various technologies that is ultimately going to make the Internet a better place for everyone.
  2. Translation is probably still a topic of active research.

I am trying to suggest an approach to translate TheAquarium into various languages so that developers around the globe can experience it and get benefited by GlassFish technologies.

Yahoo! Pipes is an interesting technology but I am not going to delve into it because there are various resources that help you get started with it. When I looked into it, the first thought that came to my mind was to use it for GlassFish and then I saw the translation module. It sort of guided me into defining the pipes as shown in attached graphic.

TheAquarium Feeds in French 

This shows the pipe that takes the English feed from feedburner (which is rather weird) and then translates the titles into French and creates the list.

Here is the English-French TheAquarium Pipe I published.

My French friends are going to be happy with me.

I bet this technique can be exploited to translate the entire posts so that Global Developers are attracted to TheAquarium. We might not need the Language Buttons any more (although they serve a slightly different purpose).

Isn't the Web (ever) burgeoning?


Welcome to my blog where mostly my work related thoughts are expressed.


« June 2016