Monday Mar 16, 2009

Using a Sailfin Cluster (Load-balancing with a Simple INVITE Application)

This blog covers the following items:

  1. Deploy a simple invite application on a sailfin cluster (Application provided).

  2. Create the SIPp INVITE scenario file (Ready scenario file provided).

  3. Run the SIPp scenario.

  4. See that the requests are load-balanced.

Before You Start
Get SIPp installed on your machine. Get it from .
b) To create the Sailfin Cluster see Quick Start with SailFin Clustering. Once the CLB enabled cluster is created you can follow the below steps to try it out.

Deploy a Simple INVITE Application on the cluster
Run asadmin deploy --target <cluster-name> <path-to-deployable-sar-file> at <sailfin-install-location>/bin
(Application source is available here. Deployable application sar file is available here. )

Create the SIPp INVITE scenario file
Get this invite.xml file available here.
\* Get the default uac scenario xml available from sipp (by running "sipp -sd uac >invite.xml) and modify it for use with sailfin.
Modifications required are:
- Replace the first <recv response="200" rtd="true"> with <recv response="200" rtd="true" rrs="true">
- Replace "ACK sip:[service]@[remote_ip]:[remote_port] SIP/2.0" with "ACK [next_url] SIP/2.0".
- Replace " BYE sip:[service]@[remote_ip]:[remote_port] SIP/2.0" with " BYE [next_url] SIP/2.0"

Now run SIPp as follows
sipp -sf invite.xml -m <number-of-calls-you-want-to-create> -p <sipp-client-port-your-pick> <host-name-any-one-instance>:<sip-port-of-that-instance>

For example:
Use "sipp -sf invite.xml -m 10 -p 7000" to send 10 requests to an instance that is running on machine and whose sip port is 35060.
Now, to find the SIP port of an instance open the domain.xml at <sailfin-install-location>/domains/<your-domain-name>/config/domain.xml and search for the instance name. Look for the SIP_PORT system property information for that instance. If you don't find it then the default 35060 port is being used.

Checking that requests are being Load Balanced

From Admin UI:
a) Open Admin UI in the browser <hostname-of-machine-with-domain>:4848. Enter default username and password (admin/adminadmin)
b) Click the arrow next to “Clusters” on the left pane. Click the arrow next to “<Your-Cluster>”. Click on one of the instances.
c) Click on the “Monitor” tab. Click on “Call Flow” sub-tab. Check the “Enabled” check box to enable Call Flow Monitoring. Click on the “Save” button. Repeat this for each instance in your cluster.
d) Run the above SIPp command again on the command line.
e) Click on the “Refresh” button. Scroll down to check on the Call Flow Data table. Do this on the Call Flow screen of each instance to find that each instance has served some requests.

From Command Line:
a) Enable SIP module logging by running "./asadmin set <cluster-name>" .
b) Run the above SIPp command again.
c) Observe the server logs for all the instances.
These logs are available at <sailfin-install-location>/nodeagents/<agent-name>/<instance-name>/logs/server.log. Look for text "The first line" and you will find the SIP messages that are processed. You will find that requests are being processed by all instances.

Friday Mar 06, 2009

GlassFish Talk at Infosys, Q and A

Talks on GlassFish and Web Space Server were presented on 27th Feb 2009 to a group of Infosys employees at their Electronic City campus. Sandeep Soni covered Web Space Server and I handled the one on GlassFish. These talks were attended by about 80 to 100 Infosys employees and were well received. The attendees had experience with the Java EE and Portal technologies and products and the purpose was to get them to learn more about Sun's Open Source offerings in that space. Some of them already knew a good deal about GlassFish and Web Space Server and the hope there was to re-inforce their belief in these products.

The GlassFish presentation was a mash up of slides from Arun Gupta and other presentations available at The presentation touched the following topics: overview of GlassFish, Sun's Open Source Strategy, GlassFish v2, v3, Metro, RESTful Web Services, Enterprise Manager, migration.

Demos covered during the talk:

  1. Arun Gupta's blog on "Netbeans 6.5 and GlassFish v3", see

  2. Creation of RESTful Web Service through Netbeans 6.5 and running it on Jersey (GlassFish).

  3. High Availability Features of GlassFish: this one hit a glitch :( (see item 1.ii at

Here's a sampling of questions that were asked during the session:

  1. Is there a way to turn off the "auto-deploy on save" functionality in netbeans?
    Yes. On Netbeans 6.5, Right Click on the project - > Properties -> Run -> uncheck the box that says "Deploy on Save"

  2. Can Felix be replace with Equinox with GlassFish v3 ?

  3. Why is it that asupgrade is required ? Why can't GlassFish take care of it and not require the customer to do anything ?
    Upgrade Functionality is quite handy for a user upgrading from one version of GlassFish to another. The asupgrade tool is optionally available when installing the new version of GlassFish. So the user only needs to inform GlassFish of its use and no more. It can also be invoked from command line after installing the new version. The tool handles upgrade/transformation of many things such as - Administrative settings, Security Roles, Cluster Settings and importantly changes to domain.xml.

  4. Can the listener port of the http service be changed without having to restart the server ?
    Yes, the user only needs to change the port information for this.

  5. Is there a way to import Netbeans Projects into Eclipse, because I like to work with Eclipse, but love the Netbeans integration with GlassFish ?
    You can try Other links also show up on a internet search. You can also try asking the question at the netbeans mailing lists [] .

  6. Can Teracotta replace clustering in GlassFish ?
    The in-memory replication module in GlassFish uses JXTA for replication and GMS for failure management. I am not sure how Terracotta provides failure management.

We now look forward to bootcamp or a workshop type activity for the Infosys Employees.

SailFin Talk at Sun Tech Days, Q and A, useful links...

I presented a session on SailFin at the recently concluded Sun Tech Days at Hyderabad. This session was a last minute replacement as Sang Shin couldn't make it for his session. The session was titled "Adding Convergence of Media to your Enterprise Application using Sun GlassFish Communication Server". This was a session that introduced SailFin to the audience by getting JavaEE and SIP together.

A similar presentation is available at A really nice introduction to Java EE and SIP (via SailFin) is also available at: .

Many students were really excited about the Call-Setup Demo that Prasad Subramanian did a day before (see A demo was not part of my session due to the short notice, but did manage to go over the recorded demo created by my colleague Bhavani (on a Loan Processing Banking Application) available at .

To complete this blog, here's a sampling of some of the questions asked during the session:

  1. How Popular is SIP ?
    SIP is very popular as a industry standard for providing telecom services like VoIP etc.

  2. What are the other competitive/comparable protocols ?
    Search the net for comparisons. There are many out there and can be confusing :-)

  3. Are users limited to SIP enabled phones when working with SailFin ?

  4. Can I deploy a SIP Servlets application on SailFin and make it talk with Skype or GTalk ?
    Although Skype does not support SIP, perhaps you still can make a SailFin application talk to it.... a quick search on the internet brings up many SIP-to-Skype (and vice-versa) functionality providing software.

  5. How do you find if a phone is SIP enabled ?
    The product must provide that information.

  6. Do application creators have to know the SIP address of the phone they need to contact, isn't that hard to remember ?
    No. The application can use the phone number of the SIP enabled phone for this purpose. The TelURL API provided by JSR 289 is available for that.

Tuesday Feb 17, 2009

Notes on Sailfin Cluster Failure Management and GMS

Here are some short notes on how a SailFin Cluster deals with instance failures. These are good for troubleshooting and helpful when debugging issues with failure scenarios. But first, if you are unfamiliar with sailfin clustering please read Quick Start with Sailfin Clustering.

Sailfin relies on Group Management Service for its failure management. This includes detection of an instance's failure and appropriate notification. Below is a list of some types of instance failure that GMS helps detect:
  1. Software Failure:
    a. Node Agent and instance process dying
    b. Instance process alone dying [Transient Failure]

  2. Hardware Failure:
    a. Network Failure [cable snap at the machine's end or at the router's end]
    b. Power Failure [of the machine hosting a sailfin instance]

Notes on how GMS works:
  1. Each instance in a sailfin cluster has an instance of GMS service running in it which starts as the instance is started. A logical GMS group is formed by the GMS services running on all the instances of a cluster.

  2. Using the GMS service each member of the group is able to send and receive signals. Using a heartbeat mechanism the GMS services are able to detect states such as addition, failure or recovery of a group member.

  3. These states are registered as events and are logged in the instance's server.log file under <sailfin-installation>/nodeagents/<agent-name>/<instance-name>/logs/server.log. For example, if an instance is shutdown using the "asadmin stop-instance <instance-name>" command, all other instances that are part of the group would detect this shutdown. You will note a PEER_STOP_EVENT registered in the server log files of these instances.

Below is a list of some important GMS events along with their significance:
  1. PEER_STOP_EVENT: Indicates a planned shutdown of an instance (using the asadmin stop-instance command).
  2. ADD_EVENT: Indicates that an instance has been started (using the asadmin start-instance command) and its GMS service joining the logical GMS group.
  3. JOINED_AND_READY_EVENT: Indicates that startup of an instance is complete.
  4. IN_DOUBT_EVENT: Indicates that GMS suspects that an instance has failed. (Try this by killing an instance and its associated node-agent's process and notice the messages in the logs of other instances)
  5. FAILURE_EVENT: Indicates confirmation of failure of an instance by GMS.
These log messages also indicate the instance associated with the event. This information is quite handy when debugging failure based scenarios.

Node-Agent as a WatchDog:
One other failure detection mechanism is a non-GMS one. The node-agent acts as a watch-dog for the instance. It detects instance process failure and attempts a restart of the instance. This is the transient failure mentioned above as item 1 (b) above.

CLB as a Listener of GMS events:
CLB is a listener of GMS events and it adjust its functonality as per the event's significance. For example, the CLB considers an instance to be available to serve requests until it receives a FAILURE_EVENT for that instance from GMS. On receiving a FAILURE_EVENT the CLB stops forwarding requests to the failed instance. The instance is added back to the CLB's list of available instances only after the CLB receives a JOINED_AND_READY_EVENT for that instance.

GMS failure detection and notification times can vary depending on the hardware used and certain GMS and sailfin configurable settings. For information on this and other functionality provided by GMS, please read documention available at and

Quick Start with Sailfin Clustering

Sailfin Clustering

[Read More]

Sailfin, Glassfish and more....


« July 2016