Friday Sep 25, 2009

How Does Sailfin (Sun GlassFish Communications Server) SIP Session Replication Module Select Replica Instances?

For scalable deployments of middleware with high availability, employing a session state persistence approach to persist session state to all instances in the cluster could be a sub-optimal solution. Replicating sessions to all instances in the cluster would result in significantly higher network traffic just for replicating state reducing bandwidth for growing application user requests. This approach of sharing sessions across all instances perhaps is suited for small clusters with limited number of concurrent requests.

One of the better approaches to use to secure scaling advantages is the approach of buddy replication. In this approach, each instance selects one (or more) other instance(s) in the cluster to replicate any and all of its sessions. This is a superior approach and in fact, works for fairly large deployments. There are factors to consider here, in terms of the overhead the replication subsystem will need to handle at the cost of performance particularly when large number of concurrent sessions are being processed and later expired. An overhead to consider is the need for instances to form ring-like replica partnerships based on a certain order in which buddies would be available and selected. When a buddy instance fails, there is the cost of re-adjusting and forming new buddy relationship with another surviving instance, and when the original buddy recovers, to re-adjust again to use this upcoming instance as a replica partner by one of the instances in the cluster. Think of this as a chain based ring with its links randomly being removed but with the consistent goal of retaining a connected chain ring with the overhead of relinking each time a link is removed or added or a new one added to the chain.

There is also cost to be considered (if such were the design approach), each time the cluster shape changes for dynamically changing/updating any cookie information pertaining to replica locations that could be sent back as part of the response headers to the LB - typically that cost should also be avoided through more efficient means.

In the case of GlassFish, we have fairly successfully used buddy replication with each instance having a single replica buddy. We use the approach of locating sessions in the cluster when a request is directed by the LB to any random instance when a failure of an instance that was processing requests occurs. This has worked well for reasonably large mission critical environments where the scalability and availability requirements are within the boundaries of this approach.

In Sailfin 2.0, scaling and reliability needs for telco applications is typically very high and we needed a scalable approach to ensure Sip Session Replication overhead sustained good performance with the added reliability and availability. We, therefore, used a consistent hashing algorithm to dynamically assign a replica instance for each new session. This we did by leveraging the consistent hashing mechanism that the Sailfin Converged Load Balancer (CLB) uses for proxying requests to a target instance using a BEKey. In the case of replication, the same logic of using a hashed key for the target instance assignment is taken a bit further.

For replica selection, for each new session, we pre-calculate the most likely target instance that the CLB would failover to, if the current target primary instance that would serve the session, were to fail in future. This gives us the instance to which, the current primary instance, should replicate to. This gave us significant benefits in that there were no client cookie updates required to include replica partner information dynamically. There was no readjustment of replica partnerships needed when a particular instance failed as the hashing algorithm would provide another instance to replicate to with just an API call. When the failed instance comes back into the cluster, the sessions that were owned by it in its prior incarnation that are unexpired, would migrate back to it to maintain a balanced set of sessions across the cluster. And the replica selection algorithm would assign the original failover instance for this primary, as the replication partner.

Since this is based on a hashed selection algorithm with predetermined failover target, replica selection is dynamic, and does not need the knowledge of a particular order of instances being ready in the cluster to point all sessions from another instance as a replication partner. And more importantly, as the failover occurs to the specifc instance where replica data is located, there is significantly less network overhead to locate any particular session in the cluster when a particular request within the session scope is sent to the CLB. This allows for more bandwidth being available for a larger number of user sessions to be served. This approach is thus superior to the buddy replication approach and helped us scale to higher throughput and sustain a larger number of long running sessions.

It must be emphasized here that system level, and application server level tuning, and sizing are essential to ensure sustained performance, scalability and reliability in addition to the improvements provided with the SSR replication scheme and other parts of the Sailfin v2 server (aka Sun GlassFish Communications Server 2.0) .

As always, we welcome your feedback and encourage you to try Sailfin and send us any inputs and questions you may have in this respect.

Sailfin Promoted Builds are available here : Sailfin Downloads


Thursday Sep 17, 2009

Project Shoal releases Shoal 1.1 Final Release

Project Shoal is announcing the release of Shoal 1.1 FCS today.  This is the third release after Shoal 1.0 and 1.0 Update Release. 

During the long time we have been working on this release, the Shoal team has accomplished many new features, and fixed many bugs providing significantly higher stability and reliability with the clustering core. A majority of the features and bug fixes were driven by the intense requirements coming out of the Telco grade application server from Project Sailfin (Sun GlassFish Communications Server) and Project GlassFish (Sun GlassFish Enterprise Server). 

Some of the highlights of this release are :

  • New event notification signal for JoinedAndReady members - applications using Shoal can call an API to report to other members that at the application level, the member is ready to begin operations - this distinguishes the member joining the group from the actual readiness of the application to start operations. 
  • New event notification signal for Group Leader changes - anytime a group leader leaves or fails, another group leader is selected and this signal allows Shoal GMS clients to be notified of this change to take any remedial actions. 
  • New health states Ready and AliveAndReady in the HealthMonitor member lifecycle state transitions to make it more closer to the finite state machine
  • Introduce capability to distinguish between group startup and individual instance startup in JOIN and JOINEDANDREADY notifications. Applications may benefit knowing whether only a single GMS member is starting vs all GMS members of the GROUP are starting at once.
  • Support for notifying failure within a specific threshold time when the failure is due to a network issue or a hardware end point failure. Blocking TCP health check connections would timeout in (default) 10 secs allowing for quicker detection of failure compared to default system settings for TCP Retransmission timeouts.
  • A new GMS Watchdog Failure Notification API to be used by external service management frameworks (i.e. like GlassFish NodeAgent) - This allows external monitoring processes to notify GMS when these external process detects a member process failure faster than GMS heartbeat based detection can assign failures. This feature allows for faster failure detection notifications than the heartbeat based unreachability algorithm can determine failure. The feature was driven by the fact that the GlassFish Node Agent would detect a member process loss faster than Shoal could detect through its heartbeats and restart the failed process, resulting in no failure notifications from Shoal GMS to its clients.  Full write up at document. Glassfish Issue 8308

This version of Shoal is already integrated into GlassFish v2.1.1 and Sailfin v2 both of which are upcoming releases slated for October 2009 end.  

Details of the release announcement are here :

https://shoal.dev.java.net/servlets/NewsItemView?newsItemID=7645

You can download Shoal 1.1 binaries and sources with Javadocs here : https://shoal.dev.java.net/downloadsindex.html

As always, we owe it to you all in the community and thank you for all the feedback and issues you have raised to help us improve. 

Thanks very much and hope to see your feedback continuing !

Tuesday Sep 08, 2009

Sun GlassFish Communications Server (Sailfin) adds High Availability Feature

Project Sailfin is building version 2.0 of the JSR 289 compliant application server. The Sailfin 2.0 release also known as Sun GlassFish Communications Server 2.0 will have a notable new feature with the addition of Sip Session Replication component. Sailfin 2.0 will provide High Availability of Sip artifacts providing resilience and availability of conversational state to Telco deployments.  Sailfin 2.0 is targeted for release around end of October/early November 2009. 

High availability through Sip Session Replication component (aka SSR component) allows for replication of SIP artifacts such as SIP Application Sessions, SIP Sessions, Timers, and Dialog Fragments in addition to Converged Sessions. Combined with existing GlassFish replication support for HTTP sessions, deployments can now be highly available covering both SIP protocol-only applications  and converged (SIP and HTTP ) applications.  To support the large scale load that can typically expected with Telco applications, the HA team employed a dynamic replica selection algorithm for each SIP artifact based on a consistent hashing algorithm thus obviating the need for buddy based replication where one would need to react to cluster shape changes and re-partner with another instance when a failure occurs - an expensive operation during high load conditions.   (see this blog entry for more details )

The SSR component along with all of Sailfin is undergoing intensive quality testing including 24x7 longevity, scalability, reliability and fault tolerance testing at this time and we are making progress every day. 

Turning on SSR in Sailfin 2.0 builds is extremely easy similar to how it is with GlassFish. You only need to deploy your SIP (JSR 289 compliant) or Converged Application with the availability-enabled option checked in the Admin GUI console or use the --availabilityenabled=true switch with the asadmin deploy command when deploy your SIP Archive sar file. 

Here's your call to action : Go ahead and download the latest promoted build of Sailfin v2, deploy your SAR archive with availablity-enabled set to true (SSR enabled) and provide us feedback. 

Sunday Jun 21, 2009

GlassFish Clustering: Meaning and impact of configuring attributes of group-management-service element ?

Based on a query from one of our field colleagues, I added an FAQ entry on the meaning of the group-management-service element attributes  and impact of changing the default values as this is not yet covered by our official documentation (it will be updated with this information soon).

These configurational attributes determine Shoal GMS's behavior with respect to Discovery and Health Monitoring of cluster instances in a GlassFish cluster. It is very useful and important to understand these underpinnings of GlassFish's runtime clustering engine.

Here's the FAQ entry :

http://wiki.glassfish.java.net/Wiki.jsp?page=FaqShoalGMSAttributesInDomainXML

Friday Oct 24, 2008

New Shoal Update : Shoal 1.1 bits updated to promoted build 10212008

Shoal 1.1 bits on the Shoal download site have been updated to reflect the promoted build dated 10/21/2008.  We are getting close to releasing Shoal 1.1 sometime in late November.

Get the latest promoted build here.

We have also added a buildable Source and Javadocs zip distribution corresponding to this promoted build on the download page based on a community user request.

A number of fixes went into Shoal during this time (from the prior promoted build of 08202008). Read the announcement here for changes that went in.

Project Shoal benefited not only from the Shoal user community feedback but also the intensive testing that is ongoing for the upcoming Sailfin v1 release in December. As a Telco appserver, Sailfin provides the ideal testing ground for intensive clustering features. Truly where the rubber hits the road.

Thanks very much to Sheetal, Joseph Fialli, Mohamed Abdelaziz, Bongjae Chang, Sailfin and GlassFish Quality Engineers Kazem, Sony and Varun, and most of all the Shoal community members.

As always, we value your feedback so please do continue sending us feedback to help improve Shoal and to address your needs with Shoal clustering. 


Tuesday Aug 26, 2008

Shoal 1.1 bits updated for a new promoted build

The Shoal team has been busy addressing the needs of major projects such as Sailfin for a while now. It had been a while since we updated our promoted build for download with the latest and greatest fixes.

We have now promoted a new build for Shoal 1.1 and download bits are available here : https://shoal.dev.java.net/downloadsindex.html

The announcement has more details on what went into this promoted build : 08202008 Promoted Build Announcement

We would welcome your feedback, issue reports and enhancement requests to make Shoal more useful for you.

Do send us your feedback at the Shoal user mailing list.

Friday May 09, 2008

JavaOne2008: Shoal Mini Talk at Java.Net Community Corner

At the JavaOne pavilion, Java.net had set up a community corner where mini talks were given by various interested project leads. I gave a short talk on Introducing Shoal clustering framework. 

The audience was \*very\* small  but I had quality questions from one very interested participant who wanted to use Shoal as a Distributed Test Harness engine. Very unique use case I thought and yes, Shoal would fit that bill.

Here's the short slide deck of the mini talk: Shoal Mini Talk JavaOne 2008

 

Friday Apr 18, 2008

New Shoal Clustering Download Page

With a view to helping our users choose the right download, we have organized the Shoal downloads page on the lines of the GlassFish download page. The new download page is located here : https://shoal.dev.java.net/downloadsindex.html

This should make it easier for users to find Shoal bits and use them in their clustering applications.We are seeing increasing interest from users with different types of questions in the Shoal user mailing list ranging from newbie questions to very advanced ones. Keep'em coming folks.

We will be posting new promoted downloads  and releases through this download page. Let us know if you need specific improvements to be made.

 

Wednesday Mar 12, 2008

GlassFish High Availability Session at Sun Tech Days Hyderabad

At the recent Sun Tech Days event at Hyderabad, I gave a talk covering GlassFish's High Availability features, particularly the In-Memory Replication support, as part of GlassFish Day (Feb 29th). 

I had the privilege of talking to a full house of around 500 people. The session covered introduction to HA, how easy it is to create, and configure a cluster of instances, and to configure the application for enabling in-memory replication based availability. The session elicited very good questions ranging from the basics to involved ones in the area of sizing the heap to sticky sessions support. I spent an hour after the session outside the hall answering questions posed by interested folks from several companies.

Many attendees wanted to get a copy of the slide deck. Look here for it.

Needless to say, we would very much appreciate any feedback or questions on GlassFish's High Availability. Please send these to us at the GlassFish user mailing alias.


Thursday Jan 17, 2008

Sailfin drives a new feature in Shoal: JoinedAndReadyNotificationSignal

As usage of Shoal's clustering framework increases across products, new feature requests are coming in to enrich Shoal's offerings for employing applications.

The Sailfin project is building a Telecommunications Application Server with contributions from Ericsson and Sun. One of the parts of Sailfin is a Java based load balancer called ConvergedLoadBalancer (CLB). The CLB load balances both Http and SIP based requests. The CLB is unique in that it is a component of the Sailfin appserver instance. Thus, any instance in the cluster can be configured as a load balancer. Such an instance can be part of a Front End LB Cluster performing load balancing of requests on a separate application server cluster tier or it can be part of a Self LoadBalancing Cluster wherein one or more instances of the application server cluster also perform the role of a load balancer while also serving requests. The CLB component and other Sailfin components employ Shoal for cluster events, messaging and health monitoring.

The CLB (as with any load balancer) needs to know when instances of a cluster have not only joined a cluster but also when the instances are ready serve requests. Shoal was up until now providing a JoinNotificationSignal which would be triggered as soon as each instance in the cluster used Shoal's GMS to join the cluster. This was sufficient for many use cases but for the LB it needed to know not just that but also when the instance had completed startup operations. This requirement helped us design a new notification called JoinedAndReadyNotificationSignal that would be disseminated to the cluster members for each instance completing startup and reporting such a completion to the group.

Shoal's GroupManagementService provides a new API called reportJoinAndReadyState() which the employing parent application can now call when the parent application has completed its own initialization and startup indicating a point where operational activities on the instance can now start.

Sheetal and I recently committed the code supporting this new feature.

This feature can be particularly useful while building Compute Grid and Cloud Computing type services using Shoal wherein each Grid node can now report when they are ready to act as nodes in the grid. BTW Shoal's concept of  a GroupLeader ties in well with the Grid's Compute Task Manager abstraction and is a good infrastructural fit.

Do please share your feedback with us at users alias about what you would like to see added in Shoal to serve your clustering and fault tolerance needs. Shoal's goal of clustering goes much beyond the realm of Data Grids (which we are looking into building ) with a wider  spread in terms of building fault tolerance solutions.


Friday Jan 11, 2008

GlassFish Hidden Nugget: Automatic Distributed Transaction Recovery Service

GlassFish v2 and v2 ur1 releases (and later) have support for transaction recovery (both manual and automated) in the sense that incomplete transactions at the time of an instance failure can be committed either manually or automatically.

Part of the new feature set in the cluster profile is a little known feature called Automated Distributed Transaction Recovery that comes out of Project Shoal's support for it. 

Essentially, Automatic Distributed Transaction Recovery in GlassFish works as follows :

Consider the following :

  • a cluster of three instances : instance1, instance2, and instance3
  • Two XA resources used by each GlassFish instance
  • a transaction starts on instance 1,
  • Transaction Manager on instance1 asks resource X to pre-commit,
  • Transaction Manager on instance1 asks resource Y to pre-commit,
  • Transaction Manager on instance1 asks resource X to do a commit,

Now, instance1 crashes

The Transaction Service component in one of the surviving members, instance2 and instance3, gets a notification signal that a failure recovery operation needs to be performed for a instance1. This signal from Shoal is called FailureRecoverySignal.

This notification signal comes to the Transaction Service component in only one particular selected instance as a result of a selection algorithm run in Shoal's GMS component that takes advantage of the identically ordered cluster view provided to it by the underlying group communication provider (default provider is Jxta).

The Transaction Service component in this instance, say instance2, would now go into its autorecovery block. It starts by waiting for a designated time (default to 60 seconds) to allow for the failed instance1 to start back up.

If instance1 is starting up, its own Transaction Service component would do self recovery to complete phase 1 transactions.

In instance2, after the wait timeout occurs, the transaction service component would now see if instance1 is part of the group view and if not try to acquire a lock for the failed instance's transaction logs through Shoal's FailureRecoverySignal and if successful (indicating that the failed instance did not startup), acquire the transaction log and start recovery of transactions i.e complete the commit operations for the pre-commit transactions. If the acquisition of the lock fails, then it gives up, and checks that the failed instance did startup through Shoal's group view and logs this fact.


If, during the recovery operations  being performed by instance2, the failed instance1 starts up, the transaction service component in this instance would first check with Shoal if a recovery operation is in progress for its resources by any other instance in the group and if yes, it waits for the recovery operations to be completed and then completes startup. This ability to check for such recovery operations in progress is through a related Shoal feature called Failure Fencing[1].  If there are no recovery operations in progress, then the startup proceeds with a self recovery which recovers any incomplete transactions in instance1's logs.

Now during recovery of instance1's transaction logs, instance2 fails, then the fact that this instance was in the process of recovering for instance1 is known to the remaining members of the group (i.e. instance3) through the failure fencing recovery state recorded in Shoal's Distributed State Cache. As a result, when instance3's transaction service gets the failure recovery signal, not only does it get it for instance2's failure, but also for instance1. This facility covers for cases where cascading failures or multiple failures occur.

Note that, for the automatic distrbuted transaction recovery to work, access to the transaction logs for all instances in the cluster for
purposes of auto recovery requires that the logs be mounted on a shared/mirrored disk[2].


[1] More on Shoal's Automated Delegated Recovery Selection
[2] Distributed Transaction Recovery

 

 

 

Saturday Dec 08, 2007

Excellent Article on Shoal by non-Sun authors

Just came across this excellent introductory article on Shoal clustering framework on Java.net which I believe is to be published on upcoming Tuesday going by the date posted (12/11/2007).

Noticeably, this is an article by authors that we, at the Shoal community, have not yet corresponded with. This is great news as it lets us know that there is a quiet adoption of this framework. The article lucidly explains salient aspects of Shoal's clustering approach and how easy it is to integrate it into your application/infrastructure. 

We hope this will make it even easier for users to adopt this technology.

Do send us your questions at the Shoal users mailing list.


Wednesday Dec 05, 2007

Shoal Whitepaper on Scalable Dynamic Clustering

Its been a while since I have blogged. I have been rather busy managing the GlassFish v2 ur1 release which is round the corner around Dec 13/14.

Meanwhile, Mohamed Abdelaziz and I put together a Whitepaper that gives details on the scalability and dynamic clustering aspects of Shoal. The paper provides a good overview of the self composing nature of Shoal and dives deeper into the set of protocols that provide the basis for building fault tolerant infrastructures.

Shoal is coming along very well in various spaces going by the increasing hits to our website from Telco and Financial majors. We are continuously improving Shoal into a robust component based on feedback from our user community. So keep'em coming to the users mailing list.

Tuesday Aug 14, 2007

Shoal Clustering User Guide Part 1

In this series of blog entries, I will provide a guide for new users on how to use Shoal for your application.

This particular blog entry provides a Shoal Clustering 101 type basic introduction. Subsequent blogs will enable the user to get more closer to the metal on how to use this technology.

What is Shoal? 

Shoal is a technology/framework that allows consuming applications the ability to participate in a group and thereby communicate with the group, register interest in being notified of group events, and share application data across group members. These functionalities enable Shoal to be used as a clustering framework in enterprise infrastructure software while also being capable of other use cases.

Shoal's core is the Group Management Service (GMS) which provides client APIs to interact with a group while allowing group communication libraries to be integrated through a service provider interface implementation. Currently with a Jxta service provider, Shoal takes advantage of advanced Jxta features for robustness, reliability and scalability.

While Shoal itself is scalable to many instances in a group (upto 64 nodes in our tests), the scaling is always determined by the size and characteristics of the employing application/product.

Shoal Downloads  

Among the first things you want to know is where to get Shoal downloads.  

Shoal download is available here. Pick the latest zip file for the latest and greatest stable version. The zip file contains two jars of interest - the shoal-gms.jar containing Shoal's client api, GMS core implementation, and a Jxta service provider implementation,  and jxta.jar which is the Jxta peer-to-peer platform.

Shoal Documentation 

Now that you have the jars, you would need documentation to see how to integrate Shoal into your product. Shoal offers APIs to consuming applications to participate in a cluster. Shoal's JavaDocs containing the APIs are available here. In the JavaDoc, select the com.sun.enterprise.ee.cms.core package.  Look in the Description section for a simple introduction to the API.

As the Description mentions, the GMSFactory class located in the com.sun.enterprise.ee.cms.core package is the entry point for getting Shoal's Group Management Service.

Code Snippet 

The following code snippet uses the GMSFactory to start the GMS module, use the GroupManagementService reference to initialize the group, register interest in events,  join the group and the api to call when the process is ready to leave the group : 


//initializes GMS and underlying group communication provider

final GroupManagementService gms = GMSFactory.startGMSModule(serverIdentifierName, groupIdentifierName, GroupManagementService.MemberType.CORE, configProperties);

//register for group events

//register to receive notification when a process joins the group
gms.addActionFactory(new JoinNotificationActionFactoryImpl(this));

//register to receive notification when a group member leaves on a planned shutdown
gms.addActionFactory(new PlannedShutdownActionFactoryImpl(this));

//register to receive notification when a group member is suspected to have failed
gms.addActionFactory(new FailureSuspectedActionFactoryImpl(this));

//register to receive notification when a group member is confirmed failed
gms.addActionFactory(new FailureNotificationActionFactoryImpl(this));

//register to receive notification when this process is selected to perform recovery operations on a failed member's resources
gms.addActionFactory(serviceName, new FailureRecoveryActionFactoryImpl(this));

//register to receive messages from other group members to this registered component

gms.addActionFactory(new MessageActionFactoryImpl(this), componentName);

//joins the group
gms.join();
 

//leaves the group gracefully
gms.shutdown(GMSConstants.shutdownType.INSTANCE_SHUTDOWN); 

As the above code snippet shows, it is extremely easy to instrument Shoal into your application and be on the road to clustering your application and take remedial actions on occurrence of group events.

A Simple Shoal Example (sources)

Look through this example code to see the full source of a Simple Shoal Example.  Look in the runSimpleSample() method to understand steps to make Shoal an in-process component.

Questions? Comments? 

If you have questions on the above, please send your questions to the Shoal users alias: users [at] shoal [dot] dev [dot] java [dot] net or post your questions as a comment to this blog entry.

In the next blog entry, I will go a bit deeper explaining specific parts of Shoal.


Tuesday Aug 07, 2007

Shoal Clustering Framework 1.0 Early Access available

Its been a fairly long time since I blogged.

Over the past few months, we (the GlassFish HA team and the Jxta Team ) have been concentrating on improving and addressing GlassFish high availability features and associated bugs. In the process, Shoal's Group Management Service benefited from intensive QE cycles on 8-node GlassFish clusters under scores of scenarios and test cases. We have been focused on fixing bugs, improving performance, and progressively gating changes in Shoal to manage risks for the upcoming GlassFish v2 FCS release. 

At the moment, Shoal is in good shape and we decided to release the Shoal 1.0 Early Access version a couple of days ago. This will be followed by the 1.0 Final release before the GlassFish v2 release once we know that we have delivered the final acceptable bits for that product.

After that our next step is to address the unique requirements that arise out of the Sailfin project which is building a SIP supported application server based on GlassFish.  

We'd love to hear feedback from our user community about your success stories, issues and enhancement requests using Shoal.

We have seen anecdotal evidence of how useful and easy-to-use this library is and it would help improve our adoption and project growth if more specific feedback is available. Our Statcounter statistics are showing companies from very interesting industry segments going through the Shoal site, downloads and documents so this is a huge boost to our commitment to build a good group communications API based library.

If you have a success story or feedback using Shoal that you can share, drop us a line at the Shoal user alias

users[at]shoal[dot]dev[dot]java[dot]net

and we will highlight it our blogs and on the Shoal web site.

Also we welcome code and design contributions from experienced clustering and distributed systems developers.

 

 

 

About

Shreedhar Ganapathy

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today