Tuesday Mar 01, 2011

Load-balancing feature in GlassFish 3.1

GlassFish 3.1 comes will all rich clustering feature, newly added in this release. GlassFish 3.1 also has load-balancing feature to complement its clustering feature.  Following sections provide details on load-balancer supported in GlassFish 3.1.


GlassFish Server Open Source Edition 3.1

Apache + mod_jk is supported for this flavour of GlassFish Server. User need to download Apache source or binary from Apache site, and build/install it with mod_jk module. GlassFish Server is build over tomcat web container supports AJP protocol, thus it can work with mod_jk module of Apache. It does require some configuration. Please refer to blog or official documentation for complete details : 

Blog : Amy Roh's Blog

Official documentation : Docs bundle

  • Refer to chapter Configuring HTTP Load Balancing in guide ha-administration-guide.pdf

Oracle GlassFish Server 3.1

GlassFish Loadbalancer plugin is available for this flavour of GlassFish Server. This load-balancer plugin in similar to one available in GlassFish 2.1. Please refer to one pager for new features added in this release.

GlassFish Loadbalancer Configurator, installer and configurator for load-balancer plugin, is officially supported in this release. We did beta for this last year, and it is now inducted in the product itself. This tool will enable users to install and configure the loadbalancer plugin on the web server. It has easy to use graphical interface which accepts user inputs, and then installs and configures load-balancer plugin based on that.

This is a IzPack based installer and requires JAVA to execute. It is bundled as a zip file which contains installer jar file. User can start the installer by running java -jar glassfish-lbconfigurator-3_1.jar. User need to provide inputs as per his/her requirement. Each screen has Help button to guide user. Please refer to Help page to understand what inputs need to be provided on that particular screen. The installation of loadbalancer plugin is two step process :

  1. In first step, load-balancer plugin is exploded and installed
  2. In second step, web server is configured to use load-balancer plugin.

Installer provides Post Installation steps, if any. On last screen of installer, user can save a automation script which can be used for silent install at a later point of time.

It also contains an uninstall script, which user can use to unconfigure and uninstall the loadbalancer plugin from the web server.

Please refer to official documentation for supported web servers and platforms.

Download Location : GlassFish Loadbalancer Configurator

Official documentation : Configuring Web Servers for HTTP Load Balancing


Tuesday Jan 12, 2010

GlassFish LoadBalancer Configurator

Please refer to blog Load-balancing feature in GlassFish 3.1 for latest details.

Till now, for GlassFish users, installing and configuring load-balancer plugin was a manual process. Apart from being cumbersome, manual editing is in general error prone. To ease out this process, we are releasing beta version of GlassFish LoadBalancer Configurator. This tool will enable users to install and configure the loadbalancer plugin on the web server. It has easy to use graphical interface which accepts user inputs, and then installs and configures loadbalancer plugin based on that.

This is a IzPack based installer and require JAVA to execute. It is bundled as a zip file which contains installer jar file, user's guide and legal documents. User can start the installer by running java -jar glassfish-lbconfigurator.jar. User need to provide inputs as per his/her requirement. Each screen has Help button to guide user. The installation of loadbalancer plugin is two step process :

  1. In first step, loadbalancer plugin is exploded and installed
  2. In second step, web server is configured to use loadbalancer plugin

Installer provides Post Installation steps, if any. On last screen of installer, user can save a automation script which can be used for silent install at a later point of time.

It also contains an uninstallation script, which user can use to unconfigure and uninstall the loadbalancer plugin from the web server.


Download location : GlassFish Loadbalancer Configurator

Bundle contains

  • glassfish-lbconfigurator.jar : Jar file which user need to execute to install and configure loadbalancer plugin
  • User_Guide.pdf : User's guide providing information on how to use GlassFish LoadBalancer Configurator
  • legal documents

Supported Platforms:

  • Solaris 9/10 SPARC/X86
  • RHEL 3/4/5
  • Windows 2003 Enterprise Edition

Supported Web Servers :

  • Sun Java System Web Server 7.X/6.X
  • Apache 2.2.X\*/2.0.X
  • IIS 5.0/6.0

\*Apache 2.2.x is not supported on Windows

 Queries and Issues :


Wednesday Oct 28, 2009

CLB Enhancements in SailFin 2.0

There are handful of features newly added for Converged Load-Balancer(CLB) in SailFin 2.0. These features are as follows :

  1. DCR plugin in JAVA
  2. CLB monitoring
  3. Increase load on recovered instance in periodic manner
  4. Reuse TCP connection between client to CLB front-end


DCR plugin in JAVA

In SailFin 1.5, user need to provide DCR plugin in propriety XML format. Developer found it difficult to write DCR as it was hard to understand propriety format. Also this format is very restrictive in nature, and difficult to enhance it to support new rules.

Hence in SailFin 2.0, we have introduced support for DCR plugin in JAVA. Please refer to my blog to get complete details on how to develop DCR plugin in JAVA.


CLB Monitoring

In SailFin 2.0, user can gather CLB monitoring data. CLB exposes both front-end and back-end statistics. Monitoring level is dynamically (re)configurable.

How to enable CLB monitoring

User can enable CLB monitoring by setting property converged-load-balancer under module-monitoring-levels to HIGH or
LOW. User can set it using admin CLI or GUI.

Front-end statistics

Front-end statistics are SIP and HTTP requests or responses
handled by CLB acting as front-end. The exposed statistics are as follows :

  • Total number of http requests received
  • Total number of http requests received to be serviced locally
  • Total number of http requests proxied to remote back-end to be serviced
  • Total number of http requests which resulted in error responses
  • Total number of failed over http requests
  • Total number of sip requests received
  • Total number of sip requests received to be serviced locally
  • Total number of sip requests proxied to remote back-end to be serviced
  • Total number of failed over sip requests
  • Total number of sip requests which resulted in error responses
  • Total number of outgoing sip requests
  • Total number of sip responses received
  • Total number of sip responses received to be consumed locally
  • Total number of sip responses proxied to remote back-end to be consumed
  • Total number of sip responses discarded as they cannot be routed to its
    consumer

Back-end statistics

Back-end statistics are SIP and HTTP requests or responses
handled by CLB acting as back-end. The exposed statistics are as follows :

  • Total number of incoming sip requests
  • Total number of incoming http requests
  • Total number of incoming sip responses
  • Total number of outgoing sip responses
  • Total number of outgoing sip requests

Increase load on recovered instance in periodic manner

In SailFin 2.0, new properties are added to increase load on recovered instance in periodic manner. This feature is mainly introduced for Sip Session Replication(SSR).

These new properties are namely load-increase-factor and load-factor-increase-period-in-seconds. These properties can be defined under converged-clb-config and are dynamically (re)configurable. Both of these properties will be used by consistent-hash algorithm in CLB. When an instance recovers, consistent-hash algorithm will not immediately include that instance for load distribution. It will increase load, from 0 to factor of 1, on recovered instance in a periodic manner. The load factor increases by load-increase-factor every load-factor-increase-period-in-seconds seconds.

The default value of load-increase-factor is 1 and load-factor-increase-period-in-seconds is 0. User can set load-increase-factor to any decimal value between 0 and 1, and load-factor-increase-period-in-seconds to positive integer.

User can set using following asadmin commands:

  • asadmin set domain.converged-lb-configs.<converged-lb-config-name>.property.load-factor-increase-period-in-seconds=120
  • asadmin set domain.converged-lb-configs.<converged-lb-config-name>.property.load-increase-factor=.05

All requests which are distributed using consistent hash algorithm will follow described load-distribution. Consistent hash algorithm is applicable for
1. SIP requests
2. HTTP requests for converged application, only if dcr xml is provided.

This does not impact round-robin algorithm at all. Thus HTTP requests belonging to pure web application(s) will have no impact in behavior what-so-ever.

This will not impact load-distribution in case of cluster startup. It will only come into effect in case of instance startup.

If load-factor-increase-period-in-seconds is set to 0, then load-increase-factor will not have any impact. However if load-factor-increase-period-in-seconds is set and load-increase-factor is not set, then recovered instance will not get any load for period specified by load-factor-increase-period-in-seconds. After period load-factor-increase-period-in-seconds  lapses, then load factor for recovered instance will immediately become 1.

Let me explain how it will work with an example. Lets start with some assumptions :
=> There are two instances, instance1 and instance2, in cluster
=> Total Call Per Second(CPS) on system is 200, so per instance it is 100 CPS
=> load-factor-increase-period-in-seconds is set to 60
=> load-increase-factor is set to .1

Now instance1 goes down and then recovers, then load on instance1 will increase in this manner :
0 - 60 : 0 CPS
60 - 120 : 10 CPS
120 - 180 : 20 CPS
180 - 240 : 30 CPS
240 - 300 : 40 CPS
300 - 360 : 50 CPS
360 - 420 : 60 CPS
420 - 480 : 70 CPS
480 - 540 : 80 CPS
540 - 600 : 90 CPS
600 - onward : 100 CPS


Reuse TCP connection between client to CLB front-end connection

In case client in behind a firewall, then SailFin cannot open new TCP connections to client. In such a scenario, SailFin must reuse existing TCP connections between client and SailFin instance.

New property defined to control this behavior is reuseClientFEConnection. The default value of this property is false. User need to set this property to true to enable this feature. User can set it using following asadmin command:

  • asadmin set domain.converged-lb-configs.<converged-lb-config-name>.property.reuseClientFEConnection=true

To achieve it, client to CLB Front-End(FE) mapping is maintained with-in CLB. However there are conditions to store client to CLB FE mapping, which are as follows :

  • Request has to come over TCP
  • Contact is a sip-uri
  • Response has a request associated with it, implies SailFin instance is either acting as UAS or B2BUA

A new mapping of client to CLB FE is created for following cases :

  • For all Register requests which result in 2xx response, a mapping is created for contact to CLB FE address mapping
  • For all dialog creating requests which result in 2xx response, CLB FE address is added as an attribute in session

Mapping created by REGISTER requests are cleaned up either by timeout or REGISTER request with expires set to 0.

For all outgoing requests, we check whether we have a mapping for contact address in map created by REGISTER request, or attribute set in session. If either is found, we push a ROUTE header for CLB FE and use that to send request out to client.


Tuesday Jul 21, 2009

Data Centric Rules in Java

In SailFin 1.0, user can provide Data Centric Rules(DCR) as xml file. User found it is difficult to define rules in xml file. Also flexibility was a major concern.

To solve above issues, in SailFinv2 user can provide DCR plugin as a java implementation. User can define rules as per their requirement in a very flexible manner.

This blog provides detail on how to write a DCR plugin in Java.


Functional Specification

For deep-dive design details on DCR plugin as jar file, user can refer to functional specification. It provides introduction to DCR Plugin, configuration and design details.


Developing DCR Plugin jar

Writing DCR plugin implementation

User implementation for DCR plugin must implement interface org.glassfish.comms.api.datacentric.DcrPlugin. A sample implementation of DCR plugin is provided below :

package mydcr;

import org.glassfish.comms.api.datacentric.DcrPlugin;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.sip.SipFactory;
import javax.servlet.sip.SipServletRequest;
import javax.servlet.sip.SipURI;
import org.glassfish.comms.api.telurl.TelUrlResolver;
import org.glassfish.comms.api.uriutils.UriTools;

public class MyDcrPluginImpl implements DcrPlugin{

    public String getKey(SipServletRequest request,
            SipFactory sipFactory, UriTools uriTools,
            TelUrlResolver telUrlResolver) {
        String key = null;
        String method = request.getMethod().toUpperCase();
        if(method.equals( "PUBLISH" ) || method.equals( "INVITE" ) || method.equals( "CANCEL" )){
           key = SipURI.class.cast(request.getTo().getURI()).getUser();
        } else if (method.equals( "REGISTER" )
                || method.equals( "SUBSCRIBE" )){
            key = SipURI.class.cast(request.getFrom().getURI()).getUser();
        }
        System.out.println("MY DCR PLUGIN - Key extracted from SIP request : " + key);
        return key;
    }

    public String getKey(HttpServletRequest request,
            SipFactory sipFactory, UriTools uriTools,
            TelUrlResolver telUrlResolver) {
        String key =  request.getParameter( "j_username" );
        System.out.println("MY DCR PLUGIN -
Key extracted from HTTP request : " + key);
        return key;
    }


}


Compiling DCR plugin

User need three jars to compile DCR plugin. These jars are ssa-api.jar, javaee.jar and comms-appserv-api.jar. They are available in SailFin installation. Below is command to compile DCR plugin :

javac -classpath <SAILFIN_INSTALL_PATH>/lib/ssa-api.jar:<SAILFIN_INSTALL_PATH>/lib/javaee.jar:<SAILFIN_INSTALL_PATH>/lib/comms-appserv-api.jar -d classes src/mydcr/MyDcrPluginImpl.java

Note: classes directory must exist. Compiled classes will reside in classes directory.

Writing Manifest file

Below is sample manifest file :

Manifest-Version: 1.0
Dcr-Plugin-Class: mydcr.MyDcrPluginImpl

Lets call this file as mymanifest

Creating DCR plugin jar

Command to create jar file using above manifest and compiled dcr plugin implementation class is provided below :

jar -cvfm mydcrplugin.jar mymanifest -C classes mydcr/MyDcrPluginImpl.class

Command to set this jar as DCR file

User can set DCR file using asadmin set-dcr-file command. One example of the command is provided below :

<SAILFIN_INSTALL_PATH>/bin/asadmin set-dcr-file --host <host-name> --port <port-number> --clbconfig  <clb-config-name> mydcrplugin.jar

User can set dcr file using admin GUI as well.


One-stop for GlassFish Load-Balancer Plugin

In this blog we will try and provide references to all information available for GlassFish load-balancer plugin. We will put in continuous efforts to keep this blog up to date.


Contents


InstallerNEW!

User can use GlassFish LoadBalancer Configurator to install and configure load-balancer plugin. It has easy to use graphical interface which accepts user inputs, and then installs and configures loadbalancer plugin based on that. You can refer to my blog for further information.


Latest Version

The latest version of load-balancer plugin is aslb-9.1.1-b13. User can download it from following download locations :


Official documentation

Latest documentation for load-balancer plugin is available under Sun GlassFish Enterprise Server 2.1's High Availability Administration Guide. The relevant information is in two chapters :

There is also an official white paper available on GlassFish load-balancer plugin.


Blogs

There are multiple blogs written on how to configure load-balancer plugin with various web-servers. Below are links for them :


Bugs fixed in a load-balancer plugin


Wednesday Feb 25, 2009

Configuring GlassFish load-balancer plugin in IIS

IIS is supported web-server for installing load-balancer plugin for a long time. Sun GlassFish Enterprise Server comes with a installer which takes care of configuring load-balancer plugin on IIS. There are still some manual steps, but they are minimal.

In this blog, user will be provided with complete set of manual steps to configure load-balancer plugin on IIS.

Platform supported

  • Windows 2003 Enterprise Edition

Currently the load-balancer plugin is well tested on above mentioned platform. It may work on other windows platform, even though it is not tested on those platform.


IIS Version supported

  • IIS 6.0
  • IIS 5.0

These are tested version of IIS with load-balancer plugin


Download Location

The load-balancer plugin library can be downloaded from http://download.java.net/javaee5/external/WINNT/aslb/jars

The latest version of load-balancer plugin available is aslb-9.1.1-b7.jar. User can download any other latest version compared to above version, if available.


Steps to configure load-balancer plugin on IIS

Let me first put down steps which is automatically taken care of by installer when using Sun GlassFish Enterprise Server :

  1. User has already installed IIS and directory C:\\Inetpub\\wwwroot(or equivalent) exists. Now create a directory sun-passthrough under C:\\Inetpub\\wwwroot. From now on directory C:\\Inetpub\\wwwroot\\sun-passthrough will be referred to as <sun-passthrough-dir>.
  2. Download load-balancer plugin and unjar it. Then unzip SUNWaspx.zip and SUNWaslb.zip. This directory will be referred to as <lbplugin-unzip-dir>.
  3. Create directory
    1. <sun-passthrough-dir>\\errorpages
    2. <sun-passthrough-dir>\\resource
    3. <sun-passthrough-dir>\\sec_db_files
  4. Copy file - <lbplugin-unzip-dir>\\lib\\webserver-plugin\\windows\\iis\\sun-passthrough.dll to <sun-passthrough-dir>. Also change permission of the file to executable for IIS run-as user(IIS_WPG).
  5. Copy file - <lbplugin-unzip-dir>\\lib\\webserver-plugin\\windows\\iis\\sun-passthrough.properties to <sun-passthrough-dir>. Append following entries to the file :
    1. lb-config-file=<sun-passthrough-dir>\\loadbalancer.xml
    2. log-file=<sun-passthrough-dir>\\lb.log
    3. log-level=INFO
  6. Copy files - <lbplugin-unzip-dir>\\lib\\webserver-plugin\\windows\\iis\\errorpages\\default-error.html and <lbplugin-unzip-dir>\\lib\\webserver-plugin\\windows\\iis\\errorpages\\sun-http-lberror.html to <sun-passthrough-dir>\\errorpages directory.
  7. Copy files - <lbplugin-unzip-dir>\\lib\\webserver-plugin\\windows\\iis\\LBPluginDefault_root.res and <lbplugin-unzip-dir>\\lib\\webserver-plugin\\windows\\iis\\LBPlugin_root.res to <sun-passthrough-dir>\\resource directory.
  8. Copy files - <lbplugin-unzip-dir>\\lib\\webserver-plugin\\windows\\iis\\secmod.db, <lbplugin-unzip-dir>\\lib\\webserver-plugin\\windows\\iis\\cert8.db and <lbplugin-unzip-dir>\\lib\\webserver-plugin\\windows\\iis\\key3.db to <sun-passthrough-dir>\\sec_db_files directory.
  9. Copy file <lbplugin-unzip-dir>\\lib\\install\\templates\\loadbalancer.xml.example to <sun-passthrough-dir>\\loadbalancer.xml directory.
  10. Copy files - <lbplugin-unzip-dir>\\lib\\dtds\\sun-loadbalancer_1_1.dtd and <lbplugin-unzip-dir>\\lib\\dtds\\sun-loadbalancer_1_2.dtd to <sun-passthrough-dir> directory.

Manual steps that are needed even when using installer are well documented and are available at http://docs.sun.com/app/docs/doc/819-4740/abdhq?a=view
Note: In step 6, use <lbplugin-unzip-dir>\\lib instead of <application-server-install-dir\\bin and <application-server-install-dir\\lib as suggested in linked documentation.

User need to edit load-balancer xml(<sun-passthrough-dir>\\loadbalancer.xml) to reflect their cluster configuration. User can export load-balancer xml in DAS and then copy it to above mentioned location using following instructions.


Monday Feb 09, 2009

Converged Load-Balancer(CLB) Whitepaper and Screencast

Pankaj has written a whitepaper on Converged Load-Balancer(CLB). This whitepaper provides following details :

  1. Overview of CLB
  2. Deployment Topology
  3. Load-balancing algorithms
  4. Functional details of CLB
  5. Cluster health monitoring
To see CLB in action, you can also view the CLB screencast. This was presented at SailFin webinar. To see the live demo, please visit SailFin booth at Mobile World Congress, 16-19 February 2009, Barcelona.

Wednesday Nov 19, 2008

Load-balancer plugin in GlassFish now supports Apache 2.2

Support for Apache 2.2 will be coming in GlassFish Server v2.1. Users can download this version of GlassFish server from GlassFish download page. The enterprise edition will even contain installer support for installing load-balancer plugin on Apache 2.2.

For the benefit  of GlassFish user, steps to manually configure load-balancer plugin with Apache2.2 is provided in this blog.


 Platform supported

  • Solaris SPARC
  • Solaris x86
  • Linux

For exact version of the platform supported, please refer to release notes of GlassFish Server 2.1

NOTE : Apache 2.2 in not supported on windows right now. However Apache 2.0.x is supported on windows.  


Download Location

The load-balancer plugin library can be downloaded from http://download.java.net/javaee5/external/<OS>/aslb/jars where OS can be SunOS, SunOS_X86 and Linux.

The latest version of load-balancer plugin available is aslb-9.1.1-b5.jar. User can download any other latest version compared to above version, if available.


Steps to configure load-balancer plugin on Apache2.2

  1. Build Apache 2.2 and install it. If user wants to use auto-apply feature, then it should be build with openssl and server certificate must be installed. You can refer to Apache2.0 documentation for building Apache2.2 with ssl and installing certificate. Use --with-included-apr option when building Apache 2.2 to build bundled apr. Apache 2.2 install directory will be referred to as <apache2.2-install-dir>.

  2. Download load-balancer plugin and unjar it. Then unzip SUNWaspx.zip and SUNWaslb.zip. This directory will be referred to as <lbplugin-unzip-dir>.

  3. Create directory

    1. <apache2.2-install-dir>/modules/errorpages
    2. <apache2.2-install-dir>/modules/resource
    3. <apache2.2-install-dir>/sec_db_files
  4. Copy file - <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/mod_loadbalancer.so to <apache2.2-install-dir>/modules. Also change permission of the file to executable.

  5. Copy files - <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/errorpages/default-error.html and <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/errorpages/sun-http-lberror.html to <apache2.2-install-dir>/modules/errorpages directory.

  6. Copy files - <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/LBPluginDefault_root.res and <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/LBPlugin_root.res to <apache2.2-install-dir>/modules/resource directory.

  7. Copy files - <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/secmod.db, <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/cert8.db and <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/key3.db to <apache2.2-install-dir>/sec_db_files directory.

  8. Copy file <lbplugin-unzip-dir>/lib/install/templates/loadbalancer.xml.example to <apache2.2-install-dir>/conf directory.

  9. Copy files - <lbplugin-unzip-dir>/lib/dtds/sun-loadbalancer_1_1.dtd and <lbplugin-unzip-dir>/lib/dtds/sun-loadbalancer_1_2.dtd to <apache2.2-install-dir>/conf directory.

  10. Changes in file <apache2.2-install-dir>/conf/httpd.conf

    1. Change
      FROM:
      ServerName www.example.com:80
      TO:
      ServerName <host-name>:80

    2. Change
      FROM:
      #Include conf/extra/httpd-mpm.conf
      TO:
      Include conf/extra/httpd-mpm.conf

    3. Change
      FROM:
      #Include conf/extra/httpd-vhosts.conf
      TO:
      Include conf/extra/httpd-vhosts.conf

    4. Change
      FROM:
      #Include conf/extra/httpd-ssl.conf
      TO:
      Include conf/extra/httpd-ssl.conf

    5. Append following entry
      ##BEGIN EE LB Plugin Parameters
      LoadFile /usr/lib/libCstd.so.1 (THIS LINE IS ONLY NEEDED ON SOLARIS SPARC)
      LoadModule apachelbplugin_module modules/mod_loadbalancer.so
      #AddModule apachelbplugin_module
      <IfModule apachelbplugin_module>
      config-file <apache2.2-install-dir>/conf/loadbalancer.xml
      locale en
      </IfModule>
      ##END EE LB Plugin Parameters

  11. Changes in file <apache2.2-install-dir>/conf/extra/httpd-mpm.conf

    1. Change prefork MPM settings
      Set StartServers to 1
      Set MaxClients to 1

  12. Changes in file <apache2.2-install-dir>/conf/extra/httpd-vhosts.conf

    1. Append
      ##BEGIN EE LB Plugin Parameters
      <VirtualHost <machine-ip-address>>
          ServerName <host-name>
          DocumentRoot "<apache2.2-install-dir>/htdocs"
      </VirtualHost>
      ##END EE LB Plugin Parameters

    2. User can remove other existing VirtualHost entries from this file

  13. Changes in file <apache2.2-install-dir>/conf/extra/httpd-ssl.conf

    1. Change
      FROM:
      <VirtualHost _default_:443>
      TO:
      <VirtualHost <machine-ip-address>:443>
      b)Change
      FROM:
      ServerName www.example.com:443
      TO:
      ServerName <host-name>:443

  14. If user want to use feature auto-apply feature

    1. Export DAS certificate using command : certutil -L -d <application-server-DAS-dir>/config -n s1as -a -o sjsas.crt

    2. Copy certificate file(sjsas.crt) to <apache2.2-install-dir>/conf/ssl.crt directory. If directory does not exist, create one.

    3. Append in file <apache2.2-install-dir>/conf/extra/httpd-ssl.conf
      ##BEGIN EE LB Plugin Parameters
      <Location /lbconfigupdate >
      SSLVerifyClient require
      SSLVerifyDepth 1
      SSLRequireSSL
      SSLCACertificateFile <apache2.2-install-dir>/conf/ssl.crt/sjsas.crt
      SSLRequire ( %{SSL_CIPHER} !~ m/\^(EXP|NULL)-/ \\
      and %{SSL_CLIENT_S_DN_O} eq "Sun Microsystems" \\
      and %{SSL_CLIENT_S_DN_OU} eq "Sun Java System Application Server" \\
      and %{SSL_CLIENT_M_SERIAL} eq "<Cert-Serial-Number>" )
      </Location>
      <Location /getmonitordata >
      SSLVerifyClient require
      SSLVerifyDepth 1
      SSLRequireSSL
      SSLCACertificateFile  <apache2.2-install-dir>/conf/ssl.crt/sjsas.crt
      SSLRequire ( %{SSL_CIPHER} !~ m/\^(EXP|NULL)-/ \\
      and %{SSL_CLIENT_S_DN_O} eq "Sun Microsystems" \\
      and %{SSL_CLIENT_S_DN_OU} eq "Sun Java System Application Server" \\
      and %{SSL_CLIENT_M_SERIAL} eq "<Cert-Serial-Number>" )
      </Location>
      ##END EE LB Plugin Parameters

      NOTE:

      1. Cert-Serial-Number need to be generated from provided DAS certificate file(sjsas.crt).  Command for that is : keytool -printcert -file sjsas.crt. Note Serial number from output of above command. Take its value and change all lower case characters to upper case.
      2. Application Server name is assumed to be Sun Java System Application Server. If it is different, then it needs to be reflected here as well. Above command will print that as well.
  15. Changes in file <apache2.2-install-dir>/bin/envvars

    1. Suffix <lbplugin-unzip-dir>/lib and <apache2.2-install-dir>/modules directory to LD_LIBRARY_PATH on unix platform

  16. For auto-apply feature to work, the run-as user for apache must have writable permission for <apache2.2-install-dir>/conf directory. If security is not an issue, user can even set <apache2.2-install-dir>/conf directory permission to 777.


Monday Mar 31, 2008

SIP Load-Balancer

Are you looking for load-balancer to handle SIP requests ? If yes, you have come to right place .

Brief history : Sun Communication Application Server, aka sailfin, is built over glassfish. Glassfish is  Java EE 5 based application server. And sailfin provides SIP Servlet Container based on JSR 289 over and above what is offered by Glassfish.

Sailfin not only provide containers to deploy SIP and JAVA EE 5 based applications. It is complete package with high-availability features like load-balancing and session-replication required by enterprise systems.

 To learn more about load-balancer available with sailfin, please refer to my blogs on Converged Load Balancer.

Monday Mar 24, 2008

Converged Load-Balancer (CLB)


Brief Overview of Converged Load-Balancer

Converged Load-Balancer, is designed to provide high-availability to applications deployed on Sun Communication Application Server, aka sailfin. It will distribute requests across the instances in cluster to increase throughput of the system. It will also fail-over request from unhealthy/inaccessible instances to healthy and available instances.

Features of Converged Load-Balancer

  1. It can handle both SIP(S) and HTTP(S) requests.
  2. It can load-balance converged application (application having web and sip components) and pure web applications
  3. It will distribute the load across healthy and available instances in cluster, thus enabling a higher throughput from the system
  4. It maintains stickiness of requests in a session to single instance till serving instance is healthy, available and enabled
  5. It will fail-over request from unhealthy/unavailable/disabled instances to instances which are healthy, available and enabled
  6. It supports round-robin and consistent-hash algorithm for load-balancing
  7. It supports Data-Centric-Rule(DCR) to extract hash-key used in consistent-hash algorithm

Deployment Topology

Converged load-balancer currently supports only self load-balancing cluster. Below figure illustrates self load-balancing topology.

Converged Load-Balancer Deployment

The above deployment contains:
  • Hardware IP sprayer : This distributes request evenly across all instances in sailfin cluster.
  • Sailfin Cluster : Sailfin cluster having converged load-balancer component.
In case user does not have Hardware IP sprayer, the request can be forwarded to any one instance in sailfin cluster. The converged load-balancer component on that instance will ensure that requests are distributed across the cluster. However that instance will be single point of failure. The presence of Hardware IP sprayer ensures high availability.

Note : Sailfin does not support two-tier deployment as of now. However there is no such restriction put on admin commands. User can still create two tier deployment using admin commands. Two-tier deployment may not function correctly.

Functioning of Converged Load-Balancer

Below illustration depicts the functioning of converged load-balancer

CLB Functionality
  • Step 1 : Client request comes to IP Sprayer
  • Step 2 : IP sprayer selects any of the sailfin instances in the cluster and forwards request to that instance, which in above illustration is instance X.
  • Step 3 : CLB on Instance X based on configured algorithm selects an instance to service the request. Then it forward the request to that instance, which in above illustration is instance Y. This can be same instance, i.e. instance X, as well. In such case Step 3 and 4 are bypassed.
  • Step 4 : CLB on Instance Y receives the request. It finds out that request is already proxied from another instance. Without any further processing, it passes the request to the container so that it can be serviced. CLB then sends the response back to CLB on Instance X.
  • Step 5 : CLB on Instance X in turn sends the response back to IP sprayer
  • Step 6 : IP sprayer sends the response back to the client

Algorithms

Converged load-balancer currently supports two load-balancing algorithm
  1. Round-robin : Instance to service new request are selected in a round robin fashion. 
  2. Consistent-hash : Instance to service new request is selected based on hash-key extracted from request.

In both cases, once a session is established, sticky information is set on response. Subsequent requests will have that sticky information. Thus subsequent requests part of same session will stick to same instance.

There are two possible configuration:

  • Configuration 1: This should be used when only pure web applications are deployed. The user does not provide a dcr-file.
    • Round-robin algorithm for all http requests
    • Consistent-hash algorithm for all sip requests, in case converged/pure-sip applications are deployed. The hash-key used to select instance is extracted using from-tag,to-tag,call-id parameters of the request.
  • Configuration 2: This should be used when converged/pure-sip applications are deployed on the application server. The user must provide a dcr-file in this case, to extract hash key from sip and http requests. If dcr-file is not provided, sip and http requests part of same session may be serviced by different instances.
    • Round-robin algorithm for http requests belonging to pure web applications
    • Consistent-hash algorithm for http requests belonging to converged applications
      • If any dcr rule matches http request, hash-key is extracted using that rule
      • If none of the rules matches http request, hash key is extracted using remote host and port of the http request
    • Consistent-hash algorithm for sip request belonging to converged/pure-sip applications
      • If any dcr rule matches sip request, hash-key is extracted using that rule
      • If none of the rules matches sip request, hash key is extracted using from-tag,to-tag,call-id parameters of the sip request

Note : For complete details of converged load-balancer, please go through functional specifications of converged load-balancer.

How to configure converged load-balancer

Some common points to remember before a user start with configuration of converged load-balancer
  1. The converged load-balancer uses xml file converged load-balancer xml to get cluster information. If this file is not present, CLB will not initialize and will return error. The request will not be serviced.
  2. The converged load-balancer xml is generated when auto-commit attribute of converged-load-balancer element under availability-service is set to true. Till it set to false, no xml file is generated.
  3. It is recommended that attribute auto-commit should be set to true, once user is done with complete configuration of cluster. This includes instances creation and application deployment. Otherwise it will result in unnecessary reconfiguration with any change to cluster.
  4. It is recommended to start cluster after auto-commit is set to true and all cluster configuration is done. Please refer to point 1 above to understand why this is recommended.
  5. Only instances with lb-enabled attribute set to true will participate in load-balancing. The disabled instances are not considered for load-balancing. The command to enable/disable instance is still not available. It will be provided shortly. Till then user can do it using set command. Any new instance created in cluster will have lb-enabled set to false. User has to set this attribute to true.
  6. The cluster heartbeat-enabled attribute must be set to true. This is set to true by default, when user creates a cluster.

Using default-cluster

A default-cluster already exists in the domain created using cluster profile. The default-cluster is pre-configured as self load-balancing cluster. User has to just create instances and deploy applications and a self load-balancing cluster is ready for use. Please follow steps below :
  1. User must ensure that all instances are created in cluster and all applications are deployed to cluster. This is not a mandatory step. However this is a recommended step.
  2. Set lb-enabled attribute for all instances of default-cluster to true
    • Command : asadmin> enable-converged-lb-server default-cluster
  3. Set auto-commit attribute to true
    • Command : asadmin > set default-cluster.availability-service.converged-load-balancer.auto-commit=true
  4. Start default-cluster. If it was already running, it will now have working converged load-balancer. There is no need to restart the cluster

Creating converged-load-balancer on already existing cluster

The converged-load-balancer element does not exist under cluster config and user is starting afresh with converged load-balancer configuration. Please follow steps below :

Option 1 : A single step process
  1. User must ensure that all instances are created in cluster and all applications are deployed to cluster. This is not a mandatory step. However this is a recommended step.
  2. Create converged load-balancer using a single command
    • Command : asadmin > create-converged-lb --target <cluster-name> --autocommit=true --selfloadbalance=true --lbenableallinstances=true --configfile <converged-load-balancer-xml> <converged-load-balancer-name>
  3. Start the cluster. If cluster was already running, please restart the cluster.
Option 2 : Multi-step process with details about what exactly happens in background with Option 1
  1. User must ensure that all instances are created in cluster and all applications are deployed to cluster. This is not a mandatory step. However this is a recommended step.
  2. Create a converged load-balancer config
    • Command : asadmin> create-converged-lb-config <clb-config-name>
    • For example : asadmin> create-converged-lb-config test-clb-config
  3. Create a converged load-balancer reference to the cluster
    • Command : asadmin> create-converged-lb-ref --clbconfig  <clb-config-name>  --selfloadbalance=true --lbenableallinstances=true <cluster-name>
    • For example : asadmin> create-converged-lb-ref --clbconfig  test-clb-config  --selfloadbalance=true --lbenableallinstances=true test-cluster
  4. Create converged load-balancer for the cluster
    • Command : asadmin> create-converged-lb --clbconfig <clb-config-name> --configfile <converged-load-balancer-xml-for-cluster> --autocommit=true --target <cluster-name> <converged-load-balancer-name>
    • For example : asadmin> create-converged-lb --clbconfig test-clb-config --configfile converged-load-balancer.xml --autocommit=true --target test-cluster test-cluster-clb
  5. Start the cluster. If cluster was already running, please restart the cluster.
Note:
  1. The above command does not show the default options to be provided with each command, for example user name, password file etc.
  2. Above mentioned way of configuration is just one way of configuring converged load-balancer. The used commands has many more options. Please look at man page of each command for all possible options.


Data Centric Rules(DCR)

Data centric rules are used extract the hash-key from the request. The extracted hash-key is used to select an instance to service the request under consistent-hash algorithm.

Sample DCR file

Below is a sample DCR file :
<?xml version="1.0" encoding="ISO-8859-1"?>
<user-centric-rules>
    <sip-rules>
        <if>
            <session-case>
                <equal>ORIGINATING</equal>
                <if> 
                    <header name="ConferenceName"
                            return="request.ConferenceName">
                        <exist/>
                    </header>  
                </if>
            </session-case>
        </if>
    </sip-rules>
    <http-rules> 
        <request-uri parameter="ConferenceName" return="parameter.ConferenceName">
            <exist/>
        </request-uri>
    </http-rules>  
</user-centric-rules>

Above sample DCR file define following rules :
  1. For SIP(S) request, for an originating request look for header named ConferenceName, and value of that header will be used as hash key.
  2. For HTTP(S) request, look for parameter named ConferenceName, and value of that parameter will be used as hash key.

Configuring DCR 

A user can setup DCR in following manner :
  1. At the time of converged load-balancer creation using a single command.
    • Command : asadmin > create-converged-lb --target <cluster-name> --autocommit=true --selfloadbalance=true --lbenableallinstances=true --configfile <converged-load-balancer-xml> --dcrfile <dcr-file-name> <converged-load-balancer-name>
  2. At the time of converged load-balancer config creation
    • Command : asadmin> create-converged-lb-config --dcrfile <dcr-file-name> <clb-config-name>
  3. After creation of converged load-balancer config
    • Command :  asadmin> set-dcr-file --dcrfile <dcr-file-name> <clb-config-name>
Note:
  1. The above command does not show the default options to be provided with each command, for example user name, password file etc.

Tuesday Apr 10, 2007

Connection Leak Tracing

Does your application suffer from connection starvation? Worked long and hard on your application  to get rid of such issues but in vain . Ever wondered what is eating up all the connections. Unable to figure out whether it is your application or someone-else's . Now you have an option in glassfish v2 to trace out all such cases of connection leaks . The new connection leak tracing feature not only provides information of connection leaks, it also allows a user to reclaim leaking connections to make it available for other applications.

Now on a more technical front a brief description of the feature is provided, and how to configure, monitor and view the connection leaks.


Brief Description

Lets start with the definition of connection leak. As of now the definition is kept really simple. An application is said to be leaking connection, if it acquires a connection and does not close it within specified time period. If this feature is enabled, the application server detects these potential connection leaks and dumps the leak tracing logs to server logs. Looking at the logs the user can figure out which application is leaking connection and fix the issues with application, if exists.

An additional feature of connection reclaim is also provided with connection leak tracing. This can only be enabled when connection leak tracing is turned on. If enabled, the leaking connection with be reclaimed by application server and made available to other applications accessing the connection pool. This feature should be used responsibly. The feature should not be enabled if user's application caches connection. In such cases, enabling this feature will result in unexpected behavior.

For more detailed description, please refer to the one pager.

This feature is not enabled by default. The user need to configure his/her connection pools in order to enable connection leak tracing. The steps to achieve this is provided in the next section.


How to Configure

The connection leak tracing is an advanced feature. Additional steps are needed to configure the connection leak tracing after creating a connection pool. These settings are picked up by application server dynamically. Hence no server restart is needed.

Using GUI


1. Browse to the pool. This can be jdbc(or connector) connection pool.
Browse to the Pool
2. Click on Advanced Tab of the pool
Advanced Tab
3. Provide appropriate inputs
  • Provide non-zero value for Leak Timeout to enable connection leak tracing
  • Click the check-box of Leak Reclaim if you want connection reclaim to be enabled
Provide Pool Inputs
4. Save the changes


Using CLI


1. Set a non-zero connection leak timeout value for connection pool
  • For jdbc connection pool
    • asadmin>set domain.resources.jdbc-connection-pool.<pool-name>.connection_leak_timeout_in_seconds = 300
  • For connector connection pool
    • asadmin>set domain.resources.connector-connection-pool.<pool-name>.connection_leak_timeout_in_seconds = 300
2. Enable connection leak reclaim if desired
  • For jdbc connection pool
    • asadmin>set domain.resources.jdbc-connection-pool.<pool-name>.connection-leak-reclaim = true
  • For connector connection pool
    • asadmin>set domain.resources.connector-connection-pool.<pool-name>.connection-leak-reclaim = true



Monitoring stats for connection leak tracing

The connection leak tracing data is available to the user when monitoring is enabled for that component. To enable monitoring please refer to monitoring manual. For getting connection leak stats of :
  • JDBC connection pool, set monitoring level for JDBC connection pool component to HIGH.
  • Connector connection pool, set monitoring level for JMS/Connector Service component to HIGH.
After configuring the monitoring level for component, the monitoring data can be viewed using GUI as well as CLI.

Using GUI


1. Browse to Monitor tab. Now click on Resources tab. Select your connection pool from the drop down box.
Browse to monitoring tab
2. Scroll down the page and you will find connection leak stats
Connection leak stats

Using CLI

The command to get monitoring stats related to connection leak using CLI is:
  • For jdbc/connector connection pool
    • asadmin> get -m server.resources.<pool-name>.numpotentialconnleak\*
  • The stats will look like
    • server.resources.<pool-name>.numpotentialconnleak-count = 3
    • server.resources.<pool-name>.numpotentialconnleak-description = Provides a count value reflecting the number of potential connection leaks
    • server.resources.<pool-name>.numpotentialconnleak-lastsampletime = 1175495050536
    • server.resources.<pool-name>.numpotentialconnleak-name = NumPotentialConnLeak
    • server.resources.<pool-name>.numpotentialconnleak-starttime = 1175494865525
    • server.resources.<pool-name>.numpotentialconnleak-unit = Count


Viewing potential connection leak traces

The connection leak traces are dumped  to server log with WARNING as log level. So, the log level of Connector component should be at least WARNING. This is default log level for out of the box installation. To enable logging, please refer to logging manual.

It is cumbersome to search through server logs manually. A utility is already provided in application server GUI to search through logs. The steps are provided below to get connection leak traces using this utility.

1. Open the Log Search page by clicking View Log files button. It will open up a new window.
open log search page
2. Click Advanced Search on this page
Click Advanced Search
3. Provide inputs
  • Select Log Level : WARNING
  • Select Logger : javax.enterprise.resource.resourceadapter
  • Provide connection pool name in Name-Value Pairs : ConnectionPoolName=<pool-name>
  • Click Search
Provide inputs
4. If connection leak records exist, they will be displayed
Log records
5. Click on individual record to view the record in detail
Record Details
6. The complete stack trace is provided in record detail. User can browse through it and find out the application which is causing connection leaks



About

kshitiz

Search

Categories
Archives
« July 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
  
       
Today