Tuesday Jun 16, 2009

TOTD #84: Using Apache + mod_proxy_balancer to load balance Ruby-on-Rails running on GlassFish


TOTD #81 explained how to install/configure nginx for load-balancing/front-ending a cluster of Rails application running on GlassFish Gem. Another popular approach in the Rails community is to use Apache HTTPDmod_proxy_balancer. A user asked the exact details of this setup on the GlassFish Gem Forum. This Tip Of The Day (TOTD) will clearly explain the steps.
  1. Create a simple Rails scaffold and run this application using GlassFish Gem on 3 separate ports as explained in TOTD #81.
  2. Setup and configure HTTPD and mod_proxy_balancer
    1. Setup and install Apache HTTPD as explained here. I believe mod_proxy_balancer and other related modules comes pre-bundled with HTTPD, at least that's what I observed with Mac OS X 10.5.7. Make sure that the "mod_proxy_balancer" module is enabled by verifying the following line is uncommented in "/etc/apache2/httpd.conf":

      LoadModule proxy_balancer_module libexec/apache2/mod_proxy_balancer.so

      Please note another similar file exists in "/etc/httpd/httpd.conf" but ignore that one.
    2. Setup a mod_proxy_balancer cluster by adding the following fragment in "httpd.conf" as:

      <Proxy balancer://glassfishgem>
      BalancerMember http://localhost:3000
      BalancerMember http://localhost:3001
      BalancerMember http://localhost:3002
      </Proxy>

      The port numbers must exactly match with those used in the first step.
    3. Specify the ProxyPass directives to map the cluster to a local path as:

      ProxyPass / balancer://glassfishgem/
      CustomLog /var/log/glassfishgem.log/apache_access_log combined

      The "/" at the end of "balancer://glassfishgem" is very important to ensure that all the files are resolved correctly.
    4. Optionally, the following directive can be added to view the access log:

      CustomLog /var/log/glassfishgem.log/apache_access_log combined

      Make sure to create the directory specified in "CustomLog" directive.
  3. Now the application is accessible at "http://localhost/runlogs". If a new GlassFish instance is started then update the <Proxy> directive and restart your HTTPD as "sudo httpd -k restart". Dynamic update of BalancerMembers can be configured as explained here.
TOTD #81 started the Rails application in root context. You can alternatively start the application in a non-root context as:

~/tools/jruby/rails/runner >../../bin/jruby -S glassfish -e production -c myapp
Starting GlassFish server at: 10.0.177.178:3000 in production environment...
Writing log messages to: /Users/arungupta/tools/jruby-1.3.0/rails/runner/log/production.log.
Press Ctrl+C to stop.
. . .
~/tools/jruby/rails/runner >../../bin/jruby -S glassfish -e production -c myapp -p 3001
Starting GlassFish server at: 10.0.177.178:3001 in production environment...
Writing log messages to: /Users/arungupta/tools/jruby-1.3.0/rails/runner/log/production.log.
Press Ctrl+C to stop.
. . .
~/tools/jruby/rails/runner >../../bin/jruby -S glassfish -e production -c myapp -p 3002
Starting GlassFish server at: 10.0.177.178:3002 in production environment...
Writing log messages to: /Users/arungupta/tools/jruby-1.3.0/rails/runner/log/production.log.
Press Ctrl+C to stop.

and then the ProxyPass directive will change to:

ProxyPass /myapp/ balancer://glassfishgem/myapp/

The changes are highlighted in bold. And the application is now accessible at "http://localhost/myapp/runlogs".

After discussing on Apache HTTP Server forum, the BalancerMember host/port can be printd in the log file using a custom log format. So add the following log format to "/etc/apache2/httpd.conf":

LogFormat "%h %l %u %t \\"%r\\" %>s %b \\"%{Referer}i\\" \\"%{User-agent}i\\" \\"%{BALANCER_WORKER_NAME}e\\"" custom

And change the format from the default "combined" to the newly defined "custom" format as:

CustomLog /var/log/glassfishgem.com/apache_access_log custom

Three subsequent invocations of "http://localhost/runlogs" then prints the following log entries:

::1 - - [17/Jun/2009:10:53:53 -0700] "GET /runlogs HTTP/1.1" 304 - "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.11) Gecko/2009060214 Firefox/3.0.11" "http://localhost:3002"
::1 - - [17/Jun/2009:10:54:04 -0700] "GET /runlogs HTTP/1.1" 200 621 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.11) Gecko/2009060214 Firefox/3.0.11" "http://localhost:3000"
::1 - - [17/Jun/2009:10:54:05 -0700] "GET /runlogs HTTP/1.1" 304 - "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.11) Gecko/2009060214 Firefox/3.0.11" "http://localhost:3001"

As evident from the last fragment of each log line, the load is distributed amongst three GlassFish Gem instances. More details on load balancer algorithm are available here.

Feel free to drop a comment on this blog if you are using GlassFish in production for your Rails applications. Several stories are already available at rubyonrails+glassfish+stories.

Technorati: glassfish rubyonrails apache httpd mod_proxy_balancer loadbalancing clustering

Thursday Jan 29, 2009

TOTD #67: How to front-end a GlassFish Cluster with Apache + mod_jk on Mac OSX Leopard ?

GlassFish provides support for High Availability by creating a cluster of server instances and session state replication. This enhances the scalability and availability of your application and is a critical piece of decision making critieria when selecting an Application Server. Clustering in GlassFish Version 2 provides comprehensive introduction to clustering, high availability and load balancing in GlassFish.

GlassFish provides out-of-the-box support for load-balancing HTTP(S), JMS, and RMI/IIOP traffic and front-ended by Sun Java System Web Server, Apache Web Server, and Microsoft IIS (more details here) using the Load Balancer plug-in. This plug-in however is not available for Mac OS X and a popular technique used on that platform for front-ending is to use Apache httpd + mod_jk. This is exactly what this TOTD (Tip Of The Day) is going to describe.

This TOTD is going to explain how to front-end a 3-instance GlassFish cluster with Apache httpd and mod_jk on Mac OS X.

This blog is using information from the following blogs:
And thanks to Vivek and Shreedhar for helping me understand the guts of GlassFish High Availability.

Without further ado, lets get started. The steps are slightly involving so strap your seatbelts!
  1. First, lets create a 3-instance cluster following the screencast at GlassFish Clustering in under 10 minutes. Use the cluster name as "cluster1" and instance names as "instance1", "instance2", "instance3". The admin console will look like:



    Deploy "clusterjsp" and make sure it works using port hopping as explained in the screencast. Click on each instance to identify their associated HTTP port.
  2. Define "jvmRoute" and "enableJK" properties on the newly created cluster as:

    ~/samples/v2/clustering/glassfish/bin >./asadmin create-jvm-options --target cluster1 "-DjvmRoute=\\${AJP_INSTANCE_NAME}"
    Command create-jvm-options executed successfully.
    ~/samples/v2/clustering/glassfish/bin >./asadmin create-jvm-options --target cluster1 "-Dcom.sun.enterprise.web.connector.enableJK=\\${AJP_PORT}"
    Command create-jvm-options executed successfully.

    These properties are required to enable "stickiness" for "mod_jk". More details about how these properties are used internally is explained here.
  3. Configure the above system properties for each instance in the cluster as shown:

    ~/samples/v2/clustering/glassfish/bin >./asadmin create-system-properties --target instance1 AJP_INSTANCE_NAME=instance1
    Command create-system-properties executed successfully.
    ~/samples/v2/clustering/glassfish/bin >./asadmin create-system-properties --target instance1 AJP_PORT=9090
    Command create-system-properties executed successfully.
    ~/samples/v2/clustering/glassfish/bin >./asadmin create-system-properties --target instance2 AJP_INSTANCE_NAME=instance2
    Command create-system-properties executed successfully.
    ~/samples/v2/clustering/glassfish/bin >./asadmin create-system-properties --target instance2 AJP_PORT=9091
    Command create-system-properties executed successfully.
    ~/samples/v2/clustering/glassfish/bin >./asadmin create-system-properties --target instance3 AJP_INSTANCE_NAME=instance3
    Command create-system-properties executed successfully.
    ~/samples/v2/clustering/glassfish/bin >./asadmin create-system-properties --target instance3 AJP_PORT=9092
    Command create-system-properties executed successfully.

    Note the value of "AJP_PORT" property for each instance, this will be used for configuring "mod_jk" later. You may have to restart the cluster in order for these properties to be synchronized for each instance. This can be easily done using the admin console as explained in the screencast above.
  4. Install httpd: Mac OS X Leopard 10.5.6 comes bundled with Apache httpd 2.2, so that's cool! Otherwise it can be downloaded from httpd.apache.org. However in the pre-installed version there are some intricacies with directory names that are explained below.
  5. Lets install and configure "mod_jk" in "httpd".
    1. The mod_jk binaries for Mac OSX are supposedly available at www.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/macosx/. But installing the available module in httpd gives the following error:

      httpd: Syntax error on line 116 of /private/etc/apache2/httpd.conf: Cannot load /usr/libexec/apache2/mod_jk-1.2.25-httpd-2.2.4.so into server: dlopen(/usr/libexec/apache2/mod_jk-1.2.25-httpd-2.2.4.so, 10): no suitable image found.  Did find:\\n\\t/usr/libexec/apache2/mod_jk-1.2.25-httpd-2.2.4.so: mach-o, but wrong architecture
    2. The alternative is to build "mod_jk". Fortunately it turned out to be a straight forward process because of this blog.
      1. Download latest Connectors source code (version 1.2.27). The file "BUILD.txt" (included in the source bundle) provide clear instructions to build and described below as well.
      2. Configure the build environment as shown:

        ~/workspaces/tomcat-connectors-1.2.27-src/native >./configure --with-apxs=/usr/sbin/apxs

        It shows the output as:

        . . .
        checking for target platform... unix
        no apache given
        no netscape given
        configure: creating ./config.status
        config.status: creating Makefile
        config.status: creating apache-1.3/Makefile
        config.status: creating apache-1.3/Makefile.apxs
        config.status: creating apache-2.0/Makefile
        config.status: creating apache-2.0/Makefile.apxs
        config.status: creating common/Makefile
        config.status: creating common/list.mk
        config.status: creating common/jk_types.h
        config.status: creating jni/Makefile
        config.status: creating common/portable.h
        config.status: executing depfiles commands
      3. Edit Edit "native/apache-2.0/Makefile.apxs" and add "-arch x86_64" as described here. Please note that this string needs to be specified twice.
      4. Invoke "make" and "mod_jk.so" is generated in "native/apache-2.0" directory.
    3. Copy the generated "mod_jk.so" as:

      ~/workspaces/tomcat-connectors-1.2.27-src/native/apache-2.0 >sudo cp mod_jk.so /usr/libexec/apache2/
    4. Load the "mod_jk" module in httpd by editing "/etc/apache2/httpd.conf". Please note another similar file exists in "/etc/httpd/httpd.conf" but ignore that one. Add the following as the last "LoadModule" line:

      LoadModule jk_module     libexec/apache2/mod_jk-1.2.25-httpd-2.2.4.so
    5. Configure "mod_jk" by adding the following lines immediately below the previously "LoadModule" line:

      JkWorkersFile /etc/apache2/worker.properties
      # Where to put jk logs
      JkLogFile /var/log/httpd/mod_jk.log
      # Set the jk log level [debug/error/info]
      JkLogLevel debug
      # Select the log format
      JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
      # JkRequestLogFormat set the request format
      JkRequestLogFormat "%w %V %T"
      # Send all jsp requests to GlassFish
      JkMount /\*.jsp loadbalancer

      The key lines in this configuration are the first and the last one. The first line informs "mod_jk" about the location of "worker.properties" file (explained later). The last line instructs to redirect only JSP requests. This will allow the static content such as images, text files, and media to be served from "httpd" itself.

      Also create the log directory specified in the configuration as:

      sudo mkdir /var/log/httpd
    6. Create a new file "/etc/apache2/worker.properties". Again, this is not in "/etc/httpd" directory. Use the following contents:

      # Define 1 real worker using ajp13
      worker.list=loadbalancer
      # Set properties for instance1
      worker.instance1.type=ajp13
      worker.instance1.host=localhost
      worker.instance1.port=9090
      worker.instance1.lbfactor=50
      worker.instance1.cachesize=10
      worker.instance1.cache_timeout=600
      worker.instance1.socket_keepalive=1
      worker.instance1.socket_timeout=300
      # Set properties for instance2
      worker.instance2.type=ajp13
      worker.instance2.host=localhost
      worker.instance2.port=9091
      worker.instance2.lbfactor=50
      worker.instance2.cachesize=10
      worker.instance2.cache_timeout=600
      worker.instance2.socket_keepalive=1
      worker.instance2.socket_timeout=300
      # Set properties for instance3
      worker.instance3.type=ajp13
      worker.instance3.host=localhost
      worker.instance3.port=9092
      worker.instance3.lbfactor=50
      worker.instance3.cachesize=10
      worker.instance3.cache_timeout=600
      worker.instance3.socket_keepalive=1
      worker.instance3.socket_timeout=300

      worker.loadbalancer.type=lb
      worker.loadbalancer.balance_workers=instance1,instance2,instance3

      Read more about worker.properties format. Essentially this file is used to load-balance a 3-instance cluster and specify configuration values for each instance. Note that the value of "worker.instanceX.port" for instance X is exactly same as specified during instance configuration earlier.
  6. Copy "tomcat-ajp.jar" from the "lib" directory of the latest Tomcat 5.5.xcommons-logging.jar (version 1.1.1), and commons-modeler.jar (version 2.0.1) to GLASSFISH_HOME/lib. This is done as:

    ~/samples/v2/clustering/glassfish/lib >cp ~/tools/apache-tomcat-5.5.27/server/lib/tomcat-ajp.jar .
    ~/samples/v2/clustering/glassfish/lib >cp ~/Downloads/commons-logging-1.1.1/commons-logging-1.1.1.jar .
    ~/samples/v2/clustering/glassfish/lib >cp ~/tools/commons-modeler-2.0.1/commons-modeler-2.0.1.jar .

    You may have to restart cluster in order for these JARs to be loaded by each instance.
  7. An "httpd" instance is already running on port# 80 in my particular instance of Mac OS X. Instead of mangling with that, I decided to change the listening port for the new instance that will be spawn for out front-end. This can be easily done by editing "/etc/apache2/httpd.conf" and looking for lines similar to:

    #Listen 12.34.56.78:80
    Listen 80

    And change "Listen 80" to "Listen 81".
That completes the configuration, phew!

Lets start "httpd" as:

sudo httpd

The "httpd" logs are available in "/private/var/log/apache2". The following message indicates a successful start of the web server:

. . .
[Thu Jan 29 11:14:16 2009] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Jan 29 11:14:16 2009] [warn] No JkShmFile defined in httpd.conf. Using default /usr/logs/jk-run
time-status
[Thu Jan 29 11:14:16 2009] [warn] No JkShmFile defined in httpd.conf. Using default /usr/logs/jk-run
time-status
[Thu Jan 29 11:14:16 2009] [notice] Digest: generating secret for digest authentication ...
[Thu Jan 29 11:14:16 2009] [notice] Digest: done
[Thu Jan 29 11:14:16 2009] [warn] pid file /private/var/run/httpd.pid overwritten -- Unclean shutdow
n of previous Apache run?
[Thu Jan 29 11:14:16 2009] [notice] Apache/2.2.9 (Unix) mod_ssl/2.2.9 OpenSSL/0.9.7l DAV/2 mod_jk/1.
2.27 configured -- resuming normal operations

OK, now show time!

If everything has been configured properly as described above then "http://localhost:81/clusterjsp/HaJsp.jsp" looks like:



Enter session attribute as "aaa" and value of the attribute as "111". After you click on "ADD SESSION DATA" button, the updated page looks like:



The highlighted part shows that the request is served from "instance1" and recently added session data. Lets stop "instance1" and see if the promise of high availability is fulfilled :)

Click on "Instances" tab, select "instance1" and click "Stop". The admin console looks like:



Notice "instance1" is shown as stopped. Clicking on "Reload Page" on "http://localhost:81/clusterjsp/HaJsp.jsp" shows:




Aha!

Even though "instance1" is not runing the session data is still available. And that is possible because of the seamless session failover from primary ("instance1") to the replica partner ("instance2"). The highlighted part indicates that the request is now indeed served by "instance2".

Here are some other useful links to consider:
Please leave suggestions on other TOTD (Tip Of The Day) that you'd like to see. A complete archive of all tips is available here.

Technorati: totd glassfish highavailability clustering loadbalancer mod_jk apache httpd mac leopard
About

profile image
Arun Gupta is a technology enthusiast, a passionate runner, author, and a community guy who works for Oracle Corp.


Java EE 7 Samples

Stay Connected

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today