Tuesday Jun 16, 2009

TOTD #84: Using Apache + mod_proxy_balancer to load balance Ruby-on-Rails running on GlassFish


TOTD #81 explained how to install/configure nginx for load-balancing/front-ending a cluster of Rails application running on GlassFish Gem. Another popular approach in the Rails community is to use Apache HTTPDmod_proxy_balancer. A user asked the exact details of this setup on the GlassFish Gem Forum. This Tip Of The Day (TOTD) will clearly explain the steps.
  1. Create a simple Rails scaffold and run this application using GlassFish Gem on 3 separate ports as explained in TOTD #81.
  2. Setup and configure HTTPD and mod_proxy_balancer
    1. Setup and install Apache HTTPD as explained here. I believe mod_proxy_balancer and other related modules comes pre-bundled with HTTPD, at least that's what I observed with Mac OS X 10.5.7. Make sure that the "mod_proxy_balancer" module is enabled by verifying the following line is uncommented in "/etc/apache2/httpd.conf":

      LoadModule proxy_balancer_module libexec/apache2/mod_proxy_balancer.so

      Please note another similar file exists in "/etc/httpd/httpd.conf" but ignore that one.
    2. Setup a mod_proxy_balancer cluster by adding the following fragment in "httpd.conf" as:

      <Proxy balancer://glassfishgem>
      BalancerMember http://localhost:3000
      BalancerMember http://localhost:3001
      BalancerMember http://localhost:3002
      </Proxy>

      The port numbers must exactly match with those used in the first step.
    3. Specify the ProxyPass directives to map the cluster to a local path as:

      ProxyPass / balancer://glassfishgem/
      CustomLog /var/log/glassfishgem.log/apache_access_log combined

      The "/" at the end of "balancer://glassfishgem" is very important to ensure that all the files are resolved correctly.
    4. Optionally, the following directive can be added to view the access log:

      CustomLog /var/log/glassfishgem.log/apache_access_log combined

      Make sure to create the directory specified in "CustomLog" directive.
  3. Now the application is accessible at "http://localhost/runlogs". If a new GlassFish instance is started then update the <Proxy> directive and restart your HTTPD as "sudo httpd -k restart". Dynamic update of BalancerMembers can be configured as explained here.
TOTD #81 started the Rails application in root context. You can alternatively start the application in a non-root context as:

~/tools/jruby/rails/runner >../../bin/jruby -S glassfish -e production -c myapp
Starting GlassFish server at: 10.0.177.178:3000 in production environment...
Writing log messages to: /Users/arungupta/tools/jruby-1.3.0/rails/runner/log/production.log.
Press Ctrl+C to stop.
. . .
~/tools/jruby/rails/runner >../../bin/jruby -S glassfish -e production -c myapp -p 3001
Starting GlassFish server at: 10.0.177.178:3001 in production environment...
Writing log messages to: /Users/arungupta/tools/jruby-1.3.0/rails/runner/log/production.log.
Press Ctrl+C to stop.
. . .
~/tools/jruby/rails/runner >../../bin/jruby -S glassfish -e production -c myapp -p 3002
Starting GlassFish server at: 10.0.177.178:3002 in production environment...
Writing log messages to: /Users/arungupta/tools/jruby-1.3.0/rails/runner/log/production.log.
Press Ctrl+C to stop.

and then the ProxyPass directive will change to:

ProxyPass /myapp/ balancer://glassfishgem/myapp/

The changes are highlighted in bold. And the application is now accessible at "http://localhost/myapp/runlogs".

After discussing on Apache HTTP Server forum, the BalancerMember host/port can be printd in the log file using a custom log format. So add the following log format to "/etc/apache2/httpd.conf":

LogFormat "%h %l %u %t \\"%r\\" %>s %b \\"%{Referer}i\\" \\"%{User-agent}i\\" \\"%{BALANCER_WORKER_NAME}e\\"" custom

And change the format from the default "combined" to the newly defined "custom" format as:

CustomLog /var/log/glassfishgem.com/apache_access_log custom

Three subsequent invocations of "http://localhost/runlogs" then prints the following log entries:

::1 - - [17/Jun/2009:10:53:53 -0700] "GET /runlogs HTTP/1.1" 304 - "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.11) Gecko/2009060214 Firefox/3.0.11" "http://localhost:3002"
::1 - - [17/Jun/2009:10:54:04 -0700] "GET /runlogs HTTP/1.1" 200 621 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.11) Gecko/2009060214 Firefox/3.0.11" "http://localhost:3000"
::1 - - [17/Jun/2009:10:54:05 -0700] "GET /runlogs HTTP/1.1" 304 - "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.11) Gecko/2009060214 Firefox/3.0.11" "http://localhost:3001"

As evident from the last fragment of each log line, the load is distributed amongst three GlassFish Gem instances. More details on load balancer algorithm are available here.

Feel free to drop a comment on this blog if you are using GlassFish in production for your Rails applications. Several stories are already available at rubyonrails+glassfish+stories.

Technorati: glassfish rubyonrails apache httpd mod_proxy_balancer loadbalancing clustering

Wednesday Apr 29, 2009

TOTD #81: How to use nginx to load balance a cluster of GlassFish Gem ?

nginx (pronounced as "engine-ex") is an open-source and high-performance HTTP server. It provides the common features such as reverse proxying with caching, load balancing, modular architecture using filters (gzipping, chunked responses, etc), virtual servers, flexible configuration and much more.

nginx is known for it's high performance and low resource consumption. It's a fairly popular front-end HTTP server in the Rails community along with Apache, Lighttpd, and others. This TOTD (Tip Of The Day) will show how to install/configure nginx for load-balancing/front-ending a cluster of Rails application running on GlassFish Gem.
  1. Download, build, and install nginx using the simple script (borrowed from dzone):

    ~/tools > curl -L -O http://sysoev.ru/nginx/nginx-0.6.36.tar.gz
    ~/tools > tar -xzf nginx-0.6.36.tar.gz
    ~/tools > curl -L -O http://downloads.sourceforge.net/pcre/pcre-7.7.tar.gz
    ~/tools > tar -xzf pcre-7.7.tar.gz
    ~/tools/nginx-0.6.36 > ./configure --prefix=/usr/local/nginx --sbin-path=/usr/sbin --with-debug --with-http_ssl_module --with-pcre=../pcre-7.7
    ~/tools/nginx-0.6.36 > make
    ~/tools/nginx-0.6.36 > sudo make install
    ~/tools/nginx-0.6.36 > which nginx
    /usr/sbin/nginx

    OK, nginx is now roaring and can be verified by visiting "http://localhost" as shown below:


  2. Create a simple Rails scaffold as:

    ~/samples/jruby >~/tools/jruby/bin/jruby -S rails runner
    ~/samples/jruby/runner >~/tools/jruby/bin/jruby script/generate scaffold runlog miles:float minutes:integer
    ~/samples/jruby/runner >sed s/'adapter: sqlite3'/'adapter: jdbcsqlite3'/ <config/database.yml >config/database.yml.new
    ~/samples/jruby/runner >mv config/database.yml.new config/database.yml
    ~/samples/jruby/runner >~/tools/jruby/bin/jruby -S rake db:migrate
  3. Run this application using GlassFish Gem on 3 separate ports as:

    ~/samples/jruby/runner >~/tools/jruby/bin/jruby -S glassfish
    Starting GlassFish server at: 192.168.1.145:3000 in development environment...
    Writing log messages to: /Users/arungupta/samples/jruby/runner/log/development.log.
    Press Ctrl+C to stop.

    The default port is 3000. Start the seond one by explicitly specifying the port using "-p" option ..

    ~/samples/jruby/runner >~/tools/jruby/bin/jruby -S glassfish -p 3001
    Starting GlassFish server at: 192.168.1.145:3001 in development environment...
    Writing log messages to: /Users/arungupta/samples/jruby/runner/log/development.log.
    Press Ctrl+C to stop.

    and the last one on 3002 port ...

    ~/samples/jruby/runner >~/tools/jruby/bin/jruby -S glassfish -p 3002
    Starting GlassFish server at: 192.168.1.145:3002 in development environment...
    Writing log messages to: /Users/arungupta/samples/jruby/runner/log/development.log.
    Press Ctrl+C to stop.

    On Solaris and Linux, you can run GlassFish as a daemon as well.
  4. Nginx currently uses a simple round-robin algorithm. Other load balancers such as nginx-upstream-fair (fair proxy) and nginx-ey-balancer (maximum connections) are also available. The built-in algorithm will be used for this blog. Edit "/usr/local/nginx/conf/nginx.conf" to specify an upstream module which provides load balancing:
    1. Create a cluster definition by adding an upstream module (configuration details) right before the "server" module:

      upstream glassfish {
              server 127.0.0.1:3000;
              server 127.0.0.1:3001;
              server 127.0.0.1:3002;
          }

      The cluster specifies a bunch of GlassFish Gem instances running at the backend. Each server can be weighted differently as explained here. The port numbers must exactly match as those specified at the start up. The modified "nginx.conf" looks like:



      The changes are highlighted on lines #35 through #39.
    2. Configure load balancing by specifying this cluster using "proxy_pass" directive as shown below:

      proxy_pass http://glassfish;

      in the "location" module. The updated "nginx.conf" looks like:



      The change is highlighted on line #52.
  5. Restart nginx by using the following commands:

    sudo kill -15 `cat /usr/local/nginx/logs/nginx.pid`
    sudo nginx
Now "http://localhost" shows the default Rails page as shown below:



"http://localhost/runlogs" now serves the page from the deployed Rails application.

Now lets configure logging so that the upstream server IP address and port are printed in the log files. In "nginx.conf", uncomment "log_format" directive and add "$upstream_addr" variable as shown:

    log_format  main  '$remote_addr - [$upstream_addr] $remote_user [$time_local] $request '
                      '"$status" $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

Also change the log format to "main" by uncommenting "access_log logs/access.log main;" line as shown above (default format is "combined"). Accessing "http://localhost/runlogs" shows the following lines in "logs/access.log":

127.0.0.1 - [127.0.0.1:3000] - [29/Apr/2009:15:27:57 -0700] GET /runlogs/ HTTP/1.1 "200" 3689 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "-"
127.0.0.1 - [127.0.0.1:3001] - [29/Apr/2009:15:27:57 -0700] GET /favicon.ico HTTP/1.1 "200" 0 "http://localhost/runlogs/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "-"
127.0.0.1 - [127.0.0.1:3002] - [29/Apr/2009:15:27:57 -0700] GET /stylesheets/scaffold.css?1240977992 HTTP/1.1 "200" 889 "http://localhost/runlogs/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "-"

The browser makes multiple requests (3 in this case) to load resources on a page and they are nicely load-balanced on the cluster. If an instance running on port 3002 is killed, then the access log show the entries like:

127.0.0.1 - [127.0.0.1:3000] - [29/Apr/2009:15:28:53 -0700] GET /runlogs/ HTTP/1.1 "200" 3689 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "-"
127.0.0.1 - [127.0.0.1:3002, 127.0.0.1:3000] - [29/Apr/2009:15:28:53 -0700] GET /favicon.ico HTTP/1.1 "200" 0 "http://localhost/runlogs/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "-"
127.0.0.1 - [127.0.0.1:3001] - [29/Apr/2009:15:28:53 -0700] GET /stylesheets/scaffold.css?1240977992 HTTP/1.1 "200" 889 "http://localhost/runlogs/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "-"

The second log line shows that server running on port 3002 did not respond and so it automatically fall back to 3000, this is nice!

But this is inefficient because a back-end trip is made even for serving a static file ("/favicon.ico" and "/stylesheets/scaffold.css?1240977992"). This can be easily solved by enabling Rails page caching as described here and here.

More options about logging are described in NginxHttpLogModule and upstream module variables are defined in NginxHttpUpstreamModule.

Here are some nginx resources:
Are you using nginx to front-end your GlassFish cluster ?

Apache + JRuby + Rails + GlassFish = Easy Deployment! shows similar steps if you want to front-end your Rails application running using JRuby/GlassFish with Apache.

Hear all about it in Develop with Pleasure, Deploy with Fun: GlassFish and NetBeans for a Better Rails Experience session at Rails Conf next week.

Please leave suggestions on other TOTD (Tip Of The Day) that you'd like to see. A complete archive of all tips is available here.

Technorati: rubyonrails glassfish v3 gem jruby nginx loadbalancing clustering

Wednesday Feb 11, 2009

TOTD #69: GlassFish High Availability/Clustering using Sun Web Server + Load Balancer Plugin on Windows Vista


TOTD #67 shows how to configure GlassFish High Availability using Apache httpd + mod_jk on Mac OS X. Even though that's a standard and supported configuration, there are several advantages for replacing Apache httpd with Sun Web Server and mod_jk with Load Balancer plugin that comes with GlassFish.

This Tip Of The Day (TOTD) shows how to configure Clustering and Load Balancing using GlassFish v2.1, Sun Web Server, Load Balancer plugin on Windows Vista. This blog is using JDK 6 U7, GlassFish v2.1 (cluster profile), Sun Web Server 7 U4, and Load Balancer plug-in with Sun GlassFish Enterprise Server 2.1 Enterprise Profile (with HADB link).

Lets get started!
  1. Install the required software
    1. Download JDK (if not already installed).
    2. Download and Install GlassFish v2.1. Make sure to configure using "ant -f setup-cluster.xml". This will ensure that the created domain is capable of creating clusters and can perform in-memory session replication for applications deployed on the cluster.
    3. Download and Install Sun Web Server. The process is very simple by unzipping the downloaded bundle, clicking on "setup.exe" and taking all the defaults.
    4. Download GlassFish Enterprise Profile for Load Balancer plugin bits. Start the install by clicking on the downloaded file and select the options as shown below:


    5. Copy the following "loadbalancer.xml" in "https-<host>" (replace <host> with the host name of your machine) directory of Sun Web Server installation directory:

      <?xml version="1.0" encoding="UTF-8"?>
      <!DOCTYPE loadbalancer PUBLIC "-//Sun Microsystems Inc.//DTD Sun Java
      System Application Server 9.1//EN"
      "file:///C:/Sun/WebServer7/https-LH-KRKZDW6CJE1V/config/sun-loadbalancer_1_2.dtd
      ">

      <loadbalancer>
       <cluster name="cluster1" policy="round-robin" policy-module="">
        <instance name="instance1" enabled="true"
      disable-timeout-in-minutes="60" listeners="http://localhost:38080" weight="100"/>
        <instance name="instance2" enabled="true"
      disable-timeout-in-minutes="60" listeners="http://localhost:38081" weight="100"/>
        <web-module context-root="/clusterjsp"
      disable-timeout-in-minutes="30" enabled="true" error-url=""/>
        <health-checker interval-in-seconds="7" timeout-in-seconds="5" url="/"/>
       </cluster>
       <property name="response-timeout-in-seconds" value="120"/>
       <property name="reload-poll-interval-in-seconds" value="7"/>
       <property name="https-routing" value="false"/>
       <property name="require-monitor-data" value="false"/>
       <property name="active-healthcheck-enabled" value="false"/>
       <property name="number-healthcheck-retries" value="3"/>
       <property name="rewrite-location" value="true"/>
      </loadbalancer>

      The parameters to be changed are highlighted in bold and explained below:
      1. Sun Web Server installation directory
      2. HTTP port of instances created in the cluster. The ports specified are the default ones and can be found by clicking on the instance as shown below:

      3. Context root of the application that will be deployed in the cluster. The Domain Administration Server (DAS) can be configured to populate this file whenever any application is deployed to the cluster.
  2. Create the cluster as explained in TOTD #67. The admin console shows the following screenshot after the cluster is created and all instances are created/started:

     

    and the following for 2 instances:



  3. Deploy "clusterjsp" as explained in TOTD #67. The admin console shows the following screenshot after "clusterjsp" is deployed:


  4. Start Sun Web Server using "startserv.bat" in "https-<host>" directory.
This concludes the installation and configuration steps, now show time!

Accessing "http://localhost/clusterjsp" shows:



The Sun Web Server is running on port 80 and uses "loadbalancer.xml" to serve the request from the configured instances in <loadbalancer> fragment. This particular page is served by "instance1" as indicated in the image. Lets add session data with property name "aaa" and value "111". The value is shown as:



The instance serving the data, "instance1" in this case, and the session data are highlighted.

Now lets stop "instance1" using the admin console and it looks like:



Click on "RELOAD PAGE" and it looks like:



Exactly same session data is served, this time by "instance2".

The sequence above proves that the session data created by the user is preserved even if the instance serving the data goes down. This is possible because of GlassFish High Availability. The session data is served by the "replica partner" where its already copied using in-memory session replication.

The following articles are also useful:
Please leave suggestions on other TOTD (Tip Of The Day) that you'd like to see. A complete archive of all tips is available here.

Technorati: totd glassfish highavailability clustering loadbalancing lbplugin sunwebserver windows vista
About

profile image
Arun Gupta is a technology enthusiast, a passionate runner, author, and a community guy who works for Oracle Corp.


Java EE 7 Samples

Stay Connected

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today