TOTD #81: How to use nginx to load balance a cluster of GlassFish Gem ?

nginx (pronounced as "engine-ex") is an open-source and high-performance HTTP server. It provides the common features such as reverse proxying with caching, load balancing, modular architecture using filters (gzipping, chunked responses, etc), virtual servers, flexible configuration and much more.

nginx is known for it's high performance and low resource consumption. It's a fairly popular front-end HTTP server in the Rails community along with Apache, Lighttpd, and others. This TOTD (Tip Of The Day) will show how to install/configure nginx for load-balancing/front-ending a cluster of Rails application running on GlassFish Gem.
  1. Download, build, and install nginx using the simple script (borrowed from dzone):

    ~/tools > curl -L -O http://sysoev.ru/nginx/nginx-0.6.36.tar.gz
    ~/tools > tar -xzf nginx-0.6.36.tar.gz
    ~/tools > curl -L -O http://downloads.sourceforge.net/pcre/pcre-7.7.tar.gz
    ~/tools > tar -xzf pcre-7.7.tar.gz
    ~/tools/nginx-0.6.36 > ./configure --prefix=/usr/local/nginx --sbin-path=/usr/sbin --with-debug --with-http_ssl_module --with-pcre=../pcre-7.7
    ~/tools/nginx-0.6.36 > make
    ~/tools/nginx-0.6.36 > sudo make install
    ~/tools/nginx-0.6.36 > which nginx
    /usr/sbin/nginx

    OK, nginx is now roaring and can be verified by visiting "http://localhost" as shown below:


  2. Create a simple Rails scaffold as:

    ~/samples/jruby >~/tools/jruby/bin/jruby -S rails runner
    ~/samples/jruby/runner >~/tools/jruby/bin/jruby script/generate scaffold runlog miles:float minutes:integer
    ~/samples/jruby/runner >sed s/'adapter: sqlite3'/'adapter: jdbcsqlite3'/ <config/database.yml >config/database.yml.new
    ~/samples/jruby/runner >mv config/database.yml.new config/database.yml
    ~/samples/jruby/runner >~/tools/jruby/bin/jruby -S rake db:migrate
  3. Run this application using GlassFish Gem on 3 separate ports as:

    ~/samples/jruby/runner >~/tools/jruby/bin/jruby -S glassfish
    Starting GlassFish server at: 192.168.1.145:3000 in development environment...
    Writing log messages to: /Users/arungupta/samples/jruby/runner/log/development.log.
    Press Ctrl+C to stop.

    The default port is 3000. Start the seond one by explicitly specifying the port using "-p" option ..

    ~/samples/jruby/runner >~/tools/jruby/bin/jruby -S glassfish -p 3001
    Starting GlassFish server at: 192.168.1.145:3001 in development environment...
    Writing log messages to: /Users/arungupta/samples/jruby/runner/log/development.log.
    Press Ctrl+C to stop.

    and the last one on 3002 port ...

    ~/samples/jruby/runner >~/tools/jruby/bin/jruby -S glassfish -p 3002
    Starting GlassFish server at: 192.168.1.145:3002 in development environment...
    Writing log messages to: /Users/arungupta/samples/jruby/runner/log/development.log.
    Press Ctrl+C to stop.

    On Solaris and Linux, you can run GlassFish as a daemon as well.
  4. Nginx currently uses a simple round-robin algorithm. Other load balancers such as nginx-upstream-fair (fair proxy) and nginx-ey-balancer (maximum connections) are also available. The built-in algorithm will be used for this blog. Edit "/usr/local/nginx/conf/nginx.conf" to specify an upstream module which provides load balancing:
    1. Create a cluster definition by adding an upstream module (configuration details) right before the "server" module:

      upstream glassfish {
              server 127.0.0.1:3000;
              server 127.0.0.1:3001;
              server 127.0.0.1:3002;
          }

      The cluster specifies a bunch of GlassFish Gem instances running at the backend. Each server can be weighted differently as explained here. The port numbers must exactly match as those specified at the start up. The modified "nginx.conf" looks like:



      The changes are highlighted on lines #35 through #39.
    2. Configure load balancing by specifying this cluster using "proxy_pass" directive as shown below:

      proxy_pass http://glassfish;

      in the "location" module. The updated "nginx.conf" looks like:



      The change is highlighted on line #52.
  5. Restart nginx by using the following commands:

    sudo kill -15 `cat /usr/local/nginx/logs/nginx.pid`
    sudo nginx
Now "http://localhost" shows the default Rails page as shown below:



"http://localhost/runlogs" now serves the page from the deployed Rails application.

Now lets configure logging so that the upstream server IP address and port are printed in the log files. In "nginx.conf", uncomment "log_format" directive and add "$upstream_addr" variable as shown:

    log_format  main  '$remote_addr - [$upstream_addr] $remote_user [$time_local] $request '
                      '"$status" $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

Also change the log format to "main" by uncommenting "access_log logs/access.log main;" line as shown above (default format is "combined"). Accessing "http://localhost/runlogs" shows the following lines in "logs/access.log":

127.0.0.1 - [127.0.0.1:3000] - [29/Apr/2009:15:27:57 -0700] GET /runlogs/ HTTP/1.1 "200" 3689 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "-"
127.0.0.1 - [127.0.0.1:3001] - [29/Apr/2009:15:27:57 -0700] GET /favicon.ico HTTP/1.1 "200" 0 "http://localhost/runlogs/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "-"
127.0.0.1 - [127.0.0.1:3002] - [29/Apr/2009:15:27:57 -0700] GET /stylesheets/scaffold.css?1240977992 HTTP/1.1 "200" 889 "http://localhost/runlogs/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "-"

The browser makes multiple requests (3 in this case) to load resources on a page and they are nicely load-balanced on the cluster. If an instance running on port 3002 is killed, then the access log show the entries like:

127.0.0.1 - [127.0.0.1:3000] - [29/Apr/2009:15:28:53 -0700] GET /runlogs/ HTTP/1.1 "200" 3689 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "-"
127.0.0.1 - [127.0.0.1:3002, 127.0.0.1:3000] - [29/Apr/2009:15:28:53 -0700] GET /favicon.ico HTTP/1.1 "200" 0 "http://localhost/runlogs/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "-"
127.0.0.1 - [127.0.0.1:3001] - [29/Apr/2009:15:28:53 -0700] GET /stylesheets/scaffold.css?1240977992 HTTP/1.1 "200" 889 "http://localhost/runlogs/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "-"

The second log line shows that server running on port 3002 did not respond and so it automatically fall back to 3000, this is nice!

But this is inefficient because a back-end trip is made even for serving a static file ("/favicon.ico" and "/stylesheets/scaffold.css?1240977992"). This can be easily solved by enabling Rails page caching as described here and here.

More options about logging are described in NginxHttpLogModule and upstream module variables are defined in NginxHttpUpstreamModule.

Here are some nginx resources:
Are you using nginx to front-end your GlassFish cluster ?

Apache + JRuby + Rails + GlassFish = Easy Deployment! shows similar steps if you want to front-end your Rails application running using JRuby/GlassFish with Apache.

Hear all about it in Develop with Pleasure, Deploy with Fun: GlassFish and NetBeans for a Better Rails Experience session at Rails Conf next week.

Please leave suggestions on other TOTD (Tip Of The Day) that you'd like to see. A complete archive of all tips is available here.

Technorati: rubyonrails glassfish v3 gem jruby nginx loadbalancing clustering
Comments:

If you are running on OpenSolaris there is no need to download or manually build nginx, it is available in the Web Stack project repository (http://pkg.opensolaris.org/webstack) and can conveniently be installed via pkg. The package includes smf integration built-in.

More info here:
http://blogs.sun.com/jyrivirkki/entry/nginx_package_updates

(The manual import step is no longer needed, that pkg bug has since been fixed.)

Posted by Jyri Virkki on April 30, 2009 at 02:47 AM PDT #

Thanks Jyri, that's good to know!

Posted by Arun Gupta on May 03, 2009 at 01:44 PM PDT #

[Trackback] This is a follow up post from David's keynote. Attended Women in Rails panel discussion. The panel, Sarah Mei, Lori Olson, and Desi McAdam (from L to R), had a very interesting discussion around the genuine problems and possible solutions...

Posted by Arun Gupta's Blog on May 05, 2009 at 10:40 PM PDT #

[Trackback] TOTD #81 explained how to install/configure nginx for load-balancing/front-ending a cluster of Rails application running on GlassFish Gem. Another popular approach in the Rails community is to use Apache HTTPD +&nbsp;mod_proxy_balancer. A user asked t...

Posted by Arun Gupta's Blog on June 17, 2009 at 04:06 AM PDT #

1.I already setup Glassfish Load-balancer plug-in with Sun Web Server 7.
My problem is that if the concurrent connection over 256 threads then it failed to process those threads.
Is that a normal case? How to configure the Web server 7.

2.Another problem is that my Glassfish server handle a 4 sec. process in my SOAP application. It have to wait some operation last about 4 sec. Then I tried to configure the Glassfish in my 4-core CPU with 4GB RAM in CentOS 5.3. The performance is around only 500 TPS. Is this a normal case?

3.I need the Load-Balancer to increase the total performance but it only response 256 TPS less than original one Glassfish performance.
Can anyone help me to slove such problem?

Thanks!!

Eric

Posted by Eric on July 01, 2009 at 08:52 PM PDT #

Eric,

Please post your question to users@glassfish.dev.java.net for a wider audience.

Posted by Arun Gupta on July 02, 2009 at 03:47 AM PDT #

[Trackback] The GlassFish High Availability allows to setup a cluster of GlassFish instances and achieve highly scalable architecture using in-memory session state replication. This cluster can be very easily created and tested using the "clusterjsp" sample bundl...

Posted by Arun Gupta's Blog on August 11, 2009 at 11:11 PM PDT #

[Trackback] The GlassFish High Availability allows to setup a cluster of GlassFish instances and achieve highly scalable architecture using in-memory session state replication. This cluster can be very easily created and tested using the "clusterjsp" sample bundl...

Posted by Arun Gupta's Blog on August 12, 2009 at 02:15 AM PDT #

Post a Comment:
Comments are closed for this entry.
About

profile image
Arun Gupta is a technology enthusiast, a passionate runner, author, and a community guy who works for Oracle Corp.


Java EE 7 Samples

Stay Connected

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today