Monday Dec 07, 2009

Load Balancing for Glassfish V2 Deployments and BIG-IP Systems: Addendum

I was recently at a customer who was having issues configuring BIG-IP to load balance to Glassfish (v2.1.1) cluster instances.   They had been following Prashanth's blog directions on how to configure this setup.   The load-balancing piece worked fine; However, BIG-IP would not recognize a Glassfish instance as running in the monitor.  The monitor is extremely simple, it just does a GET and if it gets a valid response then Glassfish is running.   The send string is typically set as:

GET /index.html HTTP/1.0

Glassfish showed no requests in the access logs.  If we used a browser to request index.html it would return correctly.  Oddly, we replaced Glassfish with Tomcat running on the same port.   The monitor send string was unchanged.   BIG-IP immediately recognized the instance as running.    The next step was to watch the traffic.   An easy way to do this was to use tcpmon.   tcpmon is a simple utility that watches TCP traffic.    With tcpmon in between the browser and Glassfish/Tomcat we were able to watch what was going on.   Requests were coming in fine, but only tomcat would respond.   Step three was to really bump up the logging level.   We added this property to <domain.xml>:

-Dcom.sun.enterprise.web.connector.grizzly.enableSnoop=true

This showed that a connection was being established to Glassfish (snip from the logs with IPs changed to protect the innocent)

[#|2009-11-20T11:09:25.515-0600|INFO|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=15;_ThreadName=SelectorThread-7780;|Handling OP_ACCEPT on SocketChannel java.nio.channels.SocketChannel[connected local=/10.99.101.29:7780 remote=/1.1.1.1:50627]|#]
[#|2009-11-20T11:09:25.518-0600|INFO|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=15;_ThreadName=SelectorThread-7780;|Handling OP_READ on SocketChannel java.nio.channels.SocketChannel[connected local=/10.99.101.29:7780 remote=/1.1.1.1:50627]|#]
[#|2009-11-20T11:09:25.529-0600|FINEST|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=21;_ThreadName=httpSSLWorkerThread-7780-0;ClassName=com.sun.enterprise.web.connector.grizzly.DefaultReadTask;MethodName=finishConnection;_RequestID=f902a22c-7228-4c91-8ef4-6ddd686d044a;|finishConnection|#]

My next step was to simplify.   Instead of requesting index.html from either a browser or BIG-IP, we switched to wget.   The http headers are much less verbose for wget.   Matching the BIG-IP send string to what wget sends still had the same effect.   That is when I started thinking back to the golden days of telnet (OK, telnet is still around and you can do this with SSH as well).   If you telnet to an http server port you can type in the GET command to request a page.   The gotcha is that after you finish typing the command you have to hit two carriage returns.   So, I changed the BIG-IP request string to:

GET /index.html HTTP/1.0\\n\\n

BIG-IP showed Glassfish as a running instance, confetti fell from the roof, and there was much rejoicing!

About

bounds

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Links
Blogroll

No bookmarks in folder