Monday Oct 05, 2009

A Creative Way Of Using The Rewrite NameTrans SAF in WS7.0

When creating a reverse proxy mapping from a pattern like "/abc" to a file resource like "http://remoteserver/abc.jsp" on the remote server, the map function creates an undesirable effect of adding a slash "/", to the end of the URL that is being rewritten.  While this is per design,  and adding the slash "/" is intentional behavior of the reverse proxy functionality, this can be problematic when the mapping is to a file resource rather than to a file path.

The first problem with this is that the path gets rewritten to "http://remoteserver/abc.jsp/", causing a "404 Not Found" response from the remote server.  Another problematic effect of this is if the request to "/abc" was for a POST request.  When the request is for a POST to "/abc", the path gets rewritten to "http://remoteserver/abc.jsp/" via a "302 Redirect" response from the map function, resulting in the POST entity body being dropped from the request.

Web Server 7.0 is highly configurable and contains configurable features that can help remedy a problem like this in our reverse proxy functionality.  Here is a quick and easy way to do it - add the following configuration to the default object in the obj.conf before your general map NameTrans line to your reverse-proxy object :

<If uri="/abc">
NameTrans fn="rewrite" path="/abc/"
NameTrans fn="map" from="/abc" to="http://remoteserver/abc.jsp"
<If defined $vars{'escaped'} and $vars{'escaped'} =~'(.\*)/$'>
Route fn="set-variable" $vars{'escaped'}="$1"

By utilizing the above configuration, the rewrite NameTrans SAF will append a slash "/" to the end of the request path so to avoid the undesired behavior of the map NameTrans SAF.  The usage of the set-variable Route SAF is to remove the slash "/" for the escaped variable.

Documentation for the rewrite SAF.

Tuesday Sep 29, 2009

WS7.0 Admin Server Does Not Detect server is up and running

In support, we often times run into a problem that is reported by multiple customers.  Here is one problem that is generating a lot of support calls this last few months.

Bug CR 6801628 - \*Synopsis\* Adminserver doesn't detect the server is up and running.


The problem affects Sun Web Server 7.0 admin servers on T5240 machines.  When trying to start the web server via the admin gui or CLI, you may see the following error:

ADMIN3584: Error while starting the server. Please check the server logs.
However, when you do a "ps -ef" and make a request to your server instance, you will see that the server actually started.


Two possible workarounds exist,
1. On Solaris 10, one can enable the extended FILE facility,

  •       stop the admin server

  •       add the following to the admin-server/bin/startserv:

  •           ulimit -n 65536
              export LD_PRELOAD

  •       start the admin server

2.  For the admin server configuration, tune keep-alive's max-connections value < 100 in server.xml. For example, in your server.xml,


This bug is targeted to be fixed in WS7.0u7 which is due out sometime in November.

Tuesday Aug 11, 2009

wshang 3.1 has been released on BigAdmin

A quick update to let folks know that there has been an update to the wshang data gathering script and it is now available for download on BigAdmin!  This script can be downloaded here:

Sun GDD - Web and Web Proxy Server - wshang version 3.1

Here are the latest changes made in wshang 3.1:
1. fix date error for file output
2. fix pid selector problems
3. fix to warn user about removing data directory
4. RFE: provide interactive user input for collecting core file, pkgapp data, and number of seconds between proc tool data gather.
5. integrate pkgapp v3.2


Monday Jul 06, 2009

HTTP3360: connection limit (1) exceeded.

A few weeks back, I helped a customer with an interesting problem.  Their symptoms was something that I've not seen with our web server before.  They were using Sun Web Server 7.0u3 and seeing a random problem where their web page was not able to load image files properly.  They were also seeing the error in their errors logs:

 [01/Jul/2009:13:57:58] failure (15481): HTTP3360: connection limit (1) exceeded.

What does the HTTP3360 error mean?

The connection limit is the same thing as the maximum number of threads that are set in the Connection Queue for incoming HTTP requests to be queued waiting for servicing.  This is configurable via the thread-pool subelement queue-size via the server.xml configuration file.  See our documentation for some additional info:

Administrator's Configuration File Reference
Performance Tuning Guide

By default in Web Server 6.1, the connection queue size is set to 4096 via the magnus.conf directive ConnQueueSize.  In Web Server 7.0 however, there is not a set default value for queue-size.

Web Server Connection Auto Detect Enhancements since 7.0u2

Since Web Server 7.0u2, the web server has been changed to automatically size the connection queue and other variables based on the number of file descriptors that are set on the OS.   The other variables affected by the auto detect feature is the keep-alive maximum number of connections and the maximum files for the file cache.  The purpose of this enhancement is to allow the server to maximize and properly partition the file descriptors among the web server subsystems for maximum performance.  Thus, if an administrator did not explicitly configure these parameters in their server.xml, the server default is to auto detect the tuning for these values.

 To find out what the server has selected for your configuration defaults, turn on the performance monitoring tool called perfdump to review your auto detected server settings for queue-size, max-connections, and file cache subelement max-open-files.

Conclusion - Auto Detect Algorithm Broken

 Upon investigating the server.xml configuration file, the queue-size had not been set to a customized setting by the administrator and the perfdump output was showing only one thread as the connection queue limit.  The only customization was made to the max-threads, which was set to a very high number of 4096.  The file descriptor settings from ulimit showed nofiles as 8192.  With these settings, you can reproduce the auto detect bug where the queue-size default to the value 1.  I have filed a bug for this problem, CR 6855513.

With only 1 thread in the connection queue, the random image files were not being displayed because those requests were getting dropped by the web server.  Dropped connections are a symptom of the connection queue being set too low.

Wednesday Jun 24, 2009

The Purpose Of My WebLog

I have been encouraged to write a weblog to provide tips and tricks for using the Sun Java System Web Server.  Whenever I come across an interesting question that I feel would be useful for our customers, I will write about it here.

I also support the Sun GlassFish Web Stack (Sun's LAMP/SAMP solution) and will blog about any tips and tricks I find using Apache or Lighttpd.


Quick tips and tricks, problems and resolutions, questions and answer, to help you with your Sun Java System Web Server.


« April 2014