HTTP3360: connection limit (1) exceeded.
By Janice on Jul 06, 2009
A few weeks back, I helped a customer with an interesting problem. Their symptoms was something that I've not seen with our web server before. They were using Sun Web Server 7.0u3 and seeing a random problem where their web page was not able to load image files properly. They were also seeing the error in their errors logs:
[01/Jul/2009:13:57:58] failure (15481): HTTP3360: connection limit (1) exceeded.
What does the HTTP3360 error mean?
The connection limit is the same thing as the maximum number of threads that are set in the Connection Queue for incoming HTTP requests to be queued waiting for servicing. This is configurable via the thread-pool subelement queue-size via the server.xml configuration file. See our documentation for some additional info:
By default in Web Server 6.1, the connection queue size is set to 4096 via the magnus.conf directive ConnQueueSize. In Web Server 7.0 however, there is not a set default value for queue-size.
Web Server Connection Auto Detect Enhancements since 7.0u2
Since Web Server 7.0u2, the web server has been changed to automatically size the connection queue and other variables based on the number of file descriptors that are set on the OS. The other variables affected by the auto detect feature is the keep-alive maximum number of connections and the maximum files for the file cache. The purpose of this enhancement is to allow the server to maximize and properly partition the file descriptors among the web server subsystems for maximum performance. Thus, if an administrator did not explicitly configure these parameters in their server.xml, the server default is to auto detect the tuning for these values.
To find out what the server has selected for your configuration defaults, turn on the performance monitoring tool called perfdump to review your auto detected server settings for queue-size, max-connections, and file cache subelement max-open-files.
Conclusion - Auto Detect Algorithm Broken
Upon investigating the server.xml configuration file, the queue-size had not been set to a customized setting by the administrator and the perfdump output was showing only one thread as the connection queue limit. The only customization was made to the max-threads, which was set to a very high number of 4096. The file descriptor settings from ulimit showed nofiles as 8192. With these settings, you can reproduce the auto detect bug where the queue-size default to the value 1. I have filed a bug for this problem, CR 6855513.
With only 1 thread in the connection queue, the random image files were not being displayed because those requests were getting dropped by the web server. Dropped connections are a symptom of the connection queue being set too low.