Too Many Open Files on Solaris Web Servers

Many Apache HTTP Server and Sun ONE Web Server administrators have seen "Too many open files" or "PR_PROC_DESC_TABLE_FULL_ERROR: file descriptor table full" errors on Solaris. These error messages indicate that some limit on the number of file descriptors (i.e. open files and network connections) has been reached.

Such messages are typically the result of reaching one of the following file descriptor limits:

1. Per-process file descriptor limit

The first limit is the easiest to avoid. In Solaris 8, there is a default limit of 1024 file descriptors per process. To increase the limit to 65536, the following can be added to the /etc/system file:

set rlim_fd_max=65536

In Solaris 9, the default limit was raised to 65536. While this is large enough for most web servers, you may wish to raise it even higher if you're using a connection-hungry web monster like Sun's T2000 server.

2. Limit on the maximum file descriptor passed to select(3C).

The second limit doesn't affect Apache HTTP Server or Sun ONE Web Server themselves. Instead of using the outdated select function to manage connections, these servers use more modern interfaces like the poll(2) system call.

However, some plugins such as PHP still use select and are limited to 1024 file descriptors by default. To raise the limit, include -DFD_SETSIZE=65536 in your CFLAGS when you compile PHP.

3. Limit on the maximum file descriptor used in stdio streams

The third and most restrictive limit is enshrined in the 32-bit Solaris ABI. The stable Solaris ABI (Application Binary Interface) is what allows programs compiled 10 years ago on SunOS 5.5 to run unchanged today on Solaris 10. Unfortunately, the 32-bit Solaris ABI restricts stdio streams (that is, the streams created by fopen(3C), et al.) to file descriptors below 256. The limit cannot be changed because increasing the limit for new programs and libraries would break binary compatibility with old programs and libraries.

This limit doesn't affect Apache HTTP Server or Sun ONE Web Server themselves because neither server uses stdio streams. However, poorly designed 3rd party modules and plugins are often ignornant of the limit, resulting in "Too many open files" errors under load. For example, it is this limit that typically causes the "failed to create/open stream: Too many open files" error in PHP versions prior to 4.3.3.

While the limit cannot be changed, there are some possible work arounds:

a) Use a 64-bit web server like Sun ONE Web Server 6.1 SP5 (64-Bit). Unlike the 32-bit Solaris ABI, the 64-bit Solaris ABI allows programs to open more than 2 billion streams. Unfortunately, existing 32-bit plugins won't work in a 64-bit web server.

b) Use only file descriptors above 255 for non-stdio files and connections, leaving the lower 256 file descriptors available for stdio streams.

libstdiolowfd.so

I've created an interposer library, libstdiolowfd.so, that implements option b. libstdiolowfd.so sits between a web server and the Solaris stdio library functions and ensures that plugins are able to make maximum use of the 256 available stdio streams.

To use libstdiolowfd.so, download it (Solaris SPARC) and copy it to some directory, e.g. /opt/stdiolowfd, then instruct ld.so.1(1) to preload it before starting your web server:

$ set LD_PRELOAD_32=/opt/stdiolowfd/libstdiolowfd.so
$ export LD_PRELOAD_32
$ /opt/SUNWwbsvr/https-mbelshe.red.iplanet.com/start

(You may wish to export the LD_PRELOAD_32 environment variable in your web server's start script.)

Because libstdiolowfd.so does not actually remove the limit on open streams, it is important to limit the number of streams a plugin will attempt to open. In iPlanet Web Server, Sun ONE Web Server, and Sun Java System Web Server, that means ensuring the RqThrottle directive in magnus.conf is set below 256. The default value of 128 works well.

I've placed the source code for stdiolowfd.c online.

Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

elving

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today