Using Solaris and SPARC Networking and Virtualization

  • Sun
    January 28, 2008

Breaking the stdio(3C) 256 file descriptor barrier in Solaris' stdio

In the mid 1990s I was trying to show a customer that Solaris can host 10,000 web sites on a single system. This was in the days of 400MHz processors and 1GB of memory cost more than a luxury automobile. One thing I tried to was to use the Apache web server and virtual hosting. However, since the customer required separate logs for each web site, each virtual host needed at least one log file and thus a file descriptor. Apache uses fopen(3C) for these files. So I had to use on hundred web servers each hosting 100 web sites.

Solaris has historically allowed only 256 stdio streams to be open, where the file descriptors are below 256. So applications can quickly run out of file descriptors when doing lots of fopen() calls. For 32-bit applications, it has not been possible to increase this limit, as it could cause binary compatibility issues for older applications (compatibility going back as far back as those compiled on SunOS 4.x). The dup(2) system call has been used to move other file descriptors above 256 to free up slots for fopen. But the application is still limited to a maximum of 256 stdio streams!

With the release of Solaris 10 8/07 (often referred to as update 4), there is a new interface to extend the FILE facility. Programming details are in the man page enable_extended_FILE_stdio(3C). And if you don't want to make any code changes, extendedFILE(5) describes how to do this for existing applications and binaries.

I am working with a customer who needs to host over 1,400 web sites. We are using portions of the coolstack, as well as customized versions of Apache and PHP. With virtual hosting, the setup quickly hit the 256 stdio file limit!

With a small change to apachectl, it is now possible to host all 1,400+ web sites within a single instance of Apache. I added the following to the configuration section of apachectl:

ulimit -n 3000
LD_PRELOAD_32=/usr/lib/extendedFILE.so.1 ; export LD_PRELOAD_32

The ulimit -n 3000 increases the number of file descriptors a process can have open to 3000, up from the default of 256. Since apachectl is run as root, or with sufficient privileges using Role Bases Access Control, this is permitted.

The LD_PRELOAD_32 setting allows me to have the library provide special versions of library functions or system calls. In this case, it does special things when fopen is called, and automatically uses dup(2) to free up the lower 256 file descriptors.

The enable_extended_FILE_stdio(3C) man pages lists some of the requirements for an application to work well with this interposition library, such as not doing direct access into the fields of the FILE structure. Since Apache is using stdio for log files, it is unlikely that Apache is accessing the structures directly.

Testing with the customer's configuration has Apache serving up all 1,400 web sites using a single instance of the httpd server! Cool, success at last!

Join the discussion

Comments ( 2 )
  • cvr Monday, January 28, 2008

    Wow, that's pretty cool!

  • patrick giagnocavo Monday, January 28, 2008

    That is neat stuff, however I feel that I should point out another way to do it is using the "logsplit" utility and piping all the logging output to that program, which then splits up the combined log into individual logfiles directly.

Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.