Apache exception hooks in GlassFish Web Stack 1.5
By Jeff Trawick on Jul 31, 2009
As I mentioned in a previous article, Apache exception hooks are now supported in Web Stack's delivery of Apache 2.2, via a change to Apache's configure options. This enables additional diagnostic capabilities.
Consider the case of a child process crash:
In order to debug this, we often need to dig into the core file and extract information about the request, the client connection, and so forth. Since crashes usually occur repeatedly until solved, we would like to know if it is the same request, or at least the same client, that triggers multiple crashes. The access log entry isn't written if a crash occurs during request
processing, so we must examine every core file. That's extra work, and it also means we have to save core files long enough to be able to examine them. (So checking more core files for similar triggers is often skipped.)
One solution for this is mod_log_forensic, which can be used to write information about each request to a special log file early in request processing before a crash is likely to occur. A unique id for the request is written to the forensic log and to the regular access log, so after a crash occurs a script is able to determine which request(s) had no access log entries (i.e., crashed before completing). Unfortunately, in order for this to work a separate log of information for each request has to be maintained, the existing access log format has to be modified to include the unique id, and the failure information isn't automatically written to the error log but instead must be determined by examining the log files with a script.
mod_whatkilledus takes a different approach to this problem. It saves the necessary information in memory in case a crash occurs but only logs the information in the event of a crash. By implementing an Apache exception hook it gains control at the point of the crash in order to log the information. When it returns from the hook, crash processing continues (writing a core file if configured, cleaning up the process, etc.).
Here are the commands to download and install mod_whatkilledus, create a tiny configuration file to enable exception hooks and load the module, then restart Apache. I'm using a non-root install of the IPS/pkg(5) delivery of Web Stack 1.5, described previously.
Solaris 10: You need Sun Studio 12 installed to compile it with Web Stack's apxs. Be sure to update PATH as necessary so that Sun Studio's cc is found first. Your Sun Studio installation may be in a different directory from the example.
Red Hat Enterprise Linux: You need gcc installed, and you don't have to change your PATH.
With mod_whatkilledus active, we'll see much more information in the error log if the crash occurs again. The information logged includes the client IP address, the request headers, the actual request URI, and the addresses of key data structures, which would be useful if we need to peruse the core file. pstack still shows us the original crash. (Note: The core file is in a different directory for this run because I added
"CoreDumpDirectory /tmp" to my Apache configuration.)
I think that's a lot better than simply
child pid nnnn exit signal Segmentation fault!