Sizing the number of DRCP connection brokers on Oracle RAC
By cj on Apr 09, 2009
The Oracle 11g DRCP connection pooling "connection broker" process handles incoming connection requests. It also retains a socket for idle connections, thus enabling faster reconnection. Large web applications with multiple mid-tier Apache servers typically need to increase the database server OS limit on file descriptors so the broker can receive all the connections and work to full capacity.
If you start RAC with the cluster services tool "srvctl", a buglet in RAC 18.104.22.168 & 22.214.171.124 (the latest version at time of writing) resets the OS limit back down on nodes. RAC DBAs might notice an alert file message:
"WARNING:Oracle instance running on a system with low open file descriptor limit. Tune your system to increase this limit to avoid severe performance degradation."
DRCP users on RAC who see connection errors, but don't notice this alert message, have compensated for the throttling by increasing the number of brokers to spread the connection load. This isn't so necessary. Looking at the DRCP scalability benchmark in the PHP Scalability and Availability white paper you can see that only one broker was needed for the 20,000 concurrent connections. And the default configuration "max connections per broker" setting is 40,000.
The recommendation is to apply the simple resource limit script patch given in Oracle bug 7483048 (also see bug 7308467) and keep the number of connection brokers to a minimum.
The number of brokers required can be calculated by dividing the maximum number of open connections by the number of file descriptors per process or other OS limit on processes. The broker's hand-off task is not intensive but having more than one broker can spread processing over multiple CPUs and will reduce the impact if one of the broker processes crashes (planning for the worst case scenario here). You could find that just two or three brokers is all that the biggest sites need.
The RAC bug has been fixed in our next major version of the DB but I don't know yet whether it will be back ported to any future 11.1 patch set. Note the impact of the bug isn't just to DRCP. If you see the alert message above, check out the patch.