DHCP server tour
By Dave Miner on Jun 13, 2005
With the launch of OpenSolaris, one thing that comes along for the ride is our DHCP server. This is somewhat significant, in that the only other widely-used implementation of a DHCP server that's open-source is the one provided by ISC. We'll leave comparisons between the two for another day, as the point today is to take a look at how the Solaris server is implemented.
First, some history, because some people probably wonder why we don't just ship ISC's server like all the Linux distributions. Development of the Solaris DHCP server started out about 12 years ago, as part of a PC-NFS spinoff product which was ultimately marketed as SolarNet PC-Admin. The idea there was to leverage PC-NFS to build a simple, TCP/IP-based workgroup computing solution which used Solaris servers as the back-end for Windows PC desktops (remember, those were the days before Windows 95 where, if you wanted to access the Internet using Windows, you had to buy a third-party product such as PC-NFS). One of the key features of the product was a way to centralize administrative control for those desktops; DHCP was a new protocol at the time, and it had the basic elements we needed to construct a centralized control solution, so we built a DOS DHCP client, a Solaris DHCP server, and some Solaris commands and a Windows GUI to administer it. The product ultimately wasn't especially successful, but DHCP had started to catch on in IP networks enough that we needed to bundle a server as part of Solaris. So in 1996 the team moved over to the Solaris organization and proceeded to integrate, with only some minor changes, the DHCP server and Solaris commands into Solaris 2.6. In Solaris 8, I added the DHCP Manager GUI.
One thing we'd realized by that time, though, was that our original server was good in the environment we'd written it for, workgroup/department-level networks, but didn't scale to the loads of large-scale enterprises or ISP's. That led to a very extensive rewrite of the server for Solaris 9 (also shipped in later Solaris 8 updates) to provide high levels of scalability, on the order of several hundred client operations per second on mid-range SPARC hardware of the day; to this day the current implementation is commonly referred to internally as the "Enterprise DHCP" server, because that was the particularly uncreative project name we used. Right about that same time, as a separate project, we also added RFC 2136 DNS dynamic update support.
The obvious place to start in scaling the server was to make it fully multi-threaded; the original server had been single-threaded, but some limited multithreading to handle ICMP (used for duplicate detection prior to offering an address to a client) had been added as part of the Solaris 2.6 work. To fully multi-thread the server, though, we needed to make the access to its lease storage thread-safe, as the original code I'd written for that in '94 (which actually was a minor evolution of code from the original Solaris 2 admintool) hadn't had that as a requirement (nowadays you wouldn't get away with writing a library that's not MT-safe). We also felt, though, that we needed to provide more flexibility in storing the DHCP data. We'd provided two options for storing the data from the beginning: traditional Unix ASCII files, and NIS+ tables. Neither was actually sufficient for the transaction rates we wanted to support. Ultimately, we settled on providing a plug-in architecture for data storage, with a private API used by the server and the administrative tools, which mediated access to a documented public layer, so that we (or anyone else) could add data stores to the server. The existing data stores were re-implemented in terms of this API, and we added a new, binary file format which would support the transaction rate we were after. The Netra HA folks subsequently wrote a high-availability data store solely from the documentation, so we know it can be done.
Well, that's the history behind what you see in the source today. The main DHCP server source is found in usr/src/cmd/cmd-inet/usr.lib/in.dhcpd (one helpful thing to know is that most IP networking utilities are in the usr/src/cmd/cmd-inet portion of the tree). A couple of interesting reads there are the comments in main.c describing the thread model, and the README on the server's in-memory caching, another part of the performance story. The data storage API is over under usr/src/lib in libdhcpsvc; the private subdirectory contains the API used by the server and tools, while the modules subdirectory contains the various data storage modules that put the bits on the disk. Other interesting libraries are libdhcpdu, which implements the DNS dynamic update feature, and libdhcputil, which contains some shared code related to DHCP option definitions that is used by the server, the admin tools, and the DHCP client. The server administration tools, including the CLI's, can all be found under usr/src/cmd/cmd-inet/usr.sadm/dhcpmgr, one of the early, and few, outposts of Java in the OS/Net source.
If this has whetted your appetite, you can get really deep into one of the more unusual aspects of our server implementation over in my buddy meem's blog.