It's now been one year since the GA release of Oracle Solaris 11.4. During that time, we've released 12 Support Repository Updates (SRUs), with the latest, SRU12 (aka 11.4.12) coming out last week. These releases have included hundreds of updates to the bundled FOSS packages, over 2500 bug fixes, and over 300 enhancements.
One area a few of us have been working on lately is making improvements to the standard C library, libc, that increase compatibility with the Linux & BSD libraries and make developing for Solaris easier. These include:
We're not stopping there. Implementations of mkostemp() and mkostemps() are currently in testing for a future SRU, and we're evaluating what to do next.
My contributions were in our common memory allocators, and the effects go beyond just increasing compatibility.
The simplest of the new functions is reallocf(), which takes the same arguments as realloc(), but if the allocation fails, it frees the old pointer, so that every caller doesn't have to replicate the same boilerplate code to save the old pointer and free it manually on failure. Forgetting to do this is a very common error, as many static analysers will report, so this makes it easier to clean your code of such issues at a low cost. You may note in the SRU 12 ReadMe that we indeed used this function to clean up a couple of our own bugs:
reallocarray() is also fairly simple - like realloc() it can either allocate new memory or change the size of an existing allocation, but instead of a single size value, it takes the size of an object and the number of objects to allocate space for, like calloc(), and makes sure there isn't an integer overflow when multiplied together. This also helps save adding integer overflow checks to many callers, and is easier to audit for correctness by having one overflow check in libc, instead of hundreds or more spread across a code base.
I first encountered reallocarray() not long after spending months working to eradicate integer overflow bugs from the protocol handling code in X11 servers and clients. Reading about OpenBSD introducing reallocarray seemed like such a better way to solve these problems than all the hand-coded checks we'd added to the X code base. (Though it couldn't replace all of them, since many didn't involve memory allocations.) Adding an implementation of reallocarray() to the Xserver and then to libX11, along with a macro to make a simple mallocarray() equivalent that just called reallocarray() with a NULL pointer for the old value, helped prove the usefulness of having this function. As adoption spread, I nominated it for standardization, and added it to the Solaris libc as well.
The interface that was most complicated and took the most work was malloc_usable_size(). The other two could be implemented as simple wrappers around any realloc() implementation without knowing the details of the allocation internals, and thus just have a single version in libc. However, since malloc_usable_size() has to look into the internal data structures of the memory allocator to determine the size of an allocated block, it has to have a different implementation for each memory allocator. As noted in the Alternative Implementations section of the malloc(3C) man page we currently ship 8 different implementations in Oracle Solaris 11.4 - in the libc, libadimalloc, libbsdmalloc, libmalloc, libmapmalloc, libmtmalloc, libumem, and watchmalloc libraries. While there are some commonalities, there are also some major differences, and this involved several completely different sets of code to implement across all these libraries.
While this seems like a lot of work for a rarely used function (though the few users are rather important, like the Mozilla applications and several memory/address sanitizers for systems without ADI support), it turned out to be a very worthwhile addition as it allowed writing a common set of unit tests across all 8 malloc implementations. These tests not only helped confirm the fixes for some long-standing bugs, but helped find bugs we hadn't known about in various corner cases, either by checking that allocation sizes are in the range expected, or by using malloc_usable_size() as a reliable way to check whether or not a pointer is free after given operations.
For instance, a test that boils down to
p2 = malloc(73); assert(p2 != NULL); s = malloc_usable_size(p2); /* request a size that realloc can't possibly allocate */ p = realloc(p2, SIZE_MAX - 4096); assert(p == NULL); assert(malloc_usable_size(p2) == s);found that some of our realloc() implementations would first increase the size of the block to include any following free blocks from the memory arena, and if that wasn't enough, tried allocating a new block — but never restored the original block size if that failed, leading to wasted memory allocation. The ancient bsdmalloc() actually freed the old block before trying to allocate a new one, and if it failed, left the old block unexpectedly unallocated.
Bugs fixed as part of this include:
So you can see we've both strengthened our compatibility story, and hardened our code in the process, making Solaris 11.4 better for both developers and users.