malloc interposition can't possibly work reliabily

In Solaris, many of the routines called from libc are "direct bound" so that references from inside libc will always find the function implementations that are inside libc. This approach prevents outside libraries from interposing (substituting) different implmentations of common functions. The largest exception to this is the malloc family of routines. The malloc routines (when called by libc, for example from strdup) MUST call an externally supplied malloc routine, if one is supplied via LD_PRELOAD or library link ordering.

There is a huge gotcha related to malloc interposing. If you get a pointer from malloc, you have to free it using the free routine in the same library that allocated it. But how do you guarantee that? If every program has libc, then every program will have at least one allocator in it. Any program that uses libumem will have at least two (one from libumu and one from libc). If the user wants to LD_PRELOAD their own memory checker library, it just gets worse.

It gets even worse because malloc libraries implement many additional routines to allocate memory. Let's say my app calls valloc in libc. Let's say I want to interpose libmymalloc because it has a spiffy new memory checker that I want to use. Now let's say libmymalloc doesn't include a definition for valloc. My app will crash, because valloc gets memory from the libc pool, and free will free it to the libmymalloc pool.

At this point there are people who will say: "Easy, just make sure they all implement the same set of functions." Well yes, that would solve the problem, if there were a way to do this. But there is no standard for what this list of routines is. Memory allocation libraries are useful because they offer additional functionality beyond the plain malloc and free in libc. So they will always be adding functions that are not in anyone else's implementation. If I write my app to use libmalloc_alpha.so, and someone interposes libmalloc_beta.so, then all the custom functions in libmalloc_alpha.so that I was calling will still go to libmalloc_alpha.so, but all the customary ones will go to libmalloc_beta.so. The result is undefined.

Unfortunately the idea that you can replace a memory allocator library by just interposing a different one is a widely known "fact". You can read about how this bit the Solaris libraries in bugid 4846556. The problem came up in comp.unix.solaris recently as well.

Comments:

Typo: comp.solaris.org -> comp.unix.solaris. I didn't remember I posted that on comp.unix.solaris until I clicked on the link :)

Posted by Seongbae Park on October 05, 2006 at 06:38 AM PDT #

Thanks Seongbae, I fixed the typo. Nice to hear from you again. Hope things are going well.

Posted by Chris Quenelle on October 05, 2006 at 07:24 AM PDT #

Well, the interposers Sun ships ought to be safe to use as interposers for libraries also shipped by Sun. Making it safe for third parties to write interposers of their own really means defining closed sets of interposable functions or defining rules for extending them that can make it safe to interpose specific sub-sets of them.

Posted by Nico on October 06, 2006 at 02:12 AM PDT #

1. I am debugging an app which crashes in free() 2. I check to see if the app calls any custom functions inside its allocation library. 3. If not, then I can try substituting a different allocation library. -- Step 2 is difficult and error prone, and our tools don't help. Also, most people will just skip step 2 and assume that step 3 will work. Nico: Sun ships many malloc libraries, and most are not supported very well or used very heavily. It's wishful thinkking to assume that this "isn't a problem" for Sun libraries, and if people use non-Sun libraries, they get what they deserve.

Posted by Chris Quenelle on October 10, 2006 at 04:23 AM PDT #

Post a Comment:
Comments are closed for this entry.
About

Chris Quenelle

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today