wierd malloc problem.
By timatworkhomeandinbetween on May 25, 2005
A collegue asked me to help debug a userland core file a couple of days ago. The stack showed the normal segv in a frame under realfree() that is so typical of a malloc arena corruption. The standard malloc keeps some housekeeping in the 8 bytes below the pointer that malloc returns, once you call free() those 8 bytes and the first few of the allocation are cast into an internal data structure and it is added to a b-tree sorted by size. It is not until another malloc/free causes a change in the tree that corruption is detected. As this can happen a long time after the free the program can segv long after the damage - another reason to use libumem or libwatchmalloc or ...
I thought it would be easy as there was a testcase, just load up libumem or my own debugging malloc library, run the testcase and point out the incorrect code. But it wouldn't fail with those. I interposed on malloc/free/calloc/valloc/memalign but I could see no obvious error like a double free. My current theory is that something is trampling over the structure after it has gone on the free list, time to become more of a dbx power user, or maybe write a mdb dcmd to verify the malloc arena at regular intervals.
Once that is done I can get back to putting the sparc emulator into my crash dump running program.