Finding Memory leaks Within Solaris Applications Using libumem

Debugging Methodology :

The malloc() and free() memory management methods are used by many application developers. An application can be written without a dependence on any particular memory management programming interface by using the standard memory management methods malloc() and free(). This section will outline the steps needed to take advantage of the libumem library to debug an application's memory transactions.

Library Interposition and libumem Flags

If the libumem library is interposed (by setting the LD_PRELOAD environment variable) when executing an application, the malloc() and free() methods defined within the libumem library will be used whenever the application calls malloc() or free(). In order to take advantage of the debugging infrastructure of the libumem library, one needs to set the UMEM_DEBUG and the UMEM_LOGGING flags in the environment where the application is being executed.  The most common values for these flags are as follows: UMEM_DEBUG=default and UMEM_LOGGING=transaction.  With these settings, a thread ID, high-resolution time stamp, and stack trace are recorded for each memory  transaction initiated by the application.

The following are examples of the commands used to set the appropriate debug flags and interpose the libumem library when executing an application.


(csh)

%(setenv UMEM_DEBUG default; setenv UMEM_LOGGING transaction;
setenv LD_PRELOAD libumem.so.1;)

or

(bash)

bash-2.04$UMEM_DEBUG=default UMEM_LOGGING=transaction LD_PRELOAD=libumem.so.1

More details about the debug flags (UMEM_DEBUG and UMEM_LOGGING) can be found in the umem_debug(3MALLOC) man page.

MDB Commands

The developer can view the debug information pertaining to an application's memory management transactions by using MDB.  The following commands within MDB can be used to provide a great deal of information about the memory transactions that took place during the execution of the application.

::umem_status

    \* Prints the status of the umem indicating if the logging features have been turned on or off

::findleaks

    \* Prints a summary of the memory leaks found within the application
::umalog

    \* Prints the memory transactions initiated by the application and the correlated stack traces
::umem_cache

    \* Prints the details about each of the umem caches
[address]::umem_log

    \* Prints the umem transaction log for the application
[address]::umem_verify

    \* Prints the integrity of the umem caches which is useful in determining if a buffer has been corrupted

address$<bufctl_audit

    \* Prints the contents of the umem_bufctl_audit structure as defined in the /usr/include/umem_impl.h header file

Example :

Traditional Memory Leak

In order to examine if SunMC has a memory leak, one can execute the following steps to narrow down the section of the code which is causing the leak.

1. The libumem library is only available on systems which are running the Solaris 9 OS, Update 3 and above.
%uname -a
SunOS hostname 5.9 Generic_112233-05

(csh)


2. Execute the application with the libumem library interposed and the appropriate debug flags set.
%(setenv UMEM_DEBUG default; setenv UMEM_LOGGING transaction;setenv LD_PRELOAD libumem.so.1; ./a.out)

(bash)

#UMEM_DEBUG=default;UMEM_LOGGING=transaction;LD_PRELOAD=libumem.so.1; ./a.out

3. Use the gcore (1) command to get an application core to analyze the application's memory transactions.

%ps -ef | grep esd
user1     970   714  0 10:42:42 pts/4    0:00 ./a.out

%gcore 970
gcore: core.970 dumped

4. Use MDB to analyze the core for memory leaks using the commands described in the previous section.

$mdb core.970
Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]

> ::umem_log
CPU ADDR     BUFADDR         TIMESTAMP THREAD  
  0 0002e0c8 00055fb8     159d27e121a0 00000001
  0 0002e064 00055fb8     159d27e0fce8 00000001
  0 0002e000 00049fc0     159d27da1748 00000001
    00034904 00000000                0 00000000
    00034968 00000000                0 00000000
    ... snip ...

  Here we can see that there have been three transactions by thread #1 on cpu #0.

 

> ::umalog

T-0.000000000  addr=55fb8  umem_alloc_32
         libumem.so.1`umem_cache_free+0x4c
         libumem.so.1`process_free+0x68
         libumem.so.1`free+0x38
         main+0x18
         _start+0x108

T-0.000009400  addr=55fb8  umem_alloc_32
         libumem.so.1`umem_cache_alloc+0x13c
         libumem.so.1`umem_alloc+0x44
         libumem.so.1`malloc+0x2c
         main+0x10
         _start+0x108

T-0.000461400  addr=49fc0  umem_alloc_24
         libumem.so.1`umem_cache_alloc+0x13c
         libumem.so.1`umem_alloc+0x44
         libumem.so.1`malloc+0x2c
         main+4
         _start+0x108

 The three transactions consist of one allocation to the 24 byte umem cache, and one memory allocation and release from the 32 byte umem cache. Note that the high resolution timestamp output in the upper left hand corner is relative to the last memory transaction initiated by the application.

> ::findleaks
CACHE     LEAKED   BUFCTL CALLER
0003d888       1 00050000 libumem.so.1`malloc+0x0
----------------------------------------------------------------------
 Total       1 buffer, 24 bytes

  This shows that there is one 24 byte buffer which has been leaked.

Comments:

[Trackback] Good introduction into memory leak debugging with libumem

Posted by c0t0d0s0.org on July 18, 2007 at 08:15 AM IST #

This is really a cool stuff :-)..

Posted by Shankar on July 18, 2007 at 10:36 AM IST #

Note that if you use gcore you are taking the core of a running process, and so an allocation that has not yet been freed is not necessarily a leak since it is still possible for the process to free it.

If you gcore a process right when it is exiting (after all atexit() handlers) then you'll know for sure that an un-freed allocation was leaked (or maybe not! think of shared object .fini sections).

Worse, it's often viewed as OK to leak some things -- things that may be needed throughout the lifetime of the process, finite sets of such things.

IOW, you have to investigate every leak (and associated call stack at allocation time) to decide what is and what isn't a leak.

Posted by Nico on July 18, 2007 at 03:42 PM IST #

How can I get such information without the libumem library.
I am getting core files from one application that is running with another library, I cannot change it to libumem for now.
Is there another way to get this info from mdb?

Thanks,

Posted by okam on November 13, 2009 at 07:35 PM IST #

The bash invocation is incorrect, the example creates shell variables, not environment variables (unless they already exist as environment variables, in which case they are modified and it will work). Shell variables are not passed to the environment, and will not be availble to libumem.

My suggested invocation would be:

UMEM_DEBUG=default UMEM_LOGGING=transaction LD_PRELOAD=libumem.so.1 ./a.out

This idiosyncratic bournism places those variables in the environment only for that subprocess, but not for the current shell.

To set them for the current shell in a persistent way, you can use:

export UMEM_DEBUG=default UMEM_LOGGING=transaction LD_PRELOAD=libumem.so.1; ./a.out

Or if you wish to be very very deliberate and compatable with old fashioned bourne:

UMEM_DEBUG=default; export UMEM_DEBUG
UMEM_LOGGING=transaction; export UMEM_LOGGING
LD_PRELOAD=libumem.so.1; export LD_PRELOAD
./a.out

Posted by Joshua Rodman on October 18, 2010 at 06:54 PM IST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

pnayak

Search

Archives
« July 2015
MonTueWedThuFriSatSun
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
  
       
Today