Creating separate debug info

In my previous post, I briefly described how to use separate debug info on Ubuntu Linux where you can easily install debug and source packages and be able to debug system libraries in source language terms. On Solaris, this scheme is not implemented; all Solaris system libraries come without debug info (albeit none stripped as of Solaris 10, as far as I remember, so you should at least have all symbols available).

So what are the advantages of separate debug info on Solaris?

For benefits of support, some developers always ship unstripped binaries to their customers; sometimes, they even compile production version without optimization and ship it with full debug info. All this greatly simplifies process of bug reproduction and fixing. However, it affects performance in several ways:
  1. binaries shipped to customers are larger (debug info doesn't affect memory image, though, so the app won't be slower just because it has debug info),
  2. when compiling with -g, no optimization is performed by compiler and therefore, run-time performance suffers.
Problem #2 can be solved by compiling with -g -O. Recent versions of Sun Studio compilers and tools support optimized code debugging so you can have [almost] full debuggability with [almost] full speed. But it still leaves problem #1: size of packages your customers have to download.

For example, standard Solaris C++ support library - - is about 2Mb large. When compiled with DWARF debug info, it grows to 14Mb; in other words, 700% (!) grows. This factor is even greater for some applications - 10x or more. Obviously, this is not acceptable in some situations. The same Solaris has thousands of libraries and you never know in advance which ones you'll have to debug.

However, it is possible to provide debug info on demand by extracting it from original load object (executable or shared library) and shipping as a separate unit. For example, if your app crashes at customer site, you can send separate debug info file(s) to that site and have customer obtain full stack trace with line numbers and parameter names.

How to create separate debug info file?

Separate debug info file is created with objcopy command (may be called gobjcopy on Solaris):
$ objcopy --only-keep-debug a.out a.out.debuginfo
$ objcopy --strip-debug a.out 
$ objcopy --add-gnu-debuglink=a.out.debuginfo a.out

First command copies only debug-related sections from a.out to a.out.debuginfo (name is arbitrary).

Second command removes debug-related sections from a.out. You can use strip command instead of objcopy --strip-debug; the latter leaves symbol table thus giving  debugger opportunity to present more meaningful stack traces even without debug info, while strip removes symtab as well. The choice is up to you.

And the last command adds a special section to a.out called .gnu_debuglink (described in earlier post) that contains name and checksum of debug info file - a.out.debuginfo.

This process is also described in details in objcopy(1) man page under --only-keep-debug option.

The file you created can be placed in the following three conventional places:

  1. in the same directory with executable,
  2. in a directory named ".debug" under (1),
  3. in /usr/lib/debug directory (configurable).

(type help separate debug info in dbx for more info on the subject).

What version of dbx supports separate debug info?

Support for separate debug info was added in Sun Studio 12 (dbx version 7.6), so current version - Sun Studio 12 update 1 (dbx 7.7) also has it. Moreover, this feature was ported to earlier releases and is available as a patch, should you need to work on Solaris 8 or even earlier (Sun Studio 12 works on Solaris 9+).

Besides DWARF debug info, where separation of debug info from the executable is most beneficial, dbx also supports stabs (generated by default in Sun Studio 11 and earlier versions). With stabs, debug info is usually kept in object (.o) files and load object only has an index built into it, so separation of debug info is not really of any use here.

However, there is a compiler option (-xs) that enables generation of all stabs into load object; with this option, you don't have to keep .o files, but resulting executable or shared library grows the same way as with DWARF. Grows factor is less as stabs tend to be more compact than DWARF. Still, one might want to use separate debug info in order to

  • not to keep .o files after build,
  • reduce size of executable or shared library.


  1. Stabs versus dwarf
  2. objcopy(1)
  3. Sun Studio 12 C++ compiler man page

Can this method be used to perform some source-level core file debugging from an optimized binary?

опубликовал Ryan Сентябрь 13, 2009 at 04:36 PM MSD #

Certainly, provided that you compiled with -g -O _beforehand_ and are using latest Sun Studio compilers and tools (dbx) - see

Here's how it is supposed to work:
1. Compile with optimization (-O) _and_ debug info (-g); latest Sun Studio compilers generate meaningful debug info even with optimization (DWARF location lists are used for this purpose).
2. Move debug info from the binary (a.out) to a separate file (a.out.debug) as I described in this post.
3. As soon as there's a need to debug core file generated by the same binary, you just place a.out.debug in the same directory with the binary and dbx should pick up debug info automatically.

If binary and debug info do not match (checksum built into the binary is different from actual checksum of separate debug info), dbx will silently ignore this file. Unfortunately, because of the way most Linux distributions organize their /usr/lib/debug, we can't issue "checksum mismatch" error reliably.

опубликовал Maxim Kartashev Сентябрь 14, 2009 at 04:08 AM MSD #

Опубликовать комментарий:
  • HTML Syntax: Отключен

Articles, news, notes on dbx, the Sun Studio debugger and other stuff.


« Апрель 2014