• Categories
  • Search
Sun | Thursday, February 1, 2018

Core File Enhancements for elfdump

By: Ali Bahrami | Principal Software Engineer

Solaris 11.4 comes with a number of enhancements that allow the elfdump utility to display a wealth of information that was previously hidden in Solaris core files. Best of all, this comes without a significant increase in core file size.

Core files are ELF files, and as such, can be examined by tools like elfdump. Historically, Solaris core files have contained the following information:

Program Headers
PT_LOAD program headers for the mappings of objects found in the process. There is no indication of which program header belongs to which mapped object, and segment, within that object.
Core Notes
Information related to the state of the process when the core file was written is recorded in a note section, referenced via a PT_NOTE program header.
Optional CTF and symbol table sections
Subject to coreadm(8), non-allocable sections containing debug data, such as the standard .symtab symbol table, and CTF, may be added. These additional sections are not part of the process mappings. and so, adding them to the core file requires copying their data from the original disk file to the end of the core file, as well as the addition of a section header.

Although elfdump has always been able to dump such core files, the information shown had only minimal value. Without an association between the PT_LOAD program headers and the objects and segments they represent, it was rather difficult to do much with the information shown. For instance, consider this elfdump output from a core file containing 637 program headers:

% elfdump -p core

Program Header[0]:
    p_vaddr:      0                   p_flags:    [ PF_R ]
    p_paddr:      0                   p_type:     [ PT_NOTE ]
    p_filesz:     0x1820c             p_memsz:    0
    p_offset:     0x1ec18             p_align:    0

Program Header[1]:
    p_vaddr:      0x100000            p_flags:    [ PF_X PF_R ]
    p_paddr:      0                   p_type:     [ PT_LOAD ]
    p_filesz:     0x4000              p_memsz:    0x4000
    p_offset:     0x36e24             p_align:    0

Program Header[2]:
    p_vaddr:      0x204000            p_flags:    [ PF_W PF_R ]
    p_paddr:      0                   p_type:     [ PT_LOAD ]
    p_filesz:     0x1000              p_memsz:    0x1000
    p_offset:     0x3ae24             p_align:    0

Program Header[3]:
    p_vaddr:      0x284000            p_flags:    [ PF_W PF_R ]
    p_paddr:      0                   p_type:     [ PT_LOAD ]
    p_filesz:     0xd131000           p_memsz:    0xd131000
    p_offset:     0x3be24             p_align:    0
...
Program Header[636]:
    p_vaddr:      0xf62af000          p_flags:    [ PF_W PF_R ]
    p_paddr:      0                   p_type:     [ PT_LOAD ]
    p_filesz:     0x24000             p_memsz:    0x24000
    p_offset:     0x1659ae24          p_align:    0

We can surmise that program headers [1] and [2] are the text and data segments for the main executable, and we might guess that the mappings for libc follow, but it's hard to make much use of that information, or to make anything at all out of the vast majority of these headers.

We added the ability for elfdump to decode core file PT_NOTE sections in Solaris 11, and this greatly increased the usefulness of elfdump with core files, but otherwise, elfdump's ability to extract useful information from core files was limited.

Improvement #1: Apply libproc

The primary tool for analyzing Solaris core files is the mdb debugger. mdb uses libproc to match the PT_LOAD headers in a core file to the on disk objects and segments within those objects. With Solaris 11.4, elfdump also uses libproc for this purpose. In addition, elfdump, via libproc, makes use of the new program header names that are now recorded in Solaris ELF objects. As a result, it provides considerably more information for the core file shown above:

% elfdump -p core

Program Header[0]:
    p_vaddr:      0                   p_flags:    [ PF_R ]
    p_paddr:      0                   p_type:     [ PT_NOTE ]
    p_filesz:     0x1820c             p_memsz:    0
    p_offset:     0x1ec18             p_align:    0

Program Header[1]:  text
    p_vaddr:      0x100000            p_flags:    [ PF_X PF_R ]
    p_paddr:      0                   p_type:     [ PT_LOAD ]
    p_filesz:     0x4000              p_memsz:    0x4000
    p_offset:     0x36e24             p_align:    0
    pr_mapname:   /usr/bin/gnome-shell

Program Header[2]:  data
    p_vaddr:      0x204000            p_flags:    [ PF_W PF_R ]
    p_paddr:      0                   p_type:     [ PT_LOAD ]
    p_filesz:     0x1000              p_memsz:    0x1000
    p_offset:     0x3ae24             p_align:    0
    pr_mapname:   /usr/bin/gnome-shell

Program Header[3]:
    p_vaddr:      0x284000            p_flags:    [ PF_W PF_R ]
    p_paddr:      0                   p_type:     [ PT_LOAD ]
    p_filesz:     0xd131000           p_memsz:    0xd131000
    p_offset:     0x3be24             p_align:    0
    pr_mflags:    [ BREAK ANON ]
...
Program Header[636]:
    p_vaddr:      0xf62af000          p_flags:    [ PF_W PF_R ]
    p_paddr:      0                   p_type:     [ PT_LOAD ]
    p_filesz:     0x24000             p_memsz:    0x24000
    p_offset:     0x1659ae24          p_align:    0
    pr_mflags:    [ STACK ANON ]

We now can see that program headers [1] and [2] are indeed the text and data segments for the main executable, /usr/bin/gnome-shell. In addition, program header [3] is the heap for the process, and [636] is the process stack.

Note that the pr_mapname and pr_mflags information shown by elfdump does not come from the program headers. Rather, that information is derived by libproc.

Improvement #2: Decode Unwind Sections

Independent of my work, my colleague Robert Harris extended Solaris 11.4 core files to include the .eh_frame and .eh_frame_hdr unwind sections, making them visible to elfdump. Shortly afterwards, I did a significant overhaul of elfdump's unwind section decoding, adding the ability to use the relocation information in the object to map the addresses found in unwind sections to their corresponding symbols, and display them by name. From there, it was a relatively short step to enable this decoding for core files:

% elfdump -u ~/core | more
/home/alib/core: core of pid 24870: /usr/bin/gnome-shell

Unwind Section:  .eh_frame_hdr (/usr/bin/gnome-shell)
Frame Header:
  Version: 1
  FramePtrEnc: [ sdata4 pcrel ]      FramePtr: 0x100280
  FdeCntEnc:   [ udata4 ]            FdeCnt: 14
  TableEnc:    [ sdata4 datarel ]    BaseAddress: 0x100200  (.eh_frame_hdr)
  Binary Search Table:
    InitialLoc    FdeLoc  symbol
      0x103060  0x1002a0  __start_crt
      0x1030f0  0x1002e0  _mcount
      0x103100  0x100310  deregister_tm_clones
...
      0x103570  0x100508  main

Unwind Section:  .eh_frame (/usr/bin/gnome-shell)
           CIE: [0x100280]
      [0]    length:       0x1c
    [0x4]    cieid:        0
    [0x8]    version:      1
    [0x9]    augmentation: 'zR'
    [0xc]    codealign:    0x1
    [0xd]    dataalign:    -8
    [0xe]    retaddr:      16
             Augmentation Data:
    [0xf]        size:     1
   [0x10]        code pointer encoding: 0x1b [ sdata4 pcrel ]
             CallFrameInstructions:
   [0x11]        DW_CFA_def_cfa: r7 (rsp), offset=8
   [0x14]        DW_CFA_offset: r16 (ra), cfa-8
   [0x16]        DW_CFA_same_value: r12
   [0x18]        DW_CFA_same_value: r13
   [0x1a]        DW_CFA_same_value: r14
   [0x1c]        DW_CFA_same_value: r15
   [0x1e]        DW_CFA_nop [2]

             FDE: [0x1002a0]
   [0x20]      length:    0x24
   [0x24]      cieptr:    0x24
   [0x28]      initloc:   0x103060 [ sdata4 pcrel ]  __start_crt
                   base: 0x1002a8  value: 0x2db8
   [0x2c]      addrrange: 0x87 [ sdata4 ]
                   endloc: 0x1030e6

Improvement #3: Add (A Lot) More Section Headers

As has been noted many times before, ELF objects contain 2 distinct type of header:

Section Header
Label and identify different sections of the object, such as text, data, symbol tables, etc.
Program Header
Collections of sections with shared basic attributes (read/write/executable), used by the runtime linker to load a given object into memory.

Program headers cater to the needs of the runtime linker, while section headers are used by link-editors, debuggers, and observability tools. Sections can be "allocable", meaning that they are mapped into process memory at runtime, or "non-allocable", meaning that they exist in the disk file, but are not mapped into process memory at runtime. Non-allocable sections are often informally called "debug sections".

Allocable sections are contained within a program header, usually text (readonly, executable), and data (writable, ideally non-executable). No program header is produced for non-allocable sections.

Historically, Solaris core files have not had many section headers. Typically, only the .symtab symbol table, and its associated string table .strtab have been included. This data is not found within the memory mappings captured in a core file, as reflected by the core file program headers, so the kernel and libproc (gcore) create them by copying the data from the on disk object file to the end of the core file, and then creating the section header that references them. This is essential information, so it is worthwhile, but it does have a cost in time, and in core file size.

In contrast, when Robert introduced the .eh_frame_hdr, and .eh_frame sections to our core files, as mentioned in the previous section, it was not necessary to add data to the end of the core file. These are allocable (SHF_ALLOC) sections, meaning that they are mapped into the process at runtime, and therefore, the data is already present. All that is necessary is to add section headers to reference the data already captured by the process mappings. A 32-bit section header is 40 bytes, and a 64-bit section header is 64 bytes. That means that this new information was unlocked in our core files, essentially for free, since the size of the section headers is a rounding error on the size of the data they describe.

The fact that SHF_ALLOC sections don't require the core file to become larger is a fairly obvious observation. However, Solaris core files had not done this before, largely because core files have been produced mainly to capture mappings. Personally, because I hadn't previously worked deeply on core files, it was a revelation. It caused me to realize that I could cheaply add access to information valuable to elfdump, without perturbing core file contents or layout. The same idea Robert used for .eh_frame_hdr and .eh_frame can be used for any SHF_ALLOC section found in the original objects. In so doing, that untapped information becomes available to observability tools such as debuggers, and elfdump.

Building on the technique used for unwind sections, I modified the Solaris core file generation code to unconditionally issue core file section headers for the .SUNW_ldynsym, .dynsym, and .dynstr sections. We also include the .SUNW_syminfo, and .dynamic sections. These are less essential, but .SUNW_syminfo augments the .dynsym with useful and easily accessible information, and .dynamic is the gateway to much of the information used by the runtime linker to manage the object.

As with .eh_frame and .eh_frame_hdr, this is unconditional, and not subject to coreadm(8), because the core file size is not increased, and no additional computational overhead is required to generate them.

Example

I used gcore to produce core files for a process running emacs-nox, one on an older system without these enhancements, and another on a new system. In each case, I started the editor, waited until it was ready for user input, and then used gcore to grab a core:

% ls -alFh core.emacs.baseline core.emacs.new
-rw-r--r--  1 alib  staff  27M Aug 23 12:03 core.emacs.baseline
-rw-r--r--  1 alib  staff  27M Aug 23 12:02 core.emacs.new

As you can see, there is no significant difference in their size. elfdump reveals that both have the same number of program headers (segments). However, the new core advertises 49 section headers (e_shnum) rather than the 24 found in the older one.

% elfdump -e core.emacs.baseline core.emacs.new

core.emacs.baseline:

core.emacs.baseline: core of pid 5706: emacs-nox

ELF Header
  ei_magic:   { 0x7f, E, L, F }
  ei_class:   ELFCLASS64          ei_data:       ELFDATA2LSB
  ei_osabi:   ELFOSABI_NONE       ei_abiversion: 0
  e_machine:  EM_AMD64            e_version:     EV_CURRENT
  e_type:     ET_CORE
  e_flags:                     0
  e_entry:                     0  e_ehsize:     64  e_shstrndx:  23
  e_shoff:                 0x660  e_shentsize:  64  e_shnum:     24
  e_phoff:                  0x40  e_phentsize:  56  e_phnum:     28

core.emacs.new:

core.emacs.new: core of pid 3838: emacs-nox

ELF Header
  ei_magic:   { 0x7f, E, L, F }
  ei_class:   ELFCLASS64          ei_data:       ELFDATA2LSB
  ei_osabi:   ELFOSABI_SOLARIS    ei_abiversion: EAV_SUNW_CURRENT
  e_machine:  EM_AMD64            e_version:     EV_CURRENT
  e_type:     ET_CORE
  e_flags:                     0
  e_entry:                     0  e_ehsize:     64  e_shstrndx:  48
  e_shoff:                 0x660  e_shentsize:  64  e_shnum:     49
  e_phoff:                  0x40  e_phentsize:  56  e_phnum:     28

elfdump can also be used to see the section headers that used to be included:

    % elfdump -c core.emacs.baseline  | grep sh_name
    Section Header[1]:  sh_name: .eh_frame_hdr (/usr/bin/emacs-nox)
    Section Header[2]:  sh_name: .eh_frame (/usr/bin/emacs-nox)
    Section Header[3]:  sh_name: .symtab (/usr/bin/emacs-nox)
    Section Header[4]:  sh_name: .strtab (/usr/bin/emacs-nox)
    Section Header[5]:  sh_name: .eh_frame_hdr (/lib/amd64/ld.so.1)
    Section Header[6]:  sh_name: .eh_frame (/lib/amd64/ld.so.1)
    Section Header[7]:  sh_name: .symtab (/lib/amd64/ld.so.1)
    Section Header[8]:  sh_name: .strtab (/lib/amd64/ld.so.1)
    Section Header[9]:  sh_name: .SUNW_ctf (/lib/amd64/ld.so.1)
    Section Header[10]:  sh_name: .eh_frame_hdr (/lib/amd64/libc.so.1)
    Section Header[11]:  sh_name: .eh_frame (/lib/amd64/libc.so.1)
    Section Header[12]:  sh_name: .symtab (/lib/amd64/libc.so.1)
    Section Header[13]:  sh_name: .strtab (/lib/amd64/libc.so.1)
    Section Header[14]:  sh_name: .SUNW_ctf (/lib/amd64/libc.so.1)
    Section Header[15]:  sh_name: .eh_frame_hdr (/usr/lib/amd64/libncurses.so.5.7)
    Section Header[16]:  sh_name: .eh_frame (/usr/lib/amd64/libncurses.so.5.7)
    Section Header[17]:  sh_name: .symtab (/usr/lib/amd64/libncurses.so.5.7)
    Section Header[18]:  sh_name: .strtab (/usr/lib/amd64/libncurses.so.5.7)
    Section Header[19]:  sh_name: .eh_frame_hdr (/lib/amd64/libm.so.2)
    Section Header[20]:  sh_name: .eh_frame (/lib/amd64/libm.so.2)
    Section Header[21]:  sh_name: .symtab (/lib/amd64/libm.so.2)
    Section Header[22]:  sh_name: .strtab (/lib/amd64/libm.so.2)
    Section Header[23]:  sh_name: .shstrtab

as well as the augmented set that is now produced:

    % elfdump -c core.emacs.new  | grep sh_name
    Section Header[1]:  sh_name: .eh_frame_hdr (/usr/bin/emacs-nox)
    Section Header[2]:  sh_name: .eh_frame (/usr/bin/emacs-nox)
    Section Header[3]:  sh_name: .dynamic (/usr/bin/emacs-nox)
    Section Header[4]:  sh_name: .SUNW_syminfo (/usr/bin/emacs-nox)
    Section Header[5]:  sh_name: .SUNW_ldynsym (/usr/bin/emacs-nox)
    Section Header[6]:  sh_name: .dynsym (/usr/bin/emacs-nox)
    Section Header[7]:  sh_name: .dynstr (/usr/bin/emacs-nox)
    Section Header[8]:  sh_name: .symtab (/usr/bin/emacs-nox)
    Section Header[9]:  sh_name: .strtab (/usr/bin/emacs-nox)
    Section Header[10]:  sh_name: .eh_frame_hdr (/lib/amd64/ld.so.1)
    Section Header[11]:  sh_name: .eh_frame (/lib/amd64/ld.so.1)
    Section Header[12]:  sh_name: .dynamic (/lib/amd64/ld.so.1)
    Section Header[13]:  sh_name: .SUNW_syminfo (/lib/amd64/ld.so.1)
    Section Header[14]:  sh_name: .SUNW_ldynsym (/lib/amd64/ld.so.1)
    Section Header[15]:  sh_name: .dynsym (/lib/amd64/ld.so.1)
    Section Header[16]:  sh_name: .dynstr (/lib/amd64/ld.so.1)
    Section Header[17]:  sh_name: .symtab (/lib/amd64/ld.so.1)
    Section Header[18]:  sh_name: .strtab (/lib/amd64/ld.so.1)
    Section Header[19]:  sh_name: .SUNW_ctf (/lib/amd64/ld.so.1)
    Section Header[20]:  sh_name: .eh_frame_hdr (/lib/amd64/libc.so.1)
    Section Header[21]:  sh_name: .eh_frame (/lib/amd64/libc.so.1)
    Section Header[22]:  sh_name: .dynamic (/lib/amd64/libc.so.1)
    Section Header[23]:  sh_name: .SUNW_syminfo (/lib/amd64/libc.so.1)
    Section Header[24]:  sh_name: .SUNW_ldynsym (/lib/amd64/libc.so.1)
    Section Header[25]:  sh_name: .dynsym (/lib/amd64/libc.so.1)
    Section Header[26]:  sh_name: .dynstr (/lib/amd64/libc.so.1)
    Section Header[27]:  sh_name: .symtab (/lib/amd64/libc.so.1)
    Section Header[28]:  sh_name: .strtab (/lib/amd64/libc.so.1)
    Section Header[29]:  sh_name: .SUNW_ctf (/lib/amd64/libc.so.1)
    Section Header[30]:  sh_name: .eh_frame_hdr (/usr/lib/amd64/libncurses.so.5.7)
    Section Header[31]:  sh_name: .eh_frame (/usr/lib/amd64/libncurses.so.5.7)
    Section Header[32]:  sh_name: .dynamic (/usr/lib/amd64/libncurses.so.5.7)
    Section Header[33]:  sh_name: .SUNW_syminfo (/usr/lib/amd64/libncurses.so.5.7)
    Section Header[34]:  sh_name: .SUNW_ldynsym (/usr/lib/amd64/libncurses.so.5.7)
    Section Header[35]:  sh_name: .dynsym (/usr/lib/amd64/libncurses.so.5.7)
    Section Header[36]:  sh_name: .dynstr (/usr/lib/amd64/libncurses.so.5.7)
    Section Header[37]:  sh_name: .symtab (/usr/lib/amd64/libncurses.so.5.7)
    Section Header[38]:  sh_name: .strtab (/usr/lib/amd64/libncurses.so.5.7)
    Section Header[39]:  sh_name: .eh_frame_hdr (/lib/amd64/libm.so.2)
    Section Header[40]:  sh_name: .eh_frame (/lib/amd64/libm.so.2)
    Section Header[41]:  sh_name: .dynamic (/lib/amd64/libm.so.2)
    Section Header[42]:  sh_name: .SUNW_syminfo (/lib/amd64/libm.so.2)
    Section Header[43]:  sh_name: .SUNW_ldynsym (/lib/amd64/libm.so.2)
    Section Header[44]:  sh_name: .dynsym (/lib/amd64/libm.so.2)
    Section Header[45]:  sh_name: .dynstr (/lib/amd64/libm.so.2)
    Section Header[46]:  sh_name: .symtab (/lib/amd64/libm.so.2)
    Section Header[47]:  sh_name: .strtab (/lib/amd64/libm.so.2)
    Section Header[48]:  sh_name: .shstrtab

As a result, elfdump is able to find and display considerably more information for each object captured within the core file:

    % elfdump core.emacs.baseline > elfdump.emacs.baseline
    % elfdump core.emacs.new > elfdump.emacs.new
    % ls -alFh elfdump.emacs.baseline elfdump.emacs.new
    -rw-r--r--  1 alib  staff  13M Aug 23 13:05 elfdump.emacs.baseline
    -rw-r--r--  1 alib  staff  15M Aug 23 13:06 elfdump.emacs.new

The new elfdump output is significantly larger than the old, even though they were generated from core files of nearly identical size. The information was always there, but thanks to the new section headers, is no longer invisible.

Join the discussion

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha
 

Visit the Oracle Blog

 

Contact Us

Oracle

Integrated Cloud Applications & Platform Services