News, tips, partners, and perspectives for the Oracle Solaris operating system

Loading Relocatable Objects at Runtime - Very Expensive

The runtime linker, ld.so.1(1), is capable of loading relocatable objects. This capability arrived somewhat by accident, and as a side effect of some other projects related to the link-editor, ld(1). But, it was thought that this capability could prove useful within a development environment. A user can preload relocatable objects, and this technique might provide a quick prototyping turnaround instead of having to first combine the relocatable objects into some huge shared object.

However, this capability doesn't come cheap, and it was never thought to be useful for production software. Recently, a customer problem uncovered a regression in relocatable object processing. But, it was a surprise to see that this technique, in this case the dlopen(3c) of a relocatable object, was being used in production code.

The runtime linker only knows how to deal with dynamic objects, that is, dynamic executables and shared objects. When a relocatable object is encountered, the runtime linker first loads the link-editor. The link-editor then converts the relocatable object into a shared object memory image within the process. The runtime linker then processes this memory image as it would any other shared object.

The link-editor is really a family of libraries, and you can see these if you inspect the link-maps of a process that has triggered the loading of a relocatable object.

   % mdb main
   > ::bp exit
   > :r
   > ::Rt_maps
   Link-map lists (dynlm_list): 0x8045f38
     Lm_list: 0xfeffa204  (LM_ID_BASE)
       Link_map\*  ADDR()     NAME()
       0xfeffc8f4 0x08050000 main
       0xfeffccc4 0xfeef0000 /lib/libc.so.1
     Lm_list: 0xfeffa1c8  (LM_ID_LDSO)
       0xfeffc590 0xfefc8000 /lib/ld.so.1
       0xfeee06e4 0xfee60000 /lib/libld.so.2
       0xfeee0af0 0xfed90000 /lib/libc.so.1
       0xfed80068 0xfed50000 /lib/libelf.so.1

As you can see, the runtime linkers link-map list (LM_ID_LDSO) contains the runtime linker itself, and all the libraries that effectively make up the link-editor.

Besides loading all these support libraries, the shared object image adds additional overhead. Relocatable objects are typically non-pic, and thus require considerable more relocation than their pic (position independent) counterparts. This relocation overhead is passed on to the shared object memory image.

And, because the image is created in memory, each process that uses this technique creates it's own private image. There is no sharing of text pages from the shared object image, as is common for shared objects. This customers application forked off a number of processes that all dlopen()'ed the same relocatable object.

So, this technique may have some use within a development environment, but there are expenses:

  • the link-editor must be loaded into the process,

  • the relocatable object is then converted into a shared object memory image,

  • there are typically many more relocations to process in the memory image than there would be from a comparable pic shared object, and

  • the resulting shared object image isn't sharable between processes.

Personally, I'd recommend not using this technique in production software.

Join the discussion

Comments ( 7 )
  • Robert Charon Wednesday, February 2, 2005
    Couple of questions:

    1. What is a relocatable object? Specifically, is a relocatable object different from a shared object that is explicitly loaded using dlopen()?

    2. Why is sharing of text pages from the shared object image not allowed? I understand the part about the image created in memory; however, it's my understanding that the Windows loading of ActiveX components which are explicitly loaded DLL objects is highly efficient and that its text pages are shared among processes. What is it that Solaris is doing differently from Windows in this regard?

    3. If a shared object image is meant to be used for a single process, other than the loading and relocation expense per relocation object, what are the other expenses? In using this technique with multiple processes, shouldn't the runtime linker be loaded only once and its text pages shared across multiple processes since the runtime linker is a shared object?
  • Rod Evans Wednesday, February 2, 2005

    A relocatable object is output from the compilers.
    The link-editor typically combines a number of
    relocatable objects so as to create a dynamic
    executable or shared object.

    Dynamic executables and shared objects are mapped
    directly from the file system, thus their read-only
    text segments can be shared between multiple
    processes. This shareablity is common to many

    The subject of this posting is the translation of
    an input relocatable object into a shared
    object memory image. As the resultant shared object image is wriiten into memory and is
    not backed by any file, the image is private to
    the process that created it.

  • Robert Charon Wednesday, February 2, 2005
    Thanks much, Rod.

    The mention of a relocatable object in the context of dlopen() threw me off. I've wondered why relocatable files were mentioned in the specification for dlopen() but always attributed this to an unintentional specification error where they meant shared objects since I couldn't see any good reason for it. Learn something new every day.

    Thanks again.
  • Vidya Shankar M. Wednesday, May 11, 2005
    this is really very very informative, Thanks, I am facing the similar issue, the system depends on LD_LIBRARY_PATH and the shared objects have lot of relocations.
    We were thinking of using crle to create directory cache to avoid LD_LIBARARY_PATH issue and alternative objects to reduce relocations.
    Our tests showed improvement in startup times.
    However we find very limited documentation on crle with respect to alternative objects, (one in man page, other in linker and libraries guide)
    Any suggestions on using alternative objects.
  • eugenet Saturday, June 18, 2005

    To answer your 2nd question (6 months later :), all Windows DLLs (including ActiveX objects) are compiled as non-pic code. DLLs are supposed to be loaded at a base adddress which was specified at their compile time.

    If a DLL cannot be loaded at its preferred base address (i.e. if another DLL is already loaded at that address), Windows loader will load the DLL at a different address and will then repatch the text segment as per the new base address (addresses that need to be repatched are specified in a DLL's relocation segment).

    This is obviously not very efficient and Rod evaded to this in one of his posts.


    P.S. Windows executables do not contain relocation information (to save space) and thus cannot be relocated at load time.
  • Joey Tuesday, June 26, 2007
    Nice explanation, can you add an entry in your blog about possible ways to do shared library interposition, especially methods that do not rely on LD_\* variables.
  • Rod Wednesday, June 27, 2007

    Interposition occurs when the symbol in one library
    is found before a symbol of the same name in another
    library, during the runtime linkers symbol search.

    You can force interposition with the initial
    dependencies that are loaded by building the
    library with <code>-z interpose</code>.

Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.