Library Bindings - let's be a little bit more precise shall we

Library Bindings - let's be a little bit more precise shall we

Library Bindings - let's be a little bit more precise shall we

Well - now that OpenSolaris is officially open for business I think its well past time for another Blog entry from me. This blog entry will give a little history on how a executable (or shared objects) on Solaris binds to its various dependencies. My goal here is to give some insight into what we do today, as well has hi-light a alternative binding technology which you may or may not be familiar with, Direct Bindings with ld -Bdirect. The Direct Bindings model is a much more precise model when resolving reference between objects.

But first a little history on what we currently do. Solaris (and \*nix's in general) does the following when a process is executed. The kernel will load the required program (a ELF object) into memory and also load the runtime linker ( ld.so.1(1) ) into memory. The kernel then transfers control initially to the runtime linker. It's the runtime linkers job to examine the. program loaded and find any dependencies it has (in the form of a shared object), load those shared objects into memory, and then bind all of the symbol bindings (function calls, data references, etc...) from the program to each of those dependencies. Of coarse, as it loads each shared object it must in turn do the same examination on each of them and load any dependencies they require. Once all of the dependencies are loaded and their symbols have been bound - the runtime linker will fire the .init sections for each shared object loaded and finally transfer control to the executable, which calls main(). Most people think a process starts with main() but amazing things happen before we even get there.

Here we will specifically look at how the runtime linker binds the various symbol reference between all of the objects loaded into memory. Let's take a simple example first - how about a application which links against a couple of shared objects and then libc.

    % more \*.c
    ::::::::::::::
    bar.c
    ::::::::::::::
    #include 

    void bar()
    {
	printf("inside of bar\\n");
    }
    ::::::::::::::
    foo.c
    ::::::::::::::
    #include 

    void foo() {
	printf("inside of foo\\n");
    }
    ::::::::::::::
    prog.c
    ::::::::::::::
    #include 

    int
    main(int argc, char \*argv[]){
	extern void foo();
	extern void bar();
	foo();
	bar();
	return (0);
    }
    % cc    -G -o foo.so -Kpic foo.c -lc
    % cc    -G -o bar.so -Kpic bar.c -lc
    % cc    -o prog prog.c ./foo.so ./bar.so
We've now got a program, prog, which is bound against three shared objects, foo.so, bar.so and libc.so. The program makes two function calls, one to foo() and one to bar() located in it's dependent shared objects, by ldd'ing the executable we can see it's dependencies and a run of it shows the execution flow:
    % ldd prog
	    ./foo.so =>      ./foo.so
	    ./bar.so =>      ./bar.so
	    libc.so.1 =>     /lib/libc.so.1
	    libm.so.2 =>     /lib/libm.so.2
	    /platform/SUNW,Sun-Blade-1000/lib/libc_psr.so.1
    % ./prog
    inside of foo
    inside of bar
    %
Nothing too fancy really - but it's an example we can use to examine what bindings are going on. First - when the program prog makes reference to foo and bar - it's up to the runtime linker to find definitions for these functions and bind the program to them. First the runtime linker will load in the dependent shared objects (listed above) - as the objects are loaded into memory we create a Link Map entry for each object, the objects are appended onto a Link Map list in the order that they are loaded. In the case above the Link Map list would contain:
    prog -> foo.so -> bar.so ->libc.so.1 -> libm.so.2 -> libc_psr.so.1
When the runtime linker needs to find a definition for a symbol it starts at the head of the list and will search each object for that symbol. If it's found, it binds to that symbol - if it's not found it proceeds to the next object on the list. The following should help demonstrate what's happening. I will run the prog program, but with some runtime linker diagnostics turned on to trace what it is doing. I'm concentrating specifically on foo and bar for this example - of course there are thousands of other bindings going on:
    % LD_DEBUG=symbols,bindings ./prog
    ...
    20579: 1: symbol=foo;  lookup in file=./prog  [ ELF ]
    20579: 1: symbol=foo;  lookup in file=./foo.so  [ ELF ]
    20579: 1: binding file=./prog to file=./foo.so: symbol `foo'
    ...
    20579: 1: symbol=bar;  lookup in file=./prog  [ ELF ]
    20579: 1: symbol=bar;  lookup in file=./foo.so  [ ELF ]
    20579: 1: symbol=bar;  lookup in file=./bar.so  [ ELF ]
    20579: 1: binding file=./prog to file=./bar.so: symbol `bar'
    ...
Not so bad really, but it's really not the most efficient way to find a symbol is it. When we were looking for the symbol bar we had to go through 3 objects until we found it. Now imagine what happens when you have a more complex application which has many more shared objects with much larger symbol tables. If I look at firefox - I can see that has over 50 shared objects loaded:
    % pldd `pgrep firefox-bin`
    28294:  /disk3/local/firefox/firefox-bin /lib/libpthread.so.1
    /lib/libthread.so.1
    /lib/libc.so.1
    /disk3/local/firefox/libmozjs.so
    /disk3/local/firefox/libxpcom.so
    /usr/sfw/lib/libgtk-1.2.so.0.9.1
    /usr/sfw/lib/libgmodule-1.2.so.0.0.10
    /usr/sfw/lib/libglib-1.2.so.0.0.10
    /usr/openwin/lib/libXext.so.0
    /usr/openwin/lib/libX11.so.4
    /lib/libsocket.so.1
    /lib/libnsl.so.1
    /lib/libm.so.2
    /usr/sfw/lib/libgdk-1.2.so.0.9.1
    /disk3/local/firefox/libssl3.so
    /disk3/local/firefox/libnss3.so
    /disk3/local/firefox/libplc4.so
    /disk3/local/firefox/libplds4.so
    /disk3/local/firefox/libnspr4.so
    /disk3/local/firefox/libsoftokn3.so
    /lib/librt.so.1
    /lib/libdl.so.1
    /lib/libaio.so.1
    /lib/libmd5.so.1
    /usr/openwin/lib/libXt.so.4
    /platform/sun4u-us3/lib/libc_psr.so.1
    /usr/lib/libCrun.so.1
    /usr/lib/libdemangle.so.1
    /disk3/local/firefox/cpu/sparcv8plus/libnspr_flt4.so
    /lib/libm.so.1
    /disk3/local/firefox/libsmime3.so
    /usr/openwin/lib/libXp.so.1
    /disk3/local/firefox/libxpcom_compat.so
    /usr/lib/libCstd.so.1
    /usr/lib/cpu/sparcv8plus/libCstd_isa.so.1
    /lib/libw.so.1
    /lib/libmp.so.2
    /lib/libscf.so.1
    /lib/libuutil.so.1
    /usr/openwin/lib/libSM.so.6
    /usr/openwin/lib/libICE.so.6
    /usr/lib/iconv/646%UTF-16BE.so
    /usr/lib/iconv/UTF-16BE%646.so
    /usr/jdk/instances/jdk1.5.0/jre/plugin/sparc/ns7/libjavaplugin_oji.so
    /platform/sun4u/lib/libmd5_psr.so.1
    /usr/jdk/instances/jdk1.5.0/jre/lib/sparc/libjavaplugin_nscp.so
    /disk3/local/firefox/components/libjar50.so
    /usr/dt/lib/libXm.so.4
    /disk3/local/firefox/libfreebl_hybrid_3.so
    /usr/sfw/lib/mozilla/libnssckbi.so
    %
And on average - each of those objects have symbol tables with over 2,500 symbols. Doing a linear search at the beginning of each link-map list until you find the symbol just doesn't seem that practical anymore. Firefox is average for modern applications these days - if you were to take a look at Star Office you would find a single program which depends upon over 90 different shared objects.

There's got to be a better way, right? There is - we call it direct bindings. Instead of doing the linear search at runtime you can simply ask the link-editor to record not only what shared objects you bound against - but what symbols you obtained from each shared object. So, if you are bound with Direct Bindings, the runtime linker changes how it looks up symbol bindings and instead will bind directly to the object that offered the symbol at runtime. A much more efficient model - here's the same prog, but this time built with direct bindings, this is done by passing the -Bdirect link-editor option on the link-line:

    % cc -Bdirect -o prog prog.c ./foo.so ./bar.so
When you link with -Bdirect the link-editor will store additional information in a object including where each symbol was seen at link time. This can be viewed with elfdump as follows:
    % elfdump -y prog

    Syminfo Section:  .SUNW_syminfo
	 index  flgs         bound to           symbol
	  ...
	  [15]  DBL      [1] ./foo.so           foo
	  [19]  DBL      [3] ./bar.so           bar
	  ...
    %
If we do the same experiment we did earlier, that of running the program and examining the actual bindings that the runtime linker is doing - we will see a much more efficient search:
    % LD_DEBUG=symbols,bindings ./prog
    ...
    20728: 1: symbol=foo;  lookup in file=./foo.so  [ ELF ]
    20728: 1: binding file=./prog to file=./foo.so: symbol `foo'
    ...
    20728: 1: symbol=bar;  lookup in file=./bar.so  [ ELF ]
    20728: 1: binding file=./prog to file=./bar.so: symbol `bar'
    ...
    %
Notice we now find each symbol in the first object we look in, much better.

This Direct Bindings has been in Solaris for a few releases now, although because it's not the default not everyone is familiar with it. It has matured quite a bit over the last few years and we are now starting to use it for some of our core shared objects. If you look at the X11 shared objects delivered with Solaris - you'll find that they are bound with direct bindings:

    % elfdump -y /usr/lib/libX11.so | head

    Syminfo Section:  .SUNW_syminfo
         index  flgs         bound to           symbol
           [1]  D            <self>             _XimXTransDisconnect
           [2]  D        [8] libc.so.1          snprintf
           [3]  D            <self>             _XcmsFreeIntensityMaps
           [4]  D            <self>             _XcmsTableSearch
           [5]  D            <self>             _XDeq
	   [6]  D            <self>             XGetWMSizeHints
	   [7]  D            <self>             XUnmapWindow
    %
Besides the fact that Direct Bindings are more efficient, they are also much more precise. It can get very tricky to control the name space when you start to combine all of the shared objects that you see in new modern applications. If two shared objects happen to offer a symbol of the same name (not by intention) using the default binding lookup - we'll bind to the first one found, which is probably not what the user intends. If - however we bind to exactly the version that was found at the time the object was built, there will be many fewer surprises.

Along these lines - it's worth giving a cautionary note for those re-linking their existing Applications with Direct Bindings enabled. As we apply Direct Bindings to more and more applications we have found a few cases where there are multiple definitions of a single symbol, by changing the binding model you can change the behavior of the application. In most, if not all cases, this was a bug in the design of the application - but a program can become dependent upon this and result in a failure of the application when run with Direct Bindings.

Further details on Direct Bindings specifically and the runtime linker (ld.so.1(1)) and link-editor (ld(1)) in general can be found in the Linker and Libraries Guide which is part of the standard Solaris Documentation.

Examples of tracing what the runtime linker is doing can found in a Blog entry by Rod here titled Tracing a link-edit.

Technorati Tag:
Technorati Tag:

Comments:

Thanks! Excellent piece of information for a solaris application developer like me. I remember getting baffled by a coredump in one of our applications even before main() was hit. (We finally resolved it by using the LD_PRELOAD environment variable).

Posted by Sanjeev on qershor 27, 2005 at 07:54 MD PDT #

This is pretty good. Finally we have choice to link/bind shared libraries differently. I hope gcc and HP's aCC will support this feature too. I think IBM's visual age compiler and linker does this direct binding by default, but can do dynamic binding by using -brtl linker option.

Posted by Michael on qershor 28, 2005 at 01:00 PD PDT #

Does usage of -Bdirect prevent using LD_PRELOAD to substitute system calls?

Posted by Jared Robinson on nëntor 09, 2005 at 12:48 PD PST #

Hi, Thanks very much for sharing such good peice of information. It really helped me in understanding the process of dynamic binding and loading. Thanks to sanjeev for guiding me to consult your Weblog.

Posted by Venkateswara Rao Katuri on qershor 15, 2006 at 10:44 MD PDT #

Post a Comment:
Comments are closed for this entry.
About

msw

Search

Categories
Archives
« prill 2014
DieHënMarMërEnjPreSht
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today