Changing ELF Runpaths (Code Included)
By Ali Bahrami-Oracle on Jun 12, 2007
This seems like a good time to talk about runpaths and the business of how the runtime linker finds dependencies. I also provide a small program named rpath that you can use to modify the runpaths in your file (assuming they were linked under Nevada or OpenSolaris).
The Runpath ProblemThe runtime linker looks in the following places, in the order listed, to find the sharable objects it loads into a process at startup time:
- If LD_LIBRARY_PATH (or the related LD_LIBRARY_PATH_32 and LD_LIBRARY_PATH_64) environment variables are defined, the directories they specify are searched to resolve any remaining dependencies.
- If the executable, or any sharable objects that are loaded, contain a runpath, the directories it specifies are searched to resolve dependencies for those objects.
- Finally, it searches two default locations for any remaining dependencies: /lib and /usr/lib (or /lib/64 and /usr/lib/64 for 64-bit code).
The above scheme offers a great deal of flexibility, and it usually works well. There is however one notable exception the "Runpath Problem". The problem is that many objects are not built with a correct runpath, and once an object has been built, it has not been possible to change it. It is common to find objects where the runpath is correct on the system the object was built on, but not on the system where it is installed. Usually, we deal with this all too common situation by setting LD_LIBRARY_PATH, or by creating a linker configuration file with crle. Such solutions have serious downsides, as detailed in an earlier blog entry by Rod Evans entitled "LD_LIBRARY_PATH - just say no".
Both approaches will cause unrelated programs to look in unnecessary additional directories for their dependencies. At best, this imposes unnecessary overhead on their operation. At worst, they may end up binding to the wrong version of a given library, leading to mysterious and hard to debug failures. The environment variable approach is simply too broad.
One important technique that people sometimes use, is to set the environment variables in a wrapper shell script, that may look something like:
This is a huge improvement over simply setting LD_CONFIG or LD_LIBRARY_PATH in your shell login config script (.profile, .cshrc, .bashrc, etc), for many reasons:#!/bin/sh # # Run myapp, setting LD_LIBRARY_PATH so it will run LD_LIBRARY_PATH="/this/that/theother:/someplace/else" export LD_LIBRARY_PATH exec /usr/local/myapp
- Reduces the scope of influence to only cover the application (and its children see below)
- Doesn't require each user to modify their login script(s)
- Can be managed in a central location
It would be far better to modify the object in question and set a runpath that accurately reflects the actual location of its dependencies. The effect of a runpath is limited to the file that contains it, so this solution does not "bleed through" to unrelated files, and it imposes no unnecessary overhead on the general operation of the system. This would be a superior solution if it were possible. However it hasn't been an option until recently.
How Runpaths Are ImplementedEvery dynamic executable contains a dynamic section. This is an array of items which convey the information required by the dynamic linker (ld.so.1) to do its work. If an object has a runpath, there will be a DT_RUNPATH and/or DT_RPATH item in the dynamic section (there is more than one of these for historical reasons). As an example, lets examine crle:
The string (in this case, "$ORIGIN/../lib") is not actually stored in the dynamic section. Rather, it is contained in the dynamic string table (.dynstr). The value 0x612 is the offset within string table at which the desired string starts.% elfdump -d /usr/bin/crle | grep 'R*PATH'  RUNPATH 0x612 $ORIGIN/../lib  RPATH 0x612 $ORIGIN/../lib
A string table is a section that contains NULL terminated strings, one immediately following the other. To access a given string, you add the offset of the string within the section to the base of the section data area. Consider a string table that contains the names of two variables "var1", and "var2" and a runpath "$ORIGIN/../lib". By ELF convention, string tables always have a 0-length NULL terminated string in the first position. In C language notation, we might declare the contents of the resulting string table section containing these 4 strings as
The indexes of the 4 strings in our table are , , , and , and any item in the dynamic section or the dynamic symbol table that needs one of these strings will specify it using the appropriate index. An interesting result of the way that string tables are designed is that that every single offset into a string table represents a usable string. Although our intent with the C string above was to represent 4 strings, it actually contains 23 potential strings (26 if you count the duplicate NULL strings), and not just the 4 we intentionally inserted. Listing them by offset, they are:"\\0var1\\0var2\\0$ORIGIN/../lib"
This is a very efficient scheme, since each string can appear once in the string table, and multiple ELF items can refer to it. Also, it allows fixed size things, like ELF symbols or dynamic section entries, to efficiently reference variable length strings. There are two things to note, however: ""  "var1"  "ar1"  "r1"  "1"  ""  "var2"  "ar2"  "r2"  "2"  ""  "$ORIGIN/../lib"  "ORIGIN/../lib"  "RIGIN/../lib"  "IGIN/../lib"  "GIN/../lib"  "IN/../lib"  "N/../lib"  "/../lib"  "../lib"  "./lib"  "/lib"  "lib"  "ib"  "b"  ""
- For a given string, there is no way to tell if it is referenced, where it is referenced from, or how many references there are.
- There is no room to add new strings to a string table.
The options for modifying a runpath in this situation are limited:
- Any string already in the string table, as with the 23 options listed in our example above, can be safely set as a runpath, by simply changing the offset in the runpath DT entries. Note that most string table strings are variable and file (not directory) names that are not likely to make useful path strings. This option is unlikely to help.
- You might overwrite the existing path string with a new string of equal or shorter length in the (usually true) belief that nothing else is accessing that particular string. It is a simple matter using a binary aware editor to locate and overwrite the existing string. This usually works, but if there is another part of the file accessing that string, this change will break it. We cannot recommend or stand behind this, even though we have done it ourselves for one-off experiments (never in a shipping product).
- A better approach would be to add a new string to the end of the section, and then change the offset in the dynamic section to use it. Traditionally, ELF files have not had any extra room in the string table section to allow this.
As a result, it has not been possible to support the modification of the runpath in an existing object up until recently.
Making RoomI recently integrated a change to Solaris Nevada (and OpenSolaris) to add a little unused space to our ELF files, in order to facilitate a limited amount of post-link modification:
This change does two things:PSARC 2007/127 Reserved space for editing ELF dynamic sections 6516118 Reserved space needed in ELF dynamic section and string table
- Adds some extra NULL bytes to the end of every dynamic string table. (The current value is 512 bytes, but this can change in the future).
- Adds a new dynamic section entry named DT_SUNW_STRPAD to keep track of the size of the unused space at the end of the dynamic string table.
- Adds some extra (currently 10) unused DT_NULL entries at the end of the dynamic section.
The SUNW_STRPAD entry tells us that the dynamic string table has 512 (0x200) bytes of unused space available at the end of its data area.% elfdump -d /usr/bin/crle | egrep 'R*PATH|STRPAD'  RUNPATH 0x612 $ORIGIN/../lib  RPATH 0x612 $ORIGIN/../lib  SUNW_STRPAD 0x200
The way this works is very simple: If a file lacks a DT_SUNW_STRPAD dynamic entry, then we know that it is an older file, and that the dynamic string table does not have any extra space. If it does have a DT_SUNW_STRPAD, then its value tells us how much room is available. In this case, we can add the string, modify the DT_RUNPATH items, and reduce the DT_SUNW_STRPAD value by the number of bytes we used.
If the value in DT_SUNW_STRPAD is too small for our new string, then we are out of luck and cannot add it. This extra room should help in the vast majority of cases, but as with any such approach, there are limits. We recommend the use of the special $ORIGIN token, both because it is a great way to organize objects, and because it is short.
The rpath UtilityEventually, Solaris will ship with a standard utility for modifying runpaths. However, there is no need to wait. I have written an unofficial test program I call 'rpath' that you can download and build. To build rpath, you will need a version of Solaris Nevada newer than build 61, or a recent version of OpenSolaris. To check your system, try:
If your grep doesn't find DT_SUNW_STRPAD, your system lacks the necessary support.% grep DT_SUNW_STRPAD /usr/include/sys/link.h #define DT_SUNW_STRPAD 0x60000019 /* # of unused bytes at the */
To build rpath, unpack the compressed tar file and type 'make'. If you are using gcc, first edit the Makefile and uncomment the CC line:
rpath is used as follows:% gunzip < rpath.tgz | tar xvpf - % cd rpath % make
NAME rpath - set/get runpath of ELF dynamic objects SYNOPSIS rpath [-dr] file [runpath] DESCRIPTION rpath can display, modify, or delete the runpath of a dynamic ELF object. If called without a runpath argument and without the -r option, the current runpath, if any, is written to stdout. If -r is specified, the existing runpath is removed. If run- path is supplied, the runpath of the object is set to the new value. OPTIONS The following options are supported: -d Cause detailed ELF information about the ELF file and the changes being made to it to be written to stderr. -r Instead of adding or modifying the file runpath, rpath removes any DT_RPATH or DT_RUNPATH entries from the dynamic section of the file. This action completely removes any existing from the file. When this option is used, rpath does not allow the runpath argument.
Using rpathLet's use rpath to look at its own runpath. We will see that it doesn't have one, something that can be verified using elfdump:
Now, let's add a runpath to it:% rpath rpath % elfdump -d rpath | egrep 'R*PATH|STRPAD'  SUNW_STRPAD 0x200
Notice that the amount of unused space reported by SUNW_STRPAD has gone down from 512 (0x200) to 494 (0x1ee) bytes, a reduction of 18 bytes. This makes sense, since we added a 17 character string, and we must add a NULL termination.% rpath rpath pointless:runpath % rpath rpath pointless:runpath % elfdump -d rpath | egrep 'R*PATH|STRPAD'  SUNW_STRPAD 0x1ee  RUNPATH 0x33f pointless:runpath
We can observe the runtime linker looking in 'pointless' and 'runpath' as it loads rpath (note: output is edited for width):
Finally, we'll remove the runpath we just added:% LD_DEBUG=libs ./rpath 13707: 13707: hardware capabilities - 0x25ff7 [ AHF SSE3 SSE2 SSE FXSR AMD_3DNowx AMD_3DNow AMD_MMX MMX CMOV AMD_SYSC CX8 TSC FPU ] 13707: 13707: 13707: configuration file=/var/ld/ld.config: unable to process file 13707: 13707: 13707: find object=libelf.so.1; searching 13707: search path=pointless:runpath (RUNPATH/RPATH from file rpath) 13707: trying path=pointless/libelf.so.1 13707: trying path=runpath/libelf.so.1 13707: search path=/lib (default) 13707: search path=/usr/lib (default) 13707: trying path=/lib/libelf.so.1 13707: 13707: find object=libc.so.1; searching 13707: search path=pointless:runpath (RUNPATH/RPATH from file rpath) 13707: trying path=pointless/libc.so.1 13707: trying path=runpath/libc.so.1 13707: search path=/lib (default) 13707: search path=/usr/lib (default) 13707: trying path=/lib/libc.so.1 13707: 13707: find object=libc.so.1; searching 13707: search path=/lib (default) 13707: search path=/usr/lib (default) 13707: trying path=/lib/libc.so.1 13707: 13707: 1: 13707: 1: transferring control: rpath 13707: 1: usage: rpath [-dr] file [runpath] 13707: 1:
Note that even though the runpath is gone, the amount of available extra space in the dynamic string section did not go back up from 494 (0x1ee) to 512 (0x200). Adding strings is a one way operation. Once they are added, they are permanent. So even though you now have the ability to add strings of moderate length, you won't want to do it indiscriminately.% rpath -r rpath % rpath rpath % elfdump -d rpath | egrep 'R*PATH|STRPAD'  SUNW_STRPAD 0x1ee
On the plus side, you can always re-add the same runpath back without using any more space:
rpath found that the string 'pointless:runpath' was already in the string table, so it used it without inserting another copy.% rpath rpath pointless:runpath % rpath rpath pointless:runpath % elfdump -d rpath | egrep 'R*PATH|STRPAD'  SUNW_STRPAD 0x1ee  RUNPATH 0x33f pointless:runpath
ConclusionsOur best advice has always been that the LD_LIBRARY_PATH environment variable should not be used to work around objects with bad or missing runpaths. It is best to rebuild such objects and set the runpath correctly. This hasn't changed, and you should always do so if you can.
The problem with that advice is that there are times when all you have is the object, and no option to rebuild. In that case, LD_LIBRARY_PATH has been a necessary evil (and one that we've been glad to have). With the advent of objects that can have their runpaths modified, we now have a better answer, and the use of LD_LIBRARY_PATH for this purpose should be allowed to slowly fade away.