Debugging shared library problems: a real-world example

There are few things I am less happy to see when trying to get some work done than something like:
$ python
>>> import pycurl
Fatal Python error: pycurl: libcurl link-time version is older than compile-time version
Aborted
"link-time" + "compile-time" + one library using another and something is broken = some kind of shared library problem.
Debugging shared library problems = sad jesstess

Nooooo I just wanted to write some code.*

But I don't have anyone to pawn this off on, and this isn't magic, it's just annoying, so how about trying to fix it.

Step 1: find out where the offended shared library lives

The error message says that pycurl is having some issue with libcurl. So let's find out where the pycurl shared library lives. First, get the actual package name for the pycurl software you're using. Then, get a list of all the files in that package and look for the location of the pycurl shared library:

rpm-based

$ rpm -qa | grep pycurl
python-pycurl-7.15.5.1-4.1.el4
$ rpm -ql python-pycurl-7.15.5.1-4.1.el4 | grep pycurl.so
/usr/lib/python2.5/site-packages/pycurl.so
dpkg-based
$ apt-cache search pycurl
python-pycurl - Python bindings to libcurl
$ dpkg -L python-pycurl | grep pycurl.so
/usr/lib/python2.5/site-packages/pycurl.so
I did just assume that the pycurl shared library would be called something like "pycurl.so”, which is not so unreasonable. To take the guesswork out of getting the name of the shared library, we could have taken advantage of python's -v flag. It says to, on module initialization, print the place from which the module is loaded:
$ python -v
…
>>> import pycurl
dlopen("/usr/lib/python2.5/site-packages/pycurl.so", 2);
Fatal Python error: pycurl: libcurl link-time version is older than compile-time version
Aborted
Either way, we have a pycurl.so in /usr/lib/python2.5/site-packages/.

Step 2: find the offended shared library's dependencies

ldd is a super-nifty tool whose job is to print out the shared library dependencies for a program or shared library, which is exactly what we want:
$ ldd /usr/lib/python2.5/site-packages/pycurl.so
       libcurl.so.3 => /lib/libcurl.so.3 (0x0000002a95693000)
       libdl.so.2 => /lib/libdl.so.2 (0x0000002a957c7000)
       libgssapi_krb5.so.2 => /lib/libgssapi_krb5.so.2 (0x0000002a958ca000)
       libkrb5.so.3 => /lib/libkrb5.so.3 (0x0000002a959e0000)
       libk5crypto.so.3 => /lib/libk5crypto.so.3 (0x0000002a95b52000)
       libcom_err.so.2 => /lib/libcom_err.so.2 (0x0000002a95c75000)
       libresolv.so.2 => /lib/libresolv.so.2 (0x0000002a95d77000)
       libidn.so.11 => /lib/libidn.so.11 (0x0000002a95e8d000)
       libssl.so.4 => /lib/libssl.so.4 (0x0000002a95fbe000)
       libcrypto.so.4 => /lib/libcrypto.so.4 (0x0000002a960fa000)
       libz.so.1 => /lib/libz.so.1 (0x0000002a9632c000)
       libpthread.so.0 => /lib/tls/libpthread.so.0 (0x0000002a9643f000)
       libc.so.6 => /lib/tls/libc.so.6 (0x0000002a96554000)
       /lib/ld-linux-x86-64.so.2 (0x000000552aaaa000)
So pycurl.so uses a libcurl shared library called libcurl.so.3, which lives in /lib.

Step 3: find out where the offending shared library should live

First, get the name of the package that installs libcurl.so.3. Then, see where the libcurl.so.3 from that package lives:

rpm-based

$ rpm -qa | grep curl
...
curl-7.15.5-2.1.el5_3.5
...
$ rpm -ql curl-7.15.5-2.1.el5_3.5 | grep libcurl.so.3
/usr/lib/libcurl.so.3
dpkg-based
$ apt-cache search curl
...
libcurl3 - Multi-protocol file transfer library (OpenSSL)
...
$ dpkg -L libcurl3 | grep libcurl.so.3
/usr/lib/libcurl.so.3

Well there's at least one problem: curl/libcurl3 puts libcurl.so.3 in /usr/lib, but pycurl.so is using a version in /lib. Sure enough, an ls in /usr/lib reveals a second libcurl.so.3.

Okay, so there are two versions. Why is the bad version the one getting used?

Step 4: Check the order in which directories are searched for shared libraries

If we read the man page for ld.so, the thing that does the actual loading of shared libraries, we see:
The necessary shared libraries needed by the program are searched for in the following order
  • Using the environment variable LD_LIBRARY_PATH (LD_AOUT_LIBRARY_PATH for a.out programs). Except if the executable is a setuid/setgid binary, in which case it is ignored.
  • From the cache file /etc/ld.so.cache which contains a compiled list of candidate libraries previously found in the augmented library path.
  • In the default path /lib, and then /usr/lib.

Alright, let's check these:

1. Check LD_LIBRARY_PATH:

$ echo $LD_LIBRARY_PATH
$ 

So LD_LIBRARY_PATH is unset or empty. Moving on:

2. Check /etc/ld.so.cache:

ldconfig will do the work of parsing ld.so.cache and printing the paths in order for us:

$ /sbin/ldconfig --print-cache | grep libcurl.so.3
       libcurl.so.3 (libc6) => /lib/libcurl.so.3
       libcurl.so.3 (libc6) => /usr/lib/libcurl.so.3
Bam. /lib/libcurl.so.3 comes before /usr/lib/libcurl.so.3 (because at some point these entries weren't in the cache, and as we see in the man page, when ld.so is down to default paths, /lib gets checked before /usr/lib).

Step 5: Fix the problem

Now, we don't want to accidentally break some other programs that expect the version of libcurl.so.3 in /lib, so we should check what package, if any, that file is in:

rpm-based

$ rpm -q -f /lib/libcurl.so.3
file /lib/libcurl.so.3 is not owned by any package
dpkg-based
$ dpkg -S /lib/libcurl.so.3
file /lib/libcurl.so.3 is not owned by any package
Revealing that /lib/libcurl.so.3 is an orphan we can pretty safely delete. Once it's been deleted, pycurl.so finds the correct libcurl.so.3 in /usr/lib, and our original python import works:
$ python
>>> import pycurl
>>>
Yay. Back to work.

* Image courtesy of Allie Brosh over at Hyperbole and a Half.

~jesstess
Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

Tired of rebooting to update systems? So are we -- which is why we invented Ksplice, technology that lets you update the Linux kernel without rebooting. It's currently available as part of Oracle Linux Premier Support, Fedora, and Ubuntu desktop. This blog is our place to ramble about technical topics that we (and hopefully you) think are interesting.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today