Debugging shared library problems: a real-world example
By Ksplice Post Importer on Sep 08, 2010
$ python >>> import pycurl Fatal Python error: pycurl: libcurl link-time version is older than compile-time version Aborted"link-time" + "compile-time" + one library using another and something is broken = some kind of shared library problem.
|Debugging shared library problems||=|
Nooooo I just wanted to write some code.*
Step 1: find out where the offended shared library lives
The error message says that
pycurl is having some issue with
libcurl. So let's find out where the pycurl shared library lives. First, get the actual package name for the pycurl software you're using. Then, get a list of all the files in that package and look for the location of the pycurl shared library:
$ rpm -qa | grep pycurl python-pycurl-220.127.116.11-4.1.el4
$ rpm -ql python-pycurl-18.104.22.168-4.1.el4 | grep pycurl.so /usr/lib/python2.5/site-packages/pycurl.sodpkg-based
$ apt-cache search pycurl python-pycurl - Python bindings to libcurl
$ dpkg -L python-pycurl | grep pycurl.so /usr/lib/python2.5/site-packages/pycurl.soI did just assume that the pycurl shared library would be called something like "
pycurl.so”, which is not so unreasonable. To take the guesswork out of getting the name of the shared library, we could have taken advantage of python's
-vflag. It says to, on module initialization, print the place from which the module is loaded:
$ python -v … >>> import pycurl dlopen("/usr/lib/python2.5/site-packages/pycurl.so", 2); Fatal Python error: pycurl: libcurl link-time version is older than compile-time version AbortedEither way, we have a
Step 2: find the offended shared library's dependencies
lddis a super-nifty tool whose job is to print out the shared library dependencies for a program or shared library, which is exactly what we want:
$ ldd /usr/lib/python2.5/site-packages/pycurl.so libcurl.so.3 => /lib/libcurl.so.3 (0x0000002a95693000) libdl.so.2 => /lib/libdl.so.2 (0x0000002a957c7000) libgssapi_krb5.so.2 => /lib/libgssapi_krb5.so.2 (0x0000002a958ca000) libkrb5.so.3 => /lib/libkrb5.so.3 (0x0000002a959e0000) libk5crypto.so.3 => /lib/libk5crypto.so.3 (0x0000002a95b52000) libcom_err.so.2 => /lib/libcom_err.so.2 (0x0000002a95c75000) libresolv.so.2 => /lib/libresolv.so.2 (0x0000002a95d77000) libidn.so.11 => /lib/libidn.so.11 (0x0000002a95e8d000) libssl.so.4 => /lib/libssl.so.4 (0x0000002a95fbe000) libcrypto.so.4 => /lib/libcrypto.so.4 (0x0000002a960fa000) libz.so.1 => /lib/libz.so.1 (0x0000002a9632c000) libpthread.so.0 => /lib/tls/libpthread.so.0 (0x0000002a9643f000) libc.so.6 => /lib/tls/libc.so.6 (0x0000002a96554000) /lib/ld-linux-x86-64.so.2 (0x000000552aaaa000)So
pycurl.souses a libcurl shared library called
libcurl.so.3, which lives in
Step 3: find out where the offending shared library should live
First, get the name of the package that installs
libcurl.so.3. Then, see where the
libcurl.so.3 from that package lives:
$ rpm -qa | grep curl ... curl-7.15.5-2.1.el5_3.5 ...
$ rpm -ql curl-7.15.5-2.1.el5_3.5 | grep libcurl.so.3 /usr/lib/libcurl.so.3dpkg-based
$ apt-cache search curl ... libcurl3 - Multi-protocol file transfer library (OpenSSL) ...
$ dpkg -L libcurl3 | grep libcurl.so.3 /usr/lib/libcurl.so.3
Well there's at least one problem:
pycurl.so is using a version in
/lib. Sure enough, an
/usr/lib reveals a second
Okay, so there are two versions. Why is the bad version the one getting used?
Step 4: Check the order in which directories are searched for shared librariesIf we read the man page for ld.so, the thing that does the actual loading of shared libraries, we see:
The necessary shared libraries needed by the program are searched for in the following order
- Using the environment variable
a.outprograms). Except if the executable is a setuid/setgid binary, in which case it is ignored.
- From the cache file
/etc/ld.so.cachewhich contains a compiled list of candidate libraries previously found in the augmented library path.
- In the default path
/lib, and then
Alright, let's check these:
$ echo $LD_LIBRARY_PATH $
LD_LIBRARY_PATH is unset or empty. Moving on:
ldconfig will do the work of parsing
ld.so.cache and printing the paths in order for us:
$ /sbin/ldconfig --print-cache | grep libcurl.so.3 libcurl.so.3 (libc6) => /lib/libcurl.so.3 libcurl.so.3 (libc6) => /usr/lib/libcurl.so.3Bam.
/usr/lib/libcurl.so.3(because at some point these entries weren't in the cache, and as we see in the man page, when
ld.sois down to default paths,
/libgets checked before
Step 5: Fix the problem
Now, we don't want to accidentally break some other programs that expect the version of
/lib, so we should check what package, if any, that file is in:
$ rpm -q -f /lib/libcurl.so.3 file /lib/libcurl.so.3 is not owned by any packagedpkg-based
$ dpkg -S /lib/libcurl.so.3 file /lib/libcurl.so.3 is not owned by any packageRevealing that
/lib/libcurl.so.3is an orphan we can pretty safely delete. Once it's been deleted,
pycurl.sofinds the correct
/usr/lib, and our original python import works:
$ python >>> import pycurl >>>Yay. Back to work.