Valgrind in Solaris
Valgrind is a framework and a set of tools for dynamic binary analysis of userspace programs. There are Valgrind tools that can automatically detect many memory management and threading bugs, avoiding hours of frustrating bug-hunting, making your programs more stable. You can also perform detailed profiling to help speed up your programs. Valgrind also supports (incremental) memory leak checking and many other advanced features. See its manual. In short: Valgrind is a debugging and profiling system for large, complex programs.
Recently a support for Oracle Solaris OS has been included in the latest Valgrind release 3.11.0. Let’s see it in action for a simple memory corruption problem.
Valgrind in Action
No system is bug free. There are always corner cases and edge conditions which developers have not thought of and testing did not cover. Let me demonstrate such a corner-case problem which exists in Oracle Solaris 11.2.
Create a local user, for example:
# useradd -d /export/home/tester -m -c "test user" -s /bin/bash tester
Setup the environment for problem reproduction:
# mkdir /foo
# cd /foo
# touch test_{1..32000}.txt
And simply watch sudo program crash:
# sudo chown tester test_*
Segmentation Fault
Now what’s going on? Before diving in with time-consuming coredump analysis, let’s try Valgrind if it readily gives us an answer:
# valgrind -q sudo chown tester test_*
valgrind: sudo: Permission denied
Hmm, sudo is a special setuid binary. Let’s copy it somewhere else and remove the setuid bit (it won’t be needed):
# cp /usr/bin/sudo /var/tmp/sudo
# chmod a-s /var/tmp/sudo
# valgrind -q sudo chown tester test_*
and voila’! Last few reported errors point to the right direction:
==6925== Invalid write of size 1
==6925== at 0x4F09FC9A: adr_short (in /lib/libbsm.so.1)
==6925== by 0x4F0A6ED0: au_to_cmd (in /lib/libbsm.so.1)
==6925== by 0x4F0A351D: adt_to_cmd (in /lib/libbsm.so.1)
==6925== by 0x4F0A336C: adt_generate_token (in /lib/libbsm.so.1)
==6925== by 0x4F0A2D5D: adt_generate_event (in /lib/libbsm.so.1)
==6925== by 0x4F0A2E6B: adt_put_event (in /lib/libbsm.so.1)
==6925== by 0x4F21FFA4: solaris_audit_success (in /usr/lib/sudo/sudoers.so)
==6925== by 0x4F229AA2: audit_success (in /usr/lib/sudo/sudoers.so)
==6925== by 0x4F21D80E: sudoers_policy_main (in /usr/lib/sudo/sudoers.so)
==6925== by 0x4F21A30F: sudoers_policy_check (in /usr/lib/sudo/sudoers.so)
==6925== by 0x8060A2C: main (in /var/tmp/sudo)
==6925== Address 0x4fbf1de4 is 16 bytes after a block of size 8,620 alloc'd
==6925== at 0x4FF8208A: malloc (vg_replace_malloc.c:310)
==6925== by 0x4F0A5F12: get_token (in /lib/libbsm.so.1)
==6925== by 0x4F0A6E4B: au_to_cmd (in /lib/libbsm.so.1)
==6925== by 0x4F0A351D: adt_to_cmd (in /lib/libbsm.so.1)
==6925== by 0x4F0A336C: adt_generate_token (in /lib/libbsm.so.1)
==6925== by 0x4F0A2D5D: adt_generate_event (in /lib/libbsm.so.1)
==6925== by 0x4F0A2E6B: adt_put_event (in /lib/libbsm.so.1)
==6925== by 0x4F21FFA4: solaris_audit_success (in /usr/lib/sudo/sudoers.so)
==6925== by 0x4F229AA2: audit_success (in /usr/lib/sudo/sudoers.so)
==6925== by 0x4F21D80E: sudoers_policy_main (in /usr/lib/sudo/sudoers.so)
==6925== by 0x4F21A30F: sudoers_policy_check (in /usr/lib/sudo/sudoers.so)
==6925== by 0x8060A2C: main (in /var/tmp/sudo)
Valgrind is telling us there is a buffer overrun and the program writes past the allocated buffer. Our focus now turns to function au_to_cmd() in libbsm.so.1 and spots immediately the problem there.
Limitations
Support for Oracle Solaris OS in Valgrind is currently limited to x86 (32-bit) and amd64 (64-bit) architectures only. We are working on sparcv9 architecture support but there is still much work ahead.
Valgrind is a dynamic binary analysis tool. It does not need access to the program source code. And it cannot spot problems in a code path which does not get run.
About the Author
Ivo Raisr works as a software engineer at Oracle, Systems. Porting of Valgrind to Solaris OS belongs to one of his most favorite hobbies. He believes that if it is useful for the Oracle engineering then must be also for partners, ISVs and customers.