Coverage testing

A couple years back, I wrote up a description of how to use the Sun Studio compiler's coverage testing features to test userland code. Now that OpenSolaris is here, I thought it might come in handy for a larger audience. Here's goes:

How do I do coverage analysis on my user-level code?

The Sun Workshop compilers we use have some pretty good profiling and
test analysis tools built in to them. One of the more useful for
user-space code is Coverage Analysis, which gives you a measure of how
complete your testing is.

Coverage analysis annotates each "block" of straight-line code with
a count of the number of times it has executed. For testing, what is
usually more interesting is which lines were never executed, and the
"Coverage", or percentage of blocks in your program or library that
were exercised in your testing. For more information, see
tcov(1), in /opt/SUNWspro/man.

Compilation and Linking

Coverage analysis requires a special compilation of your program or
library. Each .c file needs to be compiled with
-xprofile=tcov, and the final link (either to executable or
shared library) also needs -xprofile=tcov.

CFLAGS += -xprofile=tcov
CFLAGS64 += -xprofile=tcov
DYNFLAGS += -xprofile=tcov (shared libraries only)

in the appropriate Makefiles, then make clean; make install is

Generating Profile Data

The -xprofile=tcov version of your binary will generate profile
information every time the executable is run (or, in the case of a shared
library, any executable which links against it is run) and exits normally.
The output is placed (by default) in ./progname.profile/, which
will build up data from all executions as they exit. It will even join
up 32-bit and 64-bit data sets.

The tcov output location is controlled by two environment variables,
(default 'progname.profile'). So if you are testing libfoo.so,
and want to join the data from a bunch of executions into
/tmp/libfoo.profile, you would set:
% SUN_PROFDATA=libfoo.profile
% setenv SUN_PROFDATA_DIR /tmp
% setenv SUN_PROFDATA libfoo.profile

before your runs.

Processing the profile data

Once you have finished gathering data, you can use the tcov(1)
command, located in /opt/SUNWspro/bin (or wherever you keep
your compilers) to analyze it. It's syntax is pretty straightforward:
% tcov -x profile_dir sourcefile...

For example, to analyze the previous libfoo example, you might: (here I
use a seperate directory for my tcov analysis)
% cd usr/src/lib/libfoo
% mkdir tcov
% cd tcov
% tcov -x /tmp/libfoo.profile ../common/\*.c ../sparc/\*.c ../sparcv9/\*.c

Analyzing the data

Nota Bene: The counts tcov uses to generate its output are updated without
holding locks. For multi-threaded programs only, this means that some
counts may be lower than expected. Nevertheless, if a block has been
executed at least once, its count will be non-zero.

For each source file you pass in on the command line, tcov will generate
a .tcov file (for example, ../common/foo.c ->
foo.c.tcov). Each file contains the original source, annotated
with execution counts. Each line that starts a "basic block" is
prefixed with either '##### ->', indicating that it has not been
executed, or 'count ->', indicating how many times it
was executed.

After the annotated source, there is a summary of the file, including
things like total blocks, number executed, % coverage, average
executions per block, etc.

I've written a tool,
, which
takes the tcov files in the current directory and displays a summary of
the current state. The columns are "total blocks", "executed blocks",
and "% executed" (or % coverage).

Command example: cpio
% cd usr/src/cmd/cpio
% grep tcov Makefile
CFLAGS += -xprofile=tcov
% make
... (made cpio) ...
% mkdir tcov
% cd tcov
% ../cpio
cpio: One of -i, -o or -p must be specified.
cpio -i[bcdfkmrstuv@BSV6] [-C size] [-E file] [-H hdr] [-I file [-M msg]] [-R id] [patterns]
cpio -o[acv@ABLV] [-C size] [-H hdr] [-O file [-M msg]]
cpio -p[adlmuv@LV] [-R id] directory
% ls
% tcov -x cpio.profile ../\*.c
% ls
cpio.c.tcov cpio.profile/ cpiostat.c.tcov
% tcov_summarize
1818 32 1.76 cpio.c
2 0 0.00 cpiostat.c
1820 32 1.76 total
% find . | ../cpio -ocB > /dev/null
590 blocks
% tcov -x cpio.profile ../\*.c
% tcov_summarize
1818 326 17.93 cpio.c
2 0 0.00 cpiostat.c
1820 326 17.91 total

Library example: libumem
% cd usr/src/lib/libumem   
% grep tcov Makefile.com
CFLAGS += -v $(LOCFLAGS) -I$(CMNDIR) -xprofile=tcov
CFLAGS64 += -v $(LOCFLAGS) -I$(CMNDIR) -xprofile=tcov
DYNFLAGS += -M $(MAPFILE) -z interpose -xprofile=tcov
% make
... (made libumem) ...
% mkdir tcov
% cd tcov
% SUN_PROFDATA=libumem.profile
% LD_PRELOAD=../sparc/libumem.so.1 LD_PRELOAD_64=../sparcv9/libumem.so.1
% ls
% ls
% tcov -x libumem.profile ../common/\*.c ../sparc/\*.c
% /home/jwadams/bin/tcov_summarize
75 44 58.67 envvar.c
10 7 70.00 getpcstack.c
72 22 30.56 malloc.c
78 27 34.62 misc.c
592 255 43.07 umem.c
1 0 0.00 umem_agent_support.c
315 167 53.02 vmem.c
13 10 76.92 vmem_base.c
20 0 0.00 vmem_mmap.c
35 17 48.57 vmem_sbrk.c
1211 549 45.33 total
% tcov -x libumem.profile ../common/\*.c ../sparc/\*.c
% /home/jwadams/bin/tcov_summarize
77 45 58.44 envvar.c
10 7 70.00 getpcstack.c
72 28 38.89 malloc.c
78 27 34.62 misc.c
592 314 53.04 umem.c
1 0 0.00 umem_agent_support.c
315 192 60.95 vmem.c
13 10 76.92 vmem_base.c
20 0 0.00 vmem_mmap.c
35 17 48.57 vmem_sbrk.c
1213 640 52.76 total

(Note that running tcov gave us more coverage, since the library is being preloaded underneath it)

Tags: [

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.