X

Solaris serviceability and nifty tools

Testing netcat

After multiple rounds of code review the netcat (or nc) test suite is now finally in the onnv-stc2gate.
The test suite has its home in the OpenSolaris
Networking community (see
the networking tests page for the list of networking
test suites).


The source code is present in the
src/suites/net/nc/
directory and SUNWstc-netcat packages can be downloaded from
OpenSolaris Download center.

Before I go further, this is how it looks like when the test suite is run (the output is trimmed a bit):

vk:honeymooners:/opt/SUNWstc-nc$ run_test nc
Validating Arguments...
New TET_ROOT for this run : /var/tmp/honeymooners_27828
The results will be available in /var/tmp/results.27828
tcc: journal file is /var/tmp/results.27828/testlog
12:45:57 Execute /tests/dflag/tc_dflag
12:46:04 Execute /tests/hflag/tc_hflag
12:46:05 Execute /tests/kflag/tc_kflag
12:46:11 Execute /tests/nflag/tc_nflag
12:46:15 Execute /tests/portranges/tc_portranges
12:46:23 Execute /tests/pflag/tc_pflag
12:46:26 Execute /tests/sflag/tc_sflag
12:46:35 Execute /tests/Uflag/tc_Uflag
12:46:36 Execute /tests/vflag/tc_vflag
12:46:43 Execute /tests/zflag/tc_zflag
12:46:46 Execute /tests/iflag/tc_iflag
12:46:59 Execute /tests/lflag/tc_lflag
12:47:29 Execute /tests/rflag/tc_rflag
12:48:16 Execute /tests/Tflag/tc_Tflag
12:48:33 Execute /tests/uflag/tc_uflag
12:48:50 Execute /tests/wflag/tc_wflag
##################################################
TC /tests/dflag/tc_dflag
TP 1 tc_dflag PASS
##################################################
TC /tests/hflag/tc_hflag
TP 1 tc_hflag PASS
...
##################################################
SUMMARY
=======
Number of Tests : 50
PASS : 50
FAIL : 0
UNRESOLVED : 0
UNINITIATED : 0
OTHER : 0
##################################################
Test Logs are at /var/tmp/results.27828, Journal File = /var/tmp/results.27828/testlog
vk:honeymooners:/opt/SUNWstc-nc$

It's been almost a year since I started developing the test suite last Christmas (see
the initial blog entry about nc-tet).
Since then, I have lost part of the source code in
hard drive crash, had to redo the
source tree structure, fix ksh style, fix numerous bugs in test suite code and make the test suite more robust.
One might ask whether having test suite for such a simple program like nc(1) was worth the hassle.
I have only one answer to that: absolutely. First, it gives a confidence of not breaking (most of; see below)
existing things when changing/adding functionality and second it helped me (and I hope the others participating/observing
the code review on testing-discuss too) to explore what it takes to write a test suite from scratch
(I will not go here into details whether I prefer
CTI-TET over
STF and vice versa).

The Beautiful code book (which I really recommend for anyone tinkering with any source code) contains a chapter called Beautiful tests
by Alberto Savoia. I hope that at least some of the test purposes in nc test suite have some degree of beautifulness of at least one
of the ways highlighted by Alberto (1. simplicity/efficiency, 2. help making the software being tested better in terms of
quality and testability, 3. breadth/thoroughness).

One of the important questions for a test suite is code coverage level.
Obviously, for software adhering to the OpenSolaris
interface taxonomy model it is important
that the test suite exercises all of the Committed interfaces and execution paths around those interfaces.
For nc(1) this means a subset of the command line options and their arguments (see PSARC 2007/389 for the actual list).
The key is certainly to test the features which are likely to break with an intrusive code change.

Very crude view of test coverage for nc(1) test suite (counting test purposes gives only very remote idea about
real coverage but at least provides visual image) looks like this:

       rflag: +
Tflag: +++++---
pflag: +
iflag: +-
vflag: ++
kflag: +
Uflag: +-
dflag: +
uflag: ++-
sflag: +-
hflag: +
nflag: +-
wflag: +
portranges: +---
lflag: ++++++++----------

One plus character stands for one positive test purpose, minus is negative test purpose.

Side note: the above ASCII graph was produced using
test-coverage-graph.sh
script (which presumes certain naming scheme for test purpose files).
Just pipe a file listing into the script with test purpose filenames compliant to the scheme used in
ontest-stc2 gate and it will spew out graph
similar to the above.

In the above musing about code coverage I left out an important piece - why some of the features are not tested.
For nc(1) the yet untested part is the SOCKS protocol support. Basically, this is because test suite environment
does not contain SOCKS server to test against. There might not be many people using the -x/-X
options but from my own experience nothing is more frustrating than discovering some old dusty corner which had
to be fixed long time ago or removed completely. So for now, on my workstation which sits behind SOCKS proxy
I have the following in ~/.ssh/config for a server outside corporate network which hosts my personal
mailbox so it is accessed every day:

Host bar
User foo
Hostname outside.kewl.org
# NOTE: for nc(1) testing
ProxyCommand /usr/bin/nc -x socks-proxy.foothere.bar outside.kewl.org %p
ForwardAgent no
ForwardX11 no

This ensures (along with upgrades of the workstation to recent Nevada builds periodically) that SOCKS
support gets tested as well. And yes, ssh-socks5-proxy-connect(1) and ssh-http-proxy-connect(1) are not
really needed.

Now with the test suite in place, anybody modifying nc(1) (there are some RFEs for nc in the
oss-bit-size list and other bugfixes
or features are also welcome) can have pretty high confidence that his change will not break things.
Yes, this means that more nc(1) features are coming.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.