Solaris serviceability and nifty tools

netcat and network stack debugging

One of the options of the netcat program (/usr/bin/nc) available in OpenSolaris (if you don't have it installed simply run pkg install SUNWnetcat)
is the -D (aka debugging) option. Only recently I realized that not everyone out there knows how to capture the debugging data once this option
is set since this is pretty Solaris specific. What this option does is basically trigger a flag inside the ip kernel module specific for given
connection structure (conn_t). netcat does this by calling setsockopt() system call with SO_DEBUG option set for given
socket. As a result conn_debug flag is set inside conn_t structure associated with the socket. This flag is then consulted in
various functions manipulating the structure.

When there is an interesting event and the conn_debug is set the function calls strlog() kernel function to record the data.
For example, here's a snippet of usr/src/uts/common/inet/tcp/tcp_input.c:tcp_input_listener():


if (listener->tcp_conn_req_cnt_q >= listener->tcp_conn_req_max) {


TCP_STAT(tcps, tcp_listendrop);

TCPS_BUMP_MIB(tcps, tcpListenDrop);

if (lconnp->conn_debug) {

(void) strlog(TCP_MOD_ID, 0, 1, SL_TRACE|SL_ERROR,

"tcp_input_listener: listen backlog (max=%d) "

"overflow (%d pending) on %s",



tcp_display(listener, NULL, DISP_PORT_ONLY));


goto error2;


To capture the data logged via strlog it's necessary to know the STREAMS module ID, which in our case is
TCP_MOD_ID which is defined in usr/src/uts/common/inet/tcp_impl.h as 5105.

To read the data one can use either strace(1M) command line tool or strerr(1M) daemon which both produce text logs.
To read everything, one can use this command (needs read access to /dev/log so has to run under root):

# strace 5105 all all

Here are two examples using netcat: at first we try to bind to a port which we don't have privileges for:

$ nc -D -l -4 -p 23
nc: Permission denied

which produces the following entry from strace:

000004 12:11:04 19c7ff6a  1 ..E 5105 0 ip: [ID 234745 kern.debug] tcp_bind: no priv for port 23

next we try to bind to already occupied port:

$ nc -D -l -4 -p 4444
nc: Address already in use

which produces the following entry from strace:

000005 12:15:33 19c86878  1 ..E 5105 0 ip: [ID 326978 kern.debug] tcp_bind: requested addr busy

This is of course traceable via normal tools such as ppriv(1) or truss(1) but the point is that much fine grained details
can be captured from the network modules. The format of the log entries is explained in the strace(1M) man page.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.