Solaris 11.2: User, Pid and Commands in netstat(1m)

As it has been years since I've blogged, let me start with one of smallest features I added to Solaris 11.2; an option to netstat(1m), allowing administrators to figure out who is using which port and which with process or command is using a particular network connection.

As there little or no similarity between other netstat implementation, we picked our own option letter "-u". At the same time we realigned the columns as the standard width didn't fit modern TCP window sizes, the length of Unix sockets, etc. We've also removed, for unprivileged users, unusable information such as the "kernel addresses", leaving a bit more room, though an 80 width terminal isn't really enough room for all of the information. Alignment only guaranteed with -n, of course.

Our implementation doesn't use /proc like the Linux implementation uses nor does it look through /dev/kmem like lsof(1m) does; instead we get the information available direct in the kernel. While some of the information might be out of date, we can give information about sockets in TIME_WAIT or CLOSE_WAIT, even when the latter sockets haven't been accepted yet! Additionally, those sockets owned by the kernel are also listed. This works in the global zone, non-global zones, kernel zones *and* even in Solaris 10 branded zones; the latter uses the "native" Solaris 11.2 netstat command.

Here is some sample output, partially hidden by how we format blogs (so, install Solaris 11.2 and all will be revealed)

% netstat -aun

UDP: IPv4
   Local Address        Remote Address      User    Pid      Command       State
-------------------- -------------------- -------- ------ -------------- ----------
      *.50258                             root       1038 syslogd        Idle
      *.*                                 root        133 in.mpathd      Unbound
      *.*                                 root        133 in.mpathd      Unbound
      *.*                                 netadm      721 nwamd          Unbound
      *.*                                 netadm      721 nwamd          Unbound
      *.123                               root        961 ntpd           Idle
      *.123                               root        961 ntpd           Idle
127.0.0.1.123                             root        961 ntpd           Idle
10.311.249.18.123                         root        961 ntpd           Idle
      *.111                               daemon      980 rpcbind        Idle
      *.*                                 daemon      980 rpcbind        Unbound
      *.41327                             daemon      980 rpcbind        Idle
      *.111                               daemon      980 rpcbind        Idle
      *.*                                 daemon      980 rpcbind        Unbound
      *.37058                             daemon      980 rpcbind        Idle
      *.*                                 root        988 in.ndpd        Unbound
      *.*                                 root        999 statd          Unbound
      *.*                                 root        999 statd          Unbound
      *.39150                             root        999 statd          Idle
      *.43382                             root        999 statd          Idle
      *.4045                              daemon     1008 lockd          Idle
      *.4045                              daemon     1008 lockd          Idle
      *.56874                             root       1004 inetd          Idle
      *.37069                             root       1004 inetd          Idle
      *.42765                             root       1148 mountd         Idle
      *.64957                             root       1148 mountd         Idle
      *.2049                              root       1150 nfsd           Idle
      *.2049                              root       1150 nfsd           Idle

UDP: IPv6
   Local Address                     Remote Address                   User    Pid      Command       State      If
--------------------------------- --------------------------------- -------- ------ -------------- ---------- -----
      *.*                                                           root        133 in.mpathd      Unbound    
      *.*                                                           netadm      721 nwamd          Unbound    
      *.123                                                         root        961 ntpd           Idle       
::1.123                                                             root        961 ntpd           Idle       
      *.111                                                         daemon      980 rpcbind        Idle       
      *.*                                                           daemon      980 rpcbind        Unbound    
      *.41327                                                       daemon      980 rpcbind        Idle       
      *.*                                                           root        988 in.ndpd        Unbound    
      *.39150                                                       root        999 statd          Idle       
      *.4045                                                        daemon     1008 lockd          Idle       
      *.37069                                                       root       1004 inetd          Idle       
      *.42765                                                       root       1148 mountd         Idle       
      *.2049                                                        root       1150 nfsd           Idle       

TCP: IPv4
   Local Address        Remote Address      User     Pid     Command     Swind  Send-Q  Rwind  Recv-Q    State
-------------------- -------------------- -------- ------ ------------- ------- ------ ------- ------ -----------
127.0.0.1.5999             *.*            root        133 in.mpathd           0      0  128000      0 LISTEN
      *.111                *.*            daemon      980 rpcbind             0      0  128000      0 LISTEN
      *.*                  *.*            daemon      980 rpcbind             0      0  128000      0 IDLE
      *.111                *.*            daemon      980 rpcbind             0      0  128000      0 LISTEN
      *.*                  *.*            daemon      980 rpcbind             0      0  128000      0 IDLE
      *.36887              *.*            root        999 statd               0      0  128000      0 LISTEN
      *.65159              *.*            root        999 statd               0      0  128000      0 LISTEN
10.311.249.18.58810  10.312.132.13.636    root        851 nscd            49232      0  128872      0 ESTABLISHED
      *.4045               *.*            daemon     1008 lockd               0      0 1049200      0 LISTEN
      *.4045               *.*            daemon     1008 lockd               0      0 1048952      0 LISTEN
      *.22                 *.*            root       1030 sshd                0      0  128000      0 LISTEN
127.0.0.1.25               *.*            root       1068 sendmail            0      0  128000      0 LISTEN
127.0.0.1.587              *.*            root       1068 sendmail            0      0  128000      0 LISTEN
      *.47629              *.*            root       1148 mountd              0      0  128000      0 LISTEN
      *.35906              *.*            root       1148 mountd              0      0  128000      0 LISTEN
      *.2049               *.*            root       1150 nfsd                0      0 1049200      0 LISTEN
      *.2049               *.*            root       1150 nfsd                0      0 1048952      0 LISTEN
127.0.0.1.1008             *.*            pkg5srv    1600                     0      0  128000      0 LISTEN
10.311.249.18.857    10.311.246.25.2049   casper        0 <kernel>        49232      0 1049800    116 ESTABLISHED
10.311.249.18.22     10.311.249.34.64127  root       1030 sshd           263536     63  128872      0 ESTABLISHED
127.0.0.1.6010             *.*            casper     1969 sshd                0      0  128000      0 LISTEN

TCP: IPv6
   Local Address                     Remote Address                   User    Pid      Command      Swind  Send-Q  Rwind  Recv-Q   State      If
--------------------------------- --------------------------------- -------- ------ -------------- ------- ------ ------- ------ ----------- -----
::1.5999                                *.*                         root        133 in.mpathd            0      0  128000      0 LISTEN      
      *.111                             *.*                         daemon      980 rpcbind              0      0  128000      0 LISTEN      
      *.*                               *.*                         daemon      980 rpcbind              0      0  128000      0 IDLE        
      *.36887                           *.*                         root        999 statd                0      0  128000      0 LISTEN      
      *.4045                            *.*                         daemon     1008 lockd                0      0 1049200      0 LISTEN      
      *.22                              *.*                         root       1030 sshd                 0      0  128000      0 LISTEN      
::1.25                                  *.*                         root       1068 sendmail             0      0  128000      0 LISTEN      
      *.47629                           *.*                         root       1148 mountd               0      0  128000      0 LISTEN      
      *.2049                            *.*                         root       1150 nfsd                 0      0 1049200      0 LISTEN      
::1.6010                                *.*                         casper     1969 sshd                 0      0  128000      0 LISTEN      
::1.51794                         ::1.6010                          casper     1970 xterm           130880      0  139264      0 ESTABLISHED 
::1.6010                          ::1.51794                         casper     1969 sshd            139060      0  130880      0 ESTABLISHED 

Active UNIX domain sockets
Type       User        Pid Command        Local Address                           Remote Address
stream-ord casper     1969 sshd            (socketpair)                            (socketpair)
stream-ord casper     1969 sshd            (socketpair)                            (socketpair)
stream-ord casper     1969 sshd            (socketpair)                            (socketpair)
stream-ord casper     1969 sshd            (socketpair)                            (socketpair)
stream-ord casper     1969 sshd            (socketpair)                            (socketpair)
stream-ord root        372 dbus-daemon    /var/run/dbus/system_bus_socket
stream-ord root       1028 rmvolmgr                                               /var/run/dbus/system_bus_socket
stream-ord root        372 dbus-daemon    /var/run/dbus/system_bus_socket
stream-ord root        943 hald                                                   /var/run/dbus/system_bus_socket
stream-ord root       1004 inetd          /system/volatile/inetd.uds
stream-ord root        943 hald           /system/volatile/hald/dbus-TM2nMhzrpM
stream-ord root        993 hald-addon-sto                                         /system/volatile/hald/dbus-TM2nMhzrpM
stream-ord pkg5srv    1601 httpd.worker   /system/volatile/pkg/sysrepo/wsgi.1601.0.1.sock
stream-ord root        943 hald           /system/volatile/hald/dbus-TM2nMhzrpM
dgram      root        988 in.ndpd        /system/volatile/in.ndpd_mib
stream-ord root        988 in.ndpd        /system/volatile/in.ndpd_ipadm
stream-ord root        970 hald-addon-cpu                                         /system/volatile/hald/dbus-TM2nMhzrpM
stream-ord root        943 hald           /system/volatile/hald/dbus-MIhDasTVfy
stream-ord root        944 hald-runner                                            /system/volatile/hald/dbus-MIhDasTVfy
stream-ord root        943 hald           /system/volatile/hald/dbus-MIhDasTVfy
stream-ord root        943 hald           /system/volatile/hald/dbus-TM2nMhzrpM
stream-ord root        372 dbus-daemon    /var/run/dbus/system_bus_socket
stream-ord root        922 console-kit-da                                         /var/run/dbus/system_bus_socket
stream-ord root        196 rad            /system/volatile/rad/radsocket-unauth
stream-ord root        372 dbus-daemon     (socketpair)                            (socketpair)
stream-ord root        372 dbus-daemon     (socketpair)                            (socketpair)
stream-ord root        196 rad            /system/volatile/rad/radsocket
stream-ord root        372 dbus-daemon    /var/run/dbus/system_bus_socket
Adding the option -v, you also get command line:
UDP: IPv4
   Local Address        Remote Address      User    Pid     State       Command
-------------------- -------------------- -------- ------ ---------- ----------------
      *.50258                             root       1038 Idle       /usr/sbin/syslogd
      *.*                                 root        133 Unbound    /lib/inet/in.mpathd
      *.*                                 root        133 Unbound    /lib/inet/in.mpathd
      *.*                                 netadm      721 Unbound    /lib/inet/nwamd
      *.*                                 netadm      721 Unbound    /lib/inet/nwamd
      *.123                               root        961 Idle       /usr/lib/inet/ntpd -p /var/run/ntp.pid -g
...
And for half-closed connection, you'd also get the information you want:
TCP: IPv4
   Local Address        Remote Address      User     Pid     Command     Swind  Send-Q  Rwind  Recv-Q    State
-------------------- -------------------- -------- ------ ------------- ------- ------ ------- ------ -----------
127.0.0.1.55770      127.0.0.1.4321       casper     1033 closewait      130880      0  139264      0 FIN_WAIT_2
127.0.0.1.4321       127.0.0.1.55770      casper     1031 closewait      139264      0  130880      0 CLOSE_WAIT
127.0.0.1.54943      127.0.0.1.4321       casper     1033 closewait      130880      0  139264      0 FIN_WAIT_2
127.0.0.1.4321       127.0.0.1.54943      casper     1031 closewait      139264      0  130880      0 CLOSE_WAIT
127.0.0.1.41279      127.0.0.1.4321       casper     1033 closewait      130880      0  139264      0 FIN_WAIT_2
127.0.0.1.4321       127.0.0.1.41279      casper     1031 closewait      139264      0  130880      0 CLOSE_WAIT
...

PS: I used the Hollywood IP extension to masquerade the IP addresses.

Comments:

Great to see you getting back on the blogging horse. :)

I for one am wrapped to see this feature going in. No more groking through /proc with pfiles for me!

Posted by nathan on April 30, 2014 at 11:08 PM MEST #

Post a Comment:
Comments are closed for this entry.
About

casper

Search

Archives
« July 2015
MonTueWedThuFriSatSun
  
1
2
3
4
5
6
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
  
       
Today