Observe Loopback and Inter-Zone IP Packets With OpenSolaris
By user12618912 on Nov 18, 2008
I'm happy to announce that the IP Observability Devices component of the Clearview project has integrated into OpenSolaris build 103 (also see Phil Kirk's announcement to the ON community). This adds the following new capabilities to OpenSolaris:
- Network observability at the IP layer for traditional DLPI-based tools such as snoop
- Observability of loopback IP packets
- Observability of inter-zone IP packets
- Tools such as snoop can be run from within a non-global zone to observe packets associated with that zone
- Snoop filtering based on zone id
The snoop command has grown a new "-I <interface-name>" option to access this feature. Its semantics are to snoop the IP interface named <interface-name> at the IP layer. When observing a particular IP interface with this facility, packets that have a source or destination IP address assigned to that interface can be observed, as well as packets that are forwarded to or from that IP interface, and broadcast and multicast packets received by that interface. Additional internal filtering is performed to ensure that an observer from a non-global zone can only see packets that belong to that zone, with the exception of the global zone, from which packets to or from any zone that shares its stack can be observed. Any IP interface visible through "ifconfig -a" can be observed using this feature.
Here are some examples using snoop:
Example 1: Observing the Loopback Interface
bash-3.2# snoop -I lo0 Using device ipnet/lo0 (promiscuous mode) localhost -> localhost ICMP Echo request (ID: 37110 Sequence number: 0) localhost -> localhost ICMP Echo reply (ID: 37110 Sequence number: 0)
The lo0 interface has the 127.0.0.1 address assigned to it, and so any communication using the address 127.0.0.1 is seen above (in this case, I was simply doing "ping 127.0.0.1"). Snoop's verbose output mode displays a new "ipnet" header that precedes all IP packets observed:
bash-3.2# snoop -v -I lo0 Using device ipnet/lo0 (promiscuous mode) IPNET: ----- IPNET Header ----- IPNET: IPNET: Packet 1 arrived at 10:40:33.68506 IPNET: Packet size = 108 bytes IPNET: dli_version = 1 IPNET: dli_type = 4 IPNET: dli_srczone = 0 IPNET: dli_dstzone = 0 IPNET: IP: ----- IP Header ----- IP: IP: Version = 4 IP: Header length = 20 bytes ...
Note above that the source and destination zone ids are displayed. In this case, I was running "ping 127.0.0.1" in the global zone, and so both the source and destination zone ids are "0".
Example 2: Running Snoop From a Non-Global Zone
bash-3.2# zoneadm list -v ID NAME STATUS PATH BRAND IP 0 global running / native shared 4 test running /zones/test native shared bash-3.2# zlogin test [Connected to zone 'test' pts/2] ... bash-3.2# ifconfig -a lo0:1: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 bge0:1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2 inet 10.8.57.34 netmask ffffff00 broadcast 10.8.57.255 lo0:1: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1 inet6 ::1/128 bge0:2: flags=202000841<UP,RUNNING,MULTICAST,IPv6,CoS> mtu 1500 index 2 inet6 2002:a08:39f0:1::f/64 bash-3.2# snoop -I bge0 Using device ipnet/bge0 (promiscuous mode) whitestar1-2.East.Sun.COM -> mf-ubur-01.East.Sun.COM DNS C 253.57.8.10.in-addr.arpa. Internet PTR ? mf-ubur-01.East.Sun.COM -> whitestar1-2.East.Sun.COM DNS R 220.127.116.11.in-addr.arpa. Internet PTR ALL-ROUTERS.MCAST.NET. whitestar1-6.East.Sun.COM -> whitestar1-2.East.Sun.COM TCP D=22 S=62117 Syn Seq=195630514 Len=0 Win=49152 Options=<mss whitestar1-2.East.Sun.COM -> whitestar1-6.East.Sun.COM TCP D=62117 S=22 Syn Ack=195630515 Seq=195794440 Len=0 Win=49152 whitestar1-6.East.Sun.COM -> whitestar1-2.East.Sun.COM TCP D=22 S=62117 Ack=195794441 Seq=195630515 Len=0 Win=49152 whitestar1-2.East.Sun.COM -> whitestar1-6.East.Sun.COM TCP D=62117 S=22 Push Ack=195630515 Seq=195794441 Len=20 Win=491
Although not evident from the snoop output above, whitestar1-2 is 10.8.57.34 (the bge0:1 IP address in this non-global zone), and whitestar1-6 is actually an IP address in another zone on the same system. By snooping the bge0 interface, the user sees all packets associated with the bge0 IP addresses in the zone; even those that are locally delivered to other zones. Using snoop's verbose output mode allows us to see which zones these packets are flowing between:
bash-3.2# snoop -v -I bge0 whitestar1-6 Using device ipnet/bge0 (promiscuous mode) IPNET: ----- IPNET Header ----- IPNET: IPNET: Packet 1 arrived at 10:44:10.86739 IPNET: Packet size = 76 bytes IPNET: dli_version = 1 IPNET: dli_type = 4 IPNET: dli_srczone = 0 IPNET: dli_dstzone = 4 IPNET: IP: ----- IP Header ----- IP: IP: Version = 4 IP: Header length = 20 bytes ...
We can see above that the packet was from the global zone to the test zone.
Example 3: Filtering by Zone ID
Filtering by zone id can be useful on a system that has multiple zones. In this example, an administrator in the global zone observes packets being sent to or from IP addresses in the "test" zone.
bash-3.2# zoneadm list -v ID NAME STATUS PATH BRAND IP 0 global running / native shared 4 test running /zones/test native shared bash-3.2# snoop -I bge0 zone 4 Using device ipnet/bge0 (promiscuous mode) whitestar1-6.East.Sun.COM -> whitestar1-2.East.Sun.COM TCP D=22 S=61658 Syn Seq=374055417 Len=0 Win=49152 Options=<mss whitestar1-2.East.Sun.COM -> whitestar1-6.East.Sun.COM TCP D=61658 S=22 Syn Ack=374055418 Seq=374124525 Len=0 Win=49152 whitestar1-6.East.Sun.COM -> whitestar1-2.East.Sun.COM TCP D=22 S=61658 Ack=374124526 Seq=374055418 Len=0 Win=49152
This can be particularly useful with the loopback interface, as the 127.0.0.1 address is shared among all shared-stack zones, and it can be difficult to associate a loopback packet to an application in a zone.
Note that there is a pending RFE to also be able to enter a zone name as well as a zone id as the argument to the snoop "zone" filtering primitive. For now, the zone id is the only allowable argument.