X

Using Solaris and SPARC Networking and Virtualization

Using zonecfg defrouter with shared-IP zones

[Update to IPMP testing 2009.01.20]

[Minor update 2009.01.14]

When running Solaris Zones in a shared-IP configuration, all network configurations are determined by how the zone is configured using zonecfg(1M) or by what the global zone's IP determines things should be (such as routes). This has caused some trouble in situations where zones are on different subnets, and especially if the global zone is not on the subnet(s) the non-global zones are on. While exclusive IP Instances were delivered to help address these cases, using exclusive IP Instances requires a data link per zone, and if running a large number of zones there may not be enough data links available.

With Solaris 10 10/08 (Update 6), an additional network configuration parameter is available for shared-IP zones. This is the default router (defrouter) optional parameter.

Using the defrouter parameter, it is possible to set which router to use for traffic leaving the zone. In the global zone, default router entries are created the first time the zone is booted. Note that the entries are not deleted when the zone is halted.

The defrouter property looks like this for a zone with it configured.

global# zonecfg -z shared1 info net
net:
address: 10.1.14.141/26
physical: bge1
defrouter: 10.1.14.129

And it looks like this if it is not set.
global# zonecfg -z shared1 info net
net:
address: 10.1.14.141/26
physical: bge1
defrouter not specified

So I have run a variety of configurations, and some thing I observed are as follows. (Most of the configurations used a separate interface for the global zone (bge0) than for the non-global zones (bge1 and bge2). IPMP is not being used in these configurations. A comment on that at the end.) The [#] indicate examples in the outputs that follow.
  • A default route entry is create for the NIC [1] on which the zone is configured when the zone is booted. [2]
  • Entries are not deleted when a zone is halted. They persist until manually removed[3] or a reboot of the global zone.
  • It is possible to have the same default router configured for multiple zones. [4]
  • It is possible to have the same default router listed on multiple interfaces. \* [5]
  • It is possible to have multiple default routers on the same interface, even on different IP subnets. [6]
  • The interface used for outbound traffic is the one the zone is assigned to. [7]
  • It is sufficient to plumb the interface for the non-global zones in the global zone (thus it has 0.0.0.0 as its IP address in the global zone). [8]
  • The physical interface can be down in the global zone. [9]
  • If only one interface is used, and different subnets for the global and non-global zones are configured, routing works when setting defrouter [10] and does not work if it is not set.

The most interesting thing I noticed was that although two non-global zones may be on the same IP subnet, if they are configured on different interfaces, the traffic leaves the system on the interface that the zone is configured to be on. This is not the case typically when using shared IP and also having an IP address for the subnet in the global zone.

\* Note: Having two interfaces on the same IP subnet without configuring IP Multipathing (IPMP) may not be a supported configuration. I am looking for documentation that states this one way or another. [2009.01.14]

Examples

1. Single Zone, Single Interface--The Basics

Create a single non-global zone.
global# netstat -rn
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 139.164.63.215 UG 1 2 bge0
139.164.63.0 139.164.63.125 U 1 1 bge0
224.0.0.0 139.164.63.125 U 1 0 bge0
127.0.0.1 127.0.0.1 UH 1 42 lo0
global# zonecfg -z shared1 info net
net:
address: 10.1.14.141/26
physical: bge1
defrouter: 10.1.14.129
global# zoneadm -z shared1 boot [2]
global# netstat -rn
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 139.164.63.215 UG 1 2 bge0
default 10.1.14.129 UG 1 0 bge1 [1]
139.164.63.0 139.164.63.125 U 1 1 bge0
224.0.0.0 139.164.63.125 U 1 0 bge0
127.0.0.1 127.0.0.1 UH 1 42 lo0
global# zoneadm -z shared1 halt
global# zoneadm list -v
ID NAME STATUS PATH BRAND IP
0 global running / native shared
global# netstat -rn
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 10.1.14.129 UG 1 0 bge1
default 139.164.63.215 UG 1 1 bge0
139.164.63.0 139.164.63.125 U 1 1 bge0
224.0.0.0 139.164.63.125 U 1 0 bge0
127.0.0.1 127.0.0.1 UH 1 42 lo0
global# route delete default 10.1.14.129 [3]
delete net default: gateway 10.1.14.129
global# netstat -rn
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 139.164.63.215 UG 1 1 bge0
139.164.63.0 139.164.63.125 U 1 1 bge0
224.0.0.0 139.164.63.125 U 1 0 bge0
127.0.0.1 127.0.0.1 UH 1 42 lo0

2. Multiple Interfaces, Same Default Router

Three zones, where two use bge1 and the third uses bge2. All use the same default router.
global# netstat -rn
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 139.164.63.215 UG 1 1 bge0
139.164.63.0 139.164.63.125 U 1 1 bge0
224.0.0.0 139.164.63.125 U 1 0 bge0
127.0.0.1 127.0.0.1 UH 1 42 lo0
global# zonecfg -z shared1 info net
net:
address: 10.1.14.141/26
physical: bge1
defrouter: 10.1.14.129 [4]
global# zonecfg -z shared2 info net
net:
address: 10.1.14.142/26
physical: bge1
defrouter: 10.1.14.129 [4]
global# zonecfg -z shared3 info net
net:
address: 10.1.14.143/26
physical: bge2
defrouter: 10.1.14.129 [5]
global# zoneadm -z shared1 boot
global# zoneadm -z shared2 boot
global# zoneadm -z shared3 boot
global# netstat -rn
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 10.1.14.129 UG 1 0 bge1 [4]
default 139.164.63.215 UG 1 1 bge0
default 10.1.14.129 UG 1 2 bge2 [5]
139.164.63.0 139.164.63.125 U 1 1 bge0
224.0.0.0 139.164.63.125 U 1 0 bge0
127.0.0.1 127.0.0.1 UH 1 42 lo0
global# zoneadm list -v
ID NAME STATUS PATH BRAND IP
0 global running / native shared
3 shared1 running /zones/shared1 native shared
4 shared2 running /zones/shared2 native shared
5 shared3 running /zones/shared3 native shared
global# ifconfig -a4
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
lo0:1: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
zone shared1
inet 127.0.0.1 netmask ff000000
lo0:2: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
zone shared2
inet 127.0.0.1 netmask ff000000
lo0:3: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
zone shared3
inet 127.0.0.1 netmask ff000000
bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 139.164.63.125 netmask ffffff00 broadcast 139.164.63.255
ether 0:3:ba:e3:42:8b
bge1: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 0.0.0.0 netmask 0
ether 0:3:ba:e3:42:8c
bge1:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
zone shared1
inet 10.1.14.141 netmask ffffffc0 broadcast 10.1.14.191
bge1:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
zone shared2
inet 10.1.14.142 netmask ffffffc0 broadcast 10.1.14.191
bge2: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 0.0.0.0 netmask 0
ether 0:3:ba:e3:42:8d
bge2:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
zone shared3
inet 10.1.14.143 netmask ffffffc0 broadcast 10.1.14.191

3. Multiple Subnets

Add another zone, using bge2 and on a different subnet.
global# zonecfg -z shared4 info net
net:
address: 192.168.16.144/24
physical: bge2
defrouter: 192.168.16.129
global# zoneadm -z shared4 boot
global# netstat -rn
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 10.1.14.129 UG 1 0 bge1
default 10.1.14.129 UG 1 4 bge2
default 139.164.63.215 UG 1 3 bge0
default 192.168.16.129 UG 1 0 bge2 [6]
139.164.63.0 139.164.63.125 U 1 4 bge0
224.0.0.0 139.164.63.125 U 1 0 bge0
127.0.0.1

4. Interface Usage

Issue some pings within the non-global zones and see which network interfaces are used. From the global zone, I issue a ping to a remote system (on the same network as the global zone (139.164.63.0), and see which interfaces are being used. [7]
global# zlogin shared1 ping 139.164.63.38
139.164.63.38 is alive
global# zlogin shared2 ping 139.164.63.38
139.164.63.38 is alive
global# zlogin shared3 ping 139.164.63.38
139.164.63.38 is alive
global# zlogin shared4 ping 139.164.63.38
139.164.63.38 is alive

This shows the pings originating from shared1 and shared2 going out on bge1.
global1# snoop -d bge1 icmp
Using device /dev/bge1 (promiscuous mode)
10.1.14.141 -> 139.164.63.38 ICMP Echo request (ID: 4677 Sequence number: 0)
139.164.63.38 -> 10.1.14.141 ICMP Echo reply (ID: 4677 Sequence number: 0)
10.1.14.142 -> 139.164.63.38 ICMP Echo request (ID: 4681 Sequence number: 0)
139.164.63.38 -> 10.1.14.142 ICMP Echo reply (ID: 4681 Sequence number: 0)

And this shows the pings originating from shared3 and shared4 going out on bge2.
global2# snoop -d bge2 icmp
Using device /dev/bge2 (promiscuous mode)
10.1.14.143 -> 139.164.63.38 ICMP Echo request (ID: 4685 Sequence number: 0)
139.164.63.38 -> 10.1.14.143 ICMP Echo reply (ID: 4685 Sequence number: 0)
192.168.16.144 -> 139.164.63.38 ICMP Echo request (ID: 4689 Sequence number: 0)
139.164.63.38 -> 192.168.16.144 ICMP Echo reply (ID: 4689 Sequence number: 0)

Just to confirm where each zone is configured, here is the ifconfig output.
global# ifconfig -a4
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
lo0:1: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
zone shared1
inet 127.0.0.1 netmask ff000000
lo0:2: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
zone shared2
inet 127.0.0.1 netmask ff000000
lo0:3: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
zone shared3
inet 127.0.0.1 netmask ff000000
lo0:4: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
zone shared4
inet 127.0.0.1 netmask ff000000
bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 139.164.63.125 netmask ffffff00 broadcast 139.164.63.255
ether 0:3:ba:e3:42:8b
bge1: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 [9]
inet 0.0.0.0 netmask 0 [8]
ether 0:3:ba:e3:42:8c
bge1:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
zone shared1
inet 10.1.14.141 netmask ffffffc0 broadcast 10.1.14.191
bge1:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
zone shared2
inet 10.1.14.142 netmask ffffffc0 broadcast 10.1.14.191
bge2: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 0.0.0.0 netmask 0 [8]
ether 0:3:ba:e3:42:8d
bge2:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
zone shared3
inet 10.1.14.143 netmask ffffffc0 broadcast 10.1.14.191
bge2:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
zone shared4
inet 192.168.16.144 netmask ffffff00 broadcast 192.168.16.255

5. Using a Single Interface

Only using bge0 and using different subnets for the global and non-global zones. [10]

Before booting the zone.

global# netstat -nr
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 139.164.63.215 UG 1 2 bge0
139.164.63.0 139.164.63.125 U 1 2 bge0
224.0.0.0 139.164.63.125 U 1 0 bge0
127.0.0.1 127.0.0.1 UH 1 42 lo0
global# zonecfg -z shared17 info net
net:
address: 192.168.17.147/24
physical: bge0
defrouter: 192.168.17.16
global# zoneadm -z shared17 boot

Once the zone is booted, netstat shows both default routes, and a ping from the zone works.
global# netstat -rn
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 139.164.63.215 UG 1 2 bge0
default 192.168.17.16 UG 1 0 bge0
139.164.63.0 139.164.63.125 U 1 2 bge0
224.0.0.0 139.164.63.125 U 1 0 bge0
127.0.0.1 127.0.0.1 UH 1 42 lo0
global# ifconfig -a4
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
lo0:1: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
zone shared17
inet 127.0.0.1 netmask ff000000
bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 139.164.63.125 netmask ffffff00 broadcast 139.164.63.255
ether 0:3:ba:e3:42:8b
bge0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
zone shared17
inet 192.168.17.147 netmask ffffff00 broadcast 192.168.17.255
global# zlogin shared17 ping 139.164.63.38
139.164.63.38 is alive

IP Multipathing (IPMP)


I did some testing with IPMP and similar examples as above. At this time the combination of IPMP and the defrouter configuration does not work. I have filed bug 6792116 to have this looked at.

[Updated 2009.01.20]
After some addtional testing, especially with test addresses and probe based failure detection, I have seen IPMP work well only when zones are configured such that at least one zone is on each NIC in an IPMP group, including a standby NIC. For example, if you have two NICs, bge1 and bge2, at least one zone must be configured on bge1 and at least one on bge2. This is even the case when one of the NICs is in failed mode when the system or zone(s) boot. It turns out that the default route is added when the zone boot, and there is no later check for default route requirements as a zone is moved from one NIC to another based on IPMP failover or failback. Thus, I would recommend not using defrouter and IPMP together until the conbination is confirmed to work.

If this is important for your deployments, please add a service record to change request 6792116 and work with your service provide to have this addressed. Please also note that this works well with the IPMP Re-architecture coming soon to OpenSolaris.



Join the discussion

Comments ( 1 )
  • Roberto Cardoni Friday, November 5, 2010

    Hello Steffen,

    fantastic job! I applied your tips at a Customer DataCenter in my country (Italy) and it works very good. I have only one concern regardinig traffic between zones, as local zones on different subnets can "ping" each other. This is a security issue I would like to fix. I've seen some stuff regarding routes with RTF_REJECT flag, may you suggest something?

    Many thanx in advance.

    rgds


Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.