Thursday Feb 14, 2008

Patches for Using IP Instances with ce NICs are Available

The [Solaris 10] patches to be able to use IP Instances with the Cassini ethernet interface, known as ce, are available on for Solaris 10 users with a maintenance contract or subscription. (This is for Solaris 10 8/07, or a prior update patched to that level. These patches are included in Solaris 10 5/08, and also in patch clusters or bundles delivered at or around the same time, and since then.)

The SPARC patches are:

  • 137042-01 SunOS 5.10: zoneadmd patch
  • 118777-12 SunOS 5.10: Sun GigaSwift Ethernet 1.0 driver patch

The x86 patches are:

  • 137043-01 SunOS 5.10_x86: zoneadmd patch
  • 118778-11 SunOS 5.10_x86: Sun GigaSwift Ethernet 1.0 driver patch

I have not been able to try out the released patches myself, yet.


Thursday Dec 20, 2007

One Step Closer to IP Instances with ce

With the availability of Solaris Nevada build 80 [1], the ability to use IP Instances with the GigaSwift line of NICs and the ce driver becomes possible. The fix for CR 6616075 to zoneadmd(1M) has been integrated into the OpenSolaris code base and is available in build 80. The necessary fix to the ce driver, tracked in CR 6606507, has already been delivered. With this combination, a zone can have an exclusive IP Instance using a ce-based link.

Zone configuration information:

global# zonecfg -z ce1 info net
        address not specified
        physical: ce1

And the view from the non-global zone:

ce1# zonename
ce1# cat /etc/release
                  Solaris Express Community Edition snv_80 SPARC
           Copyright 2008 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                           Assembled 17 December 2007
ce1# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet netmask ff000000
ce1: flags=1000843 mtu 1500 index 2
        inet netmask ffffff00 broadcast
        ether 0:3:ba:68:1d:5f
lo0: flags=2002000849 mtu 8252 index 1
        inet6 ::1/128

More when the soak time in Nevada is complete and the backport to Solaris 10 is available.

Thanks to the engineers who put energy into these fixes!

Happy Holidays!


[1] As of 20 December 2007, build 80 is available within Sun only. Availability on will be announced on

Wednesday Dec 05, 2007

More good news for IP Instances

Continuing progress on the use of IP Instances on the full line of SPARC systems. The e1000g Intel PCI-X Gigabit Ethernet UTP and MMF adapters are now supported on the Sun Fire UltraSPARC servers. The NICs are:
  • x7285a - Sun PCI-X Dual GigE UTP Low Profile. RoHS-6 compliant
  • x7286a - Sun PCI-X GigE MMF Low Profile, RoHS-6 compliant
The NICs are supported on the V490, V890, E2900, E4900, E6900, E20K, and E25K systems. This is an alternative for those waiting for the GigaSwift (ce) NIC to be supported, or who don't need quad-port cards. Since the driver used is the e1000g, which is a GLDv3 driver, full support for IP Instances is available using these cards.

Monday Nov 05, 2007

Using IP Instances with VLANs or How to Make a Few NICs Look Like Many

[Minor editorial and clarification updates 2009.09.28]

Solaris 10 8/07 includes a new feature for zone networking. IP Instances is the facility to give a non-global zone its own complete control over the IP stack, which previously was shared with and controlled by the global zone.

A zone that has an exclusive IP Instance can set interface parameters using ifconfig(1M), put an interface into promiscuous mode to run snoop(1M), be a DHCP client or server, set ndd(1M) variables, have its own IPsec policies, etc.

One requirement for an exclusive IP Instance is that it must have exclusive access to a link name. This is any NIC, VLAN-tagged NIC component, or aggregation at this time. When they become available, virtual NICs will make this much simpler, as a single NIC can be presented to the zones using a number of VNICs, effectively multiplexing access to that NIC. A link name is an entry that can be found in /dev, such as /dev/bge0, /dev/bge321001 (VLAN tag 321 on bge1), aggr2, and so on.

To see what link names are available on a system, use dladm(1M) with the show-link option. For example:

global# dladm show-link
bge0            type: non-vlan  mtu: 1500       device: bge0
bge1            type: non-vlan  mtu: 1500       device: bge1
bge2            type: non-vlan  mtu: 1500       device: bge2
bge3            type: non-vlan  mtu: 1500       device: bge3

As folks have started to use IP Instances to isolate their zones, they have noticed that they don't have sufficient link names (I'll use just link in the rest of this) to assigned to the zones that have or wish to configure as exclusive. So, how does a global zone administrator configure a large number of zones as exclusive?

Let's consider the following situation, where there are three tiers of a web service, where each tier is on a different network.

If each server has only one NIC, the total number of switch ports required is at least eight (8). If each server has a management port, that is another eight ports, even if they are on a different, management network. Add to that at least three three switch ports going to the router.

Consolidating the servers onto a single Solaris 10 instance using exclusive IP Instances requires at least eight NICs for the services (one per service), and at least one for the global zone and management. (We'll ignore a service process requirements, since they are separate anyway, and access could be either via a serial interface or a network.)

One option to consider is using VLANs and VLAN tagging. When using VLAN tagging, additional information is put onto the ethernet frame by the sender which allows the receiver to associated that frame to a specific VLAN. The specification allow up to 4094 VLAN tags, from 1 to 4094. For more information on administering VLANs in Solaris 10, see Administering Virtual Local Area Networks in the Solaris 10 System Administrator Collection.

VLANs is a method to collapse multiple ethernet broadcast domains (whether hubs or switches) into a single network unit (usually a switch). [Typically, a single IP subnet, such as, is on a broadcast domain. Within such a switch frame, you can have a large number of virtual switches, consolidating network infrastructure and still isolating broadcast domains. Often, the use of VLANs is completely hidden from the systems tied to the switch, as a port on the switch is configured for only one VLAN. With VLAN tagging, a single port can allow a system to connect to multiple VLAns, and therefore multiple networks. Both the switch and the system must be configured for VLAN tagging for this to work properly. VLAN tagging has been used for years, and is robust and reliable.

Any one network interface can have multiple VLANs configured for it, but a single VLAN ID can only exist once on each interface. Thus it is possible to put multiple networks or broadcast domains on a single interface. It is not possible to put more than one VLAN of any broadcast domain on a single interface. For example, you can put VLANs 111, 112, and 113 on interface bge1, but you can not put VLAN 111 on bge1 more than once. You can, however, put VLAN 111 on interfaces bge1 and bge2.

Using the case shown above, if the three web servers are on the same network, say, you would want to have three interfaces that are all connected to a VLAN capable switch, and configure each interface with a VLAN tag that is the same as the VLAN ID on the switch.

For example, if the VLAN tag is 111 and the interfaces are bge1 through bge3, the link names you would assign to the three web servers would be bge111001, bge111002, and bge111003.

Introducing zones into the setup, the web servers can be run in three separate zones, and with exclusive IP Instances, they can be totally separate and each assigned a VLAN-tagged interface. Web Server 1 could have bge111001, Web Server 2 could have bge111002, and Web Server 3 could have bge111003.

global# zonecfg -z web1 info net
        address not specified
        physical: bge111001

global# zonecfg -z web2 info net
        address not specified
        physical: bge111002

global# zonecfg -z web3 info net
        address not specified
        physical: bge111003

Within the zones, you could configure IP addresses through

Similarly, for the authentication tier, using VLAN ID 112, you could assign the zones auth1 through auth3 to bge112001, bge112002, and bge112003,respectively. And for application servers app1 and app2 on VLAN ID 113, bge113001 and bge113002. This can be repeated until some limit is reached, whether it is network bandwidth, system resource limits, or the maximum number of concurrent VLANs on either the switch or Solaris.

This configuration could look like the following diagram.

Web Server 1, Auth Server 1, and Application Server 1 share the use of NIC1, yet are all on different VLANs (111, 112, and 113, respectively). The same for instances 2 and 3, except that there is no third application server. All traffic between the three web servers will stay within the switch, as will traffic between the authentication servers. Traffic between the tiers is passed between the IP networks by the router. NICg is showing that the global zone also has a network interface.

Using this technique, the maximum number of zones with exclusive IP Instances you could deploy on a single system that are on the same subnet is limited to the number of interfaces that are capable of doing VLAN tagging. In the above example, with three bge interfaces on the system, the maximum number of exclusive zones on a single subnet would be three. (I have intentionally reserved bge0 for the global zone, but it would be possible to use it as well, making sure the global zone uses a different VLAN ID altogether, such as 1 or 2.)




« April 2014