X

Using Solaris and SPARC Networking and Virtualization

VLANs and Aggregations

Every once in a while I see the question asking whether it is possible to use IEEE 802.1q VLANs together with IEEE 802.3ad Link Aggregation. I frequently have to check myself. So in order to better remind me, and share with others, here is a quick demonstration of how to get the two working together.

My test system is running build 05 of the upcoming Solaris 10 10/09 (update 8). The system has four bge interfaces, and I will use numbers 1 and 2. (This should work just as well with previous updates of Solaris 10, and with Sun Trunking in Solaris 9, except for the zones parts. I am using zones just to isolate my traffic generation and easily get it to use a specific data link.)

Starting out things like like this.

global# dladm show-dev
bge0 link: up speed: 1000 Mbps duplex: full
bge1 link: unknown speed: 0 Mbps duplex: unknown
bge2 link: unknown speed: 0 Mbps duplex: unknown
bge3 link: unknown speed: 0 Mbps duplex: unknown
global# dladm show-link
bge0 type: non-vlan mtu: 1500 device: bge0
bge1 type: non-vlan mtu: 1500 device: bge1
bge2 type: non-vlan mtu: 1500 device: bge2
bge3 type: non-vlan mtu: 1500 device: bge3
global# ifconfig -a4
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 129.154.53.125 netmask ffffff00 broadcast 129.154.53.255
ether 0:3:ba:e3:42:8b

I have my switch set up to aggregate ports 1 and 2, and here is how I do it with Solaris 10.
global# dladm create-aggr -d bge1 -d bge2 1
global# dladm show-link
bge0 type: non-vlan mtu: 1500 device: bge0
bge1 type: non-vlan mtu: 1500 device: bge1
bge2 type: non-vlan mtu: 1500 device: bge2
bge3 type: non-vlan mtu: 1500 device: bge3
aggr1 type: non-vlan mtu: 1500 aggregation: key 1

VLAN tagged interfaces are used by accessing the underlying data link by preceeding the data link ID with the VLAN tag. For bge1 and VLAN 111 that would be bge111001. For for aggr1 it would be aggr111001.

For this setup I am using zones zone111 and zone112 configured as an exclusive IP Instance. The zone configuration look like this.

global# zonecfg -z zone111 info
zonename: zone111
zonepath: /zones/zone111
brand: native
autoboot: false
bootargs:
pool:
limitpriv:
scheduling-class:
ip-type: exclusive
inherit-pkg-dir:
dir: /lib
inherit-pkg-dir:
dir: /platform
inherit-pkg-dir:
dir: /sbin
inherit-pkg-dir:
dir: /usr
net:
address not specified
physical: aggr111001
defrouter not specified

Once configured, installed, and booted, the network configuration of zone111 is:
global# zlogin zone111 ifconfig -a4
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
aggr111001: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2
inet 172.16.111.141 netmask ffffff00 broadcast 172.16.111.255
ether 0:3:ba:e3:42:8c

Turns out that configuring this was easy compared to showing that the link aggregation was really working. While the full list of links known when the zones are includes the aggregation and the VLANs on the aggregation, tools such a netstat or nicstat would not include them. As it turns out they only report on interfaces that are plumbed up in that IP Instance. It will not be possible to plumb either bge1 or bge2 since they are members of the aggregation.
global# dladm show-link
bge0 type: non-vlan mtu: 1500 device: bge0
bge1 type: non-vlan mtu: 1500 device: bge1
bge2 type: non-vlan mtu: 1500 device: bge2
bge3 type: non-vlan mtu: 1500 device: bge3
aggr1 type: non-vlan mtu: 1500 aggregation: key 1
aggr111001 type: vlan 111 mtu: 1500 aggregation: key 1
aggr112001 type: vlan 112 mtu: 1500 aggregation: key 1
global# netstat -i
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
lo0 8232 loopback localhost 98 0 98 0 0 0
bge0 1500 pinebarren pinebarren 43101 0 7181 0 0 0

So I ended up using kstat(1M) to get the values of the number of outbound packets. I an interested in outbound as that is what Solaris can affect regarding distributing traffic across links in an aggregation--the switch determines that for inbound traffic.

This example shows data on instance 2 of the bge interface for kstat value opackets.

global# kstat -m bge -i 2 -s opackets
module: bge instance: 2
name: mac class: net
opackets 2542

With kstat I can see that for different connections either bge1 or bge2 has packets going out on it. A good test for me was scp to a remote system. Neither ping nor traceroute caused the necessary hashing to use both links in the aggregation.

Steffen

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.