VXLAN in Solaris 11.2
By venu.iyer-Oracle on Aug 14, 2014
What is a VXLAN?
VXLAN, or Virtual eXtensible LAN, is essentially a tunneling mechanism used to provide isolated virtual Layer 2 (L2) segments that can span multiple physical L2 segments. Since it is a tunneling mechanism it uses IP (IPv4 or IPv6) as its underlying network which means we can have isolated virtual L2 segments over networks connected by IP. This allows Virtual Machines (VM) to be in the same L2 segment even if they are located on systems that are in different physical networks. Some of the benefits of VXLAN include:
- Better use of resources, i.e. VMs can be provisioned on systems, that span different geographies, based on system load.
- VMs can be moved across systems without having to reconfigure the underlying physical network.
- Fewer MAC address collision issues, i.e. MAC address may collide as long as they are in different VXLAN segments.
Additional details, including protocol working, can be found in the VXLAN draft IETF RFC. Note that Solaris uses the IANA specified UDP port number of 4789 for VXLAN.
The following illustrates how VXLANs are created on Solaris:
# ipadm show-addr net4ADDROBJ TYPE STATE ADDRnet4/v4 static ok 10.10.10.1/24
# dladm create-vxlan -p addr=10.10.10.1,vni=100 vxlan1# dladm create-vxlan -p addr=10.10.10.1,vni=101 vxlan2
- In the above example we explicitly provide the IP address, however, you could also:
- provide a prefix and prefixlen to use an IP address that matches it, e.g:
# dladm create-vxlan -p addr=10.10.10.0/24,vni=100 vxlan1
- provide an interface (say net4 in our case) to pick an active address on that interface, e.g:
# dladm create-vxlan -p interface=net4,vni=100 vxlan1
(you can't provide interface and addr together)
- VXLAN links can be created on an IP address over any interface, including IPoIB link, except IPMP, loopback or VNI (Virtual Network Interface).
- The IP address may belong to a VLAN segment.
# dladm show-vxlanLINK ADDR VNI MGROUPvxlan1 10.10.10.1 100 18.104.22.168vxlan2 10.10.10.1 101 22.214.171.124
# dladm show-linkLINK CLASS MTU STATE OVER...vxlan1 vxlan 1440 up --vxlan2 vxlan 1440 up --
# dladm create-vnic -l vxlan1 vnic1# dladm create-vnic -l vxlan1 vnic2# dladm create-vnic -l vxlan2 vnic3# dladm create-vnic -l vxlan2 vnic4
# dladm show-vnicLINK OVER SPEED MACADDRESS MACADDRTYPE VIDSvnic1 vxlan1 10000 2:8:20:d9:df:5f random 0vnic2 vxlan1 10000 2:8:20:72:9a:70 random 0vnic3 vxlan2 10000 2:8:20:19:c7:14 random 0vnic4 vxlan2 10000 2:8:20:88:98:6d random 0
A VXLAN can be deleted once all the VNICs over the VXLAN links have been deleted. Thus in our case:
# dladm delete-vnic vnic1# dladm delete-vnic vnic2# dladm delete-vnic vnic3# dladm delete-vnic vnic4
# dladm delete-vxlan vxlan1# dladm delete-vxlan vxlan2
- VXLAN for Solaris Kernel zone and LDom guests are not supported with direct I/O.
- Hardware capabilities such as checksum and LSO are not available for the encapsulated (inner) packet.
- Some earlier implementations (e.g. Linux) might use a pre-IANA assigned port number. If so, such implementations might have to be configured to use the IANA port number to interoperate with Solaris VXLAN.
- IP multicast must be available in the underlying network and if communicating across different IP subnets, multicast routing should be available as well.
- Modifying properties (IP address, multicast address or VNI) on a VXLAN link is currently not supported; you'd have to delete the VXLAN and re-create it.