Virtualization @ Oracle (Part 6: Network Virtualization and Network Resource Management)
By uwes on May 21, 2012
After discussing Oracle VM, OS-Virtualization and aspects of resource management in the previous articles, we cover in this article a special area of resource management and virtualization of resources,
Network Virtualization and Network Resource Management
The network is a special shared resource that glues all the VMs, zones and systems together. The network is their communication channel with the world. Thus the network is a very important layer of the virtualization stack.
Network virtualization is categorized as external or internal.
- External network virtualization combines many networks, switches, network ports, virtual ports or virtual interfaces into virtual units or networks. Those virtual units are called virtual LANs or just VLANs. VLANs are created by using VLAN tags to group networks from different ports, switches and physical networks together into one common virtual network. A VLAN tag is an identifier that is sent together with network packets to identify which packets belong to a virtual network. A virtual network can also be called a broadcast domain. That is a group of network participants that all receive a network broadcast.
- Internal network virtualization is the virtualization of a network stack, network interfaces or other networking functionality within one system. This virtualization functionality is provided by the Host-OS or the hypervisor. Internal network virtualization enables the shared usage of a limited number of network ports by many VMs, zones or containers. All of the virtualized environments need their “own” network interfaces and with network virtualization some physical network interfaces (PNIC) can be “divided” into many virtual network interfaces (VNIC). This is one of the basic functionalities of internal network virtualization.
Because of the high usage of the shared resource network by many consumers like processes, VMs, zones or containers, network resource management is very important in conjunction with network virtualization. This resource management helps to deliver powerful and stable network connections to the virtualized environments. The available network bandwidth can now be better spread between multiple virtualized environments to meet their service level agreements. Extensive usage of network virtualization should only be considered together with well-implemented network resource management.
virtualization and Solaris Zones together with network
virtualization and resource
management enables a whole new range of new capabilities to create network-based
architectures. The picture on the right shows one example, where physical systems and network components have been replaced by Solaris Zones and virtual switches.
In this article we concentrate on the functionalities and side effects of network virtualization and resource management in conjunction with hypervisors, containers and zones in one system. Here we concentrate on internal network virtualization.
Features of Internal Network Virtualization
The following base features are common across various type of hypervisors or zone technologies, however specific implementations differ.
- Virtual network interfaces are needed to share a small number of physical network ports (PNIC) by a larger number of VMs or zones - let’s call them consumers. Every consumer requires its own network interface that it can use as if it would be a physical port. It is the task of the hypervisor, the host operating system or the Global Zone to provide this network interface. The administrator can decide if this network interface is mapped to a dedicated physical port or if it is a virtual interface (VNIC) and then assigned to a shared physical port. In the latter case the physical port is shared by many virtual interfaces and resource management features are useful to limit the bandwidth each virtual interface can make use of. The picture on the right shows an example of how VNICs are built in Oracle Solaris on top of physical interfaces and then are used by Solaris Zones. In this example we also use bandwidth limitations assigned to VNICs.
network switches connect multiple virtual network interfaces that are
created on one physical interface. This makes it possible for VNICs on
one physical port to communicate with each other, but also to share the
physical interface. The feature names for this - used by various
products - differ, but the functionality is similar.
In Oracle VM for x86 this is called a 'bridge', which is automatically
created if a virtual interface is created on a physical port. For
Oracle VM for SPARC a virtual switch has to be created by the admin in
the service domain, where the network interfaces of the guest domains
connect to. The pictures on the right show the examples for Oracle VM
(x86 top, SPARC bottom).
Oracle Solaris creates a switch above the physical interface, if the first VNIC is created. Oracle VM VirtualBox creates virtual PCI Ethernet cards and assigns them to VMs as network interfaces. There are different ways as to how these interfaces communicate with the host operating system or the outside world (NAT, Bridged Networking, Internal Networking, Host-only networking)
- A special implementation of a virtual network switch that is only available in Oracle Solaris 11 is an 'etherstub'. This is a special type of data link that can be used instead of a physical NIC to create VNICs and the virtual switches that connect them. With etherstubs, complex network architectures or just network-in-a-box setups can be created and tested without needing any physical network switches.
Solaris Zones are used, IP-Interfaces, VNICs or physical interfaces are
provided by the Global Zone. An Oracle Solaris Zone can then use a
shared-IP instance or an exclusive-IP instance to communicate with the
Global Zone or the outside world. With shared-IP instance, the zones
share one IP-stack infrastructure in the kernel with i.g. its arp-cache,
routing table and IP-
configuration flags (not the IP-address).
A zone with an exclusive-IP instance has its own IP-stack. To use the latter one, a dedicated physical interface or virtual network interface is needed. Using a shared-IP instance does not require a dedicated network interface. The picture on the right shows the general difference.
Features of network resource management
The network is always a shared resource, either outside the server chassis by using central cables, switches or routers - or inside the chassis, by sharing physical ports, network stacks or just the CPUs that are handling the traffic, doing check-summing or handling the network adapter interrupts. To meet different service level agreements of network consumers in one chassis, network resource management is needed. The requirements can be based on available network bandwidth, network latency or network data loss rate. While network latency and data loss rate is typically based on the used network technology and the OS- or hypervisor-specific implementation, the available bandwidth can be controlled by resource management. Related to internal network virtualization, various product-specific implementations exist:
- Dedication of a network port enables the host or the hypervisor to assign a separate physical port to a consumer. With this the consumer gets the whole bandwidth of this port, but may need many network ports, many network adapters and may be limited by the number of available PCI slots.
- A specific CPU can be assigned to network interfaces or VNICs to handle their device interrupts, doing the data buffer handling or computing network checksums. In relation to the resource management features of the previous article of this series, we can compare these two functionalities with resource partitioning.
- During the creation of VNICs, an interface-based network bandwidth cap can be assigned. With that the useable bandwidth is capped on a configured boundary. This enables the sharing of a physical network port by many network consumers by limiting the useable bandwidth for each consumer. This setup is very flexible and can be often changed dynamically. In the previous article we discussed this functionality as resource constraints.
- While the previous network bandwidth capping is interface based, there is also a need to control the bandwidth on a network connection base. Such a network connection can be described by a source-IP, a destination-IP address and by a protocol. In Oracle Solaris this is called a 'flow'. The configured flows can be used to control network bandwidth independently of network interfaces, only on a connection base. The picture on the right side shows an example. A configured flow for the network data type “network backup” can be used, to give the “green” and the “blue” traffic more available bandwith in critical load situations. Compared to the basic resource management functionalities of the previous article we can compare this with resource scheduling, because if “green” and “blue” do not have bandwidth needs, “network backup” can get the maximum available bandwidth.
Virtual network interfaces, virtual bridges, virtual switches or virtual PCI ethernet cards are basic internal network virtualization features that are part of virtualization products. The networking 'glues' all the VMs, zones or containers together and let them communicate together or with the outside world. To enable stable communication for all of them on the shared resource network, the use of network resource management features is recommended. We have also seen that for networks, various types of resource managements like constraints, scheduling or partitioning are used.
With that we'd like to close this article on Network Virtualization and hope we've kept you eager to read the ones coming up in the following newsletters.
This series already had the following articles:
- December 2011: Introduction to Virtualization (Matthias Pfützner)
- January 2012: Oracle VM Server for SPARC (Matthias Pfützner)
- February 2012: Oracle VM Server for x86 (Matthias Pfützner)
- March 2012: Oracle Solaris Zones and Linux Containers (Detlef Drewanz)
- April 2012: Resource Management (Detlef Drewanz)
The series will continue as follows (tentative):
- June 2012: Oracle VM VirtualBox (Detlef Drewanz)
- July 2012: Oracle Virtual Desktop Infrastructure (VDI) (Matthias Pfützner)
- August 2012: OpsCenter as Management Tool for Virtualization (Matthias Pfützner)
If you have questions, feel free to contact me at: Detlef Drewanz
|<<< Part 5: Resource Management as Enabling Technology for Virtualization||>>> Part 7: Oracle VM VirtualBox