After discussing Oracle VM, OS-Virtualization and aspects of resource
management in the previous articles, we cover in this article a special
area of resource management and virtualization of resources,
Network Virtualization and Network Resource Management
The network is a special shared resource that glues all the VMs,
zones and systems together. The network is their communication channel
with the world. Thus the network is a very important layer of the
Network virtualization is categorized as external or internal.
- External network virtualization combines many
networks, switches, network ports, virtual ports or virtual interfaces
into virtual units or networks. Those virtual units are called virtual
LANs or just VLANs. VLANs are created by using VLAN tags to group
networks from different ports, switches and physical networks together
into one common virtual network. A VLAN tag is an identifier that is
sent together with network packets to identify which packets belong to a
virtual network. A virtual network can also be called a broadcast
domain. That is a group of network participants that all receive a
- Internal network virtualization is the
virtualization of a network stack, network interfaces or other
networking functionality within one system. This virtualization
functionality is provided by the Host-OS or the hypervisor. Internal
network virtualization enables the shared usage of a limited number of
network ports by many VMs, zones or containers. All of the virtualized
environments need their “own” network interfaces and with network
virtualization some physical network interfaces (PNIC) can be “divided”
into many virtual network interfaces (VNIC). This is one of the basic
functionalities of internal network virtualization.
Because of the high usage of the shared resource network by many consumers like processes, VMs, zones or containers, network resource management
is very important in conjunction with network virtualization. This
resource management helps to deliver powerful and stable network
connections to the virtualized environments. The available network
bandwidth can now be better spread between multiple virtualized
environments to meet their service level agreements. Extensive usage of
network virtualization should only be considered together with
well-implemented network resource management.
virtualization and Solaris Zones together with network
virtualization and resource
management enables a whole new range of new capabilities to create network-based
architectures. The picture on the right shows one example, where
physical systems and network components have been replaced by Solaris
Zones and virtual switches.
In this article we concentrate on the functionalities and side
effects of network virtualization and resource management in
conjunction with hypervisors, containers and zones in one system. Here
we concentrate on internal network virtualization.
Features of Internal Network Virtualization
The following base features are common across various type of
hypervisors or zone technologies, however specific implementations
- Virtual network interfaces
are needed to share a small number of physical network ports (PNIC) by a
larger number of VMs or zones - let’s call them consumers. Every
consumer requires its own network interface that it can use as if it
would be a physical port. It is the task of the hypervisor, the host
operating system or the Global Zone to provide this network interface.
The administrator can decide if this network interface is mapped to a dedicated physical port or if it is a virtual interface (VNIC) and then assigned to a shared
physical port. In the latter case the physical port is shared by many
virtual interfaces and resource management features are useful to limit
the bandwidth each virtual interface can make use of. The picture on the
right shows an example of how VNICs are built in Oracle Solaris on top
of physical interfaces and then are used by Solaris Zones. In this
example we also use bandwidth limitations assigned to VNICs.
network switches connect multiple virtual network interfaces that are
created on one physical interface. This makes it possible for VNICs on
one physical port to communicate with each other, but also to share the
physical interface. The feature names for this - used by various
products - differ, but the functionality is similar.
In Oracle VM for x86 this is called a 'bridge', which is automatically
created if a virtual interface is created on a physical port. For
Oracle VM for SPARC a virtual switch has to be created by the admin in
the service domain, where the network interfaces of the guest domains
connect to. The pictures on the right show the examples for Oracle VM
(x86 top, SPARC bottom).
Solaris creates a switch above the physical interface, if the first
VNIC is created.
Oracle VM VirtualBox creates virtual PCI Ethernet cards and assigns
them to VMs as network interfaces. There are different ways as to how
these interfaces communicate with the host operating system or the
outside world (NAT, Bridged Networking, Internal Networking, Host-only
special implementation of a virtual network switch that is only
available in Oracle Solaris 11 is an 'etherstub'. This is a special type
of data link that can be used instead of a physical NIC to create VNICs
and the virtual switches that connect them. With etherstubs, complex
network architectures or just network-in-a-box setups can be created and
tested without needing any physical network switches.
Solaris Zones are used, IP-Interfaces, VNICs or physical interfaces are
provided by the Global Zone. An Oracle Solaris Zone can then use a
shared-IP instance or an exclusive-IP instance to communicate with the
Global Zone or the outside world. With shared-IP instance, the zones
share one IP-stack infrastructure in the kernel with i.g. its arp-cache,
routing table and IP-
configuration flags (not the IP-address).
zone with an exclusive-IP instance has its own IP-stack. To use the
latter one, a dedicated physical interface or virtual network interface
is needed. Using a shared-IP instance does not require a dedicated
network interface. The picture on the right shows the general
Features of network resource management
The network is always a shared resource, either outside the server
chassis by using central cables, switches or routers - or inside the
chassis, by sharing physical ports, network stacks or just the CPUs that
are handling the traffic, doing check-summing or handling the network
adapter interrupts. To meet different service level agreements of
network consumers in one chassis, network resource management is needed.
The requirements can be based on available network bandwidth, network
latency or network data loss rate. While network latency and data loss
rate is typically based on the used network technology and the OS- or
hypervisor-specific implementation, the available bandwidth can be
controlled by resource management. Related to internal network
virtualization, various product-specific implementations exist:
- Dedication of a network port enables the host or
the hypervisor to assign a separate physical port to a consumer. With
this the consumer gets the whole bandwidth of this port, but may need
many network ports, many network adapters and may be limited by the
number of available PCI slots.
- A specific CPU can be assigned to
network interfaces or VNICs to handle their device interrupts, doing the
data buffer handling or computing network checksums. In relation to the
resource management features of the previous article of this series, we
can compare these two functionalities with resource partitioning.
- During the creation of VNICs, an interface-based network bandwidth cap
can be assigned. With that the useable bandwidth is capped on a
configured boundary. This enables the sharing of a physical network port
by many network consumers by limiting the useable bandwidth for each
consumer. This setup is very flexible and can be often changed
dynamically. In the previous article we discussed this functionality as resource constraints.
the previous network bandwidth capping is interface based, there is
also a need to control the bandwidth on a network connection base. Such a
network connection can be described by a source-IP, a destination-IP
address and by a protocol. In Oracle Solaris this is called a 'flow'.
The configured flows can be used to control network bandwidth
independently of network interfaces, only on a connection base. The
picture on the right side shows an example. A configured flow for the
network data type “network backup” can be used, to give the “green” and
the “blue” traffic more available bandwith in critical load situations.
Compared to the basic resource management functionalities of the
previous article we can compare this with resource scheduling, because if “green” and “blue” do not have bandwidth needs, “network backup” can get the maximum available bandwidth.
Virtual network interfaces, virtual bridges, virtual switches or
virtual PCI ethernet cards are basic internal network virtualization
features that are part of virtualization products. The networking
'glues' all the VMs, zones or containers together and let them
communicate together or with the outside world. To enable stable
communication for all of them on the shared resource network, the use of
network resource management features is recommended. We have also seen
that for networks, various types of resource managements like
constraints, scheduling or partitioning are used.
With that we'd like to close this article on Network Virtualization
and hope we've kept you eager to read the ones coming up in the
This series already had the following articles:
- December 2011: Introduction to Virtualization (Matthias Pfützner)
- January 2012: Oracle VM Server for SPARC (Matthias Pfützner)
- February 2012: Oracle VM Server for x86 (Matthias Pfützner)
- March 2012: Oracle Solaris Zones and Linux Containers (Detlef Drewanz)
- April 2012: Resource Management (Detlef Drewanz)
The series will continue as follows (tentative):
- June 2012: Oracle VM VirtualBox (Detlef Drewanz)
- July 2012: Oracle Virtual Desktop Infrastructure (VDI) (Matthias Pfützner)
- August 2012: OpsCenter as Management Tool for Virtualization (Matthias Pfützner)
If you have questions, feel free to contact me at: