The first articles in this series discussed general topics in virtual machine performance, and gave a brief architectural overview for Oracle VM Server for x86 and Oracle VM Server for SPARC. In this article we'll discuss performance goals for virtual machine environments, especially in the cloud, and some things that are different in the virtual world from the bare-metal world.
Performance goals for virtual machine environments include:
Some parts of performance management to achieve the above goals are unchanged from physical servers. As mentioned in the previous article in this series, you measure service levels and hunt down bottlenecks that restrict performance much like in bare-metal.
However, some rules are different with VMs or in the cloud, notably with resource management. Sometimes “less is more”, and overcommitting resources to a VM may just mean wasting them while leaving fewer resources for other VMs. Counter-intuitively, this can even reduce performance.
Normally, we think "the more CPUs, the better the performance". This is not strictly true in a non-VM environment because excess CPUs can increase VM’s SMP overhead, but having too-many virtual CPUs can visibly harm performance in virtual machines. This is more problematic in a VM since the guest OS is not generally aware of CPU and memory topology, and therefore may suffer from NUMA (Non Uniform Memory Access) latency. Many applications don’t scale with more CPUs, so there's no point in adding more CPUs, regardless of whether real or virtual.
Another serious problem is excessive multiprogramming in which CPU capacity is over-subscribed and time-sliced to too many clients. Oracle VM Server for SPARC avoid this issue by not over-subscribing CPUs. The hypervisor dedicates CPUs to guests, which can see only their CPUs, and use them without competition for CPU cycles. This insulates SPARC virtual machines from the "noisy neighbor" problem, and provides deterministic, guaranteed access to CPU capacity regardless of other VM activity. Predictable performance is an important component for production workloads.
In contrast, CPUs can be over-subscribed on Oracle VM Server for x86, subject to contention and service requirements. Jitter and performance can worsen if the aggregate number of virtual CPUs exceeds physical CPUs while under load. There's generally no problem defining more virtual CPUs than physical CPUs (some people ask what the maximum ratio should be - but there's really no limit) as long as the total demand for CPU cycles doesn't exceed capacity.
Oracle VM Server for x86 uses a share-based "credit scheduler" to allocate CPU cycles proportionally to VMs. This makes it possible to ensure performance for important VMs even under high CPU loads, and it is possible to drive servers near 100% CPU busy. This is best for workloads where there are low-priority "cycle soaker" VMs with loose SLAs that are not sensitive to timing and can absorb whatever capacity is not used by more important VMs. Such workloads can tolerate "CPU stealing" in which they were runnable (virtual CPUs had work to run), but there were no physical cycles to run them. One still has to be careful, as even a batch workload may have time-sensitive components, such as an interactive interface or network connection. An open problem since the mid-1970s is how to briefly deliver a short burst of high priority CPU access to VMs that are otherwise designated as low-priority. In the absence of relaxed quality of service requirements, the best practice is to provision enough capacity to satisfy all VMs. In practice, most customers over-provision CPU capacity because the cost of CPU capacity is much less than the cost of missing service objectives.
In any case, no individual VM should ever have more virtual CPUs than the number of physical CPUs on a server. That just results in the VM's CPUs competing against one another for real CPU cycles, and can expose problems with lock management in the guest (consider a case where a VM is spinning on a lock on one virtual CPU waiting for another virtual CPU to release it - but that vCPU isn't receiving any service).
Ideally, all of a VM's CPUs fit in a single server socket, in which case Oracle VM Server for x86 attempt to assign the VM's memory and CPUs on the same socket for local CPU-to-memory latency. This has a powerful effect on performance, as remote memory latency on multi-socket systems is much higher than local latency. A big design point of Oracle VM's Xen hypervisor is to avoid this NUMA latency.
It may be surprising, but memory considerations often play a bigger role than CPU. Even when we manage CPUs we are really managing memory performance, as shown in the previous paragraph.
This is an area in which the rules for VMs differ sharply from real machine environments. On bare metal you assign all the memory to the OS and its applications, or it just goes to waste. Most customers provision their hardware from a small number of configurations, and choose the server model that meets or exceeds an application's peak needs. Any excess memory is used for file buffering, and if there's no need for it, no harm is done. Because the penalty for oversubscribing memory is fairly low, it's not uncommon for memory requirements to be at best approximate.
It works differently in a cloud or VM environment. Excess memory assigned to a VM reduces the number of VMs that can be hosted, which increases the cost of the cloud or hosting service. Within a VM, excess memory increases NUMA latency since it might spread over sockets.
Consequently, in cloud and VM environments it is important to measure baseline memory requirements to reduce costs and retain performance. This is more difficult than CPU over-provisioning as it's easy in most hypervisors to measure CPU utilization and shift load as needed. With memory you have to ask the OS whether it has more virtual RAM than it needs. This can be done using standard OS tools common to Linux, Solaris and Windows, or can be simplified by using a product like Oracle Enterprise Manager than can introspect and instrument within VMs.
Note that neither Oracle VM Server for SPARC nor Oracle VM Server for x86 over-subscribe memory: they do not swap or page VM memory contents to a swap file, and you must have enough real memory to back the virtual machine's defined memory size. This is intentional, and avoids serious performance problems. Overcommitting VM memories is a 40+ year old performance problem that has not really been solved, despite substantial ingenuity and engineering. VMs have poor temporal or address locality of reference, and tend to touch every page defined to them, even for file buffers that may or may not ever be used. As a result the size of their workings sets (recently touched pages) tends to be near the defined memory size. Modern operating systems themselves potentially swap or page their application's virtual memories, and it running a virtual memory guest OS under a virtual memory hypervisor can cause pathological situations where the classic LRU algorithm to select a page for swap out breaks down, and the most recently used pages get selected for displacement instead.
Hypervisors that swap memory generally are used with modest overcommit ratios (1.3:1 or 1.5:1 is often cited) to avoid this issue, but it's hard to tell in advance when the ratio is just a bit too high and performance would suffer.
Oracle VM avoids this unpredictable risk entirely by not-overcommitting memory. Better to just avoid the issue entirely and give the VMs the memory they
need, just as in modern non-virtual environments we configure
sufficient memory for a server's applications without swapping.
This blog entry describes performance goals for the VM and cloud environment, and describes CPU, I/O, and memory considerations for reaching those goals, with an emphasis on performance behavior that differs from real-machine environments. The next article will drill down into specifics of how these principles are applied with Oracle VM Server for x86.