Oracle VM Performance and Tuning - Part 3

The first articles in this series discussed general topics in virtual machine performance, and gave a brief architectural overview for Oracle VM Server for x86 and Oracle VM Server for SPARC.  In this article we'll discuss performance goals for virtual machine environments, especially in the cloud, and some things that are different in the virtual world from the bare-metal world.

Performance goals for the cloud and VM environments

Performance goals for virtual machine environments include:

• Run workloads as close to bare metal performance as possible - there should be as little "virtualization tax" as possible. There typically is overhead, but Oracle virtualization technologies do a very good job of minimizing it. That includes the Oracle VM hypervisors, the extremely low-overhead environments of Solaris zones (and the new kernel zone capability in Solaris 11.2 and 11.3) and related technologies like Linux containers and Docker.
  
• Support large servers efficiently: Large servers are ideal consolidation platforms, and permit mixing small and large workloads on the same server. Driving factors for large servers are economy of scale and ability to run very large VMs. It is not uncommon to see consolidation platforms with over 100 CPUs and multi-TB memory. On the other hand, when virtual machines are smaller, especially in the cloud space where you don't see massive monolithic individual applications, you commonly see more moderate sized servers, two (very popular) to 8 socket servers with 256GB to 1TB RAM. That permits horizontal scaleout with granular acquisition costs.
  
• Support large and small VMs: This pattern is seen on SPARC systems and the largest x86 servers, and VMs may have over 100 vCPUs, and 512GB, 1TB, 2TB or even more RAM. This is for the biggest vertically scaled applications, which can perform very well under Oracle VM. More commonly, VM sizes are 4 to 32 vCPUs and 16GB to 128GB RAM. This is typical for most cloud deployments too. The goal is to support a number of very large virtual machines and a much larger number of smaller virtual machines, and provide good performance to all of them. Heterogeneous virtual machine populations goes all the way back to the earliest mainframe VM days on VM/370, where you might have a few large (for the day) MVS guests running multiple batch jobs or OLTP monitors, and a much larger number of interactive virtual machines running VM's Conversational Monitor System (CMS). Providing good (subsecond) interactive response time while servicing large batch workloads was a challenging problem.
• Drive high I/O rates to disk and network. This has always been a challenge in VM environments because of the need to map virtual I/O onto real I/O devices through an intermediary software layer. It could be very difficult to get high I/O rates and not grow latency. 10GbE networking near line rate is a performance challenge for today's networked applications, and can be difficult to achieve. High speed networks generate high interrupt rates, and servicing them in a VM environment can be difficult due to the context switch rate, cache contents displacement, and NUMA effects of multi-socket servers. We'll discuss how Oracle VM solves this problem in a following article.
• Scale to dozens of servers, hundreds of VMs, hundreds of LUNs and paths under single  pane of glass control. Modern hosting facilities provide service at high scale under a common administrative interface. One example of this is the Oracle Managed Cloud Services environment, which operates thousands of servers and many times that number of VMs. Oracle VM and Oracle Enterprise Manager provide integrated and extensive management capabilities.
  
• Do all the above without much tuning or manual intervention, as there are too many tenants to provide individual attention to all of them!
  
  

Performance concepts for the cloud and VM environments

Some parts of performance management to achieve the above goals are unchanged from physical servers. As mentioned in the previous article in this series, you measure service levels and hunt down bottlenecks that restrict performance much like in bare-metal.

However, some rules are different with VMs or in the cloud, notably with resource management. Sometimes “less is more”, and overcommitting resources to a VM may just mean wasting them while leaving fewer resources for other VMs. Counter-intuitively, this can even reduce performance.

Virtual CPUs

Normally, we think "the more CPUs, the better the performance". This is not strictly true in a non-VM environment because excess CPUs can increase VM’s SMP overhead, but having too-many virtual CPUs can visibly harm performance in virtual machines. This is more problematic in a VM since the guest OS is not generally aware of CPU and memory topology, and therefore may suffer from NUMA (Non Uniform Memory Access) latency. Many applications don’t scale with more CPUs, so there's no point in adding more CPUs, regardless of whether real or virtual.

Another serious problem is excessive multiprogramming in which CPU capacity is over-subscribed and time-sliced to too many clients. Oracle VM Server for SPARC avoid this issue by not over-subscribing CPUs. The hypervisor dedicates CPUs to guests, which can see only their CPUs, and use them without competition for CPU cycles. This insulates SPARC virtual machines from the "noisy neighbor" problem, and provides deterministic, guaranteed access to CPU capacity regardless of other VM activity. Predictable performance is an important component for production workloads.

In contrast, CPUs can be over-subscribed on Oracle VM Server for x86, subject to contention and service requirements.  Jitter and performance can worsen if the aggregate number of virtual CPUs exceeds physical CPUs while under load. There's generally no problem defining more virtual CPUs than physical CPUs (some people ask what the maximum ratio should be - but there's really no limit) as long as the total demand for CPU cycles doesn't exceed capacity.

Oracle VM Server for x86 uses a share-based "credit scheduler" to allocate CPU cycles proportionally to VMs. This makes it possible to ensure performance for important VMs even under high CPU loads, and it is possible to drive servers near 100% CPU busy. This is best for workloads where there are low-priority "cycle soaker" VMs with loose SLAs that are not sensitive to timing and can absorb whatever capacity is not used by more important VMs. Such workloads can tolerate "CPU stealing" in which they were runnable (virtual CPUs had work to run), but there were no physical cycles to run them. One still has to be careful, as even a batch workload may have time-sensitive components, such as an interactive interface or network connection. An open problem since the mid-1970s is how to briefly deliver a short burst of high priority CPU access to VMs that are otherwise designated as low-priority. In the absence of relaxed quality of service requirements, the best practice is to provision enough capacity to satisfy all VMs. In practice, most customers over-provision CPU capacity because the cost of CPU capacity is much less than the cost of missing service objectives.

In any case, no individual VM should ever have more virtual CPUs than the number of physical CPUs on a server. That just results in the VM's CPUs competing against one another for real CPU cycles, and can expose problems with lock management in the guest (consider a case where a VM is spinning on a lock on one virtual CPU waiting for another virtual CPU to release it - but that vCPU isn't receiving any service).

Ideally, all of a VM's CPUs fit in a single server socket, in which case Oracle VM Server for x86 attempt to assign the VM's memory and CPUs on the same socket for local CPU-to-memory latency. This has a powerful effect on performance, as remote memory latency on multi-socket systems is much higher than local latency. A big design point of Oracle VM's Xen hypervisor is to avoid this NUMA latency.

Memory

It may be surprising, but memory considerations often play a bigger role than CPU. Even when we manage CPUs we are really managing memory performance, as shown in the previous paragraph.

This is an area in which the rules for VMs differ sharply from real machine environments. On bare metal you assign all the memory to the OS and its applications, or it just goes to waste. Most customers provision their hardware from a small number of configurations, and choose the server model that meets or exceeds an application's peak needs. Any excess memory is used for file buffering, and if there's no need for it, no harm is done. Because the penalty for oversubscribing memory is fairly low, it's not uncommon for memory requirements to be at best approximate.

It works differently in a cloud or VM environment. Excess memory assigned to a VM reduces the number of VMs that can be hosted, which increases the cost of the cloud or hosting service. Within a VM, excess memory increases NUMA latency since it might spread over sockets.

Consequently, in cloud and VM environments it is important to measure baseline memory requirements to reduce costs and retain performance. This is more difficult than CPU over-provisioning as it's easy in most hypervisors to measure CPU utilization and shift load as needed. With memory you have to ask the OS whether it has more virtual RAM than it needs. This can be done using standard OS tools common to Linux, Solaris and Windows, or can be simplified by using a product like Oracle Enterprise Manager than can introspect and instrument within VMs.

Note that neither Oracle VM Server for SPARC nor Oracle VM Server for x86 over-subscribe memory: they do not swap or page VM memory contents to a swap file, and you must have enough real memory to back the virtual machine's defined memory size. This is intentional, and avoids serious performance problems. Overcommitting VM memories is a 40+ year old performance problem that has not really been solved, despite substantial ingenuity and engineering. VMs have poor temporal or address locality of reference, and tend to touch every page defined to them, even for file buffers that may or may not ever be used. As a result the size of their workings sets (recently touched pages) tends to be near the defined memory size. Modern operating systems themselves potentially swap or page their application's virtual memories, and it running a virtual memory guest OS under a virtual memory hypervisor can cause pathological situations where the classic LRU algorithm to select a page for swap out breaks down, and the most recently used pages get selected for displacement instead. 

Hypervisors that swap memory generally are used with modest overcommit ratios (1.3:1 or 1.5:1 is often cited) to avoid this issue, but it's hard to tell in advance when the ratio is just a bit too high and performance would suffer.

Oracle VM avoids this unpredictable risk entirely by not-overcommitting memory. Better to just avoid the issue entirely and give the VMs the memory they
need, just as in modern non-virtual environments we configure
sufficient memory for a server's applications without swapping.

Summary

This blog entry describes performance goals for the VM and cloud environment, and describes CPU, I/O, and memory considerations for reaching those goals, with an emphasis on performance behavior that differs from real-machine environments. The next article will drill down into specifics of how these principles are applied with Oracle VM Server for x86.