News, tips, partners, and perspectives for Oracle’s virtualization offerings
-
Perspectives
November 30, 2015
Oracle VM Performance and Tuning - Part 5
described Oracle VM Server for x86 CPU and memory management.
This article discusses domain types, huge pages, and how to configure scheduling.
Oracle VM domain types
Guest VMs (domains, domU) use one of the following domain types:
| Hardware virtualized machine (HVM) | HVM guests run unmodified guest operating systems as if they were on bare metal. This mode supports most x86 operating systems. This is available on servers with Intel VT or AMDV virtualization enhancements, which includes most current server models. recommended for operating systems that can run in the other domain types. |
| Hardware virtualized, with paravirtualized drivers (PVHVM) | PVHVM guests, also written as "HVM+PV", are similar to HVM guests except they include paravirtualized device drivers written for the Oracle VM environment. These device drivers are available for Oracle Linux, Oracle Solaris, and Microsoft Windows. Paravirtualized device drivers use an optimized code path to interact with the Xen hypervisor, and provide performance benefits similar to those for PVM guests described below. Together with the hardware virtualization support of recent server models, this is recommended for 64-bit guest operating systems. |
| Paravirtualized (PVM) | PVM guests use the original Xen domain type invented before hardware virtualization support was added to Intel and AMD servers. Guest operating systems supporting PVM (generally Linux) are modified to interact with Xen rather than issue native instructions for context switching, I/O and virtual memory management. PVM is still available on current hardware, and is recommended for 32-bit operating systems. It also offers flexibility advantages for dynamically adding and removing CPU and memory to a running VM. |
Note that the advice has changed: it used to be that PVM was always best. MOS Note 757719.1 still recommends PVM in all cases, but will be updated to reflect current best practice. For further information, see http://wiki.xenproject.org/wiki/Understanding_the_Virtualization_Spectrum#What_to_Choose.3F
If you run Oracle VM Server in a virtual machine, for example under Oracle VirtualBox for test or for training purposes,
the hardware virtualization capability, if present, is consumed by the first-level hypervisor.
That means that virtual machines running in an Oracle VM Server virtual machine must use PVM mode.
For people familiar with other hypervisors: this is analogous to how hardware virtualization assists are consumed by first level mainframe
VM when running VM under VM (a traditional usage for testing and training).
This is generally not done (on all hypervisors) for production workloads due to the performance cost of running "third level" virtual machines.
For the full list of supported operating systems and domain types, see Oracle VM Release Notes4.1 Oracle VM Server for x86 Supported Guest Operating Systems.
Note that there are different configuration limits for the different domain types,
as documented in Oracle VM Release Notes Chapter 5 Configuration Limits.
For best performance on modern hardware use PVHVM mode for 64-bit operating systems, and either that or PVM mode on 32-bit operating systems.
PVM was previously best, but 64 bit guests and improved hardware capabilities have changed the recommendation.
In personal testing I've obtained a dramatic performance improvement in some cases by switching from PVM to PVHVM.
Performance effects depend on workload, and are highest for I/O intensive applications that benefit the most from the enhanced I/O path.
For further information, see the Oracle VM Concepts Guide for Release 3.3
Section 7.1 What are Virtualization Modes or Domain Types?.
Huge Pages
All modern CPUs divide real and virtual memory into uniformly sized "pages" of memory.
Each virtual memory page is individually mapped to a real memory page, requiring its own page table entry (PTE) in a page translation table.
If I remember correctly, Intel and compatible servers have a page directory and a page table,
such that each entry in the page directory points to a different page table of 1,024 PTEs, with each PTE mapping one page and occupying 4 bytes.
With the default page size of 4KB, a process with an address space of 1GB requires 262,144 4-byte entries - a megabyte of memory for each gigabyte of address.
This becomes very expensive in terms of memory footprint and CPU time to search and update the tables.
This is addressed by enabling huge pages to support a page size of 2MB, yielding a 512:1 reduction in page table size and CPU overhead.
Huge pages makes a tremendous difference for large memory applications, such as Oracle Database with large SGAs.
No Oracle VM administrative effort is needed to enable huge pages for HVM and PVHVM guests.
For PVM guests, edit the VM and set the "Enable huge pages" checkbox if you are using Oracle VM 3.3. This checkbox can only be changed when the VM is stopped.
On Oracle VM Server 3.2, see the instructions in MOS note How to Enable HugePages for a Para-Virtualized Machine (PVM) on Oracle VM 3.2 (Doc ID 1529373.1).
For reference: PVM guests require that huge pages be enabled by adding the allowsuperpage option to dom0's kernel line in /boot/grub/menu.list.
This is provided by default, so no further administrative effort should be needed.
Note that huge page and default page size VMs should not be mixed on the same server, as allocation for small page guests can cause memory fragmentation
that could prevent a huge page guest from starting.
For further details, see the Oracle VM Concepts Guide 7.13 How are Huge Pages Enabled for Virtual Machines? and the Oracle VM Manager User's Guide for Release 3.3
Create VM page.
CPU scheduler control
Oracle VM Manager provides two settings you can use to manage the CPU resources a VM can consume:
- Maximum Processors / Processors - sets the number of CPU processors, up to 128, that can be allocated to the VM.
The Maximum Processors value can only be changed when the VM is not running,
and the Processors value can be changed while the VM is running, up to the Maximum Processors setting.
If hyper-threading is enabled, then "processors" is a CPU thread, otherwise it's an entire core.
Regardless of priority or cap values, a VM can consume no more CPU resources than the number of virtual CPUs it has.
VMs running single-threaded non-scalable applications will not benefit from adding more virtual CPUs, just as happens
when limited scalability applications run on physical servers.
Even reasonably scalable applications will demonstrate diminishing returns on additional CPUs. - Priority - represents a share or weight setting that expresses the VM's relative importance for CPU access compared to other VMs.
The higher the priority, the more CPU cycles it will get relative to other VMs when there is contention for CPU.
The default value is 50, so you can arrange higher and lower priorities to assign relative importance.
When there is excess CPU capacity this value has no effect. - Processor Cap % - set the maximum CPU consumption a virtual CPU can have. The default is 100%, indicating that the VM's virtual CPUs can be given
all the CPU capacity they can consume, subject to availability and their priority.
Set this to a lower number if you want to throttle the VM's CPU consumption.
If you want the VM to "eat as much as it wants" if nobody else needs CPU, then leave the default Cap value but set a low priority.
As mentioned in the previous articles, use monitoring tools like vmstat, mpstat, and iostat inside the VM
for the best indication of CPU resource needs.
Those tools report "steal" time when the VM wanted to run but was prevented because CPU cycles weren't available, so non-zero steal values
indicate that they could benefit from higher priority.
Use mpstat to see if more CPUs would help performance (mpstat would show most CPUs busy - there's no benefit to adding CPUs if mpstat shows
idle virtual CPUs).
For further details, see the Oracle VM Manager User's Guide for Release 3.3
Create VM page
Summary
This article discussed how to choose between different domain types, how to exploit huge pages for large memory VMs, and how to set Oracle VM scheduler criteria.
For best performance use PVHVM or PVM guests depending on whether 64-bit or 32-bit operating systems,
and exploit huge pages if working with large memory applications.
Finally, leverage the CPU scheduler controls to set policies on how much CPU resource a VM can get.
Next week: I/O performance, which is even more fun.