figuring out cpu topology in oracle vm

I have had a few people in the past ask about CPU topology on Oracle VM and how to figure out which CPUs, cores or threads are used by a VM... Here's a quick overview :

1) get a summary of your server

The xm info command prints out a whole bunch of information about your system.

Look for the lines below in the output for detail on your systems CPUs.

# xm info
...
nr_cpus                : 8
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 3200
...
This server has a single socket with 4 cores and 2 threads per core. Total of 8 "CPUs". so cpu0..7 really is thread 0..7.

# xm info
...
nr_cpus                : 12
nr_nodes               : 1
cores_per_socket       : 6
threads_per_core       : 2
...
This server has a single socket with 6 cores and 2 threads per core. Total of 12 "CPUs".

2) get the cpu topology

The xenpm command lets you print out the thread/core/socket topology on a given server :

# xenpm get-cpu-topology
CPU     core    socket  node
CPU0     0       0       0
CPU1     0       0       0
CPU2     1       0       0
CPU3     1       0       0
In the above example, it's a single socket machine with 2 cores and 2 threads per core. cpu 0 is thread 0 of core 0, cpu 1 is thread 1 of core 0, cpu 2 is thread 0 of core 1, cpu 3 is thread 1 of core 1. If you were to have cpus="0,1" in the VM config file, you would be running on core 0. If you were to have cpus="0-3" in the VM config file, you would actually run a VM on both cores.

# xenpm get-cpu-topology
CPU     core    socket  node
CPU0     0       0       0
CPU1     0       0       0
CPU2     1       0       0
CPU3     1       0       0
CPU4     2       0       0
CPU5     2       0       0
CPU6     3       0       0
CPU7     3       0       0
In the above example, you have a single socket server with 4 cores and 2 threads per core. cpu 0 -> thread 0 of core 0, cpu 1 -> thread 1 of core 0 etc.If you were to have cpus="4-7" in the VM config file, you would run the VM on cores 2 and 3.

# xenpm get-cpu-topology
CPU     core    socket  node
CPU0     0       0       0
CPU1     0       0       0
CPU2     1       0       0
CPU3     1       0       0
CPU4     2       0       0
CPU5     2       0       0
CPU6     8       0       0
CPU7     8       0       0
CPU8     9       0       0
CPU9     9       0       0
CPU10    10      0       0
CPU11    10      0       0
In the above example, you see a single socket server with 6 cores hyperthreaded.

3) Show the guest/VM vcpu bindings to physical threads : Use the command xm vcpu-list to list a summary of which virtual CPU is running on which physical CPU.

# xm vcpu-list 1
Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
0004fb00000600007c351fa24276c63f     1     0     5   -b-    4673.6 5-6
0004fb00000600007c351fa24276c63f     1     1     5   -b-    4534.0 5-6
If you add the domain ID to the command xm vcpu-list 1, you get the information for just that guest or VM. In the above example, you have a guest with 2 virtual CPUs both running, at this time, on physical cpu (thread in my case) 5. The column CPU Affinity show's 5-6 which means that both virtual CPUs could be running on either thread 5 or 6. This shows that the guest is pinned on those 2 threads. Combined with the information of xenpm get-cpu-topology I can then see that in my case CPU 5 is thread 1 of core 2 and CPU 6 is thread 1 of core 8. So this 2 vcpu guest is pinned to 2 separate physical cores.

# xm vcpu-list
Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
0004fb00000600007c351fa24276c63f     1     0     5   -b-    4676.8 5-6
0004fb00000600007c351fa24276c63f     1     1     5   -b-    4537.0 5-6
Domain-0                             0     0     0   -b-     932.1 any cpu
Domain-0                             0     1     6   -b-    1168.0 any cpu
Domain-0                             0     2     7   -b-    1010.8 any cpu
Domain-0                             0     3    11   -b-     903.0 any cpu
Domain-0                             0     4     8   -b-     494.2 any cpu
Domain-0                             0     5     9   r--     773.8 any cpu
Domain-0                             0     6     1   -b-     522.7 any cpu
Domain-0                             0     7     2   -b-     785.1 any cpu
Domain-0                             0     8     4   -b-     473.8 any cpu
Domain-0                             0     9     3   -b-     728.1 any cpu
Domain-0                             0    10    10   -b-     490.8 any cpu
Domain-0                             0    11     0   r--    1219.6 any cpu
This is the same system but xm vcpu-list without argument. It also shows the dom0 guest. As you can see in this example, dom0 can run on any physical thread and the CPU Affinity is any cpu which implies any virtual CPU can be scheduled on any physical thread so no pinning or partitioning.

4) using ovm_utils ovm_vmcontrol lets a user or admin set and get the cpu/vcpu bindings for a VM through Oracle VM Manager in 3.0.

# ./ovm_vmcontrol -u admin -p Manager1 -h localhost -v apitest -c vcpuget
Oracle VM VM Control utility 0.4.2.
Connected.
Command : vcpuget
Current pinning of virtual CPUs to physical threads : 5,6
In the above example, I have a VM named apitest and using the action vcpuget it shows me that virtual cpus of this guest are bound to threads 5 and 6. This happens to be the VM in the previous example which showed that this was core 2 and core 8 (3).

Now let's bind the vcpu's to core 0, I just run the following command :

# ./ovm_vmcontrol -u admin -p Manager1 -h localhost -v apitest -c vcpuset  -s 0
Oracle VM VM Control utility 0.4.2.
Connected.
Command : vcpuset
Pinning virtual CPUs
Pinning of virtual CPUs to physical threads  '0' 'apitest' completed.
Now, running the xm vcpu-list command, I can see that this immediately took effect (and also will on subsequent startups of the VM) :
# xm vcpu-list 1
Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
0004fb00000600007c351fa24276c63f     1     0     0   -b-    4687.6 0
0004fb00000600007c351fa24276c63f     1     1     0   -b-    4547.2 0
The VM now has CPU Affinity 0 for both virtual CPUs.
Comments:

Wim,

We've been asked by a customer to limit a vm (and that vm alone) to use a certain number of vcpu's. I know this has been doable for awhile now with xen-based hypervisors by setting 'cpus = "0,1"' or 'cpus = "1-3"'

That works fine. I understand the drawbacks to doing this --namely, whenever a vm migrates to another node, you will have to forcefully re-pin the cpus. The other limitation that I see is that other vm's are still able to use those vcpu's, unless I forcefully pin all vm's (and then again we run into the live migration issues...).

1) So one thing that I'm wondering is if you know if Xen (or Oracle) has any options to allow a vm to be pinned, but force other vm's to NOT use those vcpu's (without the limitations that I mentioned above).

2) The other thing that I was wondering is the following related to cpu weights. In our pSeries AIX environment there is a concept of "guaranteed" cpu usage of a server and a "shared" cpu pool. So in our AIX environment we typically set a guaranteed amount of cpu to allocate to a vm, but allow the vm to use the shared pool with a weight. I don't know if this concept is available in Xen, but I know solaris uses weights. So with those to things in mind, do you know if the Xen community is looking to incorporate some concept like this to allow the vm's to utilize available cpu power, if it is available?

One way that I think would easily be able to achieve this is to allocate 100% of the vcpu's to 100% of the vm's. Then, just allow the weights to tell the vm's who gets priority. The one problem with this, of course, is that if one vm jams away on a cpu, then cpu scheduler will get taxed pretty heavily. And since we don't offload iSCSI (because of some issues we've run into with bnx2) I could see this being potentially problematic. So I think one option might be to allow maybe 60% or 80% of the vcpu's to be used by all vm's and then allow the weights to sort out who gets priority on the cpu's.

3) I think if there were a way to combine question #1 with #2, maybe there would be a way to force dom0 to ping to 0,1 and then all other vm's could use 100% of 2-n. Then, you could allow the cpu priority to sort out who gets priority on 2-n, but vcpus 0 and 1 could be left for the dom0 (and thus also iSCSI).

So that's my two cents on Xen and cpu's. By the way, it is very nice to allow dynamic memory and cpu to grow in Xen <-- from what I remember, I don't think ESX can do that yet (could be wrong, but it couldn't when I looked a few years ago).

Thanks,
Joe

Posted by Joseph Hoot on May 03, 2012 at 07:37 AM PDT #

Good comments and suggestions and funny I was just reading an article about Xen 4.2 features :)

1) we don't manage the pinning/nonpinned so indeed it's sort of all or nothing today it's not automated.

2) in a future version when we move to xen 4.2 (we are on 4.1) http://blog.xen.org/index.php/2012/04/23/xen-4-2-cpupools/

that is exactly what you are proposing, I think :)

Posted by Wim on May 03, 2012 at 07:44 AM PDT #

One more question for you on this Wim.

I was doing some more research on the xen credit scheduler and wanted to see if I could trend the cpu runqueues.

on a non-SMP system I would typically just run `vmstat 1` and trend the 'r' and/or 'b'. But since each CPU in a dom0 has its own run queue the VCPU's can use, I wanted to find a way to trend each CPU run queue independently.

Is there somewhere in /proc where I can print out the current runqueue for each cpu?

Here's my scenario: Let's say that I have domU with id 1. It is currently pinned to cpu0,1. I have 3 other vm's (id 2,3,4) that also have 2 vcpu's, but aren't pinned. Also assume that all domU's have the same weight and they cap at 100%

vcpu1.0 and vcpu1.1 are being heavily hit (therefore reducing there credits constantly). But since they are only ever able to use CPU0 and CPU1, they will hit negative credits (naturally). So the other vm's that are running will then be able to use CPU0 and CPU1, forcing that vm with id 1 to sit and be forced to wait (instead of normally, if NOT PINNED, being load balanced to other CPUs).

I'm interested in proving to my customer that pinning CPUs (at least in this situation prior to Xen 4.2 being implemented) is actually negatively impacting the performance of that VM under high load (especially if other vm's are also under high load.

If there is no easy way to cat something out of /proc or run some kind of vmstat, it's not a big deal. I can just verbally explain this to my customer. But if a mechanism exists to gather this and toss it into graph, I would prefer to have that information available to show them.

Thanks,
Joe

Posted by Joe Hoot on May 11, 2012 at 09:27 AM PDT #

Hi Wim,

Could you comment on the availability of the "cores-per-socket" parameter in OVM as available in citrix zenserver? [Refer: http://support.citrix.com/article/CTX126524].

This is primarily used for windows vm's which consider a vcpu (which is typically a cpu thread) as a physical cpu & allocate a socket license to it. Hence, limiting the cpu power to the windows vm to 4 threads instead of 4 physical cpu's (Considering Win 2k3 STD edition as an example).

Thanks !

Posted by Umang A on May 30, 2012 at 11:51 AM PDT #

Umang - we don't do it the same was as the citrix article. we do support partitioning so if you follow the instructions in the blog and you use ovm_utils with vcpuset, you should be able to provide the same functionality.

Posted by guest on June 02, 2012 at 12:29 PM PDT #

Hi Wim,

Changed the pinned cpu set for a VM. How we canchange the setting back to "any cpu"?

Posted by guest on August 07, 2013 at 02:25 AM PDT #

just do -c vcpuset -s ''

Posted by guest on August 10, 2013 at 08:01 PM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Wim Coekaerts is the Senior Vice President of Linux and Virtualization Engineering for Oracle. He is responsible for Oracle's complete desktop to data center virtualization product line and the Oracle Linux support program.

You can follow him on Twitter at @wimcoekaerts

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
9
10
11
12
13
14
15
16
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today