Thursday May 03, 2012

understanding memory allocation in oracle vm / xen

As a follow up to my previous blog about cpu topology, I wanted to add a little bit about memory topology and memory allocation in the hypervisor. Most systems these days that are multi-socket are considered NUMA. Even though over the years, the NUMA-factor has gone down drastically,there still is a small amount of memory locality involved.

My test setup is a dual socket server with 36GB memory. You can see this in Oracle VM Manager as part of the server info or directly on the server with xm info :

# xm info 
..
total_memory           : 36852
free_memory            : 25742
..

I have a few VMs running on this server which is why you see memory be lower than total. The 16GB VM is running with tmem enabled and because of that is not using up all memory but only the base memory needed to be active for the workload it's running.

# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
0004fb00000600001668dac79108cb84             2  4096     4     -b----    129.9
0004fb0000060000804bac06a5087809             1  4096     4     -b----    129.4
0004fb0000060000db9c71d539c940ed             3 16000     4     -b----     28.3
Domain-0                                     0  1244    24     r-----    188.0

Let's start with a clean slate and look at some statistics. The following commands will dump detailed memory information on your server :

# xm debug-key u ; xm dmesg. Basically debug info for NUMA memory info. xm dmesg will show you the debug output.


(XEN) 'u' pressed -> dumping numa info (now-0xFE:A1CFFF69)
(XEN) idx0 -> NODE0 start->0 size->4980736
(XEN) phys_to_nid(0000000000001000) -> 0 should be 0
(XEN) idx1 -> NODE1 start->4980736 size->4718592
(XEN) phys_to_nid(00000004c0001000) -> 1 should be 1
(XEN) CPU0 -> NODE0
(XEN) CPU1 -> NODE0
(XEN) CPU2 -> NODE0
(XEN) CPU3 -> NODE0
(XEN) CPU4 -> NODE0
(XEN) CPU5 -> NODE0
(XEN) CPU6 -> NODE0
(XEN) CPU7 -> NODE0
(XEN) CPU8 -> NODE0
(XEN) CPU9 -> NODE0
(XEN) CPU10 -> NODE0
(XEN) CPU11 -> NODE0
(XEN) CPU12 -> NODE1
(XEN) CPU13 -> NODE1
(XEN) CPU14 -> NODE1
(XEN) CPU15 -> NODE1
(XEN) CPU16 -> NODE1
(XEN) CPU17 -> NODE1
(XEN) CPU18 -> NODE1
(XEN) CPU19 -> NODE1
(XEN) CPU20 -> NODE1
(XEN) CPU21 -> NODE1
(XEN) CPU22 -> NODE1
(XEN) CPU23 -> NODE1
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 318627):
(XEN)     Node 0: 282976
(XEN)     Node 1: 35651
The above output shows that the first 12 cpu's are bound to memory node 0 and the next 12 to memory node 1. The info shows how many pages of RAM are available on each node NODE0 start->0 size->4980736 and NODE1 start->4980736 size->4718592. the Dom0 domain is about 1.2Gb of RAM and it has some memory allocated on each NODE (it also has all of it's 24 vcpu's allocated across all threads in the box). Now let's start a VM.

# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
0004fb0000060000804bac06a5087809             4  4096     4     r-----      8.8
Domain-0                                     0  1244    24     r-----    240.9

# xm debug-key u ; xm dmesg
...
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 318627):
(XEN)     Node 0: 282976
(XEN)     Node 1: 35651
(XEN) Domain 4 (total: 1048576):
(XEN)     Node 0: 1048576
(XEN)     Node 1: 0
You can see that the newly started VM (domain 4) has 4Gb allocated on node 0.
# xm vcpu-list 4
Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
0004fb0000060000804bac06a5087809     4     0     0   -b-       4.8 0-3
0004fb0000060000804bac06a5087809     4     1     3   -b-      26.1 0-3
0004fb0000060000804bac06a5087809     4     2     2   -b-       3.5 0-3
0004fb0000060000804bac06a5087809     4     3     1   -b-       2.4 0-3
The VM also has its virtual CPUs bound to node 0. Let's start another VM.

# xm vcpu-list 6
Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
0004fb00000600001668dac79108cb84     6     0    19   r--       2.2 19-23
0004fb00000600001668dac79108cb84     6     1    23   r--      24.6 19-23
0004fb00000600001668dac79108cb84     6     2    20   -b-       1.4 19-23
0004fb00000600001668dac79108cb84     6     3    22   -b-       1.1 19-23

# xm debug-key u ; xm dmesg
...
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 318627):
(XEN)     Node 0: 282976
(XEN)     Node 1: 35651
(XEN) Domain 4 (total: 1048576):
(XEN)     Node 0: 1048576
(XEN)     Node 1: 0
(XEN) Domain 6 (total: 1048576):
(XEN)     Node 0: 0
(XEN)     Node 1: 1048576
As you can see, this domain 6 has vCPUs bound to node 1, and Xen automatically also allocates memory from node 1. To ensure memory locality. It tries hard to keep memory and CPU as local as possible. Of course when you run with many VMs with many vCPUs then memory allocation will be spread out across multiple nodes.

After starting a 16Gb VM on this server (domain 7), now that 8Gb is allocated, you will see that this 16Gb VM's memory allocation is across the 2 memory nodes :

(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 318627):
(XEN)     Node 0: 282976
(XEN)     Node 1: 35651
(XEN) Domain 4 (total: 1048576):
(XEN)     Node 0: 1048576
(XEN)     Node 1: 0
(XEN) Domain 6 (total: 1048576):
(XEN)     Node 0: 0
(XEN)     Node 1: 1048576
(XEN) Domain 7 (total: 4097012):
(XEN)     Node 0: 2524148
(XEN)     Node 1: 1572864
About

Wim Coekaerts is the Senior Vice President of Linux and Virtualization Engineering for Oracle. He is responsible for Oracle's complete desktop to data center virtualization product line and the Oracle Linux support program.

You can follow him on Twitter at @wimcoekaerts

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
9
10
11
12
13
14
15
16
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today