Changing a Domain's Memory Size Under Xen

I know I promised this yesterday, so I'm sorry if you were up all night waiting for it :)

I was planning on writing about the using the balloon thread and giving some explanations behind the code, but it was going way too long. So, I'll make one post today explaining how to change memory sizes of domains, and another one Monday explaining some of the code behind it.

Now, on to the show...

I'm in charge of a seemingly random assortment of things for the Solaris on Xen work. One of these things is called the balloon thread. The balloon thread is a kernel thread that works with the hypervisor to dynamically grow and shrink the amount of memory in a domain. To activate this code, you can use the 'xm mem-set' command. An example on dom0:

# xm list
Name                                      ID   Mem VCPUs      State   Time(s)
Domain-0                                   0  3891     2     r-----     66.5
# xm mem-set 0 3500
# xm list
Name                                      ID   Mem VCPUs      State   Time(s)
Domain-0                                   0  3500     2     r-----     67.8

This is on my desktop, with 4G of RAM. At the beginning, the control domain (dom0) is given 3891 MB of RAM. The rest of the memory is held by the hypervisor, and is not visible to any domain. I issued the command 'xm mem-set 0 3500', which means change the amount of memory in domain #0 to 3500 MB. As you can see on the second 'xm list', it worked.

The tools can do this automatically during domain creation, as well. I still have 3500 MB in my dom0, which means I have around 400 MB free. Let's say I want to create a domU with 768 MB:

# head -3
name = "example1"
vcpus = 2
memory = "768"
# xm create
Using config file "./".
Started domain example1
# xm list
Name                                      ID   Mem VCPUs      State   Time(s)
Domain-0                                   0  3251     2     r-----     83.2
example1                                   1   767     2     -b----     28.8

You can see that more memory was automatically removed from dom0 and given to domain #1.

You may ask, "Why does it say 767 MB when we asked it to have 768 MB?" Well, some of the memory for any domain is given to the hypervisor for disk and networking purposes. It will be constantly exchanged as packets and disk blocks are moved around. So, the value given in 'xm list' is usually within 1 or 2 MB of the requested amount.

Let's lower the memory in our domU and give it back to dom0:

# xm mem-set example1 500
# xm mem-set Domain-0 3500
# xm list
Name                                      ID   Mem VCPUs      State   Time(s)
Domain-0                                   0  3501     2     r-----     89.3
example1                                   1   499     2     -b----     38.3

As you can see, it's now easy to dynamically add and subtract memory from a running Xen domain.

There is one catch: a domain cannot go above the amount of memory that it originally booted with. For our above example, I can change the domU's memory allocation to anything below 768 MB, but it will not go above 768 MB. There are some complexities in the Solaris kernel VM system causing this, which I'll explain on Monday when I give more details on the code implementing the Solaris thread.

Update, 11:45 AM:I forgot to mention... if you prefer libvirt over xm commands (and really, you should, because it's easier to maintain), you can use virsh:

# virsh
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # dominfo 0
Id:             0
Name:           Domain-0
UUID:           00000000-0000-0000-0000-000000000000
OS Type:        linux
State:          running
CPU(s):         2
CPU time:       448.5s
Max memory:     no limit       
Used memory:    3984324 kB

virsh # setmem 0 3500000

virsh # dominfo 0
Id:             0
Name:           Domain-0
UUID:           00000000-0000-0000-0000-000000000000
OS Type:        linux
State:          running
CPU(s):         2
CPU time:       450.3s
Max memory:     no limit       
Used memory:    3499008 kB

virsh # 

The most noticeable difference is that the setmem takes memory sizes in kB, not MB like xm mem-set.


Hi, Thanks for your post, is there a solution to let the memory of domain pass the amount of memory in the configuration file?

Posted by jimmiinguyen on July 25, 2007 at 11:03 PM PDT #

I'm not sure if I fully understand the question, but setting the amount of memory in a .py file is as easy as specifying a memory line:

# cat 
name = "example1"
vcpus = 2
memory = "512"

#extra = "-kd"

root = "/dev/dsk/c0d0s0"
disk = ['file:/export/guests/solaris/0/disk.img,0,w']

vif = ['']

on_shutdown = "destroy"
on_reboot = "restart"
on_crash = "destroy"

Since memory is 512, this domain will have 512 MB of memory.

Posted by Ryan Scott on July 26, 2007 at 05:41 AM PDT #

But what about a Solaris domU that
was configured with memory="512",
and later on, when the domU is already
booted, you notice that the domU
could use more memory?

While you can try this in dom0...
xm mem-max solaris 1024
xm mem-set solaris 1024

... the solaris domU just refuses to use
the extra memory and complains with
something like

unix: WARNING: New balloon target (0x138800 pages) is larger than original memory size (0x100000 pages). Ballooning beyond original memory size is not allowed.

It seems that with a linux domU, you can
- specify "memory" & and "maxmem"
in the domU's config file
- pass a parameter "mem=xxxxM" to
the domU kernel (where the amount
xxxx of memory corresponds to the
maxmem config parameter)

The linux domU starts with "memory"
abount of memory, but that can be
increased later on up to "maxmem".

Posted by Jürgen Keil on November 22, 2007 at 02:14 AM PST #

I am not familiar with Solaris. But I always dedicate limited amount of memory to Linux Dom0 since Linux will allocate memory for "memory metadata" and some of the network configuration is based on the memory amount:

We have experience that sshd crashes when balloning Dom0's memory out.

Posted by Eric Ma on September 06, 2010 at 12:37 PM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed



« December 2016