X

News, tips, partners, and perspectives for the Oracle Solaris operating system

Configuring and Managing OpenStack Nova's Image Cache

Dave Miner
Sr. Principal Software Engineer
 

One piece of configuration that we had originally neglected in configuring Solaris Engineering's OpenStack cloud was the cache that Nova uses to store downloaded Glance images in the process of deploying each guest.  For many sites, this wouldn't likely be a big problem as they will often use a small set of images for a long period of time, so the cache won't grow much.  In our case, that's definitely not true; we generate new Solaris images every two weeks for each release that is under development, and we usually have two of those (an update release and the next minor release), so we are introducing new images every week (and that's multiplied by having images for SPARC and x86; yes, our Glance catalog is pretty large, a topic for a future post).  And of course, the most recent images are usually the most used, so usually a particular image will be out of favor within a month or two.  We recently reached the point where the cache became a problem on our compute nodes, so here we'll share the solution we're using until the solariszones driver in OpenStack gets the smarts to handle this.

By default, the solariszones driver uses /var/lib/nova/images as its download cache for Glance images.  This has two problems: it's not quota'ed, and it's inside the boot environment's /var dataset, meaning that as we update the compute node's OS, each new boot environment (or BE) will create a snapshot that refers to whatever is there, and so it becomes difficult to actually free up space; you need to delete both the image and any old BE snapshots that refer to it.  There's also another related problem, in that attempting to create an archive of the system using archiveadm(1m) will also capture all of these images and bloat the archive to Godzilla size.  Thus, our solution needs to move the cache outside of the boot environment hierarchy.  All together, there are three pieces:

  1. Create a dataset and set a quota on it
  2. Introduce an SMF service to manage the cache against the quota
  3. Reconfigure Nova to use the new cache dataset

The SMF Service

For this initial iteration, I've built a really simple service with lots of hard-coded policy.  A real product-quality solution would be more configurable and such, but right now I'm in quick & dirty mode as we have bigger problems we're working on.  Here's the script at the core of the service:

#!/bin/ksh
# Nova image cache cleaner
# Presently all hard-coded policy; should be configured via SMF properties
cacheds=rpool/export/nova
cachedir=/export/nova/images
quota=$(zfs get -pH -o value quota ${cacheds})
(( quota == 0 )) && exit 0
(( limit=quota-2*1024*1024*1024 ))
size=$(zfs get -pH -o value used ${cacheds})
# Delete images until we are at least 2 GB below quota; always delete oldest
while (( size > limit )); do
oldest=$(ls -tr ${cachedir}|head -1)
rm -v ${cachedir}/${oldest}
size=$(zfs get -pH -o value used ${cacheds})
exit 0

To run something like this you'd traditionally use cron, but we have the new SMF periodic services at our disposal, and that's what we'll use here; the script will be run hourly, and gets to run for however long it takes (which should be but a second or two, but no reason to be aggressive).

<?xml version="1.0" ?>
<!DOCTYPE service_bundle
SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type="manifest"
name="site/nova-cache-cleanup">
<service version="1" type="service"
name="site/nova-cache-cleanup">
<!--
The following dependency keeps us from starting unless
nova-compute is also enabled
-->
<dependency restart_on="none" type="service"
name="nova-compute" grouping="require_all">
<service_fmri value="svc:/application/openstack/nova/nova-compute"/>
</dependency>
<periodic_method
period="3600"
exec="/lib/svc/method/nova-cache-cleanup"
timeout_seconds="-1"/>
<instance enabled="false" name="default"/>
<template>
<common_name>
<loctext xml:lang="C">
OpenStack Nova Image Cache Management
</loctext>
</common_name>
<description>
<loctext xml:lang="C">
A periodic service that manages the Nova image cache
</loctext>
</description>
</template>
</service>
</service_bundle>

To deliver it, we package it up in IPS:

set name=pkg.fmri value=pkg://site/nova-cache-cleanup@0.1
set name=pkg.summary value="Nova compute node cache management"
set name=pkg.description value="Nova compute node glance cache cleaner"
file nova-cache-cleanup.xml path=lib/svc/manifest/site/nova-cache-cleanup.xml owner=root group=bin \
mode=0444 restart_fmri=svc:/system/manifest-import:default
file nova-cache-cleanup path=lib/svc/method/nova-cache-cleanup owner=root group=bin mode=0755

Use pkgsend(1) to publish all of that into your local IPS repository.

Creating the Dataset

Since we use Puppet to handle the day-to-day management of our cloud, I updated our compute_node class to create the dataset and distribute the package.  Here's what that looks like:

# Configuration custom to compute nodes
class compute_node {
# Customize kernel settings
file { "site:cloud" :
path => "/etc/system.d/site:cloud",
owner => "root",
group => "root",
mode => 444,
source => "puppet:///modules/compute_node/site:cloud",
}
# Create separate dataset for nova to cache glance images
zfs { "rpool/export/nova" :
ensure => present,
mountpoint => "/export/nova",
quota => "15G",
}
file { "/export/nova" :
ensure => directory,
owner => "nova",
group => "nova",
mode => 755,
require => Zfs['rpool/export/nova'],
}
# Install service to manage cache and enable it
package { "pkg://site/nova-cache-cleanup" :
ensure => installed,
}
service { "site/nova-cache-cleanup" :
ensure => running,
require => Pkg['pkg://site/nova-cache-cleanup'],
}
}

It's important that the directory have the correct owner and permissions; the solariszones driver will create the /export/nova/images directory within it to store the images and the nova-compute service runs as nova/nova.  Get this wrong and guests will just plain fail to deploy.  The kernel settings referred to above are unrelated to this posting, it's our setting user_reserve_hint_pct so that the ZFS ARC cache is managed appropriately for running lots of zones (we're using a value of 90 percent as our compute nodes are beefy - 256 or 512 GB).

Reconfiguring Nova

Once you have all that in place and the Puppet agent has run successfully to create the dataset and install the package, it's just a small bit of Python to enable the service on all the compute nodes:

#!/usr/bin/python
import iniparse
from subprocess import CalledProcessError, Popen, PIPE, check_call
def configure_cache():
ini = iniparse.ConfigParser()
ini.readfp(open('/etc/nova/nova.conf'))
ini.set('DEFAULT', 'glancecache_dirname', '/var/share/nova/images')
with open('/etc/nova/nova.conf', 'w') as fh:
ini.write(fh)
check_call(["/usr/sbin/svcadm", "restart", "nova-compute"])
if __name__ == "__main__":
configure_cache()

I used cssh(1) to run this at once on all the compute nodes, but you could also do this with Puppet.  Ideally we'd package nova.conf or use Puppet's OpenStack modules to do this part, but the packaging solution doesn't work due to a couple of node-specific items in nova.conf and we don't yet have Puppet OpenStack available in Solaris (stay tuned!).

The final step once you've done all the above is to take out the trash from your old image cache:

rm -rf /var/lib/nova/images

Again, if you have older BE's around this won't free up all the space yet, you'll need to remove old BE's, but be careful not to leave yourself with just one!

Join the discussion

Comments ( 1 )
  • guest Wednesday, September 16, 2015

    We don't use puppet. How can we tell nova to use /export/nova instead of /var/lib/nova? Changing the state_path in /etc/nova/nova.conf?

    Do we want everything under /var/lib/nova to move? Or just the images directory?

    Will this require a nova restart?


Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.