Using resource pools to control the amount of CPU available to zones is not exactly a new feature.  In fact, for a long time, creating pools and binding zones to them was the only way to implement resource management for zones.  In more recent versions of Solaris, properties like “capped-cpu” were added to the zone configuration, making much of this easier to configure.  In many cases, the mechanisms provide all you need.

One aspect, however, is not well covered in the documentation: Shared resource pools.  Since I was recently asked to explain this to a customer, I thought I might as well publish it here for everyone’s benefit…

Let’s imagine this scenario:

  • We have a two tier application consisting of an application and a database tier.
  • Both software tiers come with a core-based licensing scheme similar to what Oracle uses for its products.  For both, it is legal to use resource pools for license capping.
  • We own licenses worth 2 cores for each of the two tiers: 2 for the application and 2 for the database.
  • We want two environments for this application – one for production and one for testing.  Unfortunately, we need all 2 cores for production, and we can’t afford additional licenses for the testing environment.

The obvious solution is to share the resources for both test and production.  If required, we can also throttle the test environment to give priority to production.  Here’s how to do this:

In a first step, let’s look at the global zone’s CPU resources and create the two production zones.  Before we start working with pools, they’ll just share everything with the global zone.  I’ll be using the “zonecores” script from here to visualize some of this.

root@mars:~# ./zonecores  -l
#
# Socket, Core, Strand and Zone Overview
#
Socket

 

Core

 

Strands

 

Zones
2

 

0

 

0,1,2,3,4,5,6,7 none
2

 

1

 

8,9,10,11,12,13,14,15 none
2

 

2

 

16,17,18,19,20,21,22,23 none
2

 

3

 

24,25,26,27,28,29,30,31 none
2

 

4

 

32,33,34,35,36,37,38,39 none
2

 

5

 

40,41,42,43,44,45,46,47 none
2

 

6

 

48,49,50,51,52,53,54,55 none
2

 

7

 

56,57,58,59,60,61,62,63 none
2

 

8

 

64,65,66,67,68,69,70,71 none
2

 

9

 

72,73,74,75,76,77,78,79 none
2

 

10

 

80,81,82,83,84,85,86,87 none
2

 

11

 

88,89,90,91,92,93,94,95 none
2

 

12

 

96,97,98,99,100,101,102,103 none
2

 

13

 

104,105,106,107,108,109,110,111 none
2

 

14

 

112,113,114,115,116,117,118,119 none
2

 

15

 

120,121,122,123,124,125,126,127 none
2

 

16

 

128,129,130,131,132,133,134,135 none
2

 

17

 

136,137,138,139,140,141,142,143 none
2

 

18

 

144,145,146,147,148,149,150,151 none
2

 

19

 

152,153,154,155,156,157,158,159 none
root@mars:~# zoneadm list -ivc
ID NAME STATUS PATH BRAND IP
0 global running / solaris shared
3 db-prod running /system/zones/db-prod solaris excl
4 app-prod running /system/zones/app-prod solaris excl
root@mars:~# for i in db-prod app-prod; do zonecfg -z $i info pool; done
pool:
pool:
root@mars:~# ./zonecores
#
# Checking Whole Core Assignments
#
OK – Zone db-prod using default pool.
OK – Zone app-prod using default pool.
root@mars:~# zlogin app-prod ‘psrinfo |wc -l’
160

In the above example, we see the following:

  • The global zone has 20 cores (obviously some sort of LDom) which, at 8 strands/core, give 160 vCPUs.
  • Right now, none of these cores are assigned to any zone’s resource pool.
  • We have two zones up and running.  Neither of them have any resource pool assigned to them right now.
  • So obviously, the two zones share all 160 vCPUs with the global zone.  Which also means that the “CPU count” in the zones is 160. 

Next, we’ll create two resource pools, one for each application tier, and associate each with a processor set with 2 cores.  Then we’ll bind each zone to its resource pool and recount CPUs.

root@mars:~# poolcfg -c 'create pool app-pool'
root@mars:~# poolcfg -c 'create pool db-pool'
root@mars:~# poolcfg -c 'create pset app-pset (uint pset.min = 16 ; 
                                               uint pset.max = 16 )'
root@mars:~# poolcfg -c 'create pset db-pset (uint pset.min = 16 ; 
                                              uint pset.max = 16 )'
root@mars:~# poolcfg -c 'associate pool app-pool ( pset app-pset) '
root@mars:~# poolcfg -c 'associate pool db-pool ( pset db-pset) '
root@mars:~# pooladm -c
root@mars:~# poolstat
                              pset
 id pool                 size used load
  0 pool_default          128 0.00 0.58
  3 db-pool                16 0.00 0.14
  2 app-pool               16 0.00 0.15
root@mars:~# psrset
user processor set 1: processors 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
user processor set 2: processors 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
root@mars:~# psrinfo |wc -l
     128
root@mars:~# zonecfg -z app-prod set pool=app-pool
root@mars:~# zoneadm -z app-prod apply
zone 'app-prod': Checking: Setting pool=app-pool
zone 'app-prod': Applying the changes
root@mars:~# zonecfg -z app-prod info pool
pool: app-pool
root@mars:~# zonecfg -z db-prod set pool=db-pool
root@mars:~# zoneadm -z db-prod apply
zone 'db-prod': Checking: Setting pool=db-pool
zone 'db-prod': Applying the changes
root@mars:~# zonecfg -z db-prod info pool
pool: db-pool
root@mars:~# zlogin app-prod 'psrinfo |wc -l'
      16
root@mars:~# zlogin db-prod 'psrinfo |wc -l'
      16
root@mars:~# ./zonecores 
#
# Checking Whole Core Assignments
#
OK - Zone db-prod using all 8 strands of core 0.
OK - Zone db-prod using all 8 strands of core 1.
OK - Zone app-prod using all 8 strands of core 2.
OK - Zone app-prod using all 8 strands of core 3.

Things to note in the example above:

  • The resource pool facility was already enabled.  If that’s not the case, run “pooladm -e” first.
  • Whenever you change the pool configuration, you need to persist and enable those changes with “pooladm -c”.
  • The binding of the zones to their pools was done using zone live reconfiguration.  So the zones were not rebooted for this operation.
  • Once complete, each zone now has exclusive use of 2 cores.  The other 16 cores remain for the global zone.

Next, we’ll need the two zones for the testing environment.  Right after creation, they’ll share CPUs with the global zone.  We can’t afford that, so we’ll assign them to their respective tier’s pools.

root@mars:~# zoneadm list -ivc
  ID NAME             STATUS      PATH                         BRAND      IP    
   0 global           running     /                            solaris    shared
   3 db-prod          running     /system/zones/db-prod        solaris    excl  
   7 app-prod         running     /system/zones/app-prod       solaris    excl  
   8 app-test         running     /system/zones/app-test       solaris    excl  
   9 db-test          running     /system/zones/db-test        solaris    excl  
root@mars:~# for i in db-test app-test ; do zonecfg -z $i info pool ; done
pool: 
pool: 
root@mars:~# ./zonecores 
#
# Checking Whole Core Assignments
#
OK - Zone db-prod using all 8 strands of core 0.
OK - Zone db-prod using all 8 strands of core 1.
OK - Zone app-prod using all 8 strands of core 2.
OK - Zone app-prod using all 8 strands of core 3.
OK - Zone app-test using default pool.
OK - Zone db-test using default pool.
root@mars:~# zonecfg -z app-test set pool=app-pool
root@mars:~# zoneadm -z app-test apply
zone 'app-test': Checking: Setting pool=app-pool
zone 'app-test': Applying the changes
root@mars:~# zonecfg -z db-test  set pool=db-pool
root@mars:~# zoneadm -z db-test apply
zone 'db-test': Checking: Setting pool=db-pool
zone 'db-test': Applying the changes
root@mars:~# for i in db-test app-test ; do zonecfg -z $i info pool ; done
pool: db-pool
pool: app-pool
root@mars:~# zlogin db-test 'psrinfo |wc -l'
      16
root@mars:~# ./zonecores 
#
# Checking Whole Core Assignments
#
OK - Zone db-prod using all 8 strands of core 0.
OK - Zone db-prod using all 8 strands of core 1.
OK - Zone app-prod using all 8 strands of core 2.
OK - Zone app-prod using all 8 strands of core 3.
OK - Zone app-test using all 8 strands of core 2.
OK - Zone app-test using all 8 strands of core 3.
OK - Zone db-test using all 8 strands of core 0.
OK - Zone db-test using all 8 strands of core 1.
root@mars:~# ./zonecores -s
#
# Checking Core Resource Sharing
#
INFO - Core 0 used by 2 zones!

 

-> db-prod

 

-> db-test
INFO – Core 1 used by 2 zones!

 

-> db-prod

 

-> db-test
INFO – Core 2 used by 2 zones!

 

-> app-prod

 

-> app-test
INFO – Core 3 used by 2 zones!

 

-> app-prod

 

-> app-test

So now we have two pairs of zones that each share their common resource pool.  Let’s assume that we ran out of steam in the application tier and our friendly license manager granted a license upgrade to 3 cores for the application.  Of course we don’t want to spoil the party by shutting down the application for this change:

root@mars:~# poolcfg -c 'modify pset app-pset (uint pset.min = 24; 
                                               uint pset.max = 24)'
root@mars:~# pooladm -c
root@mars:~# poolstat
                              pset
 id pool                 size used load
  0 pool_default          120 0.00 0.03
  3 db-pool                16 0.00 0.00
  2 app-pool               24 0.00 0.00
root@mars:~# ./zonecores -sc
#
# Checking Core Resource Sharing
#
INFO - Core 0 used by 2 zones!

 

-> db-prod

 

-> db-test
INFO – Core 1 used by 2 zones!

 

-> db-prod

 

-> db-test
INFO – Core 2 used by 2 zones!

 

-> app-prod

 

-> app-test
INFO – Core 3 used by 2 zones!

 

-> app-prod

 

-> app-test
INFO – Core 4 used by 2 zones!

 

-> app-prod

 

-> app-test
#
# Checking Whole Core Assignments
#
OK – Zone db-prod using all 8 strands of core 0.
OK – Zone db-prod using all 8 strands of core 1.
OK – Zone app-prod using all 8 strands of core 2.
OK – Zone app-prod using all 8 strands of core 3.
OK – Zone app-prod using all 8 strands of core 4.
OK – Zone app-test using all 8 strands of core 2.
OK – Zone app-test using all 8 strands of core 3.
OK – Zone app-test using all 8 strands of core 4.
OK – Zone db-test using all 8 strands of core 0.
OK – Zone db-test using all 8 strands of core 1.

Great!  We added the core to the application environment on the fly.

Of course, we want to make sure that the test environment doesn’t starve production by using too much CPU.  There are two simple ways to achieve this:  We can, within the pool, limit the number of CPUs available to the test zones.  This is also called CPU capping.  Or we can use the Solaris Fair Share Scheduler to guarantee a certain percentage of all available CPU to production.  In the next example, we’ll limit the test database zone to just 1 core and configure CPU shares for the application environment to give production a 75% guarantee:

root@mars:~# zonecfg -z app-prod set scheduling-class=FSS
root@mars:~# zonecfg -z app-prod set cpu-shares=300      
root@mars:~# zonecfg -z app-test set scheduling-class=FSS
root@mars:~# zonecfg -z app-test set cpu-shares=100
root@mars:~# zoneadm -z app-prod apply
zone 'app-prod': Checking: Adding rctl name=zone.cpu-shares
zone 'app-prod': Checking: Setting scheduling-class=FSS
zone 'app-prod': Applying the changes
root@mars:~# zoneadm -z app-test apply
zone 'app-test': Checking: Adding rctl name=zone.cpu-shares
zone 'app-test': Checking: Setting scheduling-class=FSS
zone 'app-test': Applying the changes
root@mars:~# zonecfg -z db-test 'add capped-cpu;set ncpus=8;end;commit'
root@mars:~# zoneadm -z db-test apply
zone 'db-test': Checking: Modifying rctl name=zone.cpu-cap
zone 'db-test': Applying the changes

Again, a few things to note:

  • We changed the default scheduler class to the Fair Share Scheduler in the application zones.  Although this works with the zone running, I recommend to restart the zone at a convenient time for best results.
  • Potentially, you could also combine both controls: Capped CPU and CPU shares.  But this makes configuration rather confusing and also somewhat defeats the purpose of CPU shares.  I don’t recommend this.
  • In SuperCluster environments, the Fair Share Scheduler is not supported for database environments.  I recommend to adhere to this in other installations as well, at least whenever RAC is involved.  Of course, resource pools without the FSS are very much supported in database environments on SuperCluster.  This is how database zones are configured by default.

Finally, a word about licensing.  The above example uses license restrictions as a motivation for CPU pools.  The way it is implemented in this example should, to the best of my knowledge, also satisfy the requirements for hard partitioning and thus license capping of Oracle core based licenses.  However, only Oracle’s License Management Services is authorized to bless such a configuration, so I strongly recommend to validate any configuration with them before using.  Of course, the same is true for any other software vendor’s license capping rules.  Always get the software vendor’s blessing for such a configuration.

As always, here are some links to documentation and references for further reading: