Solaris 11 | Thursday, December 9, 2010

All New Zonestat - Part 2

By: Jeff Victor | Principal Systems Engineer

Part 2


Recently I
introduced zonestat(1)
, a new command offered in Solaris
11 Express that replaces the zonestat Perl script that I had open-sourced a couple
of years ago. Today I will complete the description of the new zonestat.

Fair Share Scheduler


One of the many useful resource controls offered by Solaris is the Fair Share
Scheduler (FSS(7)). You can use FSS to tell Solaris "make sure that zoneXYZ can
use a specific portion of the compute capacity of a set of
'Solaris CPUs'
at a minimum.
(For more information on FSS, see its man page and the Solaris 10 or Solaris 11
Express documentation.) FSS only enforces those minima if there is CPU contention.


That terse explanation used the phrase "of a set of CPUs" because the minimum
portions of compute capacity enforced by Solaris can be calculated across all
of the CPUs in the system, or across a set of CPUs that you have configured into
a processor set. For my purpose here, the point is that you can create a
resource pool - including a processor set, assign multiple zones to that
pool, and tell Solaris to use the FSS scheduling algorithm for the processes
to be scheduled on those CPUs. (See the libpool(3LIB) man page for more information.)


Zonestat will show information relating to FSS, including data that answers the
question "is there CPU contention in any of my psets, and if there is, which
zone(s) are using more than I expected?"


The next example uses two new zones, and assumes that most of the zones used
earlier have been turned off for now. A dynamic resource pool, sharedDB, has
been created. It will be the set of CPUs used by two zones which run database
software. (This method is called "capped Containers" in Oracle licensing
documents and is considered hard partitioning. It can be used to limit
software license costs.) One of
those zones, zoneDB-2, is more important than the other, zoneDB-1. To meet its
SLA, zoneDB-2 must always be able to use at least 4 of the 6 CPUs in that pset.

# zonestat -r psets 10 1
Collecting data for first interval...
Interval: 1, Duration: 0:00:10
PROCESSOR_SET TYPE ONLINE/CPUS MIN/MAX
pset_default default-pset 22/22 1/-
ZONE USED PCT CAP %CAP SHRS %SHR %SHRU
[total] 0.12 0.56% - - - - -
[system] 0.02 0.12% - - - - -
global 0.09 0.44% - - - - -
PROCESSOR_SET TYPE ONLINE/CPUS MIN/MAX
sharedDB pool-pset 6/6 6/6
ZONE USED PCT CAP %CAP SHRS %SHR %SHRU
[total] 4.01 66.8% - - 300 - -
[system] 0.00 0.00% - - - - -
zoneDB-2 3.00 50.1% - - 200 66.6% 75.1%
zoneDB-1 1.00 16.7% - - 100 33.3% 50.2%
PROCESSOR_SET TYPE ONLINE/CPUS MIN/MAX
zoneA dedicated-cpu 4/4 4/4
ZONE USED PCT CAP %CAP SHRS %SHR %SHRU
[total] 0.00 0.09% - - - - -
[system] 0.00 0.00% - - - - -
zoneA 0.00 0.09% - - - - -

In the output above, zoneDB-1 is using the equivalent processing capacity
of 1 Solaris CPU, and zoneDB-2 is using the equivalent of 3 Solaris CPUs.
The PCT column indicates that zoneDB-2 is using 50% (3 out of 6) of the
CPU capacity of the entire pset.


In addition, the SHRS column shows the number of FSS shares assigned to
each of those zones, and %SHR is that zone's proportion of the total
number of shares. In other words, zoneDB-2 is using 50% of the pset, but
hasn't even used its enforced minimum of 66.6%.


This FSS configuration ensures that 200/300ths of 6 CPUs (i.e. 4 CPUs) are
available to zoneDB-2. The %SHRU value of 75% tells us that the zone is
using 75% of those 4 CPUs.
Again, each of those two zones is allowed to use more than its share,
as long as each zone can use its specified minimum.

Sorting the Output

You may have noticed in earlier examples that the zones were not listed in
alphabetical order. By default, they are sorted by the amount of the
resource that zonestat was reporting. If a resource is not specified, the
output is sorted on CPU%. The two rows showing total and system data are
always listed first.


You can change the sort order with the -S option, for example:

GZ$ zonestat -r processes -S used 2 1
Collecting data for first interval...
Interval: 1, Duration: 0:00:02
PROCESSES SYSTEM LIMIT
system-limit 292K
ZONE USED PCT CAP %CAP
[total] 110 0.36% - -
[system] 0 0.00% - -
global 61 0.20% - -
zoneD 26 0.08% - -
zoneA 23 0.07% - -


As the man page for zonestat(1) shows, many other resources can be monitored. I will not review each of them here.

Aggregated Data


But wait! There's more! ;-)


In addition to understanding resource usage over a short interval (e.g. 10 or 60
seconds) it is often to necessary to understand the peak usage or average usage
over a longer period of time. For that purpose, zonestat provides Summary Reports.


A simple summary report is one that is appended to the per-sample data
shown earlier. Here is the output for two samples and one summary report:

GZ$ zonestat -r processes -R high -S used 10 2
Collecting data for first interval...
Interval: 1, Duration: 0:00:10
PROCESSES SYSTEM LIMIT
system-limit 292K
ZONE USED PCT CAP %CAP
[total] 109 0.36% - -
[system] 0 0.00% - -
global 61 0.20% - -
zoneD 25 0.08% - -
zoneA 23 0.07% - -
Interval: 2, Duration: 0:00:20
PROCESSES SYSTEM LIMIT
system-limit 292K
ZONE USED PCT CAP %CAP
[total] 109 0.36% - -
[system] 0 0.00% - -
global 61 0.20% - -
zoneD 25 0.08% - -
zoneA 23 0.07% - -
Report: High Usage
Start: Thu Dec 2 21:58:54 EST 2010
End: Thu Dec 2 21:59:14 EST 2010
Intervals: 2, Duration: 0:00:20
PROCESSES SYSTEM LIMIT
system-limit 292K
ZONE USED PCT CAP %CAP
[total] 109 0.36% - -
[system] 0 0.00% - -
global 61 0.20% - -
zoneD 25 0.08% - -
zoneA 23 0.07% - -

If you only need the summary report, -q will be useful: it suppresses the individual
samples of data.
GZ$ zonestat -q -r physical-memory -R high -S used 10 2
Report: High Usage
Start: Thu Dec 2 22:03:54 EST 2010
End: Thu Dec 2 22:04:14 EST 2010
Intervals: 2, Duration: 0:00:20
PHYSICAL-MEMORY SYSTEM MEMORY
mem_default 31.8G
ZONE USED PCT CAP %CAP
[total] 5205M 15.9% - -
[system] 2790M 8.54% - -
zoneD 2229M 6.83% - -
global 141M 0.43% - -
zoneA 44.5M 0.13% - -

Let's assume that, from the data above, we determine that zoneD will never need
more than 3 GB of RAM when it is operating correctly. We can add a RAM
cap so that Solaris enforces that limit in case something goes awry:
GZ$ pfexec rcapadm -z zoneD -m 3g
GZ$ zonestat -q -r physical-memory -R high -S used 10 2
Report: High Usage
Start: Thu Dec 2 22:37:15 EST 2010
End: Thu Dec 2 22:37:35 EST 2010
Intervals: 2, Duration: 0:00:20
PHYSICAL-MEMORY SYSTEM MEMORY
mem_default 31.8G
ZONE USED PCT CAP %CAP
[total] 5204M 15.9% - -
[system] 2789M 8.54% - -
zoneD 2229M 6.83% 3072M 72.5%
global 141M 0.43% - -
zoneA 44.5M 0.13% - -

In addition to a summary report at the end of the output, zonestat is able to
generate periodic summary reports based on the collection of individual samples.
For example, you might want to know what the peak memory usage is for each zone,
per hour.

(A quick side note: zonestat does not perform continuous data collection.
It collects data at an interval you specify. Therefore, the peak values
reported by zonestat are the peak values of the values which were
collected
. In other words, zonestat reports the peak observed values.)


The example below collects data every 10 seconds for 24 hours. It reports the peak
observed values every hour.

GZ$ zonestat -q -r physical-memory -R high 10 24h 60m
Report: High Usage
Start: Mon Dec 5 16:42:01 EST 2010
End: Mon Dec 5 17:42:01 EST 2010
Intervals: 360, Duration: 1:00:00
PHYSICAL-MEMORY SYSTEM MEMORY
mem_default 31.8G
ZONE USED PCT CAP %CAP
[total] 3015M 9.23% - -
[system] 2791M 8.55% - -
global 136M 0.41% - -
zoneA 44.5M 0.13% - -
zoneD 45.7M 0.13% 3072M 1.48%
Report: High Usage
Start: Mon Dec 5 17:42:01 EST 2010
End: Mon Dec 5 18:42:01 EST 2010
Intervals: 10, Duration: 1:00:00
PHYSICAL-MEMORY SYSTEM MEMORY
mem_default 31.8G
ZONE USED PCT CAP %CAP
[total] 3015M 9.23% - -
[system] 2791M 8.55% - -
global 136M 0.41% - -
zoneA 44.5M 0.13% - -
zoneD 64.3M 0.19% 3072M 2.09%
Report: High Usage
Start: Mon Dec 5 18:42:01 EST 2010
End: Mon Dec 5 19:42:01 EST 2010
Intervals: 15, Duration: 1:00:00
PHYSICAL-MEMORY SYSTEM MEMORY
mem_default 31.8G
ZONE USED PCT CAP %CAP
[total] 3015M 9.23% - -
[system] 2791M 8.55% - -
global 136M 0.41% - -
zoneA 44.5M 0.13% - -
zoneD 65.1M 0.19% 3072M 2.11%
[further output deleted]

Parseable Data


At this point (especially after reading the sub-title of this section...)
you might think that zonestat should have an option to generate output which
is easy to parse. And you won't be disappointed: CSV (colon-separated-value)
output is the result of the lower-case -p option:
GZ$ zonestat -p -q -r physical-memory -R high -S used 10 2
report-high:header:20101203T034655Z:20101203T034715Z:10:20
report-high:physical-memory:mem_default:[resource]:33423360K
report-high:physical-memory:mem_default:[total]:5329836K:15.94%:-:-
report-high:physical-memory:mem_default:[system]:2856556K:8.54%:-:-
report-high:physical-memory:mem_default:zoneD:2282968K:6.83%:3145728K:72.57%
report-high:physical-memory:mem_default:global:144608K:0.43%:-:-
report-high:physical-memory:mem_default:zoneA:45576K:0.13%:-:-
report-high:footer:20101203T034715Z10:20

Even a small number of options can generate a great deal of output:
GZ$ zonestat -p -r psets -R high 10 2
interval:header:since-last-interval:20101203T035201Z:1:10
interval:processor-set:default-pset:pset_default:[resource]:24:24:1:-:0-04-03.01
interval:processor-set:default-pset:pset_default:[total]:0.09:0.38%:-:-:-:-:-:0-00-00.92
interval:processor-set:default-pset:pset_default:[system]:0.01:0.06%:-:-:-:-:-:0-00-00.15
interval:processor-set:default-pset:pset_default:global:0.07:0.31%:-:-:-:-:-:0-00-00.77
interval:processor-set:dedicated-cpu:zoneD:[resource]:4:4:4:4:0-00-40.50
interval:processor-set:dedicated-cpu:zoneD:[total]:1.00:25.01%:-:-:-:-:-:0-00-10.13
interval:processor-set:dedicated-cpu:zoneD:[system]:0.00:0.05%:-:-:-:-:-:0-00-00.02
interval:processor-set:dedicated-cpu:zoneD:zoneD:0.99:24.95%:-:-:-:-:-:0-00-10.10
interval:processor-set:dedicated-cpu:zoneA:[resource]:4:4:4:4:0-00-40.50
interval:processor-set:dedicated-cpu:zoneA:[total]:0.00:0.01%:-:-:-:-:-:0-00-00.00
interval:processor-set:dedicated-cpu:zoneA:[system]:0.00:0.00%:-:-:-:-:-:0-00-00.00
interval:processor-set:dedicated-cpu:zoneA:zoneA:0.00:0.01%:-:-:-:-:-:0-00-00.00
interval:footer:20101203T035201Z10:10
interval:header:since-last-interval:20101203T035211Z:2:20
interval:processor-set:default-pset:pset_default:[resource]:24:24:1:-:0-04-03.08
interval:processor-set:default-pset:pset_default:[total]:0.07:0.32%:-:-:-:-:-:0-00-00.79
interval:processor-set:default-pset:pset_default:[system]:0.01:0.06%:-:-:-:-:-:0-00-00.15
interval:processor-set:default-pset:pset_default:global:0.06:0.26%:-:-:-:-:-:0-00-00.63
interval:processor-set:dedicated-cpu:zoneD:[resource]:4:4:4:4:0-00-40.51
interval:processor-set:dedicated-cpu:zoneD:[total]:1.00:25.01%:-:-:-:-:-:0-00-10.13
interval:processor-set:dedicated-cpu:zoneD:[system]:0.00:0.12%:-:-:-:-:-:0-00-00.04
interval:processor-set:dedicated-cpu:zoneD:zoneD:0.99:24.89%:-:-:-:-:-:0-00-10.08
interval:processor-set:dedicated-cpu:zoneA:[resource]:4:4:4:4:0-00-40.51
interval:processor-set:dedicated-cpu:zoneA:[total]:0.00:0.01%:-:-:-:-:-:0-00-00.00
interval:processor-set:dedicated-cpu:zoneA:[system]:0.00:0.00%:-:-:-:-:-:0-00-00.00
interval:processor-set:dedicated-cpu:zoneA:zoneA:0.00:0.01%:-:-:-:-:-:0-00-00.00
interval:footer:20101203T035211Z10:20
report-high:header:20101203T035151Z:20101203T035211Z:10:20
report-high:processor-set:default-pset:pset_default:[resource]:24:24:1:-:0-04-03.08
report-high:processor-set:default-pset:pset_default:[total]:0.09:0.38%:-:-:-:-:-:0-00-00.92
report-high:processor-set:default-pset:pset_default:[system]:0.01:0.06%:-:-:-:-:-:0-00-00.15
report-high:processor-set:default-pset:pset_default:global:0.07:0.31%:-:-:-:-:-:0-00-00.77
report-high:processor-set:dedicated-cpu:zoneD:[resource]:4:4:4:4:0-00-40.51
report-high:processor-set:dedicated-cpu:zoneD:[total]:1.00:25.07%:-:-:-:-:-:0-00-10.15
report-high:processor-set:dedicated-cpu:zoneD:[system]:0.00:0.12%:-:-:-:-:-:0-00-00.04
report-high:processor-set:dedicated-cpu:zoneD:zoneD:0.99:24.95%:-:-:-:-:-:0-00-10.10
report-high:processor-set:dedicated-cpu:zoneA:[resource]:4:4:4:4:0-00-40.51
report-high:processor-set:dedicated-cpu:zoneA:[total]:0.00:0.01%:-:-:-:-:-:0-00-00.00
report-high:processor-set:dedicated-cpu:zoneA:[system]:0.00:0.00%:-:-:-:-:-:0-00-00.00
report-high:processor-set:dedicated-cpu:zoneA:zoneA:0.00:0.01%:-:-:-:-:-:0-00-00.00
report-high:footer:20101203T035211Z10:20

With parseable output, you can easily write scripts that consume the output. Those
scripts can further analyze the data, draw colorful graphs, and perform other data
manipulation.

Miscellaneous Comments


A few parting notes:

  1. You can run zonestat in any zone. It will only receive data that should be visible
    to that zone. For example, when run in a non-global zone, zonestat will only display
    data about the processor set on which that zone's processes run.
  2. Zonestat gets all of the data from the zonestatd daemon, which is part of the
    service svc:/system/zones-monitoring:default. That service is enabled by default.
    Because that service is managed by SMF, if for any reason zonestatd stops, SMF will
    restart it.
  3. You can specify the interval at which zonestatd gathers data by setting the
    zones-monitoring:default service property config/sample_interval.

Summary


The new zonestat will provide hours of entertainment. It also provides
answers to countless questions regarding your zones' use of system resources.
Armed with that information, you can improve your understanding of the
resource consumption of those zones, and improve the use of resource controls
to ensure predictable performance of your workloads.


I hope that you have enjoyed learning about zonestat as much as I did. Check back during
the first week of January for information on other new observability tools in Solaris 11 Express!




src=""
alt="counter for tumblr" >

Join the discussion

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha
 

Visit the Oracle Blog

 

Contact Us

Oracle

Integrated Cloud Applications & Platform Services