Freitag Dez 17, 2010

Solaris knows Hardware - pgstat explains it

When Sun's engineering teams observed large differences in memory latency on the E25K, they introduced the concept of locality groups (lgrp) into Solaris 9 9/02. They describe the hierarchy of system components, which can be very different in different hardware systems. When creating processes and scheduling them onto CPUs for execution, Solaris will try to minimize the distance between CPU and memory for optimal latency. This feature, known as Memory Placement Optimization (MPO) can, depending on hardware and appliation, significantly enhance performance.

There are, among many other things, thousands of counters in the Solaris kernel. They can be queried using kstat, cpustat, or more widely used tools like mpstat or iostat. Especially the counters made available with cpustat depend heavily on the underlying hardware. The it hasn't always been easy to analyze the performance benefit of MPO and the utilization of individual parts of the hardware using these counters. For cpustat, there was only a perl-script called corestat to help understand T1/T2 core utilization. This has finally changed with Solaris 11 Express


There are now three new commands: lgrpinfo, pginfo und pgstat.

lgrpinfo shows the hierarchy of the lgroups - the NUMA-architecture of the hardware. This can be useful when configuring resource groups (for containers or standalone) to select the right CPUs.

pginfo shows a different view of this information: A tree of the hardware hierarchy. The leaves of this tree are the individual integer and floatingpoint unit of each core.  Here's a little example from a T2 LDom configured with 16 strands from different cores:


# pginfo -v
0 (System [system]) CPUs: 0-15
|-- 3 (Data_Pipe_to_memory [chip]) CPUs: 0-7
| |-- 2 (Floating_Point_Unit [core]) CPUs: 0-3
| | `-- 1 (Integer_Pipeline [core]) CPUs: 0-3
| `-- 5 (Floating_Point_Unit [core]) CPUs: 4-7
| `-- 4 (Integer_Pipeline [core]) CPUs: 4-7
`-- 8 (Data_Pipe_to_memory [core,chip]) CPUs: 8-15
`-- 7 (Floating_Point_Unit [core,chip]) CPUs: 8-15
|-- 6 (Integer_Pipeline) CPUs: 8-11
`-- 9 (Integer_Pipeline) CPUs: 12-15

As you can see, the mapping of strands to pipelines and cores is easily visible.

pgstat finally, is a worthy successor of corestat. It gives you a good overview of the utilization of all components. Again, an example, on the same LDom, which at the same time shows almost 100% core utilization, something I don't find very often...


# pgstat -Apv 1 2
PG RELATIONSHIP HW UTIL CAP SW USR SYS IDLE CPUS
0 System [system] - - - 100.0% 99.6% 0.4% 0.0% 0-15
3 Data_Pipe_to_memory [chip] - - - 100.0% 99.1% 0.9% 0.0% 0-7
2 Floating_Point_Unit [core] 0.0% 179K 1.3B 100.0% 99.1% 0.9% 0.0% 0-3
1 Integer_Pipeline [core] 80.0% 1.3B 1.7B 100.0% 99.1% 0.9% 0.0% 0-3
5 Floating_Point_Unit [core] 0.0% 50K 1.3B 100.0% 99.1% 0.9% 0.0% 4-7
4 Integer_Pipeline [core] 80.2% 1.3B 1.7B 100.0% 99.1% 0.9% 0.0% 4-7
8 Data_Pipe_to_memory [core,chip] - - - 100.0% 100.0% 0.0% 0.0% 8-15
7 Floating_Point_Unit [core,chip] 0.0% 80K 1.3B 100.0% 100.0% 0.0% 0.0% 8-15
6 Integer_Pipeline 76.4% 1.3B 1.7B 100.0% 100.0% 0.0% 0.0% 8-11
9 Integer_Pipeline 76.4% 1.3B 1.7B 100.0% 100.0% 0.0% 0.0% 12-15
PG RELATIONSHIP HW UTIL CAP SW USR SYS IDLE CPUS
0 System [system] - - - 100.0% 99.7% 0.3% 0.0% 0-15
3 Data_Pipe_to_memory [chip] - - - 100.0% 99.5% 0.5% 0.0% 0-7
2 Floating_Point_Unit [core] 0.0% 76K 1.2B 100.0% 99.5% 0.5% 0.0% 0-3
1 Integer_Pipeline [core] 79.7% 1.2B 1.5B 100.0% 99.5% 0.5% 0.0% 0-3
5 Floating_Point_Unit [core] 0.0% 42K 1.2B 100.0% 99.5% 0.5% 0.0% 4-7
4 Integer_Pipeline [core] 79.8% 1.2B 1.5B 100.0% 99.5% 0.5% 0.0% 4-7
8 Data_Pipe_to_memory [core,chip] - - - 100.0% 99.9% 0.1% 0.0% 8-15
7 Floating_Point_Unit [core,chip] 0.0% 80K 1.2B 100.0% 99.9% 0.1% 0.0% 8-15
6 Integer_Pipeline 76.3% 1.2B 1.5B 100.0% 100.0% 0.0% 0.0% 8-11
9 Integer_Pipeline 76.4% 1.2B 1.5B 100.0% 99.8% 0.2% 0.0% 12-15

SUMMARY: UTILIZATION OVER 2 SECONDS

------HARDWARE------ ------SOFTWARE------
PG RELATIONSHIP UTIL CAP MIN AVG MAX MIN AVG MAX CPUS
0 System [system] - - - - - 100.0% 100.0% 100.0% 0-15
3 Data_Pipe_to_memory [chip] - - - - - 100.0% 100.0% 100.0% 0-7
2 Floating_Point_Unit [core] 76K 1.2B 0.0% 0.0% 0.0% 100.0% 100.0% 100.0% 0-3
1 Integer_Pipeline [core] 1.2B 1.5B 79.7% 79.7% 80.0% 100.0% 100.0% 100.0% 0-3
5 Floating_Point_Unit [core] 42K 1.2B 0.0% 0.0% 0.0% 100.0% 100.0% 100.0% 4-7
4 Integer_Pipeline [core] 1.2B 1.5B 79.8% 79.8% 80.2% 100.0% 100.0% 100.0% 4-7
8 Data_Pipe_to_memory [core,chip] - - - - - 100.0% 100.0% 100.0% 8-15
7 Floating_Point_Unit [core,chip] 80K 1.2B 0.0% 0.0% 0.0% 100.0% 100.0% 100.0% 8-15
6 Integer_Pipeline 1.2B 1.5B 76.3% 76.3% 76.4% 100.0% 100.0% 100.0% 8-11
9 Integer_Pipeline 1.2B 1.5B 76.4% 76.4% 76.4% 100.0% 100.0% 100.0% 12-15

The exact meaning of these values is nicely described in the manpage for pgstat, so I'll leave the interpretation to the reader. With this little tool, performance analysis, especially on T2/T3 systems, will be even more fun ;-)

About

Neuigkeiten, Tipps und Wissenswertes rund um SPARC, CMT, Performance und ihre Analyse sowie Erfahrungen mit Solaris auf dem Server und dem Laptop.

This is a bilingual blog (most of the time). Please select your prefered language:
.
The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today