Introduction
Performance Co-Pilot (PCP) archive files are generated by the pmlogger process, which collects various PCP metrics at intervals defined in the pmlogger configuration file.
Using PCP data viewing tools, archived metric data can be viewed in a user-friendly and familiar format. However, since the data is stored in binary format in PCP archives, it cannot be viewed directly, or individual metrics’ sizes cannot be compared. As archives can grow quite large based on system size and configuration, it may sometimes be useful to get an insight into how much space each metric group is needing. This blog outlines a way to estimate that.
A quick look at the PCP metric names before diving into the blog
In PCP, metric names are organized within the Performance Metrics Namespace (PMNS), which is structured as a hierarchical tree similar to a filesystem. Each dot (.) in a metric name indicates a new level in this hierarchy. All PMNS pathnames are rooted at an implicit node called root, which is omitted from metric names.
Within the PMNS, metrics are mapped to unique identifiers known as Performance Metric Identifiers (PMIDs). A PMID is a unique code that identifies a specific metric and is assigned only to leaf nodes—those nodes that represent actual, measurable values.
For example:
root
└── kernel
└── cpu
├── user
└── idle
Actual metric names: kernel.cpu.user and kernel.cpu.idleThe PMNS consists of two types of nodes: non-leaf nodes and leaf nodes.
Non-leaf nodes: Used only for grouping metrics, not for directly measuring values that do not have associated PMIDs. In a PMNS file, non-leaf nodes appear as pathnames followed by curly braces { }. Example:
kernel.cpu {
user 60:0:0
idle 60:0:1
}Here, kernel.cpu is a non-leaf node, grouping the metrics beneath it.
Leaf nodes: These nodes represent actual, measurable metrics with associated PMIDs. They appear inside curly braces with their corresponding PMID values.
Using the previous example:
kernel.cpu {
user 60:0:0
idle 60:0:1
}In this case, user and idle are leaf nodes, each with an associated PMID (60:0:0 and 60:0:1 respectively) which collectively makes it “kernel.cpu.user” or “kernel.cpu.idle”. So, non-leaf nodes are just for structure and can be nested multiple levels to group related metrics.
This background information about PMNS should help you understand the rest of the blog better and should help in writing the config file for pmlogextract.
pmlogextract tool: usage and working principles
pmlogextract is a tool to create a reduced archive log file based on a subset of the metrics present in one or more PCP archives. The input archive log file names can be a comma-separated list given to pmlogextract tool, each of which may be the base name of an archive or the name of a directory containing one or more archives.
It also requires an input config file consisting of metric names separated in each line, that are required to be extracted from the input archive log files given with ‘-c’ option.
This is an example of a valid config file for pmlogextract:
kernel.all.cpu
kernel.percpu.cpu.sys ["cpu0","cpu1"]
disk.dev ["sda"]For instance the metric name to be given in a config file can be hinv.ncpu (leaf node) or hinv (non-leaf node), pmlogextract will recursively descend the PMNS and include all metrics corresponding to descendent leaf nodes. So, if given only the hinv metric name in the config file then all other metrics like hinv.ncpu, hinv.physmem, hinv.pagesize etc will also be considered.
Here’s an example of using pmlogextract:
- Create a config file with metrics to be extracted:
[root@localhost localhost]$ cat config
nfsclient- Run the pmlogextract command with the config file, any existing archive and the desired output archive file name:
[root@localhost localhost]# cd /var/oled/pcp/pmlogger/$(hostname)/
[root@localhost localhost]# pmlogextract -c config 20251226.00.10.1 abcd- Check generated output archive files:
[root@localhost localhost]# ls -ltrh |grep abcd
-rw-r--r--. 1 root root 4.9K Dec 26 04:43 abcd.meta
-rw-r--r--. 1 root root 292 Dec 26 04:43 abcd.index
-rw-r--r--. 1 root root 603K Dec 26 04:43 abcd.0In the earlier example, specifying only “nfsclient” in the configuration file as a metric name allowed pmlogextract to include all metrics beginning with nfsclient. This approach ensures that all metrics under that namespace are captured.
Let’s explore another example to see how pmlogextract selects all descendant leaf node metrics in the PMNS when a specific metric name is provided in the config file.
Example: In this example, we configure pmlogextract to include only metrics that start with “hinv.map” by specifying it in the config file. As a result, only metrics under this hierarchy will be present in the new archive.
- Creating config file with “hinv.map” metric:
[root@localhost localhost]# cat config
hinv.map- Running the pmlogextract command and checking metrics present in the new archive file:
[root@localhost localhost]# pmlogextract -c config 20250205.11.43.0 test1
[root@localhost localhost]# ls -ltr test1*
-rw-r--r-- 1 root root 3481 Jan 7 12:51 test1.meta
-rw-r--r-- 1 root root 192 Jan 7 12:51 test1.index
-rw-r--r-- 1 root root 6884 Jan 7 12:51 test1.0
[root@localhost localhost]# pminfo -a test1.0 hinv
hinv.map.scsi_id
hinv.map.scsi
hinv.map.cpu_node
hinv.map.dmname
hinv.map.mdname
hinv.map.cpu_numAs shown above, only metrics starting with “hinv.map” are extracted and included in the new archive file.
Note: If the metric names given in the config file are not present in the PCP archive then pmlogextract would not be able to provide expected output archive log file containing information about the metric.
Using pmlogextract to analyze space consumption by different metrics
This blog mainly focuses on the analysis of space usage of various metrics data in PCP archives using the pmlogextract tool. As mentioned earlier in the blog, there can be scenarios where one may want to understand the space being consumed by the different metrics/groups in PCP archives.
The following bash script returns the report of space utilized by various metric groups such as PCP metrics starting with mem.numa or hinv or disk.dev etc which can help in analysing the space usage by the metric groups in an automated way. Also, the script can be modified to report space usage at a desired granularity level, down to each individual metric.
Disk space requirements: The script needs temporary disk space equal to the largest individual metric group size during execution
#!/bin/bash
# Exit on error
set -e
# Validate input
if [ -z "$1" ]; then
echo "Usage: $0 <pcp-archive>"
exit 1
fi
INPUT_ARCHIVE="$1"
CWD=$(pwd)
CONFIG_FILE="$CWD/tmp_config"
REPORT_FILE="$CWD/output_report_$(date +%H%M%S).txt"
EACH_METRIC="$CWD/tmp_config_2"
TOTAL=0
COUNT=0
# Generate config file with unique metric names (depth: 2 fields if available)
pminfo -a $INPUT_ARCHIVE | awk -F. '{if (NF>1) print $1"."$2; else print $1}' | sort -u > "$CONFIG_FILE"
# For one depth field
#pminfo -a $INPUT_ARCHIVE | awk -F. '{if (NF>1) print $1; else print $1}' | sort -u > "$CONFIG_FILE"
# Create fresh report file
{
echo "Dated: $(date)"
echo "================================ REPORT OF PCP METRICS ARCHIVE LOGS ================================"
echo ""
printf "%-80s %s\n" "Metric Group" "Size(MB)"
echo "------------------------------------------------------------------------------------------------------"
} > "$REPORT_FILE"
# Process metrics
while IFS= read -r line; do
OUTPUT_ARCHIVE="$CWD/tmp_archive_${COUNT}"
echo "$line" > "$EACH_METRIC"
if pmdumplog -d "$INPUT_ARCHIVE" |grep PMID|cut -d ' ' -f 3|tr -d '()'|grep "$line" >/dev/null 2>&1; then
pmlogextract -c "$EACH_METRIC" "$INPUT_ARCHIVE" "$OUTPUT_ARCHIVE" >/dev/null 2>&1
else
continue
fi
# compute size safely
if [ -e "$OUTPUT_ARCHIVE".0 ]; then
GEN_ARCH_SIZE=$(stat -c%s "$OUTPUT_ARCHIVE".0)
GEN_ARCH_SIZE_MB=$(awk "BEGIN {printf \"%.3f\", $GEN_ARCH_SIZE/1048576}")
else
GEN_ARCH_SIZE=0
fi
rm -f "${OUTPUT_ARCHIVE}"*
TOTAL=$(awk "BEGIN {printf \"%.3f\", $TOTAL + $GEN_ARCH_SIZE_MB}")
printf "%-80s %s\n" "$line" "$GEN_ARCH_SIZE_MB" >> "$REPORT_FILE"
COUNT=$((COUNT + 1))
done < "$CONFIG_FILE"
# Print total size
printf "\n%-80s %s\n" "TOTAL" "$TOTAL" >> "$REPORT_FILE"
# Clean up
rm -f "$CONFIG_FILE" "$EACH_METRIC"
What does this script do?
Here’s the detailed explanation of the above script:
- Initialize a few variables like input archive, config file and report file:
INPUT_ARCHIVE="$1"
CWD=$(pwd)
CONFIG_FILE="$CWD/tmp_config"
REPORT_FILE="$CWD/output_report_$(date +%H%M%S).txt"
EACH_METRIC="$CWD/tmp_config_2"
TOTAL=0
COUNT=0- Create a config file having all the available PCP metrics in the input archive file:
pminfo -a $INPUT_ARCHIVE | awk -F. '{if (NF>1) print $1"."$2; else print $1}' | sort -u > "$CONFIG_FILE"- Create a report file with header initials:
{
echo "Dated: $(date)"
echo "================================ REPORT OF PCP METRICS ARCHIVE LOGS ================================"
echo ""
printf "%-80s %s\n" "Metric Group" "Size(MB)"
echo "------------------------------------------------------------------------------------------------------"
} > "$REPORT_FILE"- Now, this part of script performs below processing:
- Start a loop that reads each line of metric name from the config file.
- b). Copies first line of metric to another temporary config file.
- c). Verifies that the current metric contains recorded samples in the archive by using the pmdumplog command.
- d). Runs the pmlogextract command with the temporary config file.
- e). After pmlogextract completes, it captures the size of generated temporary archive file and copies it to the report file.
- f). At the end, the generated archive file size is copied to the report file and added to the total size variable.
while IFS= read -r line; do
OUTPUT_ARCHIVE="$CWD/tmp_archive_${COUNT}"
echo "$line" > "$EACH_METRIC"
if pmdumplog -d "$INPUT_ARCHIVE" |grep PMID|cut -d ' ' -f 3|tr -d '()'|grep "$line" >/dev/null 2>&1; then
pmlogextract -c "$EACH_METRIC" "$INPUT_ARCHIVE" "$OUTPUT_ARCHIVE" >/dev/null 2>&1
else
continue
fi
# compute size safely
if [ -e "$OUTPUT_ARCHIVE".0 ]; then
GEN_ARCH_SIZE=$(stat -c%s "$OUTPUT_ARCHIVE".0)
GEN_ARCH_SIZE_MB=$(awk "BEGIN {printf \"%.3f\", $GEN_ARCH_SIZE/1048576}")
else
GEN_ARCH_SIZE=0
fi
rm -f "${OUTPUT_ARCHIVE}"*
TOTAL=$(awk "BEGIN {printf \"%.3f\", $TOTAL + $GEN_ARCH_SIZE_MB}")
printf "%-80s %s\n" "$line" "$GEN_ARCH_SIZE_MB" >> "$REPORT_FILE"
COUNT=$((COUNT + 1))
done < "$CONFIG_FILE"- Now copy all the archive file total sizes from the report file:
# Add total size
printf "\n%-80s %s\n" "TOTAL" "$TOTAL" >> "$REPORT_FILE"
# Clean up
rm -f "$CONFIG_FILE" "$EACH_METRIC"Sample output report generated by the script:
[root@localhost localhost]# cat output_report_170956.txt
Dated: Wed Jan 28 17:09:57 GMT 2026
================================ REPORT OF PCP METRICS ARCHIVE LOGS ================================
Metric Group Size(MB)
------------------------------------------------------------------------------------------------------
disk.all 2.452
disk.dev 3.672
disk.dm 3.157
disk.md 0.153
disk.partitions 1.447
disk.wwid 0.623
filesys.avail 0.157
filesys.blocksize 0.083
filesys.capacity 0.157
filesys.free 0.157
filesys.freefiles 0.083
filesys.full 0.157
filesys.maxfiles 0.083
filesys.mountdir 0.169
filesys.used 0.157
filesys.usedfiles 0.083
hinv.cpu 2.125
hinv.hugepagesize 0.041
hinv.map 0.000
hinv.ncpu 0.000
hinv.ndisk 0.000
hinv.ninterface 0.000
hinv.nnode 0.000
hinv.pagesize 0.000
hinv.physmem 0.000
kernel.all 3.071
kernel.percpu 2.313
kernel.pernode 0.318
kernel.uname 0.000
mem.buddyinfo 0.704
...
...
<<<output snipped>>>
...
...
nfsclient.xprt 0.056
nfs.server 0.087
pmcd.pid 0.000
pmcd.pmlogger 0.000
pmcd.seqnum 0.000
proc.fd 2.093
proc.id 41.134
proc.io 30.996
proc.memory 16.600
proc.namespaces 0.029
proc.nprocs 0.041
proc.psinfo 96.769
proc.runq 0.186
proc.schedstat 5.183
rpc.client 0.133
rpc.server 0.128
swap.free 0.054
swap.in 0.087
swap.length 0.054
swap.out 0.087
swap.pagesin 0.041
swap.pagesout 0.124
swap.used 0.054
zram.capacity 0.087
zram.mm_stat 0.112
TOTAL 237.736
Once the space requirements for the metric groups are understood, we can try to reduce space consumption by PCP archives. A couple of techniques to do that are described in the next section. These should only be considered as a last resort and only if absolutely necessary – as they can reduce the capability to diagnose issues on the system, should they occur.
Techniques to control PCP archive growth
The following outlines a few techniques that can be used to reduce space consumption by PCP archives:
Modifying PCP log files retention period
Adjusting the retention period for PCP log files is an effective way to manage disk space. By configuring how long log files are retained, you can minimize the accumulation of archived data and control overall storage usage. Additionally, tuning these settings can help expedite the archiving of unarchived log files.
Steps to modify retention period:
- Edit the Configuration File: Open the /etc/sysconfig/pmlogger_timers file.
- Locate the PMLOGGER_DAILY_PARAMS Argument: Search for the line that begins with PMLOGGER_DAILY_PARAMS, for example:
PMLOGGER_DAILY_PARAMS="-x 1 -k 7 -R"- Adjust Retention Settings:
- The -x option specifies how frequently unarchived log files should be archived (by default, -x 1 compresses archives that are older than one day to save disk space). Therefore, with the default value log files are being compressed sooner. Avoid changing the value of -x to 0 which disables compression and can cause high disk usage.
- The -k option sets how many days archived log files are retained before being deleted (by default, -k 7 keeps only the last seven days of archives, deleting anything older). Decrease this value to expedite deletion of older archives more frequently.
- The -x option specifies how frequently unarchived log files should be archived (by default, -x 1 compresses archives that are older than one day to save disk space). Therefore, with the default value log files are being compressed sooner. Avoid changing the value of -x to 0 which disables compression and can cause high disk usage.
- Save and Apply Changes: Save your edits to the file. Restart the pmlogger service to apply the new settings.
Configure PCP metric logging frequency in archives
pmlogger logs PCP archives based on the configuration of logging frequency of various PCP metrics in the pmlogger configuration file. Configuring PCP metric logging frequency can have a significant impact on the size of archive files, however – decreasing this frequency may affect debuggability of the system and hence should be done with caution.
For a default install, the config file is called config.default. However, if you install the pcp-oracle-conf RPM (which we recommend), the file will be called config.ora and is located at, /var/lib/pcp/config/pmlogger/config.ora.
Looking at a snippet from a pmlogger configuration file:
log advisory on 30 sec {
proc.nprocs
proc.psinfo.processor
proc.psinfo.minflt
proc.psinfo.maj_flt
proc.memory.vmstack
proc.id.uid
proc.psinfo.cmaj_flt
proc.psinfo.cmin_flt
}These metrics provide insights into process behavior, memory usage, and CPU utilization. The logged data can be used for performance analysis, monitoring, and troubleshooting. In the above given snippet, some process metrics have been grouped to be logged at a frequency of 30 seconds in the PCP archive file. There are many other groups present in the pmlogger configuration file for various hardware and system related metrics.
The logging frequency for a particular group can be changed by editing the pmlogger configuration file after analyzing the space usage. Increasing or decreasing the frequency can make the PCP archive log file size vary.
To change the frequency of a particular metric group, follow these steps:
- Edit the file “/var/lib/pcp/config/pmlogger/config.ora”
- Change the frequency on the “log advisory” line of a metric group and save the file
- Restart pmlogger: systemctl restart pmlogger.service
After successfully restarting the pmlogger service, the new archive file will be located in /var/log/pcp/pmlogger/${hostname}/. For OCI systems, check the path /var/oled/pcp/pmlogger/${hostname}/.
Using pmdumplog for confirming PCP metric logging frequency
The pmdumplog tool can be used to inspect the contents of the resulting PCP archive in detail. It shows which metrics and instances are present, their PMIDs, timestamps, and recorded values. By examining successive timestamps, it is also possible to verify the effective logging frequency of metrics in any archive, thus it helps to examine the sampling interval that is effective for a metric.
To confirm the pmlogger logging frequency for a particular metric in an archive file using pmdumplog, we can use the following command. Metric_name can be the first word of metric or the exact name.
pmdumplog PCP_archive_file -m Metric_name|grep metric | cut -d' ' -f1 | tail -2 | awk -F'[:.]' '{ t[NR]=$1*3600+$2*60+$3 } END{ print "Frequency : " t[2]-t[1] " seconds" }'Usage example of above given command:
[root@localhost localhost]# pmdumplog 20260102.00.10.0.xz -m nfsclient|grep metric | cut -d' ' -f1 | tail -2 | awk -F'[:.]' '{ t[NR]=$1*3600+$2*60+$3 } END{ print "Frequency : " t[2]-t[1] " seconds" }'
Frequency : 30 secondsSummary
So far, we have discussed PCP archive files, PCP metrics, the usage of the pmlogextract tool, and how to analyze space usage by PCP metrics in archive files. We also covered optimizing the space usage by calibrating the logging frequency for PCP metric in the pmlogger configuration file and confirming it using the pmdumplog tool.
For more detailed overview of PCP and it’s tools, please check out the following blog link: https://blogs.oracle.com/linux/better-diagnostics-with-performance-co-pilot.