Introduction
Performance Co-Pilot (PCP) provides a powerful framework for system-level performance monitoring and management. It unifies performance data across your system and offers various tools for querying, retrieving, and processing that data. One of the best ways to visualize and analyze PCP metrics is through Grafana, a popular open-source dashboarding platform.
In this blog, we’ll delve into how you can leverage Grafana.
Analyzing Historical Metrics
PCP stores historical performance data in archive logs. These logs contain metrics collected at regular intervals. Grafana allows you to query these metrics using the pmseries query language. You can specify the metric, time range, and aggregation functions to retrieve historical data and create Grafana dashboards with line charts, bar graphs, or heatmaps to visualize trends over time.
For example, track CPU utilization, memory usage, or disk I/O patterns across days, weeks, or months.
Monitoring Real-Time Metrics
Grafana integrates with PCP’s pmwebapi to fetch real-time metrics. Set up panels in Grafana to display live data. Monitor critical metrics like system load, network traffic, or process counts. Configure Grafana alerts to notify you when specific thresholds are breached.
For instance, receive an alert if CPU usage exceeds 90% for more than 5 minutes.
Grafana’s flexibility allows you to tailor your dashboards to your specific needs. Whether you’re a sysadmin, developer, or data enthusiast, Grafana and PCP together provide a powerful toolkit for understanding and optimizing system performance.
How to setup grafana
Now that we have an understanding of the capabilities of Grafana, let’s dive into creating a dashboard in Grafana to visualize and analyze PCP metrics. This setup is performed on Oracle Linux 8.
-
You can refer to this introductory blog on PCP to setup the initial environment before proceeding further. Now, let’s start installing the additional packages that are required from the ol8_appstream repository.
$ yum install grafana grafana-pcp pcp-pmda-haproxy redis
Once the installation of packages is completed you will see the following rpms
$ rpm -qa | grep -Ei 'pcp|redis|grafana' pcp-doc-5.3.7-19.0.2.el8_8..noarch pcp-system-tools-5.3.7-19.0.2.el8_8..x86_64 pcp-5.3.7-19.0.2.el8_8..x86_64 pcp-oracle-conf-1-5.0.6.el8.x86_64 pcp-libs-5.3.7-19.0.2.el8_8..x86_64 pcp-pmda-openmetrics-5.3.7-19.0.2.el8_8..x86_64 pcp-zeroconf-5.3.7-19.0.2.el8_8..x86_64 grafana-pcp-5.1.1-2.el8_9.x86_64 grafana-9.2.10-7.el8_9.x86_64 pcp-conf-5.3.7-19.0.2.el8_8..x86_64 pcp-pmda-nfsclient-5.3.7-19.0.2.el8_8..x86_64 pcp-pmda-dm-5.3.7-19.0.2.el8_8..x86_64 perl-PCP-PMDA-5.3.7-19.0.2.el8_8..x86_64 python3-pcp-5.3.7-19.0.2.el8_8..x86_64 redis-5.0.3-5.module+el8.4.0+20382+7694043a.x86_64 pcp-pmda-haproxy-5.3.7-19.0.2.el8_8..x86_64 pcp-selinux-5.3.7-19.0.2.el8_8..x86_64 pcp-pmda-rds-5.3.7-19.0.2.el8_8..x86_64
-
Enable the required services and start them
$ systemctl enable --now pmcd pmlogger pmproxy grafana-server redis -
Firewall Service
-
If you have a firewall service enabled (such as firewalld), ensure that port 3000 (the default Grafana port) is allowed. Grafana listens on this port for incoming requests.
-
Grafana communicates over port 3000 by default. If the firewall blocks this port, you won’t be able to access Grafana from other machines on the network.
-
This can be achieved as shown below:
$ firewall-cmd --add-port=3000/tcp --permanent success $ firewall-cmd --add-port=3000/udp --permanent success $ firewall-cmd --reload success
-
-
Re-check the port where
grafana-serveris running. By default, it’s3000.$ ss -nltp State Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=856,fd=4)) LISTEN 0 5 127.0.0.1:44321 0.0.0.0:* users:(("pmcd",pid=1422,fd=0)) LISTEN 0 128 0.0.0.0:44322 0.0.0.0:* users:(("pmproxy",pid=1682,fd=11)) LISTEN 0 128 0.0.0.0:44323 0.0.0.0:* users:(("pmproxy",pid=1682,fd=13)) LISTEN 0 128 0.0.0.0:45155 0.0.0.0:* users:(("proxyt",pid=916,fd=4)) LISTEN 0 5 127.0.0.1:4330 0.0.0.0:* users:(("pmlogger",pid=3521,fd=8)) LISTEN 0 511 127.0.0.1:6379 0.0.0.0:* users:(("redis-server",pid=857,fd=6)) LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=856,fd=6)) LISTEN 0 4096 *:3000 *:* users:(("grafana-server",pid=920,fd=19)) LISTEN 0 64 *:16385 *:* LISTEN 0 5 [::1]:44321 [::]:* users:(("pmcd",pid=1422,fd=3)) LISTEN 0 128 [::]:44322 [::]:* users:(("pmproxy",pid=1682,fd=12)) LISTEN 0 128 [::]:44323 [::]:* users:(("pmproxy",pid=1682,fd=14)) LISTEN 0 128 [::]:45155 [::]:* users:(("proxyt",pid=916,fd=3)) LISTEN 0 5 [::1]:4330 [::]:* users:(("pmlogger",pid=3521,fd=9)) -
Check if the PCP plugins are installed using the
grafana-clitool$ grafana-cli plugins ls | grep pcp performancecopilot-pcp-app @ 5.1.1
-
Open your web browser and navigate to Grafana (usually at http://localhost:3000/)
By default the credentials are username :
adminand password:adminFor some reason, if the credentials are incorrect, then set a new password for admin using
$ grafana-cli --homepath "/usr/share/grafana" admin reset-admin-password <new password>
-
Enable the pcp plugin.
- Go to plugins tab under configuration
- Enable the performance co-pilot plugin (if not already enabled)
-
Now that the plugins are in place, we need to import the data source to analyse.
-
Go to the Data sources tab under configuration
-
Click on Add datasource
-
Add the datasource – “PCP redis”
-
Enter the URL http://localhost:44322
-
You can make this as your default data source as well.
-
When finished setting configuration items, you can click on
save and test.
-
-
Once this setup is completed you can now navigate to the Dashboards in the left panel. Here you get the list of prebuilt dashboards.
“PCP Redis: Host Overview” is one among them, that gives insights and trends about Load Average, Memory Utilisation, Network throughput, Network drops and many more.
Conclusion
In this blog, we have understood the capabilities of Grafana and how one can make the most out of it. We have also seen the steps involved in setting up your first Grafana dashboard.