Monitoring Kubernetes Cluster using Management Agent StatefulSet

November 20, 2022 | 8 minute read
Chaitanya Kulkarni
Senior Member of Technical Staff
Nirav Gandhi
Software Development Director
Zubair Ansari
Senior Director of Product Management
Text Size 100%:

Kubernetes has found acceptance as the solution of choice as it offers a highly robust platform with automated software deployment and auto-scaling of containerized workloads.  As more and more applications are deployed in the Kubernetes clusters, there is a need for acquiring an insight into the performance of applications both through logs as well as metrics.  Oracle Management Agent is now available to be deployed in a Kubernetes cluster as a StatefulSet and offers out-of-box monitoring of the Kubernetes Cluster.  Management Agent enables Observability and Management services to collect Logs, Metrics, and Database collections.  Management Agent can also be used to scrape metrics from applications that emit metrics in Prometheus format. 

The blog describes how Management Agent can be deployed using helm charts and visualize the metrics in O&M Dashboard.  

Kubernetes Dashboard
Figure 1:  Kubernetes Cluster Dashboard

Prerequisites for Monitoring Kubernetes Cluster using Management Agent StatefulSet

We assume that the reader is familiar with Management Agent and Dashboard and the following have been set up.

  • oci-cli:  Installed and configured with the right user principal to communicate with OCI services. 
  • Kubectl: Installed and configured with a Kubernetes cluster
  • Helm: Installed and configured with a Kubernetes cluster
  • Kubernetes Cluster: Cluster that has already been set up where the helm charts for Management Agent need to be deployed

The References section contains the documentation for all of these.

Gather the required information in the helm charts to deploy Management Agent:

Kubernetes Cluster Name The name of the Kubernetes cluster
Kubernetes Namespace The names of the Kubernetes cluster namespaces to monitor
Container Image URL URL of the Management Agent container image used for the collection of the metrics.
Container Secret Image secrets to use for pulling container image (base64 encoded content of ~/.docker/config.json file)
Management Agent Install Key input.rsp required for Management Agent registration. 
Compartment OCID The OCID of the Compartment in which the metrics to be ingested.

 

Dynamic Group and Policies allow the Management Agent to emit Kubernetes metrics to OCI Monitoring:

# Dynamic Group for Management Agent
# The compartment OCID should match the compartment of the Management Agent


ALL {resource.type='managementagent', resource.compartment.id='<management_agent_compartment_OCID>'}


# Policies required to allow the Management Agent Dynamic group to post metrics to OCI Monitoring
#  The DG-Group-Name should match the name that you give to the Dynamic Group that you create in the above step.
#  This compartment should match the name of the compartment that you have specified in the values.yml for the metrics

ALLOW DYNAMICGROUP <DG-Group-Name> TO USE METRICS IN COMPARTMENT <metrics-compartment-name> WHERE target.metrics.namespace = 'mgmtagent_kubernetes_metrics'

 

Deployment steps to setup the agent to emit metrics and import the dashboard to view the metrics:

  1. Download the Helm Charts
  2. Generate the Management Agent Install key
  3. Install the Helm Chart
  4. Verify that the Agent pod is running
  5. Import the Dashboard
  6. View the Dashboard in O&M Service

These are described in detail here:

1. Download the Helm Charts

Create a directory to hold the Helm Charts and extract the package

The helm charts are hosted on GitHub.

mkdir /scratch
 
unzip mgmtagent_helm.zip


2. Generate the Management Agent Install Key

Follow the documentation here to create an install key for the Management Agent.  Ensure to create the key in the correct compartment.  Download the key to file and name the file: input.rsp.


3. Helm Charts Install

The Management Agent Helm Chart contains templates that need values from the user

The values.yaml should be available under the directory that was created in the earlier step.

Copy the input.rsp in /scratch/mgmtagent_helm/resources.

mgmtagent:
   # Copy the downloaded Management Agent Install Key file under root helm directory as resources/input.rsp
   installKey: resources/input.rsp
   # Follow steps documented at https://github.com/oracle/docker-images/tree/main/OracleManagementAgent to build docker image.
   image:
      # Replace this value with the actual docker image URL for Management Agent.  This is the value that you have gathered earlier in the above table ('Container Image URL')
      URL: <Container Image URL>
      # Image secrets to use for pulling container image (base64 encoded content of ~/.docker/config.json file)
      #    This content would look like:  ewoJImF1dGhzIjogewoJCSJpYWQub2Npci5pbyI...lci1BZ2VudCI6ICJEb2NrZCkiCgl9Cn0=
      secret: <Container Secret>
 
# By default, this app will create a below namespace and deploy all new resources in that namespace. You can set this to false if you want to use an already existing namespace
# Update the namespace name below if required
createNamespace: true
 
# Kubernetes namespace to create and install this helm chart in
namespace: mgmtagent
 
# By default, the metric server will be deployed and used by Management Agent to collect metrics. You can set this to false if you already have a metric server installed on your cluster
deployMetricServer: true
 
# Kubernetes Cluster details to monitor
kubernetesCluster:
   # OCI Compartment Id to push Kubernetes Monitoring metrics. Enter the value that you have gathered earlier in the above table ('Compartment OCID')
   compartmentId: <Compartment OCID>
   # Kubernetes cluster name:  Enter the value that you have gathered earlier in the above table ('Kubernetes Cluster Name')
   name: <Kubernetes Cluster Name>
   # Kubernetes cluster Namespace to monitor
   namespace: kube-system


Verify the templates as follows:

helm template --values /scratch/mgmtagent_helm/values.yaml /scratch/mgmtagent_helm/

 

Install the helm chart in a 'mgmtagent' namespace

helm install mgmtagent-monitoring /scratch/mgmtagent_helm/

 

4. Verify the installation

Verifies that the stateful set for the Management Agent is ready and running

$ kubectl get statefulset -n=mgmtagent
NAME                  READY   AGE
mgmtagent             1/1     5m40s
 
$ kubectl get pods -l app=mgmtagent -n=mgmtagent
NAME             READY   STATUS    RESTARTS   AGE
mgmtagent-0      1/1     Running   0          5m35s

 

5. Import Dashboard

The GitHub project for the Helm Charts for Management Agent also includes a sample dashboard that helps the user visualize some of the cluster metrics that the agent emits.  The sample dashboard is available as a JSON document.  The sample dashboard requires 2 values that need to be supplied by the user.

dashboardId Yes OCID of the Logging Analytics Dashboard and replace the "<LOGGING_ANALYTICS_DASHBOARD_OCID>" with the actual OCID
compartmentId Yes OCID of the compartment.  This is the compartment in which the dashboard has been created.  Replace "<LOGGING_ANALYTICS_DASHBOARD_COMPARTMENT_OCID>" with the actual value for the OCID.
displayName No The sample includes a display name.  This can be changed by the user
description No The user can modify this to add more description to the name.

The user can create an empty dashboard in Logging Analytics so that it gets assigned an OCID.  This OCID can then be used when the sample dashboard is imported.

OCI-CLI can be used to import this dashboard file

$ oci management-dashboard dashboard import --from-JSON file:///scratch/mgmtagent_helm/sampleDashboard.json

 

6. View the Dashboard in O&M Service

Once the sample dashboard has been created, it will contain the visualization for some of the metrics that the agent is emitting. 

Browse the Dashboard using the O&M Menu in the OCI Console.


7. Extending the Dashboard

Management Agent emits several metrics in the 'mgmtagent_kubernetes_metrics' namespace.  The sample dashboard that was imported only contains the widgets for a few metrics.  The user can always extend the dashboard by adding more metric widgets.  The screenshot below lists the Source data available for importing more metrics. 

Additional Metrics
Figure 2:  Extend Dashboard and add metric widget

 

Downloads

Management Agent Helm Charts and the Sample Dashboard are available on GitHub and can be downloaded here: Link

 

References

OCI-CLI Link
Kubectl Link
Helm Link
Management Agent Install Keys Link
Logging Analytics Dashboard Link

Chaitanya Kulkarni

Senior Member of Technical Staff

After earning a Master's in Computer Science from The University of Texas at Dallas in 2019, I joined Oracle to explore my passion for Software and Technology. Currently, I'm working as one of the Software Developers in the Management Agent team that focuses on providing low latency interactive communication and data collection between Oracle Cloud Infrastructure and any other targets like Databases, VMs, Java applications, etc.

Show more

Nirav Gandhi

Software Development Director

Zubair Ansari

Senior Director of Product Management


Previous Post

Quick start - get more insight on databases using OCI Database Management Dashboards capability

Murtaza Husain | 5 min read

Next Post


Monitoring enterprise exceptions with Stack Monitoring’s new Enterprise summary

Aaron Rimel | 6 min read