Monitoring Oracle WebLogic Server for Oracle Container Engine for Kubernetes with Grafana and Prometheus

How to use open source tools to keep tabs on enterprise applications

August 6, 2021

Download a PDF of this article

Everyone should monitor their production system to understand how the system is behaving. Monitors help you understand the workloads and ensure you get notifications when something fails—or is about to fail.

In Java EE applications, you can choose to monitor many metrics on your servers that will identify workloads and issues with applications. For example, you could monitor the Java heap, active threads, open sockets, CPU utilization, and memory usage.

If you have a Java EE application deployed to Oracle WebLogic Server for Oracle Cloud Infrastructure Container Engine for Kubernetes, this article is for you.

(If you are not familiar with Oracle WebLogic Server for Oracle Cloud Infrastructure Container Engine for Kubernetes, my article “Oracle WebLogic Server for OKE now available on Oracle Cloud Marketplace” is a good introduction.)

Oracle WebLogic Server for Oracle Cloud Infrastructure Container Engine for Kubernetes can help you quickly create Oracle WebLogic configurations on Oracle Cloud, for example, to allocate network resources, reuse existing virtual cloud networks or subnets, configure the load balancer, integrate with Identity Cloud Manager, or configure Oracle Database.

In this article, I’ll show you how to use two open source tools—Grafana and Prometheus—to monitor an Oracle WebLogic domain deployed in Oracle WebLogic Server for Oracle Cloud Infrastructure Container Engine for Kubernetes.

By the way, this procedure will use several Helm charts to walk through the individual steps required to install and configure Prometheus and Grafana. For your own deployment, it is up to you to create a single Helm chart to deploy Prometheus or Grafana.

Prerequisites

Before you get started, you should have installed at least one of these Oracle Cloud Marketplace applications. (UCM refers to the Universal Credits model; BYOL stands for bring your own license.)

Deploy WebLogic Monitoring Exporter to your Oracle WebLogic domain

Here are the step-by-step instructions.

  1. Open a terminal window and access the administration instance that is created with Oracle WebLogic Server for Oracle Cloud Infrastructure Container Engine for Kubernetes. You can see detailed instructions here.
  2. Go to the root Oracle Cloud Infrastructure File Storage Service folder, which is /u01/shared.
    
    cd /u01/shared
    
    
  3. Download the WebLogic Monitoring Exporter war file from GitHub into the wlsdeploy folder.
    
    wget https://github.com/oracle/weblogic-monitoring-exporter/releases/download/v2.0.0/wls-exporter.war -P wlsdeploy/applications
    
    
  4. Include the sample exporter configuration file.
    
    zip -r weblogic-exporter-archive.zip wlsdeploy/
    
    wget https://raw.githubusercontent.com/oracle/weblogic-monitoring-exporter/master/samples/kubernetes/end2end/dashboard/exporter-config.yaml -O config.yml
    zip wlsdeploy/applications/wls-exporter.war -m config.yml
    
    
  5. Create a WebLogic Server Deploy Tooling archive where you’ll place the weblogic-exporter-archive.war file.
    
    zip -r weblogic-exporter-archive.zip wlsdeploy/
    
    
    
    
  6. Create a WebLogic Server Deploy Tooling model to deploy the WebLogic Monitoring Exporter application to your domain.
    
    ADMIN_SERVER_NAME=$(curl -s -H "Authorization: Bearer Oracle" http://169.254.169.254/opc/v2/instance/ | jq -r '.metadata.wls_admin_server_name')
    DOMAIN_CLUSTER_NAME=$(curl -s -H "Authorization: Bearer Oracle" http://169.254.169.254/opc/v2/instance/ | jq -r '.metadata.wls_cluster_name')
    
    cat > deploy-monitoring-exporter.yaml << EOF
    appDeployments:
      Application:
        'wls-exporter' :
          SourcePath: 'wlsdeploy/applications/wls-exporter.war'
          Target: '$DOMAIN_CLUSTER_NAME,$ADMIN_SERVER_NAME'
          ModuleType: war
          StagingMode: nostage
    EOF
    
    
  7. Deploy the WebLogic Monitoring Exporter application to your domain using the Pipeline update-domain screen.

    Access the Jenkins console as described in the documentation.
  8. From the Jenkins dashboard, open the Pipeline update-domain screen and specify the parameters, as follows (and see Figure 1):
    • For Archive_Source, select Shared File System.
    • For Archive_File_Location, enter /u01/shared/weblogic-exporter-archive.zip.
    • For Domain_Model_Source, select Shared File System.
    • For Model_File_Location, enter /u01/shared/deploy-monitoring-exporter.yaml.

    The Pipeline update-domain parameters screen

    Figure 1. The Pipeline update-domain parameters screen

    Then click the build button.

    To verify that the deployment is working, run the following commands:

    
    INGRESS_NS=$(curl -s -H "Authorization: Bearer Oracle" http://169.254.169.254/opc/v2/instance/ | jq -r '.metadata.ingress_namespace')
    SERVICE_NAME=$(curl -s -H "Authorization: Bearer Oracle" http://169.254.169.254/opc/v2/instance/ | jq -r '.metadata.service_name')
    WLS_CLUSTER_URL=$(kubectl get svc "$SERVICE_NAME-external" -n $INGRESS_NS -ojsonpath="{.status.loadBalancer.ingress[0].ip}") 
    
    

    The output should look something like the following:

    
    [opc@wlsoke-admin ~]$ curl -k https://$WLS_CLUSTER_URL/wls-exporter
    <!DOCTYPE html>
    <html lang="en">
    <head>
        <meta charset="UTF-8">
        <title>Weblogic Monitoring Exporter
    </head>
    
    

Create PersistentVolume and PersistentVolumeClaim for Grafana, Prometheus Server, and Prometheus Alertmanager

Oracle WebLogic Server for Oracle Cloud Infrastructure Container Engine for Kubernetes creates a shared file system using Oracle Cloud Infrastructure File Storage Service, which is mounted across the different pods running in the Oracle Container Engine for Kubernetes cluster and the administration host. To store data on that shared file system, the next step is to create subpaths for Grafana and Prometheus to store data.

This procedure will create a Helm chart with PersistentVolume (PV) and PersistentVolumeClaim (PVC) for Grafana, Prometheus Server, and Prometheus Alertmanager. This step doesn’t use the Prometheus and Grafana charts for creating the PVC because those don’t yet support Oracle Cloud Infrastructure Container Engine for Kubernetes with Oracle Cloud Infrastructure File Storage Service.

  1. Open a terminal window and access the administration instance that is created with Oracle WebLogic Server for Oracle Cloud Infrastructure Container Engine for Kubernetes. You can find detailed instructions here.
  2. Create folders for monitoringpv and templates. You’ll place the Helm chart here.
    
    mkdir -p monitoringpv/templates
    
    
  3. Create the Chart.yaml file in the monitoringpv folder.
    
    cat > monitoringpv/Chart.yaml << EOF
    apiVersion: v1
    appVersion: "1.0"
    description: A Helm chart for creating pv and pvc for Grafana, Prometheus and Alertmanager
    name: monitoringpv
    version: 0.1.0
    EOF
    
    
  4. Similarly, create the values.yaml file required for the chart using the administration instance metadata.
    
    cat > monitoringpv/values.yaml << EOF
    exportpath: $(curl -s -H "Authorization: Bearer Oracle" http://169.254.169.254/opc/v2/instance/ | jq -r '.metadata.fss_export_path')
    classname: $(curl -s -H "Authorization: Bearer Oracle" http://169.254.169.254/opc/v2/instance/ | jq -r '.metadata.fss_chart_name')
    serverip: $(kubectl get pv jenkins-oke-pv -o jsonpath='{.spec.nfs.server}')
    EOF
    
    
  5. Create the target folders on the shared file system.
    
    mkdir /u01/shared/alertmanager
    mkdir /u01/shared/prometheus
    mkdir /u01/shared/grafana
    
    
  6. Create template files for PV and PVC for Grafana, Prometheus Server, and Prometheus Alertmanager.
    
    cat > monitoringpv/templates/grafanapv.yaml << EOF
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-grafana
    spec:
      accessModes:
      - ReadWriteMany
      capacity:
        storage: 10Gi
      mountOptions:
      - nosuid
      nfs:
        path: {{ .Values.exportpath }}{{"/grafana"}}
        server: "{{ .Values.serverip }}"
      persistentVolumeReclaimPolicy: Retain
      storageClassName: "{{ .Values.classname }}"
      volumeMode: Filesystem
    EOF
    
    cat > monitoringpv/templates/grafanapvc.yaml << EOF
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: pvc-grafana
      namespace: monitoring
    spec:
      accessModes:
      - ReadWriteMany
      resources:
        requests:
          storage: 10Gi
      storageClassName: "{{ .Values.classname }}"
      volumeMode: Filesystem
      volumeName: pv-grafana
    EOF
    
    cat > monitoringpv/templates/prometheuspv.yaml << EOF
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-prometheus
    spec:
      accessModes:
      - ReadWriteMany
      capacity:
        storage: 10Gi
      mountOptions:
      - nosuid
      nfs:
        path: {{ .Values.exportpath }}{{"/prometheus"}}
        server: "{{ .Values.serverip }}"
      persistentVolumeReclaimPolicy: Retain
      storageClassName: "{{ .Values.classname }}"
      volumeMode: Filesystem
    EOF
    
    cat > monitoringpv/templates/prometheuspvc.yaml << EOF
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: pvc-prometheus
      namespace: monitoring
    spec:
      accessModes:
      - ReadWriteMany
      resources:
        requests:
          storage: 10Gi
      storageClassName: "{{ .Values.classname }}"
      volumeMode: Filesystem
      volumeName: pv-prometheus
    EOF
    
    cat > monitoringpv/templates/alertmanagerpv.yaml << EOF
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-alertmanager
    spec:
      accessModes:
      - ReadWriteMany
      capacity:
        storage: 10Gi
      mountOptions:
      - nosuid
      nfs:
        path: {{ .Values.exportpath }}{{"/alertmanager"}}
        server: "{{ .Values.serverip }}"
      persistentVolumeReclaimPolicy: Retain
      storageClassName: "{{ .Values.classname }}"
      volumeMode: Filesystem
    EOF
    
    cat > monitoringpv/templates/alermanagerpvc.yaml << EOF
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: pvc-alertmanager
      namespace: monitoring
    spec:
      accessModes:
      - ReadWriteMany
      resources:
        requests:
          storage: 10Gi
      storageClassName: "{{ .Values.classname }}"
      volumeName: pv-alertmanager
    EOF
    
    
  7. Install the monitoringpv Helm chart you created.
    
    helm install monitoringpv monitoringpv --create-namespace --namespace monitoring --wait
    
    
  8. Verify that the output looks something like the following:
    
    [opc@wlsoke-admin ~]$ helm install monitoringpv monitoringpv --namespace monitoring --wait
    NAME: monitoringpv
    LAST DEPLOYED: Wed Apr  15 16:43:41 2021
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    

Install the Prometheus Helm chart

These instructions are a subset of those in the Prometheus Community Kubernetes Helm Charts GitHub project. Do these steps in the same terminal window where you accessed the administration instance created with Oracle WebLogic Server for Oracle Cloud Infrastructure Container Engine for Kubernetes (here are those instructions again):

  1. Add the required Helm repositories.
    
    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
    helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
    helm repo update
    
    
    At this time, you could optionally inspect all of Helm’s available configurable options by showing Prometheus’ values.yaml file.
    
    helm show values prometheus-community/prometheus
    
    
  2. Copy the needed values from the WebLogic Monitoring Exporter GitHub project to the Prometheus directory.
    
    wget https://raw.githubusercontent.com/oracle/weblogic-monitoring-exporter/master/samples/kubernetes/end2end/prometheus/values.yaml -P prometheus
    
    
  3. To customize your Prometheus deployment with your own domain information, create a custom-values.yaml file to override some of the values from the prior step.
    
    DOMAIN_NS=$(curl -s -H "Authorization: Bearer Oracle" http://169.254.169.254/opc/v2/instance/ | jq -r '.metadata.wls_domain_namespace')
    DOMAIN_NAME=$(curl -s -H "Authorization: Bearer Oracle" http://169.254.169.254/opc/v2/instance/ | jq -r '.metadata.wls_domain_uid')
    DOMAIN_CLUSTER_NAME=$(curl -s -H "Authorization: Bearer Oracle" http://169.254.169.254/opc/v2/instance/ | jq -r '.metadata.wls_cluster_name')
    
    cat > prometheus/custom-values.yaml << EOF
    alertmanager:
      prefixURL: '/alertmanager'
      baseURL: http://localhost:9093/alertmanager
    nodeExporter:
      hostRootfs: false
    server:
      prefixURL: '/prometheus'
      baseURL: "http://localhost:9090/prometheus"
    extraScrapeConfigs: |
        - job_name: '$DOMAIN_NAME'
          kubernetes_sd_configs:
          - role: pod
          relabel_configs:
          - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_weblogic_domainUID, __meta_kubernetes_pod_label_weblogic_clusterName]
            action: keep
            regex: $DOMAIN_NS;$DOMAIN_NAME;$DOMAIN_CLUSTER_NAME
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: \$1:\$2
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: pod_name
          basic_auth:
            username: --FIX ME--
            password: --FIX ME--
    EOF
    
    
  4. Open the custom-values.yaml file and update the username and password. Use the credentials you use to log in to the administrative console.
    
          basic_auth:
            username: myadminuser
            password: myadminpwd
    
    
  5. Install the Prometheus chart.
    
    helm install --wait prometheus prometheus-community/prometheus --namespace monitoring -f prometheus/values.yaml -f prometheus/custom-values.yaml
    
    
  6. Verify that the output looks something like the following:
    
    [opc@wlsoke-admin ~]$ helm install --wait prometheus prometheus-community/prometheus --namespace monitoring -f prometheus/values.yaml -f prometheus/custom-values.yaml
    NAME: prometheus
    LAST DEPLOYED: Wed Apr  15 22:35:15 2021
    NAMESPACE: monitoring
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    . . . 
    
    
  7. Create an ingress file to expose Prometheus through the internal load balancer.
    
    cat << EOF | kubectl apply -f -
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      annotations:
        kubernetes.io/ingress.class: nginx
      name: prometheus
      namespace: monitoring
    spec:
      rules:
      - http:
          paths:
          - backend:
              serviceName: prometheus-server
              servicePort: 80
            path: /prometheus
    EOF
    
    
  8. The Prometheus dashboard should now be available at the same IP address used to access the Oracle WebLogic Server Administration Console or the Jenkins console but at the /Prometheus path (see Figure 2).

    The Prometheus dashboard

    Figure 2. The Prometheus dashboard

Install the Grafana Helm chart

The instructions described here are a subset of those in the Grafana Community Kubernetes Helm Charts GitHub project. As before, do these steps within the same terminal window where you accessed the administration instance created with Oracle WebLogic Server for Oracle Cloud Infrastructure Container Engine for Kubernetes.

  1. Add the Grafana charts repository.
    
    helm repo add grafana https://grafana.github.io/helm-charts
    helm repo update
    
    
  2. Create a values.yaml file to customize the Grafana installation.
    
    INGRESS_NS=$(curl -s -H "Authorization: Bearer Oracle" http://169.254.169.254/opc/v2/instance/ | jq -r '.metadata.ingress_namespace')
    
    SERVICE_NAME=$(curl -s -H "Authorization: Bearer Oracle" http://169.254.169.254/opc/v2/instance/ | jq -r '.metadata.service_name')
    
    INTERNAL_LB_IP=$(kubectl get svc "$SERVICE_NAME-internal" -n $INGRESS_NS -ojsonpath="{.status.loadBalancer.ingress[0].ip}")
    
    mkdir grafana
    
    cat > grafana/values.yaml << EOF
    persistence:
      enabled: true
      existingClaim: pvc-grafana
    
    admin:
      existingSecret: "grafana-secret"
      userKey: username
      passwordKey: password
    
    grafana.ini:
      server:
        domain: "$INTERNAL_LB_IP"
        root_url: "%(protocol)s://%(domain)s:%(http_port)s/grafana/"
        serve_from_sub_path: true
    EOF
    
    
  3. Create a grafana-secret Kubernetes secret file containing admin credentials for Grafana server (with your own credentials, of course).
    
    kubectl --namespace monitoring create secret generic grafana-secret --from-literal=username=your username --from-literal=password=yourpassword
    
    
  4. Install the Grafana Helm chart.
    
    helm install --wait grafana grafana/grafana --namespace monitoring -f grafana/values.yaml
    
    
  5. Verify that the output looks something like the following:
    
    [opc@wlsoke-admin ~]$ helm install --wait grafana grafana/grafana --namespace monitoring -f grafana/values.yaml
    NAME: grafana
    LAST DEPLOYED: Fri Apr  16 16:40:21 2021
    NAMESPACE: monitoring
    STATUS: deployed
    REVISION: 1
    NOTES:
    . . .
    
    
     
  6. Expose the Grafana dashboard using the ingress controller.
    
    cat <<EOF | kubectl apply -f -
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      annotations:
        kubernetes.io/ingress.class: nginx
      name: grafana
      namespace: monitoring
    spec:
      rules:
      - http:
          paths:
          - backend:
              serviceName: grafana
              servicePort: 80
            path: /grafana
    EOF
    
    
  7. The Grafana dashboard should now be available at the same IP address used to access the Oracle WebLogic Server Administration Console or the Jenkins console and Prometheus but at the /Grafana path (see Figure 3). You should log in with the credentials you configured in the secret file. This document has instructions on how to access these endpoints through a browser.
     

    The Grafana login screen

    Figure 3. The Grafana login screen

Create the Grafana data source

For this article, I’ll reuse the steps described in the WebLogic Monitoring Exporter sample. You can find the full documentation on how to create Grafana data sources in the Grafana documentation.

  1. Once you log in to the Grafana dashboard (as shown in Figure 3), go to Configuration > Data Sources (see Figure 4) and click Add data source to go to the screen where you add the new data source (see Figure 5).
     

    The Configuration menu with the Data Sources option

    Figure 4. The Configuration menu with the Data Sources option

    Screen where you add a new data source

    Figure 5. The screen where you add a new data source

  2. Select Prometheus as the data source type (see Figure 6).
     

    Choose Prometheus as the data source type.

    Figure 6. Choose Prometheus as the data source type.

  3. Set the URL to http://<INTERNAL_LB_IP>/prometheus and click the Save&Test button (see Figure 7).

Important note. INTERNAL_LB_IP is the same IP address you use to access Grafana, Prometheus, Jenkins, and Oracle WebLogic Server Administration Console. You can see how to get that address in this document.

Setting the URL for the data source; be sure to use your own IP address.

Figure 7. Set the URL for the data source; be sure to use your own IP address.

Import the Oracle WebLogic Server dashboard into Grafana

  1. Log in to the Grafana dashboard. Navigate to Dashboards > Manage and click Import (see Figure 8).
     

    The screen for importing a new dashboard

    Figure 8. The screen for importing a new dashboard

  2. Open this JSON code file in a browser. Copy the contents into the Import via panel json section of the dashboard screen and click Load (see Figure 9).
     

    This is where you’ll paste the JSON code.

    Figure 9. This is where you’ll paste the JSON code.

  3. Click the Import button and verify you can see the Oracle WebLogic Server dashboard on Grafana (see Figure 10). That’s it! You’re done!
     

    The Oracle WebLogic Server dashboard running within Grafana

    Figure 10. The Oracle WebLogic Server dashboard running within Grafana

Conclusion

In this article, you have seen how to use two open source tools, Grafana and Prometheus, to monitor your Oracle WebLogic domain deployed in Oracle WebLogic Server for Oracle Cloud Infrastructure Container Engine for Kubernetes. Alternatively, you can use Oracle Cloud Observability and Management Platform to monitor, diagnose, and manage alerts for multicloud and on-premises environments.

Credits. This article is an adaptation and actualization of the WebLogic Monitoring Exporter end-to-end sample. You can read about that sample here.

Dig deeper

Adrian Padilla Duarte

Adrian Padilla Duarte is the product manager for Oracle WebLogic Server for Oracle Cloud Infrastructure, Oracle Container Engine for Kubernetes, and Java Cloud Service. He has more than 15 years of experience working with Java-related technologies, Eclipse, Rich Client Platform, and cloud native solutions. He has the title of engineer from the Universidad de Guadalajara and a master's in computer science from the Center for Research and Advanced Studies of the National Polytechnic Institute (CINVESTAV).

Share this Page