X

Proactive insights, news and tips from Oracle WebLogic Server Support. Learn Oracle from Oracle.

  • November 15, 2017

Using Prometheus and Grafana to Monitor WebLogic Server on Kubernetes

Marina Kogan
Principal Member of Technical Staff

As part of the release of our general availability (GA) version of the WebLogic Server Kubernetes Operator, the WebLogic team has created instructions that show how to create and run a WebLogic Server domain on Kubernetes.  The README.md in the project provides all the steps. In the area of monitoring and diagnostics, for open source, we have developed a new tool, WebLogic Monitoring Exporter, which was implemented to scrape runtime metrics for specific WebLogic Server instances and feed them to the Prometheus and Grafana tools.

The Weblogic Monitoring Exporter is a web application that you can deploy on a WebLogic Server instance that you want to monitor. The exporter uses the WebLogic Server 12.2.1.x RESTful Management Interface for accessing runtime state and metrics. For a detailed description of WebLogic Monitoring Exporter configuration and usage, see The WebLogic Monitoring Exporter. In this blog you will learn how to configure Prometheus and Grafana to monitor WebLogic Server instances that are running in Kubernetes clusters.

Monitoring Using Prometheus

We’ll be using the WebLogic Monitoring Exporter to scrape WebLogic Server metrics and feed them to Prometheus. Previous blog entries have described how to start and run WebLogic Server instances in Kubernetes with the WebLogic Monitoring Exporter deployed on Managed Servers running in the cluster.

To make sure that the WebLogic Monitoring Exporter is deployed and running, click the link:

http://[hostname]:30011/wls-exporter/metrics

You will be prompted for the WebLogic user credentials that are required to access the metrics data, which are weblogic/welcome1. The metrics page should show the metrics configured for the WebLogic Monitoring Exporter:

To create a Prometheus instance in Kubernetes, create a Prometheus configuration file, prometheus-deployment.yml. A sample file is provided in our sample, which may be modified as required to match your environment.

The above example of the Prometheus configuration file specifies:

-    weblogic/welcome1 as the user credentials

-    5 seconds as the interval between updates of WebLogic Server metrics

-    32000 as the external port to access the Prometheus dashboard

-    Use of namespace: ‘monitoring’

You can change these values as required to reflect your specific environment and configuration.

To create the namespace ‘monitoring’:

    $ kubectl create -f monitoring-namespace.yaml

To create the corresponding RBAC policy to allow the required permission for the pods. See the provided sample here:

    $ kubectl create -f crossnsrbac.yaml

The RBAC policy in the sample uses the namespaces:

  • ‘weblogic-domain’ for WebLogic Server domain deployment
  • ‘weblogic-operator’ for WebLogic Server Kubernetes

Start Prometheus to monitor the Managed Server instances:

   $ kubectl create -f prometheus-deployment.yaml

Verify that Prometheus is monitoring all Managed Servers by browsing to http://[hostname]:32000.

Examine the Insert metric at cursor pull-down. It should list metric names based on the current configuration of the WebLogic Monitoring Exporter web application.

 

To check that the WebLogic Monitoring Exporter is configured correctly, connect to the web page at  http//:[hostname]:30011/wls-exporter. The current configuration will be listed there. For example:

Below is the corresponding WebLogic Monitoring Exporter configuration yml file:

metricsNameSnakeCase: true

queries:

- applicationRuntimes:

    key: name

    keyName: app

    componentRuntimes:

      type: WebAppComponentRuntime

      prefix: webapp_config_

      key: name

      values: [deploymentState, contextRoot, sourceInfo, openSessionsHighCount, openSessionsCurrentCount, sessionsOpenedTotalCount, sessionCookieMaxAgeSecs, sessionInvalidationIntervalSecs, sessionTimeoutSecs, singleThreadedServletPoolSize, sessionIDLength, servletReloadCheckSecs, jSPPageCheckSecs]

      servlets:

        prefix: weblogic_servlet_

        key: servletName

        values: [invocationTotalCount, reloadTotal, executionTimeAverage, poolMaxCapacity, executionTimeTotal, reloadTotalCount, executionTimeHigh, executionTimeLow]

- JVMRuntime:

    key: name

    values: [heapFreeCurrent, heapFreePercent, heapSizeCurrent, heapSizeMax, uptime, processCpuLoad]

The configuration listed above was embedded into the WebLogic Monitoring Exporter WAR file. To change or add more metrics data, simply connect to the landing page at http//:[hostname]:30011/wls-exporter and use the Append or Replace buttons to load the configuration file in yml format. For example, workmanager.yml:

metricsNameSnakeCase: true

queries:

- applicationRuntimes:

    key: name

    workManagerRuntimes:

      prefix: workmanager_

      key: applicationName

      values: [pendingRequests, completedRequests, stuckThreadCount]

By constructing Prometheus-defined queries, you can generate any data output you require to monitor and diagnose the servers, applications, and resources that are running in your WebLogic domain.

For example, you can enter the following into the query box, and Prometheus will return current data from all running Managed Servers in the WebLogic cluster:

weblogic_servlet_execution_time_average > 1

Prometheus also generates graphs that are based on provided data. For example, if you click on the Graph tab, Prometheus will generate a graph showing the number of servlets with an average execution time exceeding the threshold time by 1 second or more.

 

Monitoring Using Grafana

For better visual presentation and dashboards with multiple graphs, use Grafana.

Here is an example configuration file, grafana-deployment.yaml, which can be used to start Grafana in the Kubernetes environment.

To start Grafana to monitor the Managed Servers, use the following kubectl command:

 $ kubectl create -f grafana-deployment.yaml

To start Grafana to monitor the Managed Servers, use the following kubectl command:

 $ kubectl create -f grafana-kubernetes.yml

Connect to Grafana at http://[hostname]:31000. Log in to the home page with the username admin, and the password pass. The Grafana home page will be displayed.

To connect Grafana to Prometheus, select Add Data Source and then enter the following values:       

Name:    Prometheus       

Type:      Prometheus       

Url:         http://prometheus:9090       

Access:   Proxy

Select the Dashboards tab and click Import:

Now we are ready to generate a dashboard to monitor WebLogic Server. Complete the following steps:

  1. Click the Grafana symbol in the upper left corner of the home page, and select Dashboards add new.
  2. Select Graph and pull it into the empty space. It will generate an empty graph panel:
  3. Click on Click on the panel and select the edit option. It will open an editable panel where you can customize how the metrics graph will be presented.
  4. In the Graph panel, select the General tab, and enter WebLogic Servlet Execution Average Time as the title.
  5. Select the Metrics tab, then select the Prometheus option in the Panel Data Source pull-down menu.

If you click in the empty Metric lookup field, all metrics configured in the WebLogic Monitoring Exporter will be pulled in, the same way as in Prometheus. Let’s enter the same query we used in the Prometheus example, weblogic_servlet_execution_time_average > 1. The generated graphs will show data for all available servlets with an average execution time greater than 1 second, on all Managed Servers in the cluster. Each color represents a specific pod and servlet combination.

To show data for particular pod, simply click on the corresponding legend. This will remove all other pods’ data from the graph, and their legends will no longer be highlighted. To add more data, just press the shift key and click on any desired legend. To reset, just click the same legend again, and all others will be redisplayed the graph.

To customize the legend, click the desired values in the Legend Format field. For example:

{{pod_name}} :appName={{webapp}} : servletName={{servletName}}

Grafana will begin to display your customized legend. If you click the graph, you can see all values for the selected time:

Select the Graph → Legend tab to obtain more options for customizing the legend view. For example, you can move the placement of the legend, show the minimum, maximum, or average values, and more.

By selecting the Graph → Axes tab, you can switch units to the corresponding metrics data, in this example it is time(millisecs):

Grafana also provides alerting tools. For example, we can configure an alert for specified conditions. In the example below, Grafana will fire an alert if the average servlet execution time is greater than 100 msec. It will also send an email to the administrator:

Last, we want our graph to be refreshed every 5 seconds, the same refresh interval as the Prometheus scrape interval. We can also customize the time range for monitoring the data.

To do that, we need to click on right upper corner of the created dashboard. By default, it is configured to show metrics for the prior 6 hours up to the current time. Perform the desired changes. For example, switch to refresh every 5 seconds and click Apply:

When you are done, simply click the ‘save’ icon in the upper left corner of the window, and enter a name for the dashboard.

Summary

WebLogic Server today has a rich set of metrics that can be monitored using well-known tools such as the WebLogic Server Administration Console and the Monitoring Dashboard. These tools are used to monitor the WebLogic Server instances, applications, and resources running in a WebLogic deployment in Kubernetes. In this container ecosystem, tools like Prometheus and Grafana offer an alternative way of exporting and monitoring the metrics from clusters of WebLogic Server instances running in Kubernetes. It also makes monitored data easy to collect, access, present, and customize in real time without restarting the domain. In addition, it provides a simple way to create alerts and send notifications to any interested parties. Start using it, you will love it!

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha
Oracle

Integrated Cloud Applications & Platform Services