Monday, December 4, 2017

Exporting Metrics from WebLogic Server

By: Russell Gold | Consultant Member of Technical Staff

As it runs, WebLogic Server generates a rich set of metrics and runtime state information. Several thousand individual metrics are available to capture performance data, such as invocation counts, session activity, work manager threads, and so forth. These metrics are very useful for tracking activity, diagnosing problems, and ensuring sufficient resources are available. Exposed through both JMX and web services, these metrics are supported by Oracle administration tools, such as Enterprise Manager and the WebLogic Server Administration Console, as well as third-party clients. 

One of those third-party clients is Prometheus. Prometheus is an open source monitoring toolkit that is commonly used in cloud environments as a framework for gathering, storing, and querying time series data. A number of exporters have been written to scrape information from various services and feed that information into a Prometheus server. Once there, this data can be retrieved using Prometheus itself or other tools that can process Prometheus data, such as Grafana. Oracle customers have been using the generic Prometheus JMX Exporter to scrape information from WebLogic Server instances, but this solution is hampered by usability issues and scalability at larger sites.

Consider the following portion of an MBean tree:

In this tree, ServerRuntime represents the top of the MBean tree and has several ApplicationRuntime MBeans, each of which has multiple ComponentRuntime MBeans. Some of those are of type WebAppComponentRuntime, which has multiple Servlet MBeans. We can configure the JMX Exporter as follows:

jmxUrl: service:jmx:t3://@HOST@:@PORT@/jndi/weblogic.management.mbeanservers.runtime 
username: system 
password: gumby1234 
lowercaseOutputName: false 
lowercaseOutputLabelNames: false 
whitelistObjectNames: 
  - "com.bea:ServerRuntime=*,Type=ApplicationRuntime,*" 
  - "com.bea:Type=WebAppComponentRuntime,*" 
  - "com.bea:Type=ServletRuntime,*" 
 
rules: 
  - pattern: "^com.bea<ServerRuntime=.+, Name=(.+), ApplicationRuntime=(.+), Type=ServletRuntime, WebAppComponentRuntime=(.+)><>(.+): (.+)" 
    attrNameSnakeCase: true 
    name: weblogic_servlet_$4 
    value: $5 
    labels: 
      name: $3 
      app: $2 
      servletName: $1 
 
  - pattern: "^com.bea<ServerRuntime=(.+), Name=(.+), ApplicationRuntime=(.+), Type=WebAppComponentRuntime><>(.+): (.+)$" 
    attrNameSnakeCase: true 
    name: webapp_config_$4 
    value: $5 
    labels: 
      app: $3 
      name: $2

This selects the appropriate MBeans and allows the exporter to generate metrics such as:

webapp_config_open_sessions_current_count{app="receivables",name="accounting"} 3 
webapp_config_open_sessions_current_count{app="receivables",name="inventory"} 7 
weblogic_servlet_invocations_total_count{app="receivables",name="accounting",servletName="Balance"} 0 
weblogic_servlet_invocations_total_count{app="receivables",name="accounting",servletName="Login"} 0 
weblogic_servlet_invocations_total_count{app="receivables",name="inventory",servletName="Count"} 0 
weblogic_servlet_invocations_total_count{app="receivables",name="inventory",servletName="Reorder"} 0

However, this approach has challenges. The JMX Exporter can be difficult to set up because it must run as a Java agent. In addition, because JMX is built on top of RMI, and JMX over RMI/IIOP has been removed from the JRE as of Java SE 9, the exporter must be packaged with a platform-specific RMI implementation.

The JMX Exporter is also somewhat processor-intensive. It requires a separate invocation of JMX to obtain each bean in the tree, which adds to the processing that must be done by the server. And configuring the exporter can be difficult because it relies on MBean names and regular expressions. While it is theoretically possible to select a subset of the attributes for a given MBean, in practice that adds further complexity to the regular expressions, thereby making it impractical. As a result, it is common to scrape everything and incur the transport and storage costs, and then to apply filtering only when the data is eventually viewed.

The WebLogic Monitoring Exporter

Along with JMX, Oracle WebLogic Server 12.2.1 and later provides a RESTful Management Interface for accessing runtime state and metrics. Included in this interface is a powerful bulk access capability that allows a client to POST a query that describes exactly what information is desired and to retrieve a single response that includes only that information.

Oracle has now created the WebLogic Monitoring Exporter, which takes advantage of this interface. This exporter is implemented as a web application that is deployed to the WebLogic Server instance being monitored. Its configuration explicitly follows the MBean tree, starting below the ServerRuntime MBean.

To obtain the same result as in the previous example, we could use the following:

metricsNameSnakeCase: true  
queries:  
  - applicationRuntimes: 
     key: name 
     keyName: app 
     componentRuntimes: 
       type: WebAppComponentRuntime 
       prefix: webapp_config_ 
       key: name 
       values: [openSessionsCurrentCount, openSessionsHighCount] 
       servlets: 
         prefix: weblogic_servlet_ 
         key: servletName

This exporter can scrape the desired metrics with a single HTTP query rather than multiple JMX queries, requires no special setup, and provides an easy way to select the metrics that should be produced for an MBean, while defaulting to using all available fields.

Note that the exporter does not need to specify a URL because it always connects to the server on which it is deployed, and does not specify username and password, but rather requires its clients to specify them when attempting to read the metrics.

Managing the Application

Because the exporter is a web application, it includes a landing page:

Not only does the landing page include the link to the metrics, but it also displays the current configuration. When the app is first loaded, the configuration that’s used is the one embedded in the WAR file. However, the landing page contains a form that allows you to change the configuration by selecting a new yaml file.

Only the queries from the new file are used, and we can combine queries by selecting the Append button before submitting.

For example, we could add some JVM metrics:

The new metrics will be reported the next time a client accesses the metrics URL. The new elements above will produce metrics such as:

jvm_heap_free_current{name="myserver"} 285027752
jvm_heap_free_percent{name="myserver"} 71
jvm_heap_size_current{name="myserver"} 422051840

Metrics in a WebLogic Server Cluster

In a WebLogic Server cluster, of course, it is of little value to change the metrics collected by a single server instance; because all cluster members are serving the same applications, we want them to report the same metrics. To do this, we need a way to have all the servers respond to the changes made to any one of them. The exporter does this by using a separate config_coordinator process to track changes.

To use it, we need to add a new top-level element to the initial configuration that describes the query synchronization:

query_sync: 
  url: http://coordinator:8099 
  refreshInterval: 10  

This specifies the URL of the config_coordinator process, which runs in its own Docker container. When the exporter first starts, and its configuration contains this element, it will contact the coordinator to see if it already has a new configuration. Thereafter, it will do so every time either the landing page or the metrics page is queried. The optional refreshInterval element limits how often the exporter looks for a configuration update. When it finds one, it will load it immediately without requiring a server restart.

When you update the configuration in an exporter that is configured to use the coordinator, the new queries are sent to the coordinator where other exporters can load them. In this fashion, an entire cluster of Managed Servers can have its metrics configurations kept in sync.

Summary

The WebLogic Monitoring Exporter greatly simplifies the process of exporting metrics from clusters of WebLogic Server instances in a Docker/Kubernetes environment. It does away with the need to figure out MBean names and work with regular expressions. It also allows metric labels to be defined explicitly from field names, and then automatically uses those definitions for metrics from subordinate MBeans, ensuring consistency.

In our testing, we have found enormous improvements in performance using it versus the JMX Exporter. It uses less CPU and responds more quickly. In the graphs below, the green lines represent the JMX Exporter, and the yellow lines represent the WebLogic Monitoring Exporter.

We expect users who wish to monitor WebLogic Server performance will gain great benefits from our efforts. See Using Prometheus and Grafana to Monitor WebLogic Server on Kubernetes for more information.

Join the discussion

Comments ( 1 )
  • j schneider Friday, November 10, 2017
    your link to the Weblogic Monitoring Exporter is broken besides nothing is found in the official documentation regarding this feature
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha
 

Visit the Oracle Blog

 

Contact Us

Oracle

Integrated Cloud Applications & Platform Services