X

Proactive insights, news and tips from Oracle WebLogic Server Support. Learn Oracle from Oracle.

Automatic Scaling of WebLogic Clusters on Kubernetes

Lenny Phan
Principal Member of Technical Staff

Elasticity (scaling up or scaling down) of a WebLogic Server cluster provides increased reliability of customer applications as well as optimization of resource usage.  Elasticity was introduced in WebLogic Server 12.2.1 and was built on the concepts of the elastic services framework and dynamic clusters:

 

 

Elasticity in WebLogic Server is achieved by either:

 

·      Manually adding or removing a running server instance in a dynamic WebLogic Server cluster using the WebLogic Server Administration Console, or the WebLogic Scripting Tool (WLST). This is known as on-demand scaling.

 

·      Establishing WLDF scaling policies that set the conditions under which a dynamic cluster should be scaled up or down, and actions that define the scaling operations themselves. When the conditions defined in the scaling policy occur, the corresponding scaling action is triggered automatically.

 

When a scaling action occurs, Managed Server instances are started and stopped through the use of WebLogic Server Node Managers.  Node Manager is a WebLogic Server utility that manages the lifecycle (startup, shutdown, and restart) of Managed Server instances.

 

The WebLogic Server team is investing in running WebLogic Server in Kubernetes cloud environments.  A WebLogic Server cluster can be automatically scaled by increasing (or decreasing) the number of pods based on resource metrics provided by the WebLogic Diagnostic Framework (WLDF).  We will use the sample demo from WebLogic on Kubernetes, Try It! to illustrate automatic scaling of a WebLogic Server cluster in a Kubernetes cloud environment. There are a few key differences between how elasticity works in the sample demo for a Kubernetes cloud environment versus in traditional WebLogic Server deployment environments:

 

1.     The sample demo uses statically-configured clusters, whereas elasticity works with dynamic clusters in a traditional deployment.  We’ll discuss elasticity of WebLogic Server clusters in a Kubernetes cloud environment in a future blog.

2.     In the sample demo, scaling actions invoke requests to the Kubernetes API server to scale pods, versus requests to Node Manager in traditional deployments.

 

In this blog entry, we will show you how a WebLogic Server cluster can be automatically scaled up or down in a Kubernetes environment based on metrics provided by WLDF. 

 

WebLogic on Kubernetes Sample Demo

We will use the WebLogic domain running on Kubernetes described in the following blog entry, WebLogic on Kubernetes, Try It!. 

 

 

 

The WebLogic domain, running in a Kubernetes cluster, consists of:

 

1.     An Administration Server (AS) instance, running in a Docker container, in its own pod (POD 1).

2.     A webhook implementation, running in its own Docker container, in the same pod as the Administration Server (POD 1).

 

What is a webhook? A webhook is a lightweight HTTP server that can be configured

with multiple endpoints (hooks) for executing configured commands, such as shell

scripts. More information about the webhook used in the sample demo, see

adnanh/webhook/.

 

NOTE: As mentioned in WebLogic on Kubernetes, Try It!, a prerequisite for running

 WLDF initiated scaling is building and installing a Webhook Docker image

 (oow-demo-webhook).

 

 

3.     A WebLogic Server cluster composed of a set of Managed Server instances in which each instance is running in a Docker container in its own pod (POD 2 to POD 6).

WebLogic Diagnostic Framework

The WebLogic Diagnostics Framework (WLDF) is a suite of services and APIs that collect and surface metrics that provide visibility into server and application performance.  To support automatic scaling of a dynamic cluster, WLDF provides the Policies and Actions component, which lets you write policy expressions for automatically executing scaling operations on a dynamic cluster. These policies monitor one or more types of WebLogic Server metrics, such as memory, idle threads, and CPU load.  When the configured threshold in a policy is met, the policy is triggered, and the corresponding scaling action is executed. For more information about WLDF and diagnostic policies and actions, see Configuring Policies and Actions in Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

 

Policies can be based on the following types of data:

·       Trends over time, or historical data, such as changes in average values during a specific time interval. For example, a policy can be based on average JVM heap usage above a certain threshold.

·       Runtime metrics relevant to all server instances in a cluster, not just one server instance.

·       Data from multiple services that are considered together. For example, a policy can be based on response-time metrics reported by a load balancer and message-backlog metrics from a message queue.

·       Calendar-based schedules. Policies can identify a specific calendar time, such as time of day or day of week, when a scaling action must be executed.

·       Log rules or event data rules.

Automatic Scaling of a WebLogic Server Cluster in Kubernetes

Here is how we can achieve automatic scaling of a WebLogic Server cluster in a Kubernetes environment using WLDF.  I’ll be discussing only the relevant configuration changes for automatic scaling. You can find instructions for setting up and running the sample demo in WebLogic on Kubernetes, Try It!.

 

First, I’ll quickly describe how automatic scaling of a WebLogic Server cluster in Kubernetes works.

 

In the sample demo, we have a WebLogic Server cluster running in a Kubernetes cluster with a one-to-one mapping of WebLogic Server Managed Server instances to Kubernetes pods. The pods are managed by a StatefulSet controller.  Like ReplicaSets and Deployments, StatefulSets are a type of replication controller that can be scaled by simply increasing or decreasing the desired replica count field.  A policy and scaling action is configured for the WebLogic Server cluster. While the WebLogic Server cluster is running, WLDF collects and monitors various runtime metrics, such as the OpenSessionsCurrentCount attribute of the WebAppComponentRuntimeMBean.  When the conditions defined in the policy occur, the policy is triggered, which causes the corresponding scaling action to be executed. For a WebLogic Server cluster running in a Kubernetes environment, the scaling action is to scale the corresponding StatefulSet by setting the desired replica count field.  In turn, this causes the StatefulSet controller to increase or decrease the number of pods (that is, the WebLogic Server Managed Server instances) to match the desired replica count.

 

Because StatefulSets are managing the pods in which the Managed Server instances are running, a WebLogic Server cluster can also be scaled on-demand by using tools such as kubectl:

 

For example:

$ kubectl scale statefulset ms --replicas=3

 

WLDF Policies and Actions

 

For information about configuring the WLDF Policies and Actions component, see Configuring Policies and Actions.  For this sample, the policy and action is configured in a WLDF diagnostic system module, whose corresponding resource descriptor file, Module-0-3905.xml, is shown below:

 

 

<?xml version='1.0' encoding='UTF-8'?>

<wldf-resource xmlns="http://xmlns.oracle.com/weblogic/weblogic-diagnostics" xmlns:sec="http://xmlns.oracle.com/weblogic/security" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:wls="http://xmlns.oracle.com/weblogic/security/wls" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-diagnostics http://xmlns.oracle.com/weblogic/weblogic-diagnostics/2.0/weblogic-diagnostics.xsd">

  <name>Module-0</name>

  <watch-notification>

    <watch>

      <name>myScaleUpPolicy</name>

      <enabled>true</enabled>

      <rule-type>Harvester</rule-type>

      <rule-expression>wls:ClusterGenericMetricRule("DockerCluster",

"com.bea:Type=WebAppComponentRuntime,ApplicationRuntime=OpenSessionApp,*",

"OpenSessionsCurrentCount","&gt;=",0.01,5,"1 seconds","10 seconds")

      </rule-expression>

      <expression-language>EL</expression-language>

      <alarm-type>AutomaticReset</alarm-type>

      <schedule>

        <minute>*</minute>

        <second>*/15</second>

      </schedule>

      <alarm-reset-period>60000</alarm-reset-period>

      <notification>RestScaleUpAction</notification>

    </watch>

    <rest-notification>

      <name>RestScaleUpAction</name>

      <enabled>true</enabled>

      <timeout>0</timeout>

      <endpoint-url>http://${OPERATOR_ENDPOINT}/hooks/scale-up</endpoint-url>

      <rest-invocation-method-type>PUT</rest-invocation-method-type>

      <accepted-response-type>application/json</accepted-response-type>

      <http-authentication-mode>None</http-authentication-mode>

      <custom-notification-properties></custom-notification-properties>

    </rest-notification>

  </watch-notification>

</wldf-resource>

 

The base element for defining policies and actions is <watch-notification>. Policies are defined in <watch> elements. Actions are defined in elements whose names correspond to the action type. For example, the element for a REST action is <rest-notification>.

 

Here are descriptions of key configuration details regarding the policies and actions that are specified in the preceding resource descriptor file.  For information about all the available action types, see Configuring Actions.

 

Policies:

 

·      The sample demo includes a policy named myScaleUpPolicy, which has the policy expression shown below as it would appear in the WebLogic Server Administration Console:

 

 

 

·      The policy expression for myScaleUpPolicy uses the smart rule, ClusterGenericMetricRule. The configuration of this smart rule can be read as:

 

For the cluster DockerCluster, WLDF will monitor the OpenSessionsCurrentCount attribute of the WebAppComponentRuntimeMBean for the OpenSessionApp application.  If the OpenSessionsCurrentCount is greater than or equal to 0.01 for 5 per cent of the Managed Server instances in the cluster, then the policy will be evaluated as true. Metrics will be collected at a sampling rate of 1 second, and the sample data will be averaged out over the specified 10 second period of time of the retention window.

 

For more information about smart rules, see Smart Rule Reference.

 

Actions:

 

An action is an operation that is executed when a policy expression evaluates to true.  In a traditional WebLogic Server deployment, scaling actions (scale up and scale down) are associated with policies for scaling a dynamic cluster.  Elastic actions scale Managed Server instances in a dynamic cluster by interacting with Node Managers.

 

WLDF also supports many other types of diagnostic actions:

 

·       Java Management Extensions (JMX)

·       Java Message Service (JMS)

·       Simple Network Management Protocol (SNMP)

·       Simple Mail Transfer Protocol (SMTP)

·       Diagnostic image capture

·       REST

·       WebLogic logging system

·       WebLogic Scripting Tool (WLST)

·       Heap dump

·       Thread dump

 

 

For our sample, we use a REST action to show invoking a REST endpoint to initiate a scaling operation.  We selected a REST action, instead of an elastic action, because we are not running Node Manager in the Kubernetes environment, and we’re scaling pods by using the Kubernetes API and API server.  For more information about all the diagnostic actions supported in WLDF, see Configuring Actions.

 

·      The REST action, associated with the policy myScaleUpPolicy from earlier, was configured in the Actions tab of the policy configuration pages in the WebLogic Server Administration Console:

 

 

 

·      The REST endpoint URL in which to send the notification is established by the <endpoint-url> element in the diagnostic system module’s resource descriptor file.

 

By looking at the configuration elements of the REST action, you can see that the REST invocation will send an empty PUT request to the endpoint with no authentication.  If you prefer, you can also send a Basic Authentication REST request by simply setting the <http-authentication-mode> attribute to Basic.

 

Other WLDF resource descriptor configuration settings worth noting are:

 

1.     The file name of a WLDF resource descriptor can be anything you like.  For our sample demo, Module-0-3905.xml was generated when we used the WebLogic Server Administration Console to configure the WLDF policy and REST action.

 

2.     In the demo WebLogic domain, the WLDF diagnostic system module was created using the container-scripts/add-app-to-domain.py script:

 

# Configure WLDF

# ============-=

as_name = sys.argv[3]

 

print('Configuring WLDF system resource');

cd('/')

 

create('Module-0','WLDFSystemResource')

cd('/WLDFSystemResources/Module-0')

set('DescriptorFileName', 'diagnostics/Module-0-3905.xml')

 

cd('/')

assign('WLDFSystemResource', 'Module-0', 'Target', as_name)

 

In the script, you can see that:

 

·      A WLDF diagnostic system module is created named Module-0.

·      The WLDF resource descriptor file, Module-0-3905.xml, is associated with Module-0.

·      The diagnostic system module is targeted to the Administration Server, specified as as_name, which is passed in as a system argument.  This diagnostic system module was targeted to the Administration Server because its policy contains the ClusterGenericMetricRule smart rule, which must be executed from the Administration Server so that it can have visibility across the entire cluster. For more information about smart rules and their targets, see Smart Rule Reference.

Demo Webhook

 

In the sample demo, a webhook is used to receive the REST notification from WLDF and to scale the StatefulSet and, by extension, the WebLogic Server cluster.  The following hook is defined in webhooks/hooks.json:

 

[

  {

    "id": "scale-up",

    "execute-command": "/var/scripts/scaleUpAction.sh",

    "command-working-directory": "/var/scripts",

    "response-message": "scale-up call ok\n"

  }

]

 

This hook named scale-up corresponds to the <endpoint-url> specified in the REST notification:

 

<endpoint-url>http://${OPERATOR_ENDPOINT}/hooks/scale-up</endpoint-url>

 

Notice that the endpoint URL contains the environment variable ${OPERATOR_ENDPOINT}. This environment variable will be replaced with the correct host and port of the webhook when the Administration Server is started.

 

When the hook endpoint is invoked, the command specified by the “execute-command” property is executed, which in this case is the shell script "/var/scripts/scaleUpAction.sh:

 

#!/bin/sh

 

echo "called" >> scaleUpAction.log

 

num_ms=`curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" -X GET https://kubernetes/apis/apps/v1beta1/namespaces/default/statefulsets/${MS_STATEFULSET_NAME}/status | grep -m 1 replicas| sed 's/.*\://; s/,.*$//'`

 

echo "current number of servers is $num_ms" >> scaleUpAction.log

 

new_ms=$(($num_ms + 1))

 

echo "new_ms is $new_ms" >> scaleUpAction.log

 

curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" -X PATCH -H "Content-Type: application/strategic-merge-patch+json" -d '{"spec":{"replicas":'"$new_ms"'}}' https://kubernetes/apis/apps/v1beta1/namespaces/default/statefulsets/${MS_STATEFULSET_NAME}

 

In the script, we are issuing requests to the Kubernetes API server REST endpoints with ‘curl’ and then parsing the JSON response.  The first request is to retrieve the current replica count for the StatefulSet.  Then we scale up the StatefulSet by incrementing the replica count by one and sending a PATCH request with the new value for the replicas property in the request body.

 

Wrap Up

With a simple configuration to use the Policies and Actions component in WLDF, we can provide automatic scaling functionality for a statically-configured WebLogic Server cluster in a Kubernetes environment.  WLDF’s tight integration with WebLogic Server provides a very comprehensive set of WebLogic domain-specific (custom) metrics to be used for scaling decisions.  Although we used a webhook as our REST endpoint to receive WLDF notifications, we could have just as easily implemented another Kubernetes object or service running in the Kubernetes cluster to scale the WebLogic Server cluster in our sample demo. For example, the WebLogic Server team is also investigating the Kubernetes Operator pattern for integrating WebLogic Server in a Kubernetes environment.  A Kubernetes Operator is "an application-specific controller that extends the Kubernetes API to create, configure, and manage instances of complex applications".  For more information on Operators,  see Introducing Operators: Putting Operational Knowledge into Software. Stay tuned for future blog updates on WebLogic Server and its integration with Kubernetes. The next blog related to WebLogic Server Clustering will be in the area of dynamic clusters for WebLogic Server on Kubernetes.

Join the discussion

Comments ( 4 )
  • Senthil Kumar Nandagopalan Thursday, January 11, 2018
    Looking for end to end IQ document to scale up/down weblogic servers using kubernetes
  • Lenny Phan Friday, January 12, 2018
    The WebLogic development team is developing a WebLogic Kubernetes Operator that will be released soon. The WebLogic Kubernetes Operator will manage the the scaling of a WebLogic cluster. Also, we will be publishing a white paper that documents the details.
  • Danny Monday, November 26, 2018
    Thanks Lenny for sharing this auto scaling wl cluster on k8s.

    Is there an update to the whitepaper? as the link to sample demo, https://blogs.oracle.com/weblogicserver/weblogic-on-kubernetes,-try-it -- produces "Page Not Found"
  • Lenny Phan Tuesday, November 27, 2018
    Please take a look at "WebLogic Dynamic Clusters on Kubernetes" at https://blogs.oracle.com/weblogicserver/weblogic-dynamic-clusters-on-kubernetes. This blog illustrates scaling WebLogic Dynamic Clusters with the WebLogic Operator. Documentation can also be found in the github project: https://github.com/oracle/weblogic-kubernetes-operator. More blog write-ups will be coming out soon.
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.