Proactive insights, news and tips from Oracle WebLogic Server Support. Learn Oracle from Oracle.

WebLogic Dynamic Clusters on Kubernetes

Lenny Phan
Principal Member of Technical Staff


A WebLogic Server cluster consists of multiple Managed Server instances running simultaneously and working together to provide increased scalability and reliability.  WebLogic Server supports two types of clustering configurations, configured and dynamic clustering.  Configured clusters are created by manually configuring each individual Managed Server instance.  In dynamic clusters, the Managed Server configurations are generated from a single, shared template.  Using a template greatly simplifies the configuration of clustered Managed Servers and allows for dynamically assigning servers to Machine resources, thereby providing a greater utilization of resources with minimal configuration.  With dynamic clusters, when additional server capacity is needed, new server instances can be added to the cluster without having to manually configure them individually. Also, unlike configured clusters, scaling up of dynamic clusters is not restricted to the set of servers defined for a cluster, but can be increased based on runtime demands.


For more information on how to create, configure, and use dynamic clusters in WebLogic Server, see Dynamic Clusters.


Support for Dynamic Clusters by Oracle WebLogic Server Kubernetes Operator

Previously, the WebLogic Server Kubernetes Operator supported configured clusters only.  That is, the operator could only manage and scale Managed Servers defined for a configured cluster.  Now, this limitation has been removed. By supporting dynamic clusters, the operator can easily scale the number of Managed Server instances based on a server template instead of requiring that you first manually configure them.


Creating a Dynamic Cluster in a WebLogic Domain in Kubernetes


The WebLogic Server team has been actively working to integrate WebLogic Server in Kubernetes, WebLogic Server Certification on Kubernetes.  The Oracle WebLogic Server Kubernetes Operator provides a mechanism for creating and managing any number of WebLogic domains, automates domain startup, allows scaling of WebLogic clusters, manages load balancing for web applications deployed in WebLogic clusters, and provides integration with Elasticsearch, Logstash, and Kibana. The operator is currently available as an open source project at https://oracle.github.io/weblogic-kubernetes-operator.  To create a WebLogic domain, the recommended approach is to use the provided create-weblogic-domain.sh script, which automates the creation of a WebLogic domain within a Kubernetes cluster.  The create-weblogic-domain.sh script takes an input file, create-weblogic-domain-inputs.yaml, which specifies the configuration properties for the WebLogic domain. The following parameters of the input file are used when creating a dynamic cluster:






The name of the WebLogic cluster instance to generate for the domain.



The type of WebLogic cluster. Legal values are "CONFIGURED" or "DYNAMIC".




The number of Managed Server instances to generate for the domain.



The number of Managed Servers to start initially for the domain.



Base string used to generate Managed Server names.  Used as the server name prefix in a server template for dynamic clusters.





The following example configuration will create a dynamic cluster named ‘cluster-1’ with four defined Managed Servers (managed-server1 … managed-server4) in which the operator will initially start up two Managed Servers instances, managed-server1and managed-server2:


# Type of WebLogic Cluster

# Legal values are "CONFIGURED" or "DYNAMIC"

clusterType: DYNAMIC


# Cluster name

clusterName: cluster-1


# Number of Managed Servers to generate for the domain

configuredManagedServerCount: 4


# Number of Managed Servers to initially start for the domain

initialManagedServerReplicas: 2


    # Base string used to generate Managed Server names

managedServerNameBase: managed-server


To create the WebLogic domain, you simply run the create-weblogic-domain.sh script specifying your input file and output directory for any generated configuration files:


#> create-weblogic-domain.sh  –i create-domain-job-inputs.yaml  -o /path/to/weblogic-domain-output-directory


There are some limitations when creating WebLogic clusters using the create domain script:


  • The script creates the specified number of Managed Server instances and places them all in one cluster.
  • The script always creates one cluster.

Alternatively, you can create a WebLogic domain manually as outlined in Manually Creating a WebLogic Domain.


How WebLogic Kubernetes Operator Manages a Dynamic Cluster


A Kubernetes Operator is “an application-specific controller that extends the Kubernetes API to create, configure, and manage instances of complex applications.”  For more information on operators, see Introducing Operators: Putting Operational Knowledge into Software. The Oracle WebLogic Server Kubernetes Operator extends Kubernetes to create, configure, and manage any number of WebLogic domains running in a Kubernetes environment.  It provides a mechanism to create domains, automate domain startup, and allow scaling of both configured and dynamic WebLogic clusters. For more details about the WebLogic Server Kubernetes Operator, see the blog, Announcing the Oracle WebLogic Server Kubernetes Operator.


Because the WebLogic Kubernetes Operator manages the life cycle of Managed Servers in a Kubernetes cluster, it provides the ability to start up and scale (up or down) WebLogic dynamic clusters. The operator manages the startup of a WebLogic domain based on the settings defined in a Custom Resource Domain (CRD).


The number of WLS pods/Managed Server instances running in a Kubernetes cluster, for a dynamic cluster, is represented by the ‘replicas’ attribute value of the ClusterStartup entry in the following domain custom resource YAML file:



  - desiredState: "RUNNING"

    clusterName: "cluster-1"

    replicas: 2


    - name: JAVA_OPTIONS

      value: "-Dweblogic.StdoutDebugEnabled=false"

    - name: USER_MEM_ARGS

      value: "-Xms64m -Xmx256m"


For the above example entry, during WebLogic domain startup, the operator would start two pod/Managed Server instances for the dynamic cluster ‘cluster-1’. Details of a domain custom resource YAML file can be found in Starting a WebLogic Domain.


Scaling of WebLogic Dynamic Clusters on Kubernetes


There are several ways to initiate scaling through the operator, including:


  • On-demand, updating the Custom Resource Domain specification directly (using kubectl).
  • Calling the operator's REST scale API, for example, from curl.
  • Using a WLDF policy rule and script action to call the operator's REST scale API.
  • Using a Prometheus alert action to call the Operator's REST scale API.

On-Demand, Updating the Custom Resource Domain Directly

Scaling a dynamic cluster can be achieved by editing the Custom Resource Domain directly by using the ‘kubectl edit’ command and modifying the ‘replicas’ attribute value:


#> kubectl edit domain domain1 -n [namespace]


This command will open an editor which will allow you to edit the defined Custom Resource Domain specification.  Once committed, the operator will be notified of the change and will immediately attempt to scale the corresponding dynamic cluster by reconciling the number of running pods/Managed Server instances with the ‘replicas’ value specification.


Calling the Operator's REST Scale API

Alternatively, the WebLogic Server Kubernetes Operator exposes a REST endpoint, with the following URL format, that allows an authorized actor to request scaling of a WebLogic cluster:




  • <version> denotes the version of the REST resource.
  • <domainUID> is the unique ID that will be used to identify this particular domain. This ID must be unique across all domain in a Kubernetes cluster.
  • <clusterName> is the name of the WebLogic cluster instance to be scaled.


For example:



The /scale REST endpoint:

  • Accepts an HTTP POST request.
  • The request body supports the JSON "application/json" media type.
  • The request body will be a simple name-value item named managedServerCount:


      ”managedServerCount": 3



The managedServerCount value designates the number of WebLogic Server instances to scale to.


Note: An example use of the REST API, using the curl command, can be found in scalingAction.sh.


Using a WLDF Policy Rule and Script Action to Call the Operator's REST Scale API

A WebLogic Server dynamic cluster can be automatically scaled by increasing (or decreasing) the number of pods based on resource metrics provided by the WebLogic Diagnostic Framework (WLDF). WLDF is a suite of services and APIs that collect and surface metrics that provide visibility into server and application performance. WLDF provides a Policies and Actions component to support the automatic scaling of dynamic clusters.  There are two types of scaling supported by WLDF:


  • Calendar-based scaling — Scaling operations on a dynamic cluster that are executed on a particular date and time.
  • Policy-based scaling — Scaling operations on a dynamic cluster that are executed in response to changes in demand.

In this blog, we will focus on policy-based scaling which lets you write policy expressions for automatically executing configured actions when the policy expression rule is satisfied. These policies monitor one or more types of WebLogic Server metrics, such as memory, idle threads, and CPU load. When the configured threshold in a policy is met, the policy is triggered, and the corresponding scaling action is executed.


Example Policy Expression Rule


The following is an example policy expression rule that was used in Automatic Scaling of WebLogic Clusters on Kubernetes:



ApplicationRuntime=OpenSessionApp,*","OpenSessionsCurrentCount","&gt;=",0.01,5,"1 seconds","10 seconds"


This ‘ClusterGenericMetricRule’ smart rule is used to observe trends in JMX metrics that are published through the Server Runtime MBean Server and can be read as:


For the cluster, ‘cluster-1’, WLDF will monitor the OpenSessionsCurrentCount attribute of the WebAppComponentRuntime MBean for the OpenSessionApp application.  If the OpenSessionsCurrentCount is greater than or equal to 0.01 for 5% of the servers in the cluster, then the policy will be evaluated as true. Metrics will be collected at a sampling rate of 1 second and the sample data will be averaged out over the specified 10 second period of time of the retention window.


You can use any of the following tools to configure policies for diagnostic system modules:


  • WebLogic Server Administration Console
  • WLST
  • REST
  • JMX application


Below is an example configuration of a policy, named ‘myScaleUpPolicy’, shown as it would appear in the WebLogic Server Administration Console:




Example Action


An action is an operation that is executed when a policy expression rule evaluates to true. WLDF supports the following types of diagnostic actions:

  • Java Management Extensions (JMX)
  • Java Message Service (JMS)
  • Simple Network Management Protocol (SNMP)
  • Simple Mail Transfer Protocol (SMTP)
  • Diagnostic image capture
  • Elasticity framework
  • REST
  • WebLogic logging system
  • Script


The WebLogic Server team has an example shell script, scalingAction.sh, for use as a Script Action, which illustrates how to issue a request to the operator’s REST endpoint.  Below is an example screen shot of the Script Action configuration page from the WebLogic Server Administration Console:




Important notes about the configuration properties for the Script Action:


  • Working Directory and Path to Script configuration entries specify the volume mount path (/shared) to access the WebLogic domain home.
  • The scalingAction.sh script requires access to the SSL certificate of the operator’s endpoint and this is provided through the environment variable ‘INTERNAL_OPERATOR_CERT’.  The operator’s SSL certificate can be found in the ‘internalOperatorCert’ entry of the operator’s ConfigMap weblogic-operator-cm:

For example:


#> kubectl describe configmap weblogic-operator-cm -n weblogic-operator


Name:         weblogic-operator-cm

Namespace:    weblogic-operator

Labels:       weblogic.operatorName=weblogic-operator

Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","data":{"externalOperatorCert":"","internalOperatorCert":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t...








  • The scalingAction.sh script accepts a number of customizable parameters:

       action - scaleUp or scaleDown (Required)

       domain_uid - WebLogic domain unique identifier (Required)

       cluster_name - WebLogic cluster name (Required)

       kubernetes_master - Kubernetes master URL, default=https://kubernetes

       access_token - Service Account Bearer token for authentication and authorization for access to REST Resources

       wls_domain_namespace - Kubernetes namespace in which the WebLogic domain is defined, default=default

       operator_service_name - WebLogic Operator Service name of the REST endpoint, default=internal-weblogic-operator-service

       operator_service_account - Kubernetes Service Account name for the WebLogic Operator, default=weblogic-operator

       operator_namespace – Namespace in which the WebLogic Operator is deployed, default=weblogic-operator

       scaling_size – Incremental number of WebLogic Server instances by which to scale up or down, default=1


For more information about WLDF and diagnostic policies and actions, see Configuring Policies and Actions in Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

Note: A more detailed description of automatic scaling using WLDF can be found in WebLogic on Kubernetes, Try It! and Automatic Scaling of WebLogic Clusters on Kubernetes.


There are a few key differences between the automatic scaling of WebLogic clusters described in this blog and my previous blog, Automatic Scaling of WebLogic Clusters on Kubernetes:


  • In the previous blog, as in the earlier release, only scaling of configured clusters was supported.
  • In this blog:

    • To scale the dynamic cluster, we use the WebLogic Server Kubernetes Operator instead of using a Webhook.
    • To scale the dynamic cluster, we use a Script Action, instead of a REST action.
    • To scale pods, scaling actions invoke requests to the operator’s REST endpoint, instead of the Kubernetes API server.

Using a Prometheus Alert Action to Call the Operator's REST Scale API


In addition to using the WebLogic Diagnostic Framework, for automatic scaling of a dynamic cluster, you can use a third party monitoring application like Prometheus.  Please read the following blog for details about Using Prometheus to Automatically Scale WebLogic Clusters on Kubernetes.


What Does the Operator Do in Response to a REST Scaling Request?


When the WebLogic Server Kubernetes Operator receives a scaling request through its scale REST endpoint, it performs the following actions:

  • Performs an authentication and authorization check to verify that the specified user is allowed to perform the specified operation on the specified resource.
  • Validates that the specified domain, identified by the domainUID, exists. The domainUID is the unique ID that will be used to identify this particular domain. This ID must be unique across all domains in a Kubernetes cluster.
  • Validates that the WebLogic cluster, identified by the clusterName, exists. The clusterName is the name of the WebLogic cluster instance to be scaled.
  • Verifies that the scaling request’s ‘managedServerCount’ value does not exceed the configured maximum cluster size for the specified WebLogic cluster.  For dynamic clusters, ‘MaxDynamicClusterSize’ is a WebLogic attribute that specifies the maximum number of running Managed Server instances allowed for scale up operations.  See Configuring Dynamic Clusters for more information on attributes used to configure dynamic clusters.
  • Initiates scaling by setting the ‘Replicas’ property within the corresponding domain custom resource, which can be done in either:


  • A clusterStartup entry, if defined for the specified WebLogic cluster.

For example:




  Cluster Startup:

    Cluster Name:   cluster-1

    Desired State:  RUNNING


      Name:     JAVA_OPTIONS

      Value:    -Dweblogic.StdoutDebugEnabled=false

      Name:     USER_MEM_ARGS

      Value:    -Xms64m -Xmx256m

    Replicas:   2



  • At the domain level, if a clusterStartup entry is not defined for the specified WebLogic cluster and the startupControl property is set to AUTO

For example:



    Domain Name:  base_domain

    Domain UID:   domain1

    Export T 3 Channels:

    Image:              store/oracle/weblogic:

    Image Pull Policy:  IfNotPresent

    Replicas:           2

    Server Startup:

      Desired State:  RUNNING


        Name:         JAVA_OPTIONS

        Value:        -Dweblogic.StdoutDebugEnabled=false

        Name:         USER_MEM_ARGS

        Value:        -Xms64m -Xmx256m

      Server Name:    admin-server

    Startup Control:  AUTO


Note: You can view the full WebLogic Kubernetes domain resource with the following command:

#> kubectl describe domain <domain resource name>

  • In response to a change to the ‘Replicas’ property in the Custom Resource Domain, the operator will increase or decrease the number of pods (Managed Servers) to match the desired replica count.

Wrap Up

The WebLogic Server team has developed an Oracle WebLogic Server Kubernetes Operator, based on the Kubernetes Operator pattern, for integrating WebLogic Server in a Kubernetes environment.  The operator is used to manage the life cycle of a WebLogic domain and, more specifically, to scale a dynamic cluster.  Scaling a WebLogic dynamic cluster can be done, either on-demand or automatically, using either the WebLogic Diagnostic Framework or third party monitoring applications, such as Prometheus.  In summary, the advantages of using WebLogic dynamic clusters over configured clusters in a Kubernetes cluster are:


  • Managed Server configuration is based on a single server template.
  • When additional server capacity is needed, new server instances can be added to the cluster without having to manually configure them individually.
  • Unlike configured clusters, scaling up of dynamic clusters is not restricted to the set of servers defined in the cluster but can be increased based on runtime demands.

I hope you’ll take the time to download and take the Oracle WebLogic Server Kubernetes Operator for a spin and experiment with the automatic scaling feature for dynamic clusters. Stay tuned for more blogs on future features that are being added to enhance the Oracle WebLogic Server Kubernetes Operator.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.