X

Proactive insights, news and tips from Oracle WebLogic Server Support. Learn Oracle from Oracle.

Portable WebLogic Domains Using Kubernetes Operator Configuration Overrides

Introduction

A WebLogic domain often needs to be customized when porting it between environments. For example, a domain usually needs to reference a different database in a QA environment versus a production environment. 

This blog discusses leveraging Oracle WebLogic Kubernetes Operator 2.2 to fully encapsulate a domain, and then using its custom overrides feature to inject modifications. The blog concludes with an end-to-end demonstration which alters an existing Operator sample so that it leverages custom overrides.

 

References

This blog uses version 2.2.1 of the WebLogic Kubernetes Operator and version 2.2.0 of its documentation. 

 

Overview

The Oracle WebLogic Kubernetes Operator supports running arbitrary WebLogic domains in arbitrary Kubernetes clusters, and is a convenient framework for deploying the same WebLogic domain to both cloud and on-premises environments. The Operator lets you package together a WebLogic install, a Java install, domain configuration, domain networking, and Java EE applications. This package is simply a set of Docker images and Kubernetes Resource(s) files -- let's call it a ‘bundle’.

After you’ve packaged up a bundle, it's often desirable to slightly customize its WebLogic configuration XML and system module XML files so you can port it between Kubernetes environments. For example, you may want to change your bundle’s database system module configuration for its user name, password, URL so that your WebLogic domain correctly wires into a particular environment’s database. 

The Operator provides a configuration overrides feature to help with this bundle customization. You can use overrides to customize bundles as they're moved from development, to testing, and finally on to production. You can also use overrides to customize bundles when they’re deployed to different sites or are deployed to multiple locations within the same site. Overrides alter the original WebLogic domain’s configuration found in its config.xml and system resource files, and are specified via XML templates and Kubernetes Secrets which are deployed in combination with a bundle. 

Tip: Before you start.

This blog assumes that you’re already familiar with Kubernetes and Oracle WebLogic Kubernetes Operator concepts, including the Kubernetes concepts of ‘pods’, ‘services’, and ‘secrets’, as well as the Operator concepts of 'domain custom resource', ‘domain UID’, and 'domain in image'.

This blog also uses a modified version of the Operator project's 'Quick Start' sample to demo the Operator custom override feature on a JDBC data source that references a MySQL database. If you haven't already run the Quick Start sample, I recommend doing so before trying to run the sample in this blog (see References).

 

What may I override and when?

Custom overrides can externally alter almost any WebLogic configuration but there are some exceptions and limitations. Here are the highlights: 

Typical, supported, and unsupported overrides:

Overrides are typically used to override the configuration of credentials, URLs, tuning parameters, and external DNS addresses. These may include values in a data source, remote JMS references, and WebLogic Server tuning. 

Custom overrides do not support modifying, adding, or deleting 'internal' listen addresses, listen ports, or log locations, and must not be used to alter existing MBean names (including server names, etc). The behavior when attempting to use unsupported overrides is undefined.

Note that while overriding listen addresses and ports isn't supported, it's important to understand that overriding public or external listen addresses and ports is acceptable, and often desirable. For example, you may want to override a WebLogic Server network-access-point external listen address to enable an external WLST client to access the WebLogic Server inside a remote Kubernetes cluster. (The public address configured on a network-access-point needs to match the address an RMI client uses in its JNDI Context URL to access the cluster - not the internal address the WebLogic Server actually happens to be listening on locally.)

Overrides can’t create new modules – but don’t panic - there’s an easy workaround:

The Operator doesn’t provide a way to add new resource modules (such as Data Sources or JMS Modules) via overrides; instead, you can create 'skeleton' modules in your original configuration and then use overrides to alter the existing modules. 

For example, you can configure a skeleton Data Source with a min-pool-size of 0 and its driver set to valid values, but its other values invalid - such as user name and password. This allows the Data Source to start without errors at runtime even if no overrides are specified, but ensures the Data Source cannot be accidentally used at runtime without overrides that are correct for the current environment. 

There’s a complete sample that uses this technique later in this blog.

A Domain must be shut down and restarted before override updates will take effect:

It's required to shut down any running servers in a domain before any custom overrides changes can take effect – custom overrides are not suitable for propagating changes to a running cluster.

For a complete discussion of supported and unsupported overrides, see Configuration Overrides in References.

 

A Brief Side Trip – ‘Internal Overrides’ and ‘Service Ports’

I mentioned above that custom overrides must not override listen addresses and log locations. There’s a very good reason for this: the operator is already performing some pre-defined overrides for you and its important not to interfere with these pre-defined overrides.

Specifically, the Operator automatically generates pre-defined ‘internal overrides’ for listen address and log location fields to correctly wire networking and logging into the Kubernetes environment, and a custom override of the same fields would interfere with this automation. For example, a WebLogic Server named admin-server with an Operator domain-UID domain1 will run in a Kubernetes pod with a service address domain1-admin-server , so the Operator generates internal overrides for its config.xml listen addresses so that they're also domain1-admin-server .  

I also mentioned that custom overrides must not override ports – this is because the operator reads the configuration and generates Kubernetes services for ports before any overrides can be applied.

 

How Do Custom Overrides Work?

Operator configuration overrides internally leverage a relatively new WebLogic feature that’s also called ‘configuration overrides’ or is sometimes called 'situational configuration'. This feature was introduced in WebLogic Server 12.2.1. It allows you to specify configuration override files which a WebLogic Server automatically detects and loads, and which override WebLogic configuration at runtime.

To specify custom overrides using the WebLogic Kubernetes Operator, you put situational configuration templates in a Kubernetes config map and reference Kubernetes Secrets via macros within your templates, and finally you make sure that your Domain Resource references your override config map and secret(s).  

The following diagram walks through the override flow at a high level:

The following file is an example of a configuration override template that overrides a datasource, and that has macros that reference override secrets:


<?xml version="1.0" encoding="UTF-8"?>
<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source"  xmlns:f="http://xmlns.oracle.com/weblogic/jdbc-data-source-fragment"  xmlns:s="http://xmlns.oracle.com/weblogic/situational-config">
  <name>mysqlDS</name>
    <jdbc-driver-params>
      <url f:combine-mode="replace">${secret:sample-domain1-mysql-secret.url}</url>
      <properties>
         <property>
            <name>user</name>
            <value f:combine-mode="replace">${secret:sample-domain1-mysql-secret.root-user}</value>
         </property>
      </properties>
      <password-encrypted f:combine-mode="add">${secret:sample-domain1-mysql-secret.root-password:encrypt}</password-encrypted>
    </jdbc-driver-params>
</jdbc-data-source>

We'll discuss the above template in more detail in the demo that's at the end of this blog.  Now let's walk through the override flow in a little more detail:

  1. Create one or more XML override templates to override WebLogic configuration. These templates can reference macros that in turn reference Kubernetes secrets or environment variables. Using macros lets you re-use the same override file for different Kubernetes environments. Put your templates in a Kubernetes configmap along with a ‘version.txt’ file that contains exactly the string ‘2.0’.  This is demonstrated in 'Step 13' of the demo at the end of this blog.
  1. Alter your Domain Custom Resource to specify the name of your Kubernetes configmap from step (1), and to list the secrets referenced by your override template.  This is demonstrated in 'Step 12' of the demo at the end of this blog.
  1. Shut down your domain if it’s already running. One way to do this is to call ‘kubectl -n MYNAMESPACE delete Domain MYDOMAIN’ .  For more information, see Domain Lifecycle in References.
  1. Deploy your Kubernetes secrets for any of your override macros to the same namespace as you plan to use for your domain. You can use different secrets for different environments.
  1. Deploy your overrides Kubernetes config map with the same name from step (1) to the same namespace you plan to use for your domain. 
  1. Deploy your Domain Custom Resource from step (2).
  1. The Operator pod will automatically detect the deployment of the Domain Custom Resource and run an 'introspection job' (a Kubernetes job) prior to starting the domain's WebLogic pods. This introspection job runs in the same namespace as your WebLogic domain, and is named DOMAIN_UID-introspect-domain-job . The introspection job loads the WebLogic domain home configuration plus combines your overrides templates and macros into override XML files.
  1. The Operator pod automatically takes the files generated by the job and makes them available to WebLogic Server pods via a Kubernetes configmap named DOMAIN_UID-weblogic-domain-introspect-cm.
  1. Once introspection completes successfully, the Operator starts your WebLogic Server pods.  Each pod will then copy the situational configuration supplied by the DOMAIN_UID-weblogic-domain-introspect-cm configmap to the pod's  internal DOMAIN_HOME/optconfig directory, and the pod’s WebLogic Server will automatically load the situational configuration from this location.
  1. Wait for your WebLogic Server pods to start and verify your overrides took effect. 

Tip:  Always check that your overrides are taking effect even if your WebLogic Server pods reach s running state.

After the Operator tries to start your WebLogic Servers, we reach step (10) above, which is you, the developer or administrator, verifying that your overrides took effect.

This is a very important testing step especially when you're developing new overrides or otherwise altering your WebLogic domain. This testing must never be skipped before deploying your domain in production, since your WebLogic Server pods may still successfully reach a running state even though there's been a mistake while specifying your overrides.

 

Viewing and Verifying Overrides at Runtime

It’s essential to verify overrides are taking effect once you’ve deployed your domain:

  • Check serverConfig MBean trees:  Because Operator overrides leverage WebLogic situational configuration, overrides can be checked by examining WebLogic's online 'serverConfig' MBean tree. The serverConfig MBean tree is a dynamically generated MBean tree that's only available on running WebLogic Servers.
  • Check runtime behavior:  It’s always a good idea to verify that your overridden fields are honored at runtime.  For example, if you’re overriding Data Source fields, you can check that your Data Source is working by using its testPool() runtime MBean method.
  • Check server pod logs:  Finally, you can search your WebLogic pod logs for the keyword ’situation’ via ‘grep -i situation’ to both (a) ensure your overrides file was loaded, and (b) ensure the WebLogic Server isn’t reporting any errors.

The sample below demonstrates all three of these checks in Step 15.

Tip:  You cannot use the console to view overrides.

You can view overrides in the serverConfig tree using online JMX, REST, or WLST scripting, but not on the WebLogic console. The console only reflects the static 'domainConfig' MBean tree, which is your configuration before any overrides have been applied.

Tip:  The Operator documentation provides a step-by-step discussion for debugging overrides.

For a step-by-step discussion of viewing and verifying overrides, consult the Debugging section of the configuration overrides documentation (https://oracle.github.io/weblogic-kubernetes-operator/2.2.0/userguide/managing-domains/configoverrides/).

 

Putting it all together: An End-To-End Sample

Overview

The following sample builds on the Kubernetes Operator ‘quick start’ sample. It additionally starts a MySQL database, adds a data source to its WebLogic configuration, deploys configuration overrides for this resource, and finally verifies that the overrides took effect.

Before we start, let’s review the unaltered quick start sample’s steps:

  • Initial Cleanup
    • Delete any running Kubernetes resources that might interfere with the sample.
  • Prerequisites
    • Ensure you have Kubernetes, Helm, and Tiller installed.
    • Download (git clone) the Operator project.
  • Get Images
    • Obtain Operator, Traefik, and base WebLogic Docker images.
  • Install the operator and load balancer
    • Deploy the Operator and Traefik to Kubernetes.
  • Prepare for a domain
    • Create a namespace for a WebLogic domain.
    • Tell Traefik and Operator to monitor this namespace.
  • Create a domain
    • Create a WebLogic domain-home-in-image using offline WLST.
    • Create a Domain Custom Resource YAML file that references the image and the domain home location within the image (among other settings). 
    • Deploy the Domain Custom Resource YAML file.  
    • Check that the operator detected the resource and launched the domain’s pods.
  • Final Cleanup
    • Remove the domain, the operator, Traefik load balancer, etc.

Now, to demonstrate custom configuration overrides, we’ll alter the quick start sample to additionally:

  • Ensure it uses a 2.2. version of the Operator.
  • Set up a WebLogic JDBC Data Source targeted to the domain's admin server that is deliberately configured with an invalid URL, user name, and password.
  • Deploy a MySQL database in a Kubernetes pod, plus deploy a Kubernetes secret that contains the database's URL, user name, and password.
  • Deploy a custom overrides config map with overrides for the Data Source’s URL, user name, and password. These overrides embed macros that reference the Kubernetes secret.
  • Modify its Domain Custom Resource to reference the secret and configmap.
  • Verify that the WebLogic Server has loaded the overridden configuration and can access the MySQL database.

Got it? OK. Here we go!

 

Step 1 – Tidy Up!

First clean up all WebLogic, Traefik, and Operator related Kubernetes resources that are already running in your Kubernetes cluster. This will prevent the sample from conflicting with any WebLogic Kubernetes resources from any previous experimentation.  For example, if you recently ran the quick-start sample, you can clean up its resources by following the quick sample’s final ‘Cleanup’ step (see References).

Tip: The Operator project includes a convenient 'one-step' brute force cleanup script.

If you have a dedicated Kubernetes cluster that is only intended to run the sample, then a quick way to for developers to clean up is to run the Operator project’s integration‑test ./src/integration-tests/bash/cleanup.sh script. This script aggressively deletes all Kubernetes resources that have names or labels associated with the Operator project’s samples and tests – it checks every namespace.

Be careful though! The script is very aggressive so don't use it unless you're certain that your Kubernetes cluster is solely intended for your personal use. 

Note that the script will attempt to delete directories generated on PVs or by the Operator integration test framework -- you can ignore any failures it reports if these attempts fail because the quick start sample doesn’t use these directories and PVs.

Sometimes I've need to run this script twice in a row to ensure everything is cleaned up.

Don’t have the script yet? You can defer the 'Tidy Up!' step until after steps 4 & 5 (which download Images and Operator source respectively).

 

Step 2 – Check Prerequisites

The sample assumes you’ve installed Kubernetes, Docker, Tiller, and Helm.  So first check the ‘Prerequisites’ section of the User Guide Introduction (https://oracle.github.io/weblogic-kubernetes-operator/2.2.0/userguide/introduction/introduction/) to verify that you’re using supported versions.   

You can ignore the following prerequisites for the moment as we’ll download images and cover them in a future step:

  • WebLogic image version and patches
  • Operator image
  • `You must have the cluster-admin role to install the operator.`

If you need help installing, check out our Cheat Sheet (https://oracle.github.io/weblogic-kubernetes-operator/2.2.0/userguide/overview/k8s-setup/#cheat-sheet-for-setting-up-kubernetes) and also see Install Helm and Tiller (https://oracle.github.io/weblogic-kubernetes-operator/2.2.0/userguide/managing-operators/#install-helm-and-tiller).

 

Step 3 – Create an Empty Working Directory “WORKINGDIR”.

Let’s start with an empty directory and set the WORKINGDIR environment variable to its location.  


 mkdir sample 

 cd sample

 export WORKINGDIR=`pwd`

 

Step 4 – Download the Operator Project source into “WORKINGDIR” and set “SAMPLEDIR”

Here we obtain the project source and set the SAMPLEDIR environment variable to reference the quick start sample located within the project.


 cd $WORKINGDIR

 git clone --branch release/2.2.1 --single-branch \
    https://github.com/oracle/weblogic-kubernetes-operator

 export SAMPLEDIR=$WORKINGDIR/weblogic-kubernetes-operator/kubernetes/samples/scripts/create-weblogic-domain/domain-home-in-image

Note that we’re assuming Operator version 2.2.1 in this blog, so we’re using a 2.2.1 version of its source and sample.

 

Step 5 – Get Your Operator, WebLogic, Traefik, and MySQL Images plus Check WebLogic Image Version

(5.1) Follow the steps in Quick Start - Get Images(https://oracle.github.io/weblogic-kubernetes-operator/2.2.0/quickstart/get-images/) to get your Operator 2.2.1 image, Traefik image, and WebLogic Server 12.2.1.3 image.  

NOTE: Substitute 2.2.1 for 2.2.0 when you pull the Operator image (e.g.: docker pull oracle/weblogic-kubernetes-operator:2.2.1).

(5.2) In addition, get a mysql:5.6 image.

(5.3) After downloading the images, make sure that your WebLogic Server 12.2.1.3 image includes one-off patch 29135930:


 $ docker run store/oracle/weblogic:12.2.1.3 sh -c '$ORACLE_HOME/OPatch/opatch lspatches'

 29135930;One-off
 27117282;One-off
 26355633;One-off
 26287183;One-off
 26261906;One-off
 26051289;One-off

(5.4) Make sure your Kubernetes cluster has access to all four images.  If it doesn’t, you will need to push them to a Docker repository that’s accessible from your Kubernetes cluster.

 

Step 6 – Grant Helm cluster-admin role, Deploy Operator, Deploy Load Balancer, Deploy Namespace for Domain

6.1) Grant Helm cluster-admin role, Deploy Operator, Deploy Load Balancer

Change your directory to your source tree (cd $WORKINGDIR/weblogic-kubernetes-operator), and follow all of the steps in Quick Start - Install the Operator and Load Balancer (https://oracle.github.io/weblogic-kubernetes-operator/2.2.0/quickstart/install/).

NOTE: Substitute 2.2.1 for 2.2.0 when you use helm to deploy the Operator image (e.g.: change parameter --set image=oracle/weblogic-kubernetes-operator:2.2.0 to --set image=oracle/weblogic-kubernetes-operator:2.2.1).

 

6.2) Create a namespace and let the Operator and Load Balancer know about it

It’s time to create a Kubernetes namespace that you will use to run the WebLogic domain, and then register the namespace with the Operator and Load Balancer.  

Follow all of the steps in Quick Start - Prepare for a Domain (https://oracle.github.io/weblogic-kubernetes-operator/2.2.0/quickstart/prepare/).

 

Step 7 – Deploy MySQL and verify it works.

Let’s deploy a sample MySQL database pod, a Kubernetes service to make the database’s port available to other pods, and a Kubernetes secret that contains the database’s URL, user name, and password.

First, create a file called mysql.yaml in $WORKINGDIR :


cd $WORKINGDIR

cat << EOF > mysql.yaml
apiVersion: v1
kind: Secret
metadata:
  name: sample-domain1-mysql-secret
  labels:
    weblogic.domainUID: sample-domain1
  namespace: sample-domain1-ns
data:
  # echo -n "root" | base64
  root-user: cm9vdA==
  # echo -n "password" | base64 
  root-password: cGFzc3dvcmQ=
  # echo -n "jdbc:mysql://sample-domain1-mysql:3306/mysql" | base64
  url: amRiYzpteXNxbDovL3NhbXBsZS1kb21haW4xLW15c3FsOjMzMDYvbXlzcWw=
---
apiVersion: v1
kind: Pod
metadata:
  name: sample-domain1-mysql
  labels:
    weblogic.domainUID: sample-domain1
    app: sample-domain1-mysql
  namespace: sample-domain1-ns
spec:
  terminationGracePeriodSeconds: 5
  containers:
  - image: mysql:5.6
    name: mysql
    env:
    - name: MYSQL_ROOT_PASSWORD
      valueFrom:
        secretKeyRef:
          name: sample-domain1-mysql-secret
          key: root-password
    ports:
    - containerPort: 3306
      name: mysql
---
apiVersion: v1
kind: Service
metadata:
  name: sample-domain1-mysql
  labels:
    weblogic.domainUID: sample-domain1
  namespace: sample-domain1-ns
spec:
  ports:
  - port: 3306
  selector:
    app: sample-domain1-mysql
  clusterIP: None
EOF

Then deploy the MySQL pod, service, and secret to Kubernetes by deploying the mysql.yaml resource file:


 cd $WORKINGDIR
 
 kubectl create -f mysql.yaml

Finally, verify the database is working by running some MySQL commands inside the database pod.  It can help to wait a few seconds before verifying: 



cd $WORKINGDIR

cat << EOF > mysql.ddl

SELECT 'Start of sample' AS '';
SELECT VERSION(), CURRENT_DATE;
CREATE TABLE IF NOT EXISTS mynames (name VARCHAR(255) NOT NULL);
INSERT INTO mynames VALUES ('foo');
INSERT INTO mynames VALUES ('bar');
SELECT NAME FROM mynames;
DROP TABLE IF EXISTS mynames;
SELECT 'End of sample' AS '';

EOF

# 
# run mysql.ddl to verify mysql pod is working
#
# notes:
#   The ‘mysql’ at the end of the following command is the name of the db
#   (mysql is the default DB if none specified when creating the database pod).
#
#   The ‘—table’ argument causes table output format to be decorated
#

kubectl -n sample-domain1-ns \
        exec -it sample-domain1-mysql \
        -- mysql -h localhost -u root -ppassword --table mysql \
      < mysql.ddl

The output from the above exec command should look something like this:


$ kubectl -n sample-domain1-ns \
>         exec -it sample-domain1-mysql \
>         -- mysql -h localhost -u root -ppassword --table mysql \
>       < mysql.ddl
Unable to use a TTY - input is not a terminal or the right kind of file
Warning: Using a password on the command line interface can be insecure.
+-----------------+
|                 |
+-----------------+
| Start of sample |
+-----------------+
+-----------+--------------+
| VERSION() | CURRENT_DATE |
+-----------+--------------+
| 5.6.43    | 2019-04-12   |
+-----------+--------------+
+------+
| NAME |
+------+
| foo  |
| bar  |
+------+
+---------------+
|               |
+---------------+
| End of sample |
+---------------+
$

Notes about the MySQL Kubernetes resources in this sample:

  • The database pod’s storage is set up to be ephemeral – so all database tables are erased whenever the pod is restarted.
  • We’re going to make additional use of the Kubernetes secret we just deployed when we set up overrides in a future step.  It contains the database URL, user name, and password and can be viewed by the command “kubectl -n sample-domain1-ns get secret sample-domain1-mysql-secret -o yaml” .
  • Each resource’s namespace is set so that it matches the namespace we’re going to use for the WebLogic domain (the namespace from step 6.2 above). This is necessary to allow the domain to access these resources.
  • As an optional convenience, each resource is given a weblogic.domainUID label that matches the domain UID we’re going to use for the WebLogic domain. This is helpful for cleaning up these resources later because the Operator project’s clean up scripts look for this label.

 

Step 8 - There is no Step 8

I lie. Or do I? Move along, there's nothing to see here.

 

Step 9 – Create your WebLogic Domain Secret

Create a Kubernetes secret that contains your WebLogic domain admin credentials by following step 1 (and step 1 only!) in Quick start – Create a domain(https://oracle.github.io/weblogic-kubernetes-operator/2.2.0/quickstart/create-domain/).   NOTE:  Perform only step 1 - ignore the other steps. Specifically, perform these actions:



cd $WORKINGDIR/weblogic-kubernetes-operator

kubernetes/samples/scripts/create-weblogic-domain-credentials/create-weblogic-credentials.sh \
 -u weblogic -p welcome1 -n sample-domain1-ns -d sample-domain1

 

Step 10 – Create your sample inputs

The quick start sample includes a multi-purpose script called create-domain.sh which reads its options from a file called the ‘create domain inputs file’. We need an inputs file that matches the domain UID, namespace, and WebLogic credentials secret that we’re using in this blog, and also that requests a Kubernetes node port for the admin port. Let’s create such an input file by modifying the input file that’s already included with the sample:


cd $WORKINGDIR

cat $SAMPLEDIR/create-domain-inputs.yaml \
 | sed 's/\(domainUID:\).*/\1 sample-domain1/g' \
 | sed 's/\(exposeAdminNodePort:\).*/\1 true/g' \
 | sed 's/\(weblogicCredentialsSecretName:\).*/\1 sample-domain1-weblogic-credentials/g' \
 | sed 's/\(namespace:\).*/\1 sample-domain1-ns/g' \
 > my-inputs.yaml

This should create an inputs file named $WORKINGDIR/my-inputs.yaml , with a diff versus the original that looks something like this:


$ diff $SAMPLEDIR/create-domain-inputs.yaml $WORKINGDIR/my-inputs.yaml
16c16
< domainUID: domain1
---
> domainUID: sample-domain1
70c70
< weblogicCredentialsSecretName: domain1-weblogic-credentials
---
> weblogicCredentialsSecretName: sample-domain1-weblogic-credentials
104c104
< exposeAdminNodePort: false
---
> exposeAdminNodePort: true
107c107
< namespace: default
---
> namespace: sample-domain1-ns

Tip: 

Exposing a WebLogic console port via a nodeport can be helpful for demo purposes because this makes the port directly accessible from outside of a Kubernetes cluster, but keep in mind that this isn’t always a good idea in production for security reasons.

 

Step 11 - Create a WebLogic image with a Data Source and create an initial Domain Custom Resource

11.0 Vive la difference!

Be careful! Step 11 and the following steps are modeled on Quick start – Create a domain(https://oracle.github.io/weblogic-kubernetes-operator/2.2.0/quickstart/create-domain/), but aren’t exactly the same. 

What’s the difference? 

  1. The quick start sample uses a single invocation of the create-domain.sh sample script to download an image project from GitHub, create a domain home image, create a Domain Custom Resource, and deploy the Domain Custom Resource.  But instead, in this blog we modify the create-domain.sh command-line parameters to skip deploying the Domain Custom Resource. This gives us the opportunity to later modify the domain home configuration in order to add a Data Source and the opportunity to modify the Domain Custom Resource to specify overrides. We’ll deploy the Domain Custom Resource itself separately in a future step.
     
  2. In this blog we do all our work at the top level of a dedicated directory we created in Step 3 (WORKINGDIR), instead of generating output directories within the Operator source itself.

11.1 Initial run of create-domain.sh

First, let’s run create-domain.sh so that it downloads the WebLogic image project into a directory called $WORKINGDIR/docker-images , creates a Domain Custom Resource file in the $WORKINGDIR/output directory tree (which we will update in a later step), and builds a WebLogic image (which we will also update in a later step). 


cd $WORKINGDIR

$SAMPLEDIR/create-domain.sh -i my-inputs.yaml -o ./output -u weblogic -p welcome1

Second, you can verify that the script created a Docker image that contains a domain home by running the “docker images” command and checking for an image named “domain-home-in-image” .

Notes:

  • We deliberately ran create-domain.sh without the ‘-e’ parameter.  This tells the script to skip deploying the Domain Custom Resource that it creates for you.  We will deploy in a later step.
  • The image from this first run will not have the Data Source configuration that we plan to demonstrate overriding. Don’t worry, – we’ll ignore this image and generate a new image with the Data Source configuration in a future step.  

11.2 Modify domain home creation WLST to add a Data Source

Now let’s modify the domain home creation WLST in the image project that create-domain.sh downloaded for us.  We want to modify it to add a Data Source.  

First, create a file named blogSnippet.txt with a WLST snippet.  We’ll use this later when we edit the WLST script:


cd $WORKINGDIR

cat << EOF > blogSnippet.txt

def createDataSource(dsName, dsJNDI, dsDriver, dsGlobalTX, dsXAInterface, dsURL, dsUser, dsPass, dsMinSize, dsMaxSize, dsTest, dsTarget):
  print 'Creating DataSource ' + dsName

  cd('/')

  # JDBCSystemResource

  jdbcSystemResource = create(dsName, "JDBCSystemResource")
  assign('JDBCSystemResource', dsName, 'Target', dsTarget)

  # JDBCDataSourceParams

  cd('/JDBCSystemResource/' + dsName + '/JdbcResource/' + dsName)
  create('dataSourceParams', 'JDBCDataSourceParams')
  cd('JDBCDataSourceParams/NO_NAME_0')

  set('GlobalTransactionsProtocol',java.lang.String(dsGlobalTX))
  set('JNDIName',java.lang.String(dsJNDI))

  # JDBCConnectionPoolParams

  cd('/JDBCSystemResource/' + dsName + '/JdbcResource/' + dsName)
  create('connPoolParams', 'JDBCConnectionPoolParams')
  cd('JDBCConnectionPoolParams/connPoolParams')

  set('InitialCapacity',dsMinSize)
  set('MinCapacity',dsMinSize)
  set('MaxCapacity',dsMaxSize)
  set('CapacityIncrement',1)
  set('TestConnectionsOnReserve',true)
  set('TestTableName',dsTest)

  # JDBCDriverParams

  cd('/JDBCSystemResource/' + dsName + '/JdbcResource/' + dsName)
  create('driverParams', 'JDBCDriverParams')
  cd('JDBCDriverParams/NO_NAME_0')

  set('Url',dsURL)
  set('DriverName',dsDriver)
  set('PasswordEncrypted',dsPass)
  set('UseXaDataSourceInterface',dsXAInterface)

  create('testProperties','Properties')
  cd('Properties/NO_NAME_0')

  property = create('user','Property')
  property.setValue(dsUser)

  cd('/')

  print 'DataSource ' + dsName + ' successfully created.'

def createMySQLDataSource(dsName,dsJNDI,dsGlobalTX,dsXAInterface,dsHost,dsPort,dsDB,dsUser,dsPass,dsMinSize,dsMaxSize,dsTarget):
  # dsDriver='com.mysql.cj.jdbc.Driver'
  dsDriver='com.mysql.jdbc.Driver'
  dsURL='jdbc:mysql://' + dsHost + ':' + dsPort + '/' + dsDB
  dsTest='SQL SELECT 1'
  createDataSource(dsName, dsJNDI, dsDriver, dsGlobalTX, dsXAInterface, dsURL, dsUser, dsPass, dsMinSize, dsMaxSize, dsTest, dsTarget)

createMySQLDataSource('mysqlDS','mysqlDS','None',false,'invalid-host','3306','invalid-db-name','invalid-user','invalid-pass',0,10,admin_server_name)

EOF

Second, let’s modify the Docker image project sample domain creation WLST script so that it creates a Data Source.  

This requires inserting the above blogSnippet.txt file into the ./docker-images/OracleWebLogic/samples/12213-domain-home-in-image/container-scripts/create-wls-domain.py WLST script.  The file should be inserted just after the script’s assign( command and just before its writeDomain( command.  You can use your favorite editor to do this, or try the following commands:


cd $WORKINGDIR
	
MYWLST=./docker-images/OracleWebLogic/samples/12213-domain-home-in-image/container-scripts/create-wls-domain.py

cp $MYWLST ./create-wls-domain.py.backup`date '+%Y%m%d%H%M%S'`

cat $MYWLST | sed '/assign(/r./blogSnippet.txt' > tmptmp

mv tmptmp $MYWLST

Tip: 

Configuring a Data Source with an invalid URL, user name, and password helps demonstrate that overrides are working when we apply them, and, more importantly, this is a best practice that prevents a domain-home-in-image from accidentally accessing a database until the exact desired overrides are applied.

 

11.3 Final run of create-domain.sh – generate your final WebLogic image and ensure that your Kubernetes cluster has access to the image

Now let's rerun the create-domain.sh script with a special ‘-k’ parameter so that it uses the WebLogic WLST script we modified in the previous step. This will build our final WebLogic image which contains a WebLogic 12.2.1.3 install and a WebLogic domain home configuration, and will also implicitly create a Domain Custom Resource yaml file that we’ll modify in the next step and then deploy.

Note that we must pass ‘-k’ to the script this time around:  the ‘-k’ tells it to use the $WORKINGDIR/docker-images project that was already downloaded in step 11.1 (the first time we ran the script) and that we modified in step 11.2. If we didn’t pass the ‘-k’, then the script would download a new copy of the image project and erase all of our hard work so far!


cd $WORKINGDIR

$SAMPLEDIR/create-domain.sh -i my-inputs.yaml -o ./output -u weblogic -p welcome1 -k

Important: 

This step locally creates a Docker image named domain-home-in-image  and tagged 12.2.1.3 .  

Make sure your Kubernetes cluster has access to this image. One way to check is by calling kubectl run one-off --rm --restart=Never -it --image=domain-home-in-image:12.2.1.3 -- bash -c 'echo Hi There' which should cause your Kubernetes cluster to run 'echo Hi There' in the image and locally print Hi There. If the run command stalls for more than a few seconds, ^C out of it, and call kubectl get pods to examine its status. If you want to rerun the command, you may need to call kubectl delete pod one-off to delete the failing one-off pod first. 

If your Kubernetes cluster doesn’t have access, then you’ll see pod statuses like ErrImagePull when you try the above one-off  test or when you try to deploy the sample’s Domain Custom Resource in a future step. To correct the problem, push the image to a Docker repository that’s accessible from your Kubernetes cluster.  

Note that if you need to push the new image to a remote registry, then you may also need to make the following adjustments:

  • Set the image  property in the Domain Custom Resource file $WORKINGDIR/output/weblogic-domains/sample-domain1/domain.yaml to the target image name (including the registry hostname/port, and the tag if needed).
  • If you want Kubernetes to pull the image from a private registry, deploy a Kubernetes secret that hold your registry credentials in the sample-domain1-ns namespace, and uncomment/set the imagePullSecrets  property in your Domain Custom Resource so that it references the name of this secret.

 

Step 12 Modify the Domain Custom Resource to reference your Data Source overrides

Now let’s modify the Domain Custom Resource that create-domain.sh created for us so that it references our custom overrides configuration map and secret. Use your favorite editor to make the following edits:

After the following lines in $WORKINGDIR/output/weblogic-domains/sample-domain1/domain.yaml :


  webLogicCredentialsSecret:
    name: sample-domain1-weblogic-credentials

Add the following two lines:


  configOverrides: sample-domain1-overrides-cm
  configOverrideSecrets: [sample-domain1-mysql-secret]

Important:

  • Make sure that configOverrides and configOverridesSecrets are indented exactly two spaces (the same number of spaces as webLogicCredentialsSecret ).
  • The Domain Custom Resource configOverridesSecrets field is an array that allows you to specify more than one secret as needed – but we only need to specify one secret for the purposes of this blog.
  • Don't deploy the Domain Custom Resource yet! The override secret(s) and config map both must be deployed before the Domain Custom Resource itself is deployed. We've already deployed the secret in Step 7, and will deploy the config map in the next step.

 

Step 13 Create and deploy your override config map 

Any override secrets and config maps must be deployed before you deploy a Domain Custom Resource that references them. We already deployed the sample-domain1-mysql-secret in Step 7.  Now it’s time to create and deploy your Data Source overrides Kubernetes config map using the same name as we specified in Step 12 as a Domain Custom Resource entry, and using the same namespace as we're using for our Domain Custom Resource and its pods. This config map must contain your overrides template and also a version.txt file that contains the exact string "2.0".


cd $WORKINGDIR

# An overrides config map must contain a version.txt file. Currently, the only
# supported version in this file is "2.0".

echo "2.0" > version.txt

# Note!  If you manually create the following file instead of 
# using this shell script, then be sure to remove 
# the ‘\’ before each ${secret .  (The 
# backslashes are added here to prevent the shell from
# assuming the secret macro is a shell macro.)

cat << EOF > jdbc-mysqlDS.xml
<?xml version="1.0" encoding="UTF-8"?>
<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source"  xmlns:f="http://xmlns.oracle.com/weblogic/jdbc-data-source-fragment"  xmlns:s="http://xmlns.oracle.com/weblogic/situational-config">
  <name>mysqlDS</name>
    <jdbc-driver-params>
      <url f:combine-mode="replace">\${secret:sample-domain1-mysql-secret.url}</url>
      <properties>
         <property>
            <name>user</name>
            <value f:combine-mode="replace">\${secret:sample-domain1-mysql-secret.root-user}</value>
         </property>
      </properties>
      <password-encrypted f:combine-mode="add">\${secret:sample-domain1-mysql-secret.root-password:encrypt}</password-encrypted>
    </jdbc-driver-params>
</jdbc-data-source>
EOF

kubectl -n sample-domain1-ns create cm sample-domain1-overrides-cm \
  --from-file version.txt \
  --from-file jdbc-mysqlDS.xml

kubectl -n sample-domain1-ns label cm/sample-domain1-overrides-cm \
  weblogic.domainUID=sample-domain1

Let's take a closer look at what we just did:

  • We've deployed a Data Source override template and given it a specially formatted name jdbc-DATASOURCENAME.xml. This format ensure that the Operator can correlate it with the Data Source we created in the Domain Home.
  • The template demonstrates macros.
    • Override template macros usefully allow reusing the same override template across different stages of your CI/CD process, as well as avoid directly encoding plain-text passwords into text files in your pod.  
    • Macros can reference Kubernetes Secrets. This demo override template's user name, password, and URL macros reference the secret we deployed along with the MySQL database as part of Step 7.
    • We use a special ‘encrypted macro’ syntax for the password macro. This enables it to override an encrypted type WebLogic MBean field, and is also more secure than overriding a plain-text type field.
  • The override config map resource’s namespace matches the namespace we’re going to use for the WebLogic domain (the namespace from Step 6.2 above). This is necessary to allow the domain to access these resources.
  • As a convenience, the config map override resource is given a weblogic.domainUID label that matches the domain UID we’re going to use for the WebLogic domain. This is helpful for cleaning up these resources later because the Operator project’s clean up scripts look for this particular label.

For a complete discussion of the syntax of templates and macros, see Configuration Overrides in the Operator User Guide:  https://oracle.github.io/weblogic-kubernetes-operator/2.2.0/userguide/managing-domains/configoverrides/.

 

Step 14 Deploy your Domain Custom Resource and verify the domain starts

First, let’s deploy our domain and verify it started. To deploy the domain, we use the Domain Custom Resource file that we modified in Step 12 above:


cd $WORKINGDIR

kubectl create -f output/weblogic-domains/sample-domain1/domain.yaml

Second, let’s verify the domain started.

  • You may need to wait some time for the Operator to detect the Domain Custom Resource and start its pods.  
  • Cd to the top of your source tree cd $WORKINGDIR/weblogic-kubernetes-operator  and follow these exact steps in Quick start – Create a domain(https://oracle.github.io/weblogic-kubernetes-operator/2.2.0/quickstart/create-domain/): 3, 4, 5, 6b, and 6c.
  • Notes:  
    • We are skipping items 1, 2, & 6a because we've already performed them in earlier steps and modified them for this particular demo. 
    • If Step 5 fails with a 'Error: failed to download "kubernetes/samples/charts/ingress-per-domain" (hint: running `helm repo update` may help)' error, this likely indicates you're not in the $WORKINGDIR/weblogic-kubernetes-operator directory.
    • Step 6b has an incorrect console URL for accessing the WebLogic console ' http://your.server.com:30701', the URL should be 'http://your.sever.com:30701/console'.  The username and password are 'weblogic' and 'welcome1'.
    • If you cannot get steps 4, 5, 6b, or 6c to work, it's OK to skip them - they're not necessary for demonstrating that your data source overrides are taking effect.  (We will demonstrate that overrides are taking effect in the next step.)

Third, after all of your WebLogic Servers are running then you should be able to see three WebLogic pods and the MySQL pod in the sample’s namespace:


$ kubectl -n sample-domain1-ns get pods
NAME                             READY     STATUS    RESTARTS   AGE
sample-domain1-admin-server      1/1       Running   0          7m
sample-domain1-managed-server1   1/1       Running   0          6m
sample-domain1-managed-server2   1/1       Running   0          6m
sample-domain1-mysql             1/1       Running   0          2h

 

Step 15 Verify your overrides are working!

If all goes well, your pod log on each WebLogic Server should report that your overrides file was loaded and should not report any situational config errors, your serverConfig MBean tree should reflect that the overrides have taken effect, and finally your Data Source should be able to contact the database.

First, let’s check your WebLogic admin server’s pod log:


kubectl -n sample-domain1-ns logs sample-domain1-admin-server | grep -i situation

The output should look something like the following, and should not include any error or warning messages. Note that the first config file is the internally generated situational configuration file describe in the ‘Internal Overrides’ discussion earlier in this blog:


<Mar 27, 2019 7:04:29 PM GMT> <Info> <Management> <BEA-141330> <Loading situational config file: /u01/oracle/user_projects/domains/sample-domain1/optconfig/introspector-situational-config.xml>

<Mar 27, 2019 7:04:29 PM GMT> <Info> <Management> <BEA-141330> <Loading situational config file: /u01/oracle/user_projects/domains/sample-domain1/optconfig/jdbc/mysqlDS-7969-jdbc-situational-config.xml>

Now let’s check your serverConfig bean tree to see if it reflects the overrides you've specified, and let’s also test your Data Source by running an online WLST script:


kubectl -n sample-domain1-ns exec -i sample-domain1-admin-server wlst.sh <<EOF

connect("weblogic","welcome1",url="t3://sample-domain1-admin-server:7001" )

domainconfig()

domainConfig()
get("/JDBCSystemResources/mysqlDS/JDBCResource/mysqlDS/JDBCDriverParams/mysqlDS/Url")
get("/JDBCSystemResources/mysqlDS/JDBCResource/mysqlDS/JDBCDriverParams/mysqlDS/Properties/mysqlDS/Properties/user/Value")

serverConfig()
get("/JDBCSystemResources/mysqlDS/JDBCResource/mysqlDS/JDBCDriverParams/mysqlDS/Url")
get("/JDBCSystemResources/mysqlDS/JDBCResource/mysqlDS/JDBCDriverParams/mysqlDS/Properties/mysqlDS/Properties/user/Value")

serverRuntime()
cd("/JDBCServiceRuntime/admin-server/JDBCDataSourceRuntimeMBeans/mysqlDS")
cmo.testPool()
exit()

EOF

The output should look something like the following (some lines have been deleted and some whitespace added for clarity). Note how the domainConfig tree reflects the original values of fields before overrides take effect while the serverConfig tree reflects overridden values.  Note that if the testPool call fails then it will display an exception, otherwise it just silently succeeds.


Welcome to WebLogic Server Administration Scripting Shell

wls:/offline> connect('weblogic','welcome1',url='t3://sample-domain1-admin-server:7001')

Connecting to t3://sample-domain1-admin-server:7001 with userid weblogic ...
Successfully connected to Admin Server "admin-server" that belongs to domain "sample-domain1".

wls:/sample-domain1/serverConfig/> domainConfig()

wls:/sample-domain1/domainConfig/> get('/JDBCSystemResources/mysqlDS/JDBCResource/mysqlDS/JDBCDriverParams/mysqlDS/Url')

'jdbc:mysql://invalid-host:3306/invalid-db-name'

wls:/sample-domain1/domainConfig/> get('/JDBCSystemResources/mysqlDS/JDBCResource/mysqlDS/JDBCDriverParams/mysqlDS/Properties/mysqlDS/Properties/user/Value')

'invalid-user'

wls:/sample-domain1/domainConfig/> serverConfig()

wls:/sample-domain1/serverConfig/> get('/JDBCSystemResources/mysqlDS/JDBCResource/mysqlDS/JDBCDriverParams/mysqlDS/Url')

'jdbc:mysql://sample-domain1-mysql:3306/mysql'

wls:/sample-domain1/serverConfig/> get('/JDBCSystemResources/mysqlDS/JDBCResource/mysqlDS/JDBCDriverParams/mysqlDS/Properties/mysqlDS/Properties/user/Value')

'root'

wls:/sample-domain1/serverConfig/> serverRuntime()

wls:/sample-domain1/serverRuntime/> cd('/JDBCServiceRuntime/admin-server/JDBCDataSourceRuntimeMBeans/mysqlDS')
wls:/sample-domain1/serverRuntime/JDBCServiceRuntime/admin-

server/JDBCDataSourceRuntimeMBeans/mysqlDS> cmo.testPool()
wls:/sample-domain1/serverRuntime/JDBCServiceRuntime/admin-

server/JDBCDataSourceRuntimeMBeans/mysqlDS> exit()

Exiting WebLogic Scripting Tool.

If you run into any errors, consult the Debugging section of the configuration overrides documentation. See the second Tip below for the link.

Tip:  You cannot use the console to view overrides.

You can view overrides in the serverConfig tree using online JMX, REST, or WLST scripting, but not on the WebLogic console. The console only reflects the static 'domainConfig' MBean tree, which is your configuration before any overrides have been applied.

Tip:  The Operator documentation provides a step-by-step discussion for debugging overrides.

For a step-by-step discussion of viewing and verifying overrides, consult the Debugging section of the configuration overrides documentation (https://oracle.github.io/weblogic-kubernetes-operator/2.2.0/userguide/managing-domains/configoverrides/).

Step 16 -- Tidy Up!

If you want to clean up, see Step 1.

 

End Game

We’ve demonstrated using the WebLogic Kubernetes Operator and its configuration overrides feature. This feature enables portably deploying the same WebLogic domain to different Kubernetes environments without modifying the domain configuration itself. If your WebLogic domain is contained within a Docker image, configuration overrides furthermore allow you to customize its WebLogic configuration without creating a new Docker image. This portability is particularly useful for CI/CD use cases.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.