Dynamic Debug Patches in WebLogic Server 12.2.1

Introduction

Whether
we like it not, we know that no software is perfect. Bugs happen, in
spite of the best efforts by the developers. Worse, in many
circumstances, they show up in unexpected ways. They can also be
intermittent and hard to reproduce in some cases. In such cases, there
is often not enough information even to understand the nature of the
problem if the product is not sufficiently instrumented to reveal the
underlying causes. Direct access to a customer’s production environment
is usually not an option. To get better understanding of the underlying
problem, instrumented debug patches are usually created with the hope
that running the applications with debug patches would provide more
insight. This can be a trial and error method and can take several
iterations before hitting upon the actual cause. The folks creating a
debug patch (typically Support or Development teams in the software
provider organization) and the customers running the application are
almost always different groups, often in different companies. Thus, each
iteration of creating a debug patch, providing it to the customer,
getting it applied in customer environment and getting the results back
can take substantial time. In turn, it can result in delays in problem
resolution.

In addition, there can be other significant issues
with deploying such debug patches. Applying patches in a Java EE
environment requires bouncing servers and domains or at least redeploying applications. In mission critical
deployments, it may not be possible to immediately apply patches.
Moreover, when a server is bounced, its state is lost. Thus, vital
failure data in memory may be lost. Also, an intermittent failure may
not show up for a long time after restarting servers, making quick
diagnosis difficult.

Dynamic Debug Patches

In the WebLogic Server 12.2.1 release, a new feature called Dynamic Debug Patches
is introduced which aims to simplify the process of capturing
diagnostic data for quicker problem resolution. With this feature, debug
patches can be dynamically activated without having to restart servers or clusters or redeploy applications in a WebLogic domain. It leverages the JDK’s instrumentation feature to hot-swap classes from specified debug patches using run-time WLST
commands. With provided WLST commands (as described below), one or more
debug patches can be activated within the scope of selected servers,
clusters, partitions and applications. Since no server restart  or
application redeployment is needed, associated logistical impediments
are a non-issue. For one, since the applications and services continue
to run, there is less of a barrier to activate these patches in
production environments. Also, there is no loss of state. Thus, the
instrumented code in newly activated debug patches have a better chance
at revealing erroneous transient state and providing meaningful
diagnostic information.

Prerequisites

Dynamic
debug patches are ordinary jar files containing patched classes with
additional instrumentation such as debug logging, print statements, etc.
Typically, product development or support teams build these patch jars
and make them available to system operations teams for their activation
in the field. To make them available to the WebLogic Server’s dynamic debug patches feature, system administrators need to copy them to a
specific directory in a domain. By default, this directory is the debug_patches
subdirectory under the domain root. However, it can be changed by
reconfiguring the DebugPatchDirectory attribute of the
DebugPatchesMBean.

Another requirement is to start the servers in
the domain with the debugpatch instrumentation agent with the following
option in the server’s startup command. It is automatically added by the
startup scripts created for WebLogic Server 12.2.1 domains.

-javaagent:${WL_HOME}/server/lib/debugpatch-agent.jar

Using Dynamic Debug Patches Feature

We will illustrate the use of this feature by activating and deactivating debug patches on a simple toy application.

The Application

We will use a minimalist toy web application which computes the factorial value of an input integer and returns it to the browser.

FactorialServlet.java:

package example;

import java.io.IOException;
import javax.servlet.GenericServlet;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.annotation.WebServlet;

import java.util.Map;
import java.util.HashMap;
import java.util.concurrent.ConcurrentHashMap;

/**
 * A trivial servlet: Returns addition of two numbers.
 */
@WebServlet(value=”/factorial”, name=”factorial-servlet”)
public class FactorialServlet extends GenericServlet {

  public void service(ServletRequest request, ServletResponse response)
      throws ServletException, IOException {
    String n = request.getParameter(“n”);
    System.out.println(“FactorialServlet called for input=” + n);
    int result = Factorial.getInstance().factorial(n);
    response.getWriter().print(“factorial(” + n + “) = ” + result);
  }
}

The servlet delegates to the Factorial singleton to
compute the factorial value. As an optimization, the Factorial class
maintains a Map of previously computed values which serves as an
illustration of retaining stateful information while
activating or deactivating dynamic debug patches.

Factorial.java:

package example;

import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

class Factorial {
  private static final Factorial SINGLETON = new Factorial();
  private Map<String, Integer> map = new ConcurrentHashMap<String, Integer>();

  static Factorial getInstance() {
    return SINGLETON;
  }

  public int factorial(String n) {
    if (n == null) {
      throw new NumberFormatException(“Invalid argument: ” + n);
    }
    n = n.trim();
    Integer val = map.get(n);
    if (val == null) {
      int i = Integer.parseInt(n);
      if (i < 0)
        throw new NumberFormatException(“Invalid argument: ” + n);
      int fact = 1;
      while (i > 0) {
        fact *= i;
        i–;
      }
      val = new Integer(fact);
      map.put(n, val);
    }
    return val;
  }
}

Building and Deploying the Application

To
build the factorial.war web application, create FactorialServlet.java
and Factorial.java files as above in an empty directory. Build the
application war files with following commands:

mkdir -p WEB-INF/classes
javac -d WEB-INF/classes FactorialServlet.java Factorial.java
jar cvf factorial.war WEB-INF

Deploy the application using WLST (or WebLogic Server Administration Console):

$MW_HOME/oracle_common/common/bin/wlst.sh
Initializing WebLogic Scripting Tool (WLST) …
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
connect(username, password, adminUrl)  # e.g. connect(‘weblogic’, ‘weblogic’, ‘t3://localhost:7001’)
Connecting to t3://localhost:7001 with userid weblogic …
Successfully connected to Admin Server “myserver” that belongs to domain “mydomain”.
Warning: An insecure protocol was used to connect to the server.
To ensure on-the-wire security, the SSL port or Admin port should be used instead.
deploy(‘factorial’, ‘factorial.war’, targets=’myserver’)

Note that in the illustration above, we targeted the
application only to the administration server. It may have been targeted to other
managed servers or clusters in real world. We will discuss how to
activate and deactivate debug patches over multiple managed servers and
clusters in a subsequent article.

Invoke the web application from your browser. For example: http://localhost:7001/factorial/factorial?n=4 You should see the result in the browser and a message in the server’s stdout window such as:

FactorialServlet called for input=4

The Debug Patch

The
application as written does not perform lot of logging and does not
reveal much about its functioning. Perhaps there is a problem and we
need more information when it executes. We can create a debug patch from
the application code and provide it to the system administrator so
he/she can activate it on the running server/application. Let us modify
above code to put additional print statements for getting additional
information (i.e. the lines with “MYDEBUG” below).

Updated (version 1)  Factorial.java:

class Factorial {
  private static final Factorial SINGLETON = new Factorial();
  private Map<String, Integer> map = new ConcurrentHashMap<String, Integer>();
  static Factorial getInstance() {
    return SINGLETON;
  }
  public int factorial(String n) {
    if (n == null) {
      throw new NumberFormatException("Invalid argument: " + n);
    }
    n = n.trim();
    Integer val = map.get(n);
    if (val == null) {
      int i = Integer.parseInt(n);
      if (i < 0)
        throw new NumberFormatException("Invalid argument: " + n);
      int fact = 1;
      while (i > 0) {
        fact *= i;
        i--;
      }
      val = new Integer(fact);
      System.out.println("MYDEBUG> saving factorial(" + n + ") = " + val);
      map.put(n, val);
    } else {
      System.out.println("MYDEBUG> returning saved factorial(" + n + ") = " + val);
    }
    return val;
  }
}

Build the debug patch jar. Note that this a plain jar
file, that is, not built as an application archive. Also note that we
need not compile the entire application (although it would not hurt).
The debug patch jar should contain only the classes which have changed
(in this case, Factorial.class).

mkdir patch_classes
javac -d patch_classes Factorial.java
jar cvf factorial_debug_01.jar -C patch_classes

Activating Debug Patches

In
most real world scenarios, creators (developers) and activators (system
administraors) of debug patches would be different people. For the
purpose of illustration, we will wear multiple hats here. Assuming that
we are using the default configuration for the location of the debug
patches directory, create the debug_patches directory under the domain directory if it is not already there. Copy factorial_debug_01.jar debug patch jar into the debug_patches directory.  Connect to the server with WLST as above.

First, let us check which debug patches are available in the domain. This can be done with the listDebugPatches command.

Hint:
To see available diagnostics commands, issue help(‘diagnostics’)
command. To get information on specific command, issue
help(commandName), e.g. help(‘activateDebugPatch’).

wls:/mydomain/serverConfig/> listDebugPatches()         
myserver:
Active Patches:
Available Patches:
    factorial_debug_01.jar
    app2.0_patch01.jar
    app2.0_patch02.jar 

factorial_debug_01.jar is the newly created debug patch.
app2.0_patch01.jar and app2.0_patch02.jar were created in the past to
investigate issues with some other application. The listing above shows
no “active” patches since none have been activated so far.

Now, let us activate the debug patch with the activateDebugPatch command.

tasks=activateDebugPatch('factorial_debug_01.jar', app='factorial', target='myserver')
wls:/mydomain/serverConfig/> print tasks[0].status                                                                 
FINISHED
wls:/mydomain/serverConfig/> listDebugPatches()     
myserver:
Active Patches:
    factorial_debug_01.jar:app=factorial
Available Patches:
    factorial_debug_01.jar
    app2.0_patch01.jar
    app2.0_patch02.jar

The command returns an array of tasks which can be used
to monitor the progress and status of activation command. Multiple
managed servers and/or clusters can be specified as targets if
applicable. Corresponding to each applicable target server, there is a
task in the returned tasks array. The command can also be used to
activate debug patches at the server and middleware level as well. Such
patches will be typically created by Oracle Support as needed. Output
of listDebugPatches() command above shows that factorial_debug_01.jar is
now activated on application “factorial”.

Now, let us send some requests to the application: http://localhost:7001/factorial/factorial?n=4 and http://localhost:7001/factorial/factorial?n=5

Server output:

FactorialServlet called for input=4
MYDEBUG> returning saved factorial(4) = 24
FactorialServlet called for input=5
MYDEBUG> saving factorial(5) = 120

Notice that for input=4, saved results were returned
since the values were computed and saved in the map due to a prior
request. Thus, the debug patch was activated without destroying existing
state in the application. For input=5, values were not previously
computed and saved, thus a different debug message showed up.

Activating Multiple Debug Patches

If
needed, multiple patches which potentially overlap can be activated. A
patch which is activated later would mask the effects of a previously
activated patch if there is an overlap. Say, in the above case, we need
more detailed information from the factorial() method as it is executing
its inner loop. Let us create another debug patch, copy it to
debug_patches directory and activate it.

Updated (version 2) Factorial.java:

class Factorial {
  private static final Factorial SINGLETON = new Factorial();
  private Map<String, Integer> map = new ConcurrentHashMap<String, Integer>();
  static Factorial getInstance() {
    return SINGLETON;
  }
  public int factorial(String n) {
    if (n == null) {
      throw new NumberFormatException("Invalid argument: " + n);
    }
    n = n.trim();
    Integer val = map.get(n);
    if (val == null) {
      int i = Integer.parseInt(n);
      if (i < 0)
        throw new NumberFormatException("Invalid argument: " + n);
      int fact = 1;
      while (i > 0) {
        System.out.println("MYDEBUG> multiplying by " + i);
        fact *= i;
        i--;
      }
      val = new Integer(fact);
      System.out.println("MYDEBUG> saving factorial(" + n + ") = " + val);
      map.put(n, val);
    } else {
      System.out.println("MYDEBUG> returning saved factorial(" + n + ") = " + val);
    }
    return val;
  }
}

Build factorial_debug_02.jar

javac -d patch_classes Factorial.java
jar cvf factorial_debug_02.jar  -C patch_classes .
cp factorial_debug_02.jar $DOMAIN_DIR/debug_patches

Activate factorial_debug_02.jar

wls:/mydomain/serverConfig/> listDebugPatches()     
myserver:
Active Patches:
    factorial_debug_01.jar:app=factorial
Available Patches:
    factorial_debug_01.jar
    factorial_debug_02.jar
    app2.0_patch01.jar
    app2.0_patch02.jar
wls:/mydomain/serverConfig/> tasks=activateDebugPatch('factorial_debug_01.jar', app='factorial', target='myserver')
wls:/mydomain/serverConfig/> listDebugPatches()                                                                    
myserver:
Active Patches:
    factorial_debug_01.jar:app=factorial
    factorial_debug_02.jar:app=factorial
Available Patches:
    factorial_debug_01.jar
    factorial_debug_02.jar
    servlet3.0_patch01.jar
    servlet3.0_patch02.jar

Now, let us send some requests to the application: http://localhost:7001/factorial/factorial?n=5 and http://localhost:7001/factorial/factorial?n=6

FactorialServlet called for input=5
MYDEBUG> returning saved factorial(5) = 120
FactorialServlet called for input=6
MYDEBUG> multiplying by 6
MYDEBUG> multiplying by 5
MYDEBUG> multiplying by 4
MYDEBUG> multiplying by 3
MYDEBUG> multiplying by 2
MYDEBUG> multiplying by 1
MYDEBUG> saving factorial(6) = 720

We see the additional information printed due to code in factorial_debug_02.jar.

Deactivating Debug Patches

When
the debug patch is not needed any more, it can be deactivated
deactivateDebugPatches command. To get help on it, execute
help(‘deactivateDebugPatches’).

wls:/mydomain/serverConfig/> tasks=deactivateDebugPatches('factorial_debug_02.jar', app='factorial', target='myserver')            
wls:/mydomain/serverConfig/> listDebugPatches()                                                                        
myserver:
Active Patches:
    factorial_debug_01.jar:app=factorial
Available Patches:
    factorial_debug_01.jar
    factorial_debug_02.jar
    servlet3.0_patch01.jar
    servlet3.0_patch02.jar

Now, executing http://localhost:7001/factorial/factorial?n=2 gets us the following output in server’s stdout window:

FactorialServlet called for input=2
MYDEBUG> saving factorial(2) = 2

Note that when we had activated factorial_debug_01.jar
and factorial_debug_02.jar in that order, the classes in
factorial_debug_02.jar masked those in factorial_debug_01.jar. After
deactivating factorial_debug_02.jar, the classes in
factorial_debug_01.jar got unmasked and became effective again. A list
of comma separated list of debug patches may be specified with the
deactivateDebugPatches command. To deactivate all active debug patches
on applicable target servers, deactivateAllDebugPatches() command may be
used.

WLST Commands

Followingdiagnostic WLST commands are provided to interact with the Dynamic Debug Patches
feature. As noted above, help(command-name) shows help for that command.

Limitations

The
Dynamic Debug Patch features leverages JDK’s hot-swap feature. The
hot-swap feature has a limitation that how-swapped classes cannot have a
different shape than the original classes. This means that the classes
which are swapped in cannot add, remove or update constructors, methods,
fields, super classes, implemented interfaces, etc. Only changes in
method bodies are allowed. It should be noted that debug patches
typically only gather additional information and not attempt to “fix”
the problems as such. Minor fixes which would not change the shape of
classes may be tried, but that is not the main purpose of this feature.
Therefore, we don’t expect this to be a big limitation in practice.

One
issue, however is that, in some cases the new debug code may need to
maintain some state. For example, perhaps we want to collect some data
in some map and only dump it out on some threshold. The JDK limitation
regarding shape-change creates problems in that case. The Dynamic Debug
Patches provides a DebugPatchHelper utility class to help address some
of those concerns. We will discuss that in a subsequent article. Please
visit us back to read about it.