Wednesday Jun 29, 2016

Connection Initialization Callback on WLS Datasource

WebLogic Server is now available. You can see the blog article announcing it at Oracle WebLogic Server is Now Available.

One of the WLS datasource features that appeared quite a while ago but not mentioned much is the ability to define a callback that is called during connection initialization.  The original intent of this callback was to provide a mechanism that is used with the Application Continuity (AC) feature.  It allows for the application to ensure that the same initialization of the connection can be done when it is reserved and also later on if the connection is replayed.  For the latter case, the original connection has some type of "recoverable" error and is closed, a new connection is reserved under the covers, and all of the operations that were done on the original connection are replayed on the new connection.  The callback allows for the connection to be re-initialized with whatever state is needed by the application.

The concept of having a callback to allow for the application to initialize all connections without scattering this processing all over the application software wherever getConnection() is called is very useful, even without replay being involved.  In fact, since the callback can be configured in the datasource descriptor, which I recommend, there is no change to the application except to write the callback itself.  

Here's the history of support for this feature, assuming that the connection initialization callback is configured.

WLS 10.3.6 - It is only called on an Active GridLink datasource when running with the replay driver (replay was only supported with AGL).

WLS  12.1.1, 12.1.2, and 12.1.3 - It is called if used with the replay driver and any datasource type (replay support was added to GENERIC datasources).

WLS 12.2.1 - It is called with any Oracle driver and any datasource type. 

WLS - It is called with any driver and any datasource type.  Why limit the goodness to just the Oracle driver?

The callback can be configured in the application by registering it on the datasource in the Java code. You need to ensure that you only do this once per datasource.  I think it's much easier to register it in the datasource configuration.   

Here's a sample callback.

package demo;
import oracle.ucp.jdbc.ConnectionInitializationCallback;

public class MyConnectionInitializationCallback implements
  ConnectionInitializationCallback {
  public MyConnectionInitializationCallback()  {
  public void initialize(java.sql.Connection connection)
    throws java.sql.SQLException {
     // Re-set the state for the connection, if necessary

This is a simple Jython script using as many defaults as possible to just show registering the callback.

import sys, socket
hostname = socket.gethostname()
jdbcSR = create(dsname, 'JDBCSystemResource')
jdbcResource = jdbcSR.getJDBCResource()
dsParams = jdbcResource.getJDBCDataSourceParams()
driverParams = jdbcResource.getJDBCDriverParams()
driverProperties = driverParams.getProperties()
userprop = driverProperties.createProperty('user')
oracleParams = jdbcResource.getJDBCOracleParams()
oracleParams.setConnectionInitializationCallback('demo.MyConnectionInitializationCallback')  # register the callback

 Here are a few observations.  First, to register the callback using the configuration, the class must be in your classpath.  It will need to be in the server classpath anyway to run but it needs to get there earlier for configuration.  Second, because of the history of this feature, it's contained in the Oracle parameters instead of the Connection parameters; there isn't much we can do about that.  In the WLS administration console, the entry can be seen and configured in the Advanced parameters of the Connection Pool tab as shown in the following figure (in addition to the Oracle tab).  Finally, note that the interface is a Universal Connection Pool (UCP) interface so that this callback can be shared with your UCP application (all driver types are supported starting in Database

This feature is documented in the Application continuity section of the Administration Guide.   See .

You might be disappointed that I didn't actually do anything in the callback.  I'll use this callback again in my next blog to show how it's used in another new WLS feature.

Tuesday Dec 08, 2015

Monitoring FAN Events

fanWatcher is a sample program to print the Oracle Notification Service (ONS) Fast Application Notification (FAN) event information. These events provide information regarding load balancing, and service and instance up and down events. This information is automatically processed by WebLogic Server Active GridLink and UCP on the mid-tier. For more information about FAN events, see this link.  The program described here is an enhancement of the earlier program described in that white paper  This program can be modified to work as desired to monitor events and help diagnose problems with configuration. The code is available this link (rename it from .txt to .java).

To run this Java application, you need to be set up to run a JDK and you need ons.jar and ojdbcN.jar in the CLASSPATH. The CLASSPATH is set differently depending on whether you are running on the database server or on the mid-tier with WebLogic Server or UCP. Make sure to use the correct path separator for CLASSPATH on your platform (';' for Windows, ':' otherwise).

The general format for the command line is

java fanWatcher config_type [eventtype … ]

Event Type Subscription

The event type sets up the subscriber to only return limited events. You can run without specifying the event type to see what types of events are returned. When you specify an event name on the command line, the program sets up the subscriber to have a simple match on the event. If the specified pattern occurs anywhere in a notification's header, then the comparison statement evaluates true. The most basic pattern match is an empty string (not null), which matches all notifications. The pattern is enclosed in double quotes (required) and prefixed with “%” to be case insensitive.

Event processing is more complete than shown in this sample. The subscription string is generally composed of one or more comparison statements, each logically related to another with the boolean operators '|' for an OR relationship or '&' for an AND relationship. Parentheses are used to group these comparison statements, and the '!' operator placed before an opening parenthesis negates the evaluated value within.

Each individual comparison statement must be enclosed within double quotes ('"'), and can take one of two basic forms: "pattern" or "name=value". A "pattern" is a simple string match of the notification header: if the specified "pattern" occurs anywhere in a notification's header, then the comparison statement evaluates true. The most basic pattern match is an empty string (not NULL) which matches all notifications.

The "name=value" format compares the ONS notification header or property name with the name against the specified value, and if the values match, then the comparison statement evaluates true. If the specified header or property name does not exist in the notification the comparison statement evaluates false. A comparison statement will be interpreted as a case insensitive when a percent character ('%') is placed before the opening quote. Note that for "name=value" comparison statements, only the value is treated as case insensitive with this option: the name lookup will always be case sensitive. A comparison statement will be interpreted as a regular expression when a dollar sign character ('$') is placed before the opening quote. Standard POSIX regular expressions are supported. To specify a regular expression that is also case insensitive, place the dollar sign and percent sign together and in that order ("$%") before the opening quote.

A special case subscription string composed of only the exclamation point character ('!') signifies that the subscription will not match any notifications.

You might want to modify the event to select on a specific service by using something like

%"eventType=database/event/servicemetrics/<serviceName> "

Running with Database Server 10.2 or later

This approach runs on the database server and connects directly to the local ONS daemon available in the Grid Infrastructure cluster. The FANwatcher utility must be run as a user that has privilege to access the $CRS_HOME/opmn/conf/ons.config, which is used by the ons daemon to start and accessed by this program. The configuration type on the command line is set to “crs”.

# CRS_HOME should be set for your Grid infrastructure
echo $CRS_HOME
java -Doracle.ons.oraclehome=$CRS_HOME fanWatcher crs

Running with WLS 10.3.6 or later using an explicit node list

There are two ways to run in a client environment – with an explicit node list and using auto-ONS. It’s necessary to have ojdbcN.jar and ons.jar that are available when configured for WLS. If you are set up to run with UCP directly, these should also be in your CLASSPATH.

In the first approach, it will work with Oracle driver and database 11 and later (SCAN support came in later versions of Oracle including the jar files that shipped with WLS 10.3.6).

# Set the WLS environment using wlserver*/server/bin/setWLSEnv
CLASSPATH="$CLASSPATH:." # add local directory for sample program
java fanWatcher "nodes=rac1:6200,rac2:6200" database/event/service

The node list is a string of one or more values of the form name=value separated by a newline character (\n).

There are two supported formats for the node list.

The first format is available for all versions of ONS. The following names may be specified.

nodes – This is required. The format is one or more host:port pairs separated by a comma.

walletfile – Oracle wallet file used for SSL communication with the ONS server.

walletpassword – Password to open the Oracle wallet file.

The second format is available starting in database It supports more complicated topologies with multiple clusters and node lists. It has the following names.—this value is a list of nodes representing a unique topology of remote ONS servers. id specifies a unique identifier for the node list. Duplicate entries are ignored. The list of nodes configured in any list must not include any nodes configured in any other list for the same client or duplicate notifications will be sent and delivered. The list format is a comma separated list of ONS daemon listen addresses and ports pairs separated by colon.— this value specifies the maximum number of concurrent connections maintained with the ONS servers. id specifies the node list to which this parameter applies. The default is 3. If true, the list is active and connections are automatically established to the configured number of ONS servers. If false, the list is inactive and is only be used as a fail over list in the event that no connections for an active list can be established. An inactive list can only serve as a fail over for one active list at a time, and once a single connection is re-established on the active list, the fail-over list reverts to being inactive. Note that only notifications published by the client after a list has failed over are sent to the fail over list. id specifies the node list to which this parameter applies. The default is true.

remotetimeout —The timeout period, in milliseconds, for a connection to each remote server. If the remote server has not responded within this timeout period, the connection is closed. The default is 30 seconds.

The walletfile and walletpassword may also be specified (note that there is one walletfile for all ONS servers). The nodes attribute cannot be combined with attributes.

Running with WLS using auto-ONS

Auto-ONS is available starting in Database Before that, no information is available. Auto-ONS only works with RAC configurations; it does not work with an Oracle Restart environment.  Since the first version of WLS that ships with Database 12.1 is WLS 12.1.3, this approach will only work with upgraded database jar files on versions of WLS earlier than 12.1.3. Auto-ONS works by getting a connection to the database to query the ONS information from the server. For this program to work, a user, password, and URL are required. For the sample program, the values are assumed to be in the environment (to avoid putting them on the command line). If you want, you can change the program to prompt for them or hard-code the values into the java code.

# Set the WLS environment using wlserver*/server/bin/setWLSEnv
# Set the credentials in the environment. If you don't like doing this,
# hard-code them into the java program
export password url user
java fanWatcher autoons

fanWatcher Output

The output looks like the following. You can modify the program to change the output as desired. In this short output capture, there is a metric event and an event caused by stopping the service on one of the instances.

** Event Header **
Notification Type: database/event/servicemetrics/otrade
Delivery Time: Fri Dec 04 20:08:10 EST 2015
Creation Time: Fri Dec 04 20:08:10 EST 2015
Generating Node: rac1
Event payload:
VERSION=1.0 database=dev service=otrade { {instance=inst2 percent=50 flag=U
NKNOWN aff=FALSE}{instance=inst1 percent=50 flag=UNKNOWN aff=FALSE} } timestam
p=2015-12-04 17:08:03

** Event Header **
Notification Type: database/event/service
Delivery Time: Fri Dec 04 20:08:20 EST 2015
Creation Time: Fri Dec 04 20:08:20 EST 2015
Generating Node: rac1
Event payload:
VERSION=1.0 event_type=SERVICEMEMBER service=otrade instance=inst2 database=dev db_domain= host=rac2 status=down reason=USER timestamp=2015-12-04 17:

Thursday Nov 19, 2015

WLS Replay Statistics

Starting in the Oracle thin driver, the replay driver has statistics related to replay. This is useful to understand how many connections are being replayed. It should be completely transparent to the application so you won’t know if connection replays are occurring unless you check.

The statistics are available on a per connection basis or on a datasource basis. However, connections on a WLS datasource don’t share a driver-level datasource object so the latter isn’t useful in WLS. WLS 12.2.1 provides another mechanism to get the statistics at the datasource level.

The following code sample shows how to print out the available statistics for an individual connection using the oracle.jdbc.replay.ReplayableConnection interface, which exposes the method to get a oracle.jdbc.replay.ReplayStatistics object. See for a description of the statistics values.

if (conn instanceof ReplayableConnection) {
  ReplayableConnection rc = ((ReplayableConnection)conn);
  ReplayStatistics rs = rc.getReplayStatistics(
  System.out.println("Individual Statistics");

Besides a getReplayStatistics() method, there is also a clearReplayStatistics() method.

To provide for a consolidated view of all of the connections associated with a WLS datasource, the information is available via a new operation on the associated runtime MBean. You need to look-up the WLS MBean server, get the JDBC service, then search for the datasource name in the list of JDBC datasource runtime MBeans, and get the JDBCReplayStatisticsRuntimeMBean. This value will be null if the datasource is not using a replay driver, if the driver is earlier than, or if it’s not a Generic or AGL datasource. To use the replay information, you need to first call the refreshStatistics() operation that sets the MBean values, aggregating the values for all connections on the datasource. Then you can call the operations on the MBean to get the statistics values, as in the following sample code. Note that there is also a clearStatistics() operation to clear the statistics on all connections on the datasource. The following code shows an example of how to print the aggregated statistics from the data source.

public void printReplayStats(String dsName) throws Exception {
  MBeanServer server = getMBeanServer();
  ObjectName[] dsRTs = getJdbcDataSourceRuntimeMBeans(server);
  for (ObjectName dsRT : dsRTs) {
    String name = (String)server.getAttribute(dsRT, "Name");
    if (name.equals(dsName)) {
      ObjectName mb =(ObjectName)server.getAttribute(dsRT,  
      server.invoke(mb,"refreshStatistics", null, null);
      MBeanAttributeInfo[] attributes = server.getMBeanInfo(mb).getAttributes();
      for (int i = 0; i <attributes.length; i++) {
        if(attributes[i].getType().equals("java.lang.Long")) {
            (Long)server.getAttribute(mb, attributes[i].getName()));

MBeanServer getMBeanServer() throws Exception {
  InitialContext ctx = new InitialContext();
  MBeanServer server = (MBeanServer)ctx.lookup("java:comp/env/jmx/runtime");
  return server;
ObjectName[] getJdbcDataSourceRuntimeMBeans(MBeanServer server) 
  throws Exception {
  ObjectName service = new ObjectName( "com.bea:Name=RuntimeService,Type=\");
  ObjectName serverRT = (ObjectName)server.getAttribute(service,  "ServerRuntime");
  ObjectName jdbcRT = (ObjectName)server.getAttribute(serverRT,  "JDBCServiceRuntime");
  ObjectName[] dsRTs = (ObjectName[])server.getAttribute(jdbcRT,
  return dsRTs;

Now run an application that gets a connection, does some work, kills the session, replays, then gets a second connection and does the same thing. Each connection successfully replays once. That means that the individual statistics show a single replay and the aggregated statistics will show two replays. Here is what the output might look like.

Individual Statistics


Looking carefully at the numbers, you can see that the individual count was done before the connections were closed (TotalCompletedRequests=0) and the roll-up was done after both connections were closed. 

You can also use WLST to get the statistics values for the datasource. The statistics are not visible in the administration console or FMWC in WLS 12.2.1.

Thursday Nov 12, 2015

WLS UCP Datasource

WebLogic Server (WLS) 12.2.1 introduces a new datasource type that uses the Oracle Universal Connection Pool (UCP) as an alternative connection pool.  The UCP datasource allows for configuration, deployment, and monitoring of the UCP connection pool as part of the WLS domain.  It is certified with the Oracle Thin driver (simple, XA, and replay drivers). 

The product documentation is at .  The goal of this article  is not to reproduce that information but to summarize the feature and provide some additional information and screen shots for configuring the datasource.

A UCP data source is defined using a jdbc-data-source descriptor as a system resource.  With respect to multi-tenancy, these system resources can be defined at the domain, partition, resource group template, or resource group level. 

The configuration  for a UCP data source is pretty simple with the standard datasource parameters.  You can  name it, give it a URL, user, password and JNDI name.  Most of the detailed configuration and tuning comes in the form of UCP connection properties.  The administrator can configure values for any setters supported by oracle.ucp.jdbc.PoolDataSourceImpl except LogWriter  (see oracle.ucp.PoolDaaSourceImpl) by just removing the "set" from the attribute name (the names are case insensitive).  For example,


Table 8-2 in the documentation lists all of the UCP attributes that are currently supported, based on the UCP jar that ships with WLS 12.2.1.

There is some built-in validation of the (common sense) combinations of driver and connection factory:


Factory (ConnectionFactoryClassName)

oracle.ucp.jdbc.PoolDataSourceImpl (default)






To simplify the configuration, if the "driver-name" is not specified, it will default to oracle.ucp.jdbc.PoolDataSourceImpl  and the ConnectionFactoryClassName connection property defaults to the corresponding entry from the above table.

Example 8.1 in the product documentation gives a complete example of creating a UCP data source using WLST.   WLST usage is very common for application configuration these days.

Monitoring is available via the  This MBean extends JDBCDataSourceRuntimeMBean so that it can be returned with the list of other JDBC MBeans from the JDBC service for tools like the administration console or your WLST script.  For a UCP data source, the state and the following attributes are set: CurrCapacity, ActiveConnectionsCurrentCount, NumAvailable, ReserveRequestCount, ActiveConnectionsAverageCount, CurrCapacityHighCount, ConnectionsTotalCount, NumUnavailable, and WaitingForConnectionSuccessTotal.

The administration console and FMWC make it easy to create, update, and monitor UCP datasources.

The following images are from the administration console. For the creation path, there is a drop-down that lists the data source types; UCP is one of the choices.  The resulting data source descriptor datasource-type set to "UCP".

The first step is to specify the JDBC Data Source Properties that determine the identity of the data source. They include the datasource names, the scope (Global or Multi Tenant Partition, Resource Group, or Resource Group Template) and the JNDI names.

The next page handles the user name and password, URL, and additional connection properties. Additional connection properties are used to configure the UCP connection pool. There are two ways to provide the connection properties for a UCP data source in the console. On the Connection Properties page, all of the available connection properties for the UCP driver are displayed so that you only need to enter the property value.  On the next page for Test Database Connection, you can enter a propertyName=value directly into the Properties text box.  Any values entered on the previous Connection Properties page will already appear in the text box.  This page can be used to test the specified values including the connection properties.

The Test Database Connection page allows you to enter free-form values for properties and test a database connection before the data source configuration is finalized. If necessary, you can provide additional configuration information using the Properties, System Properties, and Encrypted Properties attributes.

The final step is to target the data source. You can select one or more targets to which to deploy your new UCP data source. If you don't select a target, the data source will be created but not deployed. You will need to deploy the data source at a later time before you can get a connection in the application.

For editing the data source, minimal tabs and attributes are exposed to configure, target, and monitor this data source type.

The capabilities in FMWC are similar to the administrative console but with a different look and feel.

If you select JDBC Data Sources from the WebLogic Domain drop-down, you will see a list of existing data sources with their associated data source type, scope, and if applicable RG, RGT and Partition.