Monday Mar 09, 2015

WLS JDBC Driver Patching

The handling of Oracle driver jar patches is complicated but getting sorted out. This article tries to gather the information in one place with pointers to more details.  There are a few patches that are still not available, marked as TBA (To Be Available) in the tables below.  As these files become available, this page will be updated.

WLS 10.3.6, 12.1.1, and 12.1.1 shipped Database jar files.  However these are non-standard versions of the jars with additional bug fixes and enhancements to support WLS.  That means that you can't just drop in an patch or upgrade to using standard released jar files. Although support is required to provide patches as needed, it will be difficult and the recommendation is to upgrade to a special patch that contains and all of the patches and enhancements in the database jar files shipped with WLS. It's further complicated because WLS started using the Oracle Universal Installer in 12.1.2, requiring a different patch format.

WLS 10.3.6, 12.1.1, 12.1.2, and 12.1.3 also support running with Oracle Database 12c client jar files. For WLS 10.3.6 through 12.1.2, the jar files must be manually installed; there is no installer or patch to automate this upgrade. To get patches, you must be running with the Database jar files; WLS patches will not be generated for the Database jar files. WLS 12.1.3 ships with a pre-release version of Database driver jar files and a patch will be available to upgrade to the production version of these files. After this upgrade, standard database Oracle patch files will work as expected for WLS 12.1.3 (and WLS 12.1.2 with a manual upgrade to database jar files).

Patching the installed Oracle Driver

WLS Release

Oracle Driver Install

Database Jar

Patch Strategy


10.3.6 WLS patch

12.1.1 WLS patch

12.1.2 opatch

Patch Request 18557114 for bug 19477203


Pre- opatch to bring up to shipping; standard opatch for additional bug fixes

Patch 20741228: 12.1.3 WLS UPGRADE TO JDBC RELEASE

Running with the Database 12c Driver

WLS Release installation

Database Jar Patch Strategy

Documentation for Installation
Documentation for patching


Manual installation of WLS patch


Manual installation of WLS patch


Manual installation of opatch

Standard patch procedure


Pre- installed; Patch to bring up to shipping opatch

Patch 20741228: 12.1.3 WLS UPGRADE TO JDBC RELEASE

Standard patch procedure

On a related topic, updating non-Oracle driver jar files is covered by the following note.

This includes the DataDirect and MySQL drivers that are shipped in the kit. The jar file is backed up and removed, the new file installed, and the CLASSPATH adjusted if the jar name changes.

You'll notice that releases earlier than WLS 10.3.6 are not discussed.  For releases earlier than WLS 10.3.4, they depend only on the ojdbcN.jar file.  It's possible that they will work with the jar file but that hasn't been certified and they are not still in error correction support.  For WLS 10.3.4 or 10.3.5, it depends not only on a specific ojdbc jar file but also ONS/UCP jars that have the package names renamed.  They will likely not work correctly with the jar file (certainly not Active GridLink).  Since these releases ended error correction support in May 2012, you will need to upgrade to WLS 10.3.6 or 12.1.x to use later driver jar files.

Monday Aug 04, 2014

Setting V$SESSION for a WLS Datasource

Every Oracle database connection runs in the context of a database process called a session.  There is a v$session view that contains a lot of information about the database sessions that are active.. By default when you use the Oracle Thin Driver, the value v$session.program is set to "JDBC Thin Client".  That separates out the Java applications from sqlplus and the scores of database background programs but doesn't provide much additional information since all of the Java connections look the same.  It's easy to set this and some other values on v$session using connection properties on the Oracle Thin driver.  The following connection properties are supported:  v$session.osuser, v$session.process, v$session.machine, v$session.terminal, and v$session.program.  Setting these will set the corresponding value on the session on the database side.  These values are then available from the v$session view.

The simple approach is to hard-code a value into a normal connection Property.  That's fine if you want to associate a fixed value with a data source.  It's more interesting if you dynamically set the value at runtime. For example, if there are multiple servers running within a domain and the information needs to be server specific, a normal cluster deployment with one fixed value is not useful, and the option of deploying the DataSource to every server individually and then hand-editing each one's descriptor with unique values for these properties is not manageable. You can easily handle this using a System Property.  The value that is specified is taken to be a Java system property that you set on the command line of the application server.  It is retrieved using System.getProperty() and set as the value of the connection property.    There's a new Encrypted Property in WLS 12.1.3; I'll write another article about that.

If you use bin/ to start the server, it will put -Dweblogic.Name=${SERVER_NAME}on the command line.  If you set the v$session.program System Property connection property to "weblogic.Name", your session program value will match the WLS server that is making the connection. 

You can set connection properties by editing the data source configuration on the "Connection Pool" tab in the WebLogic administration console.  Properties are set in the Properties and System Properties text boxes.  Let's say that I set four of the values to test values and one to a system property, generating the descriptor fragment as follows.


Alternatively, you could set these values using on-line or off-line WLST.

Here's a fragment of an off-line WLST script.

cd('/JDBCSystemResource/myds/JdbcResource/myds') cd('JDBCDriverParams/NO_NAME_0') cd('Properties/NO_NAME_0') create('v$session.program','Property') cd('Property') cd('v$session.program') set('SysPropValue', 'weblogic.Name')

If $SERVER_NAME is myserver and I then go to run a query, here is the resulting output.

SQL> select program, osuser, process, machine, terminal 
  from v$session where program = 'myserver';
myserver test1 test2 test3 test4

If the server names aren't obvious enough, you could set the program to "WebLogic Server $SERVER_NAME".  You could set -Djdbc.process=<PID> to tie connections to a specific WLS server process. You might want to add the WLS data source name to the program value.  You could set osuser to the Java value "".

 Using system properties can make this a powerful feature for tracking information about the source of the connections, especially for large configurations.

Friday Aug 01, 2014

Using 3rd party JDBC Jar Files

If you look at the documentation for adding a 3rd party JDBC jar file, it recommends adding it to the system classpath by updating  See  However, there more options with some advantages and disadvantages.  To use some of these options, you need to know something about application archive files.

The general precedence of jar files in the classpath is the following:

system classpath >> DOMAIN_HOME/lib >> APP-INF/lib >> WEB-INF/lib

The highest priority is given to jar files on the system classpath.  The advantage of using the system classpath is that it works for all environment including standalone applications. 

The rest of the options only work for applications deployed in the application server.  That includes the WLS administration console and WLST but not an application like 'java utils.Schema'.  These approaches work on WLS 10.3.6 and later.

The simplest way to add a jar file when you don't need it accessed in a standalone application is to drop it in DOMAIN_HOME/lib.  This is an easy way to access a driver for a data source in the application server. It won't work for jar files that are shipped with WLS and already part of the system classpath because the system classpath takes precedence.  That's why this article focuses on "3rd party" jar files.  More information and recommendations about using DOMAIN_HOME/lib can be found at .

APP-INF/lib will only work in an EAR file and applies to all applications in the EAR file.  APP-INF is located at the root directory in the EAR file at the same level as META-INF.  This approach and the next have the advantage of packaging the driver with the application archive.  However, it's not efficient if the driver is needed by more than one archive and you probably should use DOMAIN_HOME/lib.

WEB-INF/lib only works in a WAR file and must be located at the root directory in the WAR file.  It only applies to the corresponding web application.  By using a WEB-INF/weblogic.xml file that has "<prefer-web-inf-classes>true</prefer-web-inf-classes>", the jar files in WEB-INF/lib take precedence over others (that is, the default precedence defined above is overridden).  This is useful if you want different versions of the driver for different applications in various WAR files.   Note that starting in WLS 12.1.1 with Java EE 6 support, it's possible to have a standalone WAR file that is not embedded in an EAR file.

Here's an indented list that attempts to capture the hierarchy of the files and directories described above in EAR and WAR files.

Application (ear)

  • Web module (war)
    • WEB-INF/lib
    • WEB-INF/web.xml
    • WEB-INF/weblogic.xml
  • EJB module
  • APP-INF/lib

There is sample code for a WAR file at this link.  The data source definition can appear either as a descriptor in web.xml or as an annotation in the java file.  weblogic.xml can be used to set prefer-web-inf-classes appropriately.  The @Resource annotation is used to reference the data source in the application code.  This program prints the version of the driver.  You can play with two versions of the Derby driver to see how the precedence works.

Note:  You can't have two versions of the same jar in both domain/lib or the system classpath and  WEB-INF/lib or APP-INF/lib with prefer-web-inf-classes or prefer-application-packages set.  That is, either you use domain/lib or system classpath to get it into all applications in the domain or you use it embedded in the application but not both.  If you don't follow this restriction, it's possible (depending on the jar, the version changes, and the order in which the jars are referenced) that a ClassCastException will occur in the application.

The choice of where you locate the JDBC jar file depends on your application.  For standalone application access, use the system classpath.  For simple access in all or most applications, use DOMAIN_HOME/lib.  For access across an EAR file, use APP-INF/lib.  For access within a single WAR file, use WEB-INF/lib.

Monday Jun 16, 2014

Detailed Analysis of a Stuck Weblogic Execute Thread Running JDBC Code

The following thread was extracted from a thread dump taken on a JVM instance running WebLogic Server.

In this post I will deconstruct this thread and describe the data it contains and the potential issues it may illuminate.

[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' id=73 idx=0x128 nid=13410 prio=1 alive, in native, daemon Method) Source) Source) Source) Source)









java.sql.DriverManager.getConnection( Source)

This thread is considered Stuck by WebLogic because it's been running for over the time defined in MaxStuckThreadTime (600 seconds by default). Weblogic Server waits for this time to be reached before marking a thread as stuck if the thread is still working after this time.  If you deem that 600 seconds is too long before a running thread is considered stuck then you can change the value of the this parameter using the WebLogic Console (as shown below), or use setMaxStuckThreadTime from the ServerFailureTriggerMBean interface.

An error including BEA-000337 will be logged in the server log file when the thread changes its status to stuck but the server won't take further action on this thread.  However, you might want to investigate why this thread is taking such a long time to process the work assigned to it.

Lets now look at the thread itself.  From its header, you can spot the thread identifier (2 in this example) and the queue where it originated.  The term Self-tuning indicates that the associated thread pool consistently checks the overall throughput to determine if the thread count should change.

id (or tid) is the thread identifier, a unique process-wide number that identifies this thread within the JVM process.  This id is unique but can be reused by another thread once this thread is terminated. 

nid is the OS-level native thread identifier.  It can be used effectively to correlate with high CPU usage threads identified at the OS level (e.g. with Linux watch command).  See Unexpected High CPU Usage with WebLogic Server (WLS) Support Pattern (Doc ID 779349.1) for detailed steps.

idx is the thread index in the threads array.

prio refers to the thread priority, a number inherited from the thread that created it.  You can learn more about thread priorities at Class Thread but basically threads with higher priority are executed in preference to threads with lower priority.

alive refers to the fact that this thread has not ended yet and is still active.  

in native means that the thread uses the operating system's native ability to manage multi-threaded processes.  

daemon indicates that this thread can't prevent the JVM from exiting.

The thread header is accompanied with a full java stack which lists each method and class invoked since the first assignement to the thread up to its most recent action. This thread consists of obtaining a connection to an Oracle database using a Type 4 JDBC driver and then issuing a call but getting no response from the back end database server.  The database failed to respond, and the thread has probably been in the same waiting mode (unchanged and not progressing java stack) for a long time since it's now considered stuck; the most recent invocation being

At this point the back end database needs to be checked to understand why it's not responding to the java thread request.  A starting point could be to query v$session to find potential blocking sessions at the database level.

Blocking sessions occur when one session holds an exclusive lock on an object and doesn't release it.

Needless to say, the communication with the database needs to be confirmed as healthy with none to very limited latency.  Firewall issues should be ruled out as well.  Firewalls could time out idle sockets used by JDBC connections to the database and lead to not closing the socket the JDBC driver is using.

Monday Nov 18, 2013

Navigating Through Diagnostic Data Options

In this new post I will talk about diagnostic data and more importantly what data and source of data are needed to troubleshoot and resolve some specific common problems.

WebLogic Server along with the JVM offers various ways to collect logs, debug traces, diagnostic images, thread dumps, and much more. However, useful data can easily get buried with lots of noise, unneeded data that won't help resolving a particular problem.

Performance Issues

We will start with performance issues such as long running requests or server hangs.  The best way to narrow down where the issue lies is to dump the java threads and parse them through a thread dump analyzer tool such as ThreadLogic.  Looking at the server log files, or enabling some debugging features would be premature since the first step is to identify the thread(s) that are stuck to determine what is being executed and relevant JEE resources. The log could also alert about stuck threads once they have been stuck for over the maximum allowed time (see MaxStruckThreadTime) but unless thread dumps are examined, it won't be possible to determine possible patterns of issues and threads' relationship around locked objects.

Memory Leaks

Another common performance issue involves memory leaks.  If you encounter OutOfMemory java.lang exceptions you should issue a heap dump and analyze it with a tool such as VisualVM or the Eclipse MAT.  See my post on Heap Dumps analysis for more details. Heap dumps are snapshots of memory certain times.  Heap dumps are to the Java process memory (non native) what thread dumps are to Java process threads.  Enabling -verbose:gc is also an effective way to monitor heap usage at runtime.  The output of the GC stats can be redirected to a file so that not to clog the server logs.  This can be done by setting -Xloggc with HotSpot JVM.  It's recommended to include -XX:+PrintGCDetails, XX:+PrintGCDateStamps (for Java 6 Update 4 and later JVMs) and -XX:+PrintGCTimeStamps to log time stamps and detailed GC activities as well.  Troubleshooting guide for HotSpot VM details these flags among others useful flags for diagnosis purposes.

Applications, 3rd parties, and configuration issues

The following chart summarizes what could be collected to debug each of the listed WLS component.  This list is by no means exhaustive.  Many more debug flags could be used for these listed components.  This chart contains examples of useful debug options and links to additional resources.

For the following, Redirect stdout and stderr logging should be enabled.  This can be done via the Administration Console in the advanced section of the servers logging tab or at startup in the java command line with -Dweblogic.StdoutDebugEnabled=true.  In addition, the severity level of the log file should be set to debug.

WLS comp. or 3rd party resource

Examples of Debug options and flags

MyOracleSupport  related resources

(login required)

 EJB/Web Serv.



To log SOAP requests and response messages

Trace in servers log files 

Troubleshooting EJBs Issues

How to enable Debugging for Weblogic EJB

How To Debug WLS Application Container Problems

Troubleshooting Oracle Web Services 11g



To log info on load, store events and transaction records into servers log files

See Debugging JMS for more JMS debugging options

Troubleshooting JMS Info Center



Trace in servers log files 

Troubleshooting SSL Security

-Dweblogic.debug.DebugReplication for high level replication info

-Dweblogic.debug.DebugClusterAnnouncements to log info on announcement, StateDump and attributes messages sent or received by multicast

Trace in servers log files  

Debug RJVM or Intra-Cluster Communication Problems

weblogic.JTAXA & weblogic.JTA2PC for runtime issues, example:

JAVA_OPTIONS="-Dweblogic.Debug=weblogic.JTAXA, weblogic.JTA2PC"

Trace in servers log files  

Investigating Transaction Problems


to print information about JDBC methods invoked with their arguments, return values, and thrown exceptions

See JDBC Debugging Scopes under Monitoring JDBC Resources

Trace in servers log files  

Investigating JDBC Issues

JDBC Debugging Scopes

Oracle JDBC Driver (*)

-Doracle.jdbc.Trace=true to enable jdbc logging


Trace is file defined in oracle.jdbc.LogFile

Enable Oracle JDBC Driver Debug in WLS
Deployment (Classloader) Set the following options to true to report which classes are getting loaded







Trace in servers log files

Deployment Problems - Enabling Debug

Proxy plug-in

Debug="ALL" in the proxy configuration file

(iisproxy.ini, httpd.conf or obj.conf)

Trace in file set in WLLogFile defined inside proxy configuration file

Common Diagnostic Process for Proxy Plug-In Problems

(*)  Enabling JDBC Driver debug flags is verbose so it's recommended to set logging properties to appropriate levels such as FINE, SEVERE, FINEST, INFO etc. based upon the debugging requirement.


Diagnosing and troubleshooting also require a good understanding of the Weblogic domain environment, its configuration, patching level, JVM parameters and also an easy navigation through various logs and configuration files.  For this, RDA is a great diagnosing tool because it collects all this data indiscriminately.  One of the numerous benefits of RDA is that it doesn't require any complex setting and can collect data from each distinctive log file, xml repository and scripts.  So, RDA reports are of great assistance while working directly with Oracle support because that allow the engineer assigned to a service request to quickly understand the environment under which a specific issue is occurring.  This generally helps expedite the resolution of the SR.


The Weblogic Diagnostic Framework can be used to collect metrics, setup watch and notifications, and to define instrumentation.  With WLS 12.1.2, you can set the level (Low, Medium or High) of built-in modules that will gather metrics, or disable them completely.  WLDF collects runtime statistics on JDBC, JTA and JVM and more.  You can then use the Monitoring Dashboard to view and navigate through the collected data.  WLDF is not an alternative to debug options.  Debug options activate tracing that is coded as part of WLS whereas WLDF collects and monitors runtime mbeans values and can activate notifications with defined rules.  They are used for different purposes.  

The WLDF built-in configurations (Low, Medium, & High) can be cloned into a new configuration, and used as templates for creating custom WLDF configurations.  The Low volume built-in is enabled by default in production mode.  One new feature with WLS 12.1.2 is Runtime control which gives the ability to activate or deactivate diagnostic system modules dynamically at runtime wihtout making any domain configuration change.


JFR recording data can also be captured through WLDF diagnostic images based on WLDF watch rules so that JVM runtime system information can be analyzed along with recording of WLS components diagnostic data.  Once extracted from the diagnostic image capture, the JFR file can be analyzed using Java Mission Control.


WLST diagnostic commands can be used to extract data from the diagnostic archive (event and harvested metric data) in either XML form or CSV, based on the function you use.  You can also dump the diagnostics data from a harvester to a local file.

The following WLST functions allow to capture an image and to download image files from a remote system over WLST:

One major difference between the diagnostic framework and the debug options is that debugs are enabled so that an issue can be reproduced and traced whereas the diagnostic image capture is used as a server-level dump for post-failure analysis in a similar way heap dumps are used post memory leak failures.

Monday Sep 16, 2013

Ensuring high level of performance with WebLogic JDBC

In this post you will find some common best practices aimed at ensuring high levels of performance with WebLogic JDBC.  However, rather than just throwing some tips, I will detail why each recommendation is beneficial to JDBC and Weblogic Server performance.

  • Use Oracle JDBC thin driver (Type 4) rather than OCI driver (Type 2)

The Oracle JDBC thin driver is lightweight (easy to install and administrate), platform-independent (entirely written in Java), and provides slightly higher performance than the JDBC OCI (Oracle Call Interface) driver.  The thin driver does not require any additional software on the client side.  Oracle JDBC FAQ stipulates that the performance benefit with the thin driver is not consistent and that the OCI driver can even deliver better performances in some scenarios.

  • Use PreparedStatements objects rather than Statement 
With PreparedStatements, compiled SQL statements will be kept in cache and only be executed once against the database.  As a result, unnecessary parsing and round-trips to the database will be avoided when the same statement is used later within the same connection.  The statement cache size defines the total number of prepared statement that can be made with a single connection from the Datasource.

  • Close all JDBC resources in a finally Block

This include ResultSet, Connection, Statement and Prepared Statement objects and to avoid potential memory leaks.  The connection.close() won't necessarily automatically clean up all the other objects because the implementation of close() may differ between JDBC drivers.  Also, JDBC objects not properly closed could lead to this error:

java.sql.SQLException: ORA-01000: maximum open cursors exceeded.

If you don't explicitly close Statements and ResultSets right away, cursors may accumulate and exceed the maximum number allowed in your DB before the Connection is closed. 

  • Set Shrink frequency to 0 in production environments

By disabling Shrink frequency you will not allow waits before shrinking a connection pool that has incrementally increased to meet demand.  The Shrink Frequency parameter is used to specify the number of seconds to wait before shrinking a connection pool, given that ShrinkingEnabled is kept at its default value, or set to true.  Weblogic shrinks a connection pool by reducing the number of connections to the greater of either the minimum capacity or the number of connection in use and thus frees up some resources.  In development we can afford to keep the no-longer-used connection active rather than immediately returning them to the pool.  

  • Consider skipping the SQL-query test when Test Connections on Reserve is enabled

When Test Connections on Reserve is enabled (see Advanced Connection Pool Configuration in the console), the Weblogic Server checks a database connection prior to returning the connection to a client to avoid passing an invalid connection in the application.  Since this operation could impact the performance, it's recommended to use Seconds to Trust an Idle Pool Connection (set to 10 seconds by default) that defines how long WebLogic Server will trust the connection and therefore skip the connection test after a connection has been proven valid.

  • Enable Pinned-to-Thread
Disabled by default, this option can improve performance by enabling threads to keep a pooled database connection even after the application closes the logical connection.  This eliminates potential contention between threads getting a database connection.  However, this parameter should be used with great care because the connection pool maximum capacity is ignored when pinned-to-thread is enabled.  A connection no longer used will not return to the pool but will stay linked to the thread, and no shrinking can apply to that pool.
  • Ensure that Maximum Thread Constraint property doesn't exceed the number of database connection

This property (See Environment Work Manager in the console) will set a maximum number of possible concurrent requests. If it exceeds the number of database connection then some threads might end up waiting until existing database connection are made available.

Visit Tuning Data Source Connection Pools and Tuning Data Sources for additional parameters tuning in JDBC data sources and connection pools to improve system performance with Weblogic Server 12.1.2, and Performance Tuning Your JDBC Application for application-specific design and configuration. 

Wednesday Jul 31, 2013

JMS JDBC Store Performance Using Multiple Connections

This article is a bit different than the normal data source articles I have been writing because it's focused on an application use of WebLogic Server (WLS) data sources, although the application is still part of WLS.  Java Messaging Service (JMS) supports the use of either a file store or a JDBC store to store JMS persistent messages (the JDBC store can also be used for Transaction Log information, diagnostics, etc.).  The file store is easier to configure,generates no network traffic, and is generally faster.  However, the JDBC store is popular because most customers have invested in High Availability (HA) solutions, like RAC, Data Guard or Golden Gate,  for their database so using a JDBC store on the database makes HA and migration much easier (for the file store, the disk must be shared or migrated). Some work has been done in recent releases to improve the JDBC store performance and take advantage of RAC clusters.

It's obvious from the JDBC store configuration that JMS uses just a single table in the database.  JMS use of this table is sort of like a queue so there are hot spots at the beginning and end of the table as messages are added and consumed - that might get fixed in WLS in the future but it is a consideration for the current store performance.  JMS since the beginning has been single threaded on a single database connection.  Starting in WLS 10.3.6 (see this link), the store can run with multiple worker threads each with its own connection by setting Worker Count on the JDBC Persistent Store page in the console.  There are no documented restrictions or recommendations about how to set this value  Should we set it to the maximum allowed of 1000 so we get a lot of work done?  Not quite ...

Since we have contention between the connections, using too many connections is not good.  To begin with, there is overhead in managing the work among the threads so if JMS is lightly loaded, it's worse to use multiple connections.  When you have a high load, we found that for one configuration, 8 threads gave the best performance but 4 was almost as good at half the resources using the Oracle Thin driver on an Oracle database (more about database vendors below).  Optimization for queues with multiple connections is  a big win with some gains as high as 200%.  Handling a topic is another ... well, topic.  It's complicated by the fact that a message can go to a single or multiple topics and we want to aggregate acknowledgements to reduce contention and improve performance.  Topic testing saw more modest gains of  around 20%, depending on the test.

How about different data source types? It turns out that when using a RAC cluster and updates are scattered across multiple instances, there is too much overhead in locking and cache fusion across the RAC instances.  That makes it important that all of the reserved connections are on a single RAC instance.   For a generic data source, there is nothing to worry about - you have just one node.  In the case of multi data source (MDS), you can get all connections on a single instance by using the AlgorithmType set to "Failover" (see this link ).   All connections will be reserved on the first configured generic data source within the MDS until a failure occurs, then the failed data source will be marked as suspended and all connections will come from the next generic data source in the MDS.  You don't want to use AlgorithmType set to "Load-Balancing".  In the case of Active GridLink (AGL), it's actually difficult to get connection affinity to a single node and without it, performance can seriously degrade.  Some benchmarks saw performance loss of 50% when using multiple connections on different instances.  For WLS 10.3.6 and 12.1.1, it is not recommended to use AGL with multiple connections.  In WLS 12.1.2, this was fixed so that JMS will reserve all connections on the same instance.  If there is a failure, all of the reserved connections need to be closed, a new connection is reserved using Connection Runtime Load Balancing (RCLB), hopefully on a lightly loaded instance), and then the rest of the connections are reserved on the same instance.  In one benchmark, performance improved by 200% when using multiple connections on the same instance.

How about different database vendor types?  Your performance will vary based on the application and the database.  The discussion above regarding RAC cluster performance is interesting and may have implications for any application that you are moving to a cluster.  Another thing that is specific to the Oracle database is indexing the table for efficient access by multiple connections.  In this case, it is recommended to use a reverse key index for the primary key.  The bytes in the key are reversed such that keys that normally would be grouped because the left-most bytes are the same are now distributed more evenly (imagine using a B-tree to store a bunch of sequential numbers with left padding with 0's, for example). 

Bottom line: this feature may give you a big performance boost but you might want to try it with your application, database, hardware, and vary the worker count.

Monday Jul 22, 2013

Part 2 - 12c Database and WLS - Application continuity

I introduced WLS support for 12c database in this blog link and described one of the prerequisites for Application Continuity (AC) at this blog link.

When an error occurs on a connection, it would be nice to be able to keep processing on another connection - that's what AC does.  When on a single node database, getting another connection means that the database and network are still available, e.g., maybe just a network glitch.  However, in the case of a Real Application Cluster (RAC), chances are good that even if you lost a connection on one instance you can get a connection on another instance for the same database.  For this feature to work correctly, it's necessary to ensure that all of the work that you did on the connection before the error is replayed on the new connection - that's why AC is also called replay. To replay the operations, it's necessary to keep a list of the operations that have been done on the connection so they can be done again.  Of course, when you replay the operations, data might have changed since the last time so it's necessary to keep track of the results to make sure that they match.  So AC may fail if replaying the operations fail.

You can read the WLS documentation, an AC white paper, and the Database documentation to get the details. This article gives an overview of this feature (and a few tidbits that the documentation doesn't cover).

To use this feature, you need to use both the 12c driver and a 12c database.  Note that if you use an driver and/or database, it might appear to work but replay will only happen for read-only transactions (the 11g mode is not supported but we don't have a mechanism to prevent this).

You need to configure a database service to run with AC.  To do that, you need to set the new 12c service attributes FAILOVER_TYPE=TRANSACTION and COMMIT_OUTCOME=true on the server side.

There's not much to turning on AC in WLS - you just use the replay driver "oracle.jdbc.replay.OracleDataSourceImpl" instead of "oracle.jdbc.OracleDriver"  when configuring the data source.  There's no programming in the application needed to use it.  Internal to WLS, when you get a connection we call the API to start collecting the operations and when you close a connection we call the API to clear the operation history.

WLS introduced a labeling callback for applications that use the connection labeling feature.  If this callback is registered, it will be called when a new connection is being initialized before replay.  Even if you aren't using labeling, you might still want to be called and there is a new connection initialization callback that is for replay (labeling callback trumps initialization callback if both exist).

It sounds easy and perfect - what's the catch?   I've already mentioned that you need to get rid of references to concrete classes and use the new Oracle interfaces.  For some applications that might be some significant work.  I've mentioned that if replaying the operations fails or is inconsistent, AC fails.  There are a few other operations that turn off AC - see the documentation for details. One of the big ones is that you can't use replay with XA transactions (at least for now).   Selecting from V$instance or sys_context  or other test traces for test instrumentation needs to be in callouts as the values change when replayed.  If you use sysdate or systimestamp, you need to grant keep date time to your user.

We are also tracking two defects - AC doesn't work with Oracle proxy authentication and it doesn't work with the new DRCP feature. 

There's another more complex topic to consider (not currently in the current WLS documentation).  By default, when a local transaction is completed on the connection, replay ends.  You can keep working on the connection but failure from that point on will return an error to the application. This default is based on the service attribute SESSION_STATE_CONSISTENCY with a value of DYNAMIC.  You can set the value to STATIC if your application does not modify non-transactional session state (NTSS) in the transaction.  I'm not sure how many applications fall into this trap but the safe thing is to default to dynamic.  I'll include such a code fragment below.  A related common problem that people run into is forgetting to disable autocommit, which defaults to true, and the first (implicit) commit turns off replay if SESSION_STATE_CONSISTENCY is set to DYNAMIC.

It's important to know how to turn on debugging so that if a particular sequence doesn't replay, you can understand why.  You simply need to run with the debug driver (ojdbc6_g.jar or ojdbc7_g.jar) and run with -Dweblogic.debug.DebugJDBCReplay (or turn this debug category on in the configuration).

AC won't replay everything and you still need to have some application logic to deal with the failures or return them to the end user.  Also, there's some overhead in time and memory to keep the replay data.  Still, it seems like a great feature for a lot of applications where you don't need to change anything but the driver name and you can avoid showing an error to the end user or simplify some recovery logic.

P.S. Confused about NTSS? So was I.  Examples of non-transactional session state that can change at run-time are ALTER SESSION, PL/SQL global variables, SYS_CONTEXT, and temporary table contents.  Here's an example of a PL/SQL global variable.  Imagine a package with the following body:

 current_order number := null;
 current_line number;
 procedure new_order (customer_id number) is
  current_order := order_seq.nextval;
  insert into orders values (current_order, customer_id);
  current_line := 0;
 end new_order;
 procedure new_line (product_id number, count number) is
  current_line := current_line + 1;
  insert into order_lines values (current_order, current_line,product_id, count);
 end new_line;
end order;

and a psuedo-code sequence in WLS like this:
exec "begin order.new_order(:my_customer_id); end;"
exec "begin order.new_line(:my_product_id, :my_count); end;"
<DB server failure and failover>

In this scenario, we won't replay the first transaction, because it's already committed and we'd end up with two orders in the database. But if we don't replay it, we lose the order_id, so the new_line call won't work. So we have to disable replay after the commit.  If you can guarantee that no such scenario exists, you can mark the service session-state as static.

Thursday Jul 11, 2013

Using Oracle JDBC Type Interfaces

One of the hot new features in Oracle database 12c is Application Continuity (AC).  The feature basically will detect that a connection has gone bad and substitute a new one under the covers (I'll talk about it more in another article).  To be able to do that, the application is given a connection wrapper instead of the real connection.  Wrappers or dynamic proxies can only be generated for classes based on interfaces.  The Oracle types, like REF and ARRAY, were originally introduced as concrete classes.   In WLS 10.3.6 and/or the JDBC driver, there are new interfaces for the Oracle types that you will need to use to take advantage of this new AC feature.

First, some history so that you understand the needed API changes.  In the early days of WebLogic data source, any references to vendor proprietary methods were handled by hard-coded references.  Keeping up with adding and removing methods was a significant maintenance problem. At the peak of the insanity, we had over one thousand lines of code that referenced Oracle-proprietary methods and the server could not run without an Oracle jar in the classpath (even for DB2-only shops).  In release 8.1 in March 2003, we introduced wrappers for all JDBC objects such that we dynamically generated proxies that implement all public interface methods of, and delegate to, the underlying vendor object.  The hard-coded references and the maintenance nightmare went away and just as importantly, we could provide debugging information, find leaked connections, automatically close objects when the containing object closed, replace connections that fail testing, etc.  The Oracle types were concrete classes so proxies were generated for these classes implementing the WLS vendor interfaces*.  Applications can cast to the WLS vendor interfaces or use getVendorObj to access the underlying driver object. Later, we added an option to unwrap data types, with a corresponding loss of functionality like no debug information.

Although the focus of this article is Oracle types, the dynamic proxies work for any vendor.   For example, you can cast a DB2 connection to use a proprietary method  (("myname").

Starting with Oracle driver, the database team needed wrappers for the new AC feature and introduced new interfaces.  For WebLogic data source users, that's good news - no more unwrapping, the weblogic.jdbc.vendor package is no longer needed, and it's all transparent.  Before you go and change your programs to use the new Oracle proprietary interfaces, the recommended approach is to first see if you can just use standard JDBC API's.  In fact, as part of defining the new interfaces, Oracle proprietary methods were dropped if there was an equivalent standard JDBC API or the method was not considered to add significant value.  This table defines the mapping.  The goal is to get rid of references to the first and second columns and replace them with the third column.

" align="center">Old Oracle types
Deprecated WLS Interface
New interfaces

This is a job for a shell hacker!  Much of it can be automated and the compiler can tell you if you are referencing a method that has gone away - then check if the missing method is in the equivalent jdbc.sql interface (e.g., getARRAY() becomes the JDBC standard getArray()).

You can take a look at a sample program that I wrote to demonstrate all of these new interfaces at (note that this is actually a ".java" program). It covers programming with all of these Oracle types.  While use of Blob and Clob might be popular, Ref and Struct might not be used as much.  The sample program shows how to create, insert, update, and access each type using both standard and extension methods.  Note that you need to use the Oracle proprietary createOracleArray() instead of the standard createArrayOf(). Although the sample program doesn't use the standard createBlob() or createClob(), these are supported for the Oracle driver.

The API's can be reviewed starting with the version of the Javadoc.  A copy is available at .

This is a first step toward using Application Continuity.  But it's also a good move to remove Oracle API's that will eventually go away and use standard JDBC interfaces and new Oracle interfaces.

Tuesday Jul 02, 2013

Part 1 - 12c Database and WLS - Overview

The download of Oracle 12c database became available on June 25, 2013.  There are some big new features in 12c database and WebLogic Server will take advantage of them. Immediately, we will support using 12c database and drivers with WLS 10.3.6 and 12.1.1.  With WLS 12.1.2, available July 11, 2013 (join the Online Launch Event on July 31st), additional functionality is supported (those rows in the table below with all "No" values will get a "Yes).  The following table maps the Oracle 12c Database features supported with various combinations of currently available WLS releases, 11g and 12c Drivers, and 11g and 12c Databases.

Feature WebLogic Server 10.3.6/12.1.1 with 11g drivers and 11gR2 DB WebLogic Server 10.3.6/12.1.1 with 11g drivers and 12c DB WebLogic Server 10.3.6/12.1.1 with 12c drivers and 11gR2 DB WebLogic Server 10.3.6/12.1.1 with 12c drivers and 12c DB
JDBC replay No No No Yes (Active GridLink only in 10.3.6, add generic in 12.1.1)
Multi Tenant Database No Yes (except set container) No Yes (except set container)
Dynamic switching between Tenants No No No No
Database Resident Connection pooling (DRCP) No No No No
Oracle Notification Service (ONS) auto configuration No No No No
Global Database Services (GDS) No Yes (Active GridLink only) No Yes (Active GridLink only)
JDBC 4.1 (using ojdbc7.jar files & JDK 7) No No Yes Yes

 The My Oracle Support (MOS) document covering this is "WebLogic Server 12.1.1 and 10.3.6 Support for Oracle 12c Database [ID 1564509.1]" at the link

The following documents are also key references:
12c Oracle Database Developer Guide
12c Oracle Database Administrator's Guide .

I plan to write some related blog articles not to duplicate existing product documentation but to introduce the features, provide some examples, and tie together some information to make it easier to understand.

How do you get started with 12c?  The easiest way is to point your data source at a 12c database.  The only change on the WLS side is to update the URL in your data source (assuming that you are not just upgrading your database).  You can continue to use the driver jar files that shipped with WLS 10.3.6 or 12.1.1.  You shouldn't see any changes in your application.  You can take advantage of enhancements on the database side that don't affect the mid-tier.  On the WLS side, you can take advantage of using Global Data Service or connecting to a tenant in a multi-tenant database transparently.

If you want to use the 12c client jar files, it's a bit of work because they aren't shipped with WLS and you can't just drop in ojdbc6.jar as in the old days.  You need to use a matched set of jar files and they need to come before existing jar files in the CLASSPATH.  The MOS article is written from the standpoint that you need to get the jar files directly - download almost 1G and install over 600M footprint to get 15 jar files.  Assuming that you have the database installed and you can get access to the installation (or ask the DBA), you need to copy the 15 jar files to each machine with a WLS installation and get them in your CLASSPATH.  You can play with setting the PRE_CLASSPATH but the more practical approach may be to just update WL_HOME/common/bin/ directly.  There's a change in the transaction completion behavior (read the MOS) so if you think you might run into that, you will want to set -Doracle.jdbc.autoCommitSpecCompliant=false.  Also if you are running with Active GridLink, you must set -Doracle.ucp.PreWLS1212Compatible=true (how's that for telling you that this is fixed in WLS 12.1.2). 

Once you get the configuration out of the way, you can start using the new ojdbc7.jar in place of the ojdbc6.jar to get the new JDBC 4.1 API's.  You can also start using Application Continuity.  This feature is also known as JDBC Replay because when a connection fails you get a new one with all JDBC operations up to the failure point automatically replayed.  As you might expect, there are some limitations but it's an interesting feature.

 Obviously I'm going to focus on the 12c database features that we can leverage in WLS data source.  You will need to read other sources or the product documentation to get all of the new features.

PS Since this was written, WLS 12.1.2 is now available.  See blog entry Oracle WebLogic Server 12.1.2 is available for more information. There is an Online Launch Event on July 31st.

Thursday Jun 20, 2013

JDBC in Java Mission Control

In case you haven't tried them before, JRockit Flight Recorder (JFR) and JRockit Mission Control (JMC) are useful tools to analyze what's happening in the Java Virtual Machine (JVM) and your running application. Basically, you make a record of events and information in a defined time period and then analyze the generated jfr file with JMC.  WebLogic Server 10.3.3 added support in the diagnostics component to work tightly with JMC. Part of that support is analysis of what is happening on the JDBC front and that's what I'm going to focus on here.  If you want some background, read about JFR at  and JMC at

The command name to get started is jrmc.If the JVM browser is not visible, click Window -> Show View -> JVM Browser. Right click your WebLogic Server JVM in the list and select Start Console.  It should look something like the following screen shot.

Now you need to record a sample for analysis.  Right click the JVM connection again and select Start Recording.  Select a timer period like one minute and click start.  For this article, I ran a demo program OTRADE that was used at Oracle Open World 2012.  It has a simple load generator that simulates updating stock values for Oracle stock.  The variety of SQL statements is limited but enough to demonstrate how JFR works.  A window opens to the JFR analysis when the time period is complete.  I'm going to focus the analysis on JDBC events. Select the Events button on the left.  Select Window -> Show View -> Event Types. Unselect Java Application, expand WebLogic, and select JDBC (minimize the Event Type window).  This is what the overview tab looks like with just the JDBC events showing.

The Log tab allows you to drill into individual records.

The Trace tab allows you to see the stack trace where the SQL call was made.  This can be useful if there is a misbehaving SQL statement and you want to know who is calling it.

There are a lot of other interesting areas for analysis that I won't dive into.  You might want to look at Code -> Hot methods to see where time is being spent or Memory -> object statistics to see the big memory users.

It's possible to load some "unsupported" plug-ins. I'd recommend the one for WebLogic.  Select Help, Install Plugins, Expand, WebLogic Tab Pack, Next, Finish, and Install All.  You will need to restart jrmc and select the WebLogic button on the left.

Then select the Database tab.

The default WLDF diagnostic volume is Low.  You can reset it from the administrative console by selecting Environment -> Servers, clicking on the name of your server, selecting Configuration -> General, and changing the Diagnostic Volume value to Medium or High.

If you are aware of what's happening on the JVM front, you will realize that the last version of JRockit is JDK6.  That's fine for WLS 10.3.6, which supports the server running on JDK6 or JDK7, but 12.1.1 moves to JDK7, which is based on HotSpot.  The early versions of  JDK7 on HotSpot don't have the ability to run JFR but that's coming back in Oracle JDK7 with updates greater than or equal to 7u4.  I tested using the latest publicly available update at this time, 7u25, with WLS 10.3.6.  You need to have a Java Advanced or Java Suite license (included in WebLogic Server EE and WebLogic Suite license). You can download it from "All Java SE Downloads on MOS [ID 1439822.1]". Of course, all of the "JRockit" names now become "Java".  JRMC is now JMC (JFR remains the same).  The JFR component is now gated behind switches and needs to be actively turned on.  Also, HotSpot still needs to set PermGen (that isn't fixed until JDK8).  So your command line will be something like

java -Xmx1024m -XX:MaxPermSize=350m -XX:+UnlockCommercialFeatures -XX:+FlightRecorder   weblogic.Server

This article is a bit long but I've only scratched the surface on how to use this tool to analyze your WebLogic data source usage among the many other WLS components.  I realize the pictures in this format of the blog appear small - use whatever mechanism your browser has to zoom in on them - the detail is available.

P.S. The JFR recording data is included in WebLogic Diagnostic Framework (WLDF) Diagnostic Images, and is also included in DFW incident reports. Those mechanisms can be used to capture JFR recording data as well.  See Using WLDF with Oracle JRockit Flight Recorder and Configuring and Capturing Diagnostic Images.

Friday Jun 07, 2013

Using try-with-resources with JDBC objects

There is a new language construct that is introduced in JDK7 that applies to JDBC.  It’s called try-with-resource.  There is a tutorial at
It allows you to automatically close an object when it is no longer needed.  Theoretically, the object must extend java.lang.AutoCloseable.  In JDK7, that list includes java.sql.CallableStatement, Connection, PreparedStatement, Statement, ResultSet, and *RowSet.  The following code

Statement stmt = null;
try {
  stmt = con.createStatement();
} catch (Exception ignore) {
} finally {
  if (stmt != null) stmt.close()
becomes a shorter version using the new syntax
try (Statement stmt = con.createStatement()) {
} catch (Exception ignore) {

Here’s what a larger example looks like that I embedded in a WebLogic servlet for testing.  Note that there are two resources in the first try-with-resource, separated by a semi-colon.

private String doit() {
  String table2 = "test222";
  String dropsql = "drop table " + table2 + "";
  String createsql = "create table " + table2 + " ( col1 int, col2 int )";
  String insertsql = "insert into " + table2 + " values (1,2)";
  String selectsql = "select col1, col2 from " + table2 + "";
  try {
    ds = getDS();
  } catch(Exception e) {
    return("failed to get datasource");
  try (Connection conn = ds.getConnection();
    Statement stmt = conn.createStatement()) {
    try {
   } catch (Exception ignore) {} // ignore if table not dropped
   try (ResultSet rs = stmt.executeQuery(selectsql)) {;
   } catch (Exception e2) {
 } catch(Exception e) {
 return "DONE";

I’m not sure whether or not I like the programming paradigm – you can decide that for yourself.

How do I know that the connection, statement, and result set get closed?  I can turn on JDBCSQL debugging in the server and look at the server log output (this is a good trick to use to see what is happening under the covers).  Here is a stripped down version of the output.

 Connection@1 CreateStatement() 
 Connection@1 CreateStatement returns StatementWrapper@2 
 StatementWrapper@2 execute(drop table test222) 
 StatementWrapper@2 execute(drop table test222) throws java.sql.SQLSyntaxErrorException: table or view does not exist
 StatementWrapper@2 execute(create table test222 ( col1 int, col2 int )) 
 StatementWrapper@2 execute returns false 
 StatementWrapper@2 execute(insert into test222 values (1,2)) 
 StatementWrapper@2 execute returns false 
 StatementWrapper@2 executeQuery(select col1, col2 from test222) 
 StatementWrapper@2 executeQuery returns ResultSetImpl@3 
 ResultSetImpl@3 next() 
 ResultSetImpl@3 next returns true 
 ResultSetImpl@3 close() 
 ResultSetImpl@3 close returns 
 StatementWrapper@2 close() 
 StatementWrapper@2 close returns 
 Connection@1 close() 
 Connection@1 close returns

You might be thinking that there are not a lot of JDBC 4.1 compliant drivers out there and that’s true (the Oracle thin driver will have an ojdbc7.jar in version 12c, when it is released).  However, an object like java.sql.Statement implements AutoClosable so when running on JDK7 the driver Statement also is assumed to implement AutoClosable.  The object just needs to have a close() method so all of these JDBC objects have met the criteria since the initial specification.  WebLogic Server started supporting JDK7 in version 10.3.6.  Testing with all drivers that are shipped with WebLogic server, plus some drivers from various vendors, show that this JDK7 language feature works just fine with pre-JDBC 4.1 drivers. 

Note that this trick doesn't work for new JDK7 methods - you will get an AbstractMethodError.

So if you like this programming paradigm, start using it now with WebLogic Server 10.3.6 or later.

Wednesday Apr 24, 2013

Getting Connections with Active GridLink

As a follow-up to an earlier blog, someone asked "Can you please tell us more about gravitationShrinkFrequencySeconds and DRCP ?"   DRCP support isn’t shipping yet (wait until the next version of the Database and of WLS) so I can’t talk about it.

I think that people understand how connections are allocated in generic data sources and multi data sources (MDS).   Let me talk generally about how connections are allocated in Active GridLink (AGL). With the former (generic and MDS), connection allocation is pretty much dependent on WLS.  With AGL, it is also very dependent on the database configuration and runtime statistics.  The runtime information is provided to WLS via Oracle Notification Service (ONS) in the form of up and down events and Runtime Load Balancing events.

Connections are added to the pool initially based on the configured initial capacity. Connect time load balancing based on the listener(s) is used to spread the connections across the instances in the RAC cluster. For that to work correctly, you must either specify a Single Client Access Name (SCAN) address or use LOAD_BALANCE=ON for multiple non-SCAN addresses.  If you have multiple addresses and you forget to use LOAD_BALANCE=on, then all of the connections will end up on the first RAC instance – not what you want.  Connection load balancing is intended to give out connections based on load on the available RAC instances.  It’s not perfect.  If you have two instances and they are evenly loaded, there should be approximately 50% of the connections on each instance (but don’t enter a bug if they split up 49/51).  There a general problem with load balancing that the statistics can lag reality.  If you have big swings in demand on the two instances, then you can end up creating a connection on the more heavily loaded instance.

If you go to get a connection from the pool, runtime load balancing is used to pick the instance.  That’s again based on the load on the various instances and the information is updated by default every 30 seconds.  If a connection is available on the desired instance, then you have a hit on the pool and you get the available connection.

If you go to get a connection from the pool and one doesn’t exist on the desired instance, then you have a miss.  In this case, a connection is added to the pool on demand based on connection load balancing.  That is, if you go to reserve a connection and there isn’t one available in the pool, then a new one is created and it is done using connection load balancing as described above.  There’s an assumption here that the connect time and runtime load balancing match up.

If you are running within an XA transaction, then that takes priority in determining the connection that you get.  There are restrictions on XA support within a RAC cluster such that all processing for an XA transaction branch must take place on one instance in the cluster (the restrictions are complicated so the one-instance limitation is the only safe approach).  Additionally, performance is significantly better using a single instance so you want that to happen anyway.  The first time that you get a connection, it’s via the mechanisms described above (runtime load balancing or connection load balancing).  After that, any additional requests for connections within the same XA transaction will get a connection from the same instance.  If you can’t get a connection with “XA affinity,” the application will get an exception.

Similar to XA affinity, if you are running within a Web session, you will get affinity for multiple connections created by the same session.  Unlike XA affinity, “Session Affinity” is not mandatory.  If you can’t get a connection to the same instance, it will go to another instance (but there will be a performance penalty).

When you take down a RAC instance, it generates a “planned down event”.  Any unused connections for that instance are released immediately and connections in use are released when returned to the pool.  If a RAC instance fails, it generates an “unplanned down event.”  In that case, all connections for that instance are destroyed immediately.

When a RAC instance becomes available, either a new instance or an instance is restarted, it generates an “up event.”  When that occurs, connections are proactively created on the new instance.

The pool can get filled up with connections to the “wrong” instance (heavily loaded) if load changes over time.  To help that situation, we release unused connections based on runtime load balancing information.  When gravitation shrinking occurs, one unused connection is destroyed on a heavily loaded instance.  The default period for gravitation shrinking is 30 seconds but you can control it by setting the system property “weblogic.jdbc.gravitationShrinkFrequencySeconds” to an integer value.  We don’t actively create a connection at this point.  We wait until there is demand for a new connection and then the connection load balancing will kick in.

Finally, normal shrinking happens if not disabled.  When this occurs, half of the unused connections down to minimum capacity are destroyed.  The algorithm currently just takes the first set of connections on the list without regard to load balancing (it’s random with respect to instances).  The default period is 900 seconds and you can configure this using ShrinkFrequencySeconds.

There are possible improvements that could be made with respect to pool misses, and gravitational and normal shrinking.  And the database team is working on improving the load balancing done on their end. Still, it works pretty well with instances dynamically coming and going and the load changing over time.

Monday Oct 15, 2012

Data Source Security Part 4

So far, I have covered Client Identity and Oracle Proxy Session features, with WLS or database credentials.  This article will cover one more feature, Identify-based pooling.  Then, there is one more topic to cover - how these options play with transactions.

Identity-based Connection Pooling

An identity based pool creates a heterogeneous pool of connections.  This allows applications to use a JDBC connection with a specific DBMS credential by pooling physical connections with different DBMS credentials.  The DBMS credential is based on either the WebLogic user mapped to a database user or the database user directly, based on the “use database credentials” setting as described earlier. Using this feature enabled with “use database credentials” enabled seems to be what is proposed in the JDBC standard, basically a heterogeneous pool with users specified by getConnection(user, password).

The allocation of connections is more complex if Enable Identity Based Connection Pooling attribute is enabled on the data source.  When an application requests a database connection, the WebLogic Server instance selects an existing physical connection or creates a new physical connection with requested DBMS identity.

The following section provides information on how heterogeneous connections are created:
1. At connection pool initialization, the physical JDBC connections based on the configured or default “initial capacity” are created with the configured default DBMS credential of the data source.
2. An application tries to get a connection from a data source.
3a. If “use database credentials” is not enabled, the user specified in getConnection is mapped to a DBMS credential, as described earlier.  If the credential map doesn’t have a matching user, the default DBMS credential is used from the datasource descriptor.
3b. If “use database credentials” is enabled, the user and password specified in getConnection are used directly.
4. The connection pool is searched for a connection with a matching DBMS credential.
5. If a match is found, the connection is reserved and returned to the application.
6. If no match is found, a connection is created or reused based on the maximum capacity of the pool:
- If the maximum capacity has not been reached, a new connection is created with the DBMS credential, reserved, and returned to the application.
- If the pool has reached maximum capacity, based on the least recently used (LRU) algorithm, a physical connection is selected from the pool and destroyed. A new connection is created with the DBMS credential, reserved, and returned to the application.

It should be clear that finding a matching connection is more expensive than a homogeneous pool.  Destroying a connection and getting a new one is very expensive.  If you can use a normal homogeneous pool or one of the light-weight options (client identity or an Oracle proxy connection), those should be used instead of identity based pooling.

Regardless of how physical connections are created, each physical connection in the pool has its own DBMS credential information maintained by the pool. Once a physical connection is reserved by the pool, it does not change its DBMS credential even if the current thread changes its WebLogic user credential and continues to use the same connection.

To configure this feature, select Enable Identity Based Connection Pooling.  See  "Enable identity-based connection pooling for a JDBC data source" in Oracle WebLogic Server Administration Console Help.

You must make the following changes to use Logging Last Resource (LLR) transaction optimization with Identity-based Pooling to get around the problem that multiple users will be accessing the associated transaction table.
- You must configure a custom schema for LLR using a fully qualified LLR table name. All LLR connections will then use the named schema rather than the default schema when accessing the LLR transaction table. 
- Use database specific administration tools to grant permission to access the named LLR table to all users that could access this table via a global transaction. By default, the LLR table is created during boot by the user configured for the connection in the data source. In most cases, the database will only allow access to this user and not allow access to mapped users.

Connections within Transactions

Now that we have covered the behavior of all of these various options, it’s time to discuss the exception to all of the rules.  When you get a connection within a transaction, it is associated with the transaction context on a particular WLS instance.

When getting a connection with a data source configured with non-XA LLR or 1PC (using the JTS driver) with global transactions, the first connection obtained within the transaction is returned on subsequent connection requests regardless of the values of username/password specified and independent of the associated proxy user session, if any. The connection must be shared among all users of the connection when using LLR or 1PC.

For XA data sources, the first connection obtained within the global transaction is returned on subsequent connection requests within the application server, regardless of the values of username/password specified and independent of the associated proxy user session, if any.  The connection must be shared among all users of the connection within a global transaction within the application server/JVM.

Friday Oct 12, 2012

Data Source Security Part 3

In part one, I introduced the security features and talked about the default behavior.  In part two, I defined the two major approaches to security credentials: directly using database credentials and mapping WLS user credentials to database credentials.  Now it's time to get down to a couple of the security options (each of which can use database credentials or WLS credentials).

Set Client Identifier on Connection

When "Set Client Identifier" is enabled on the data source, a client property is associated with the connection.  The underlying SQL user remains unchanged for the life of the connection but the client value can change.  This information can be used for accounting, auditing, or debugging.  The client property is based on either the WebLogic user mapped to a database user using the credential map or is the database user parameter directly from the getConnection() method, based on the “use database credentials” setting described earlier.

To enable this feature, select “Set Client ID On Connection” in the Console.  See "Enable Set Client ID On Connection for a JDBC data source" in Oracle WebLogic Server Administration Console Help.

The Set Client Identifier feature is only available for use with the Oracle thin driver and the IBM DB2 driver, based on the following interfaces.

For pre-Oracle 12c, oracle.jdbc.driver.OracleConnection.setClientIdentifier(client) is used.  See for more information about how to use this for auditing and debugging.   You can get the value using getClientIdentifier()  from the driver.  To get back the value from the database as part of a SQL query, use a statement like the following.
“select sys_context('USERENV','CLIENT_IDENTIFIER') from DUAL”.

Starting in Oracle 12c, java.sql.Connection.setClientInfo(“OCSID.CLIENTID", client) is used.  This is a JDBC standard API, although the property values are proprietary.  A problem with setClientIdentifier usage is that there are pieces of the Oracle technology stack that set and depend on this value.  If application code also sets this value, it can cause problems. This has been addressed with setClientInfo by making use of this method a privileged operation. A well-managed container can restrict the Java security policy grants to specific namespaces and code bases, and protect the container from out-of-control user code. When running with the Java security manager, permission must be granted in the Java security policy file for
permission "oracle.jdbc.OracleSQLPermission" "clientInfo.OCSID.CLIENTID";
Using the name “OCSID.CLIENTID" allows for upward compatible use of

“select sys_context('USERENV','CLIENT_IDENTIFIER') from DUAL”

or use the JDBC standard API java.sql.getClientInfo(“OCSID.CLIENTID") to retrieve the value.

This value in the Oracle USERENV context can be used to drive the Oracle Virtual Private Database (VPD) feature to create security policies to control database access at the row and column level. Essentially, Oracle Virtual Private Database adds a dynamic WHERE clause to a SQL statement that is issued against the table, view, or synonym to which an Oracle Virtual Private Database security policy was applied.  See Using Oracle Virtual Private Database to Control Data Access for more information about VPD.  Using this data source feature means that no programming is needed on the WLS side to set this context; it is set and cleared by the WLS data source code.

For the IBM DB2 driver, is used for older releases (prior to version 9.5).  This specifies the current client user name for the connection. Note that the current client user name can change during a connection (unlike the user).  This value is also available in the CURRENT CLIENT_USERID special register.  You can select it using a statement like


When running the IBM DB2 driver with JDBC 4.0 (starting with version 9.5), java.sql.Connection.setClientInfo(“ClientUser”, client) is used.  You can retrieve the value using java.sql.Connection.getClientInfo(“ClientUser”) instead of the DB2 proprietary API (even if set setDB2ClientUser()). 

Oracle Proxy Session

Oracle proxy authentication allows one JDBC connection to act as a proxy for multiple (serial) light-weight user connections to an Oracle database with the thin driver.  You can configure a WebLogic data source to allow a client to connect to a database through an application server as a proxy user. The client authenticates with the application server and the application server authenticates with the Oracle database. This allows the client's user name to be maintained on the connection with the database.

Use the following steps to configure proxy authentication on a connection to an Oracle database.
1. If you have not yet done so, create the necessary database users.
2. On the Oracle database, provide CONNECT THROUGH privileges. For example:
where “connectionuser” is the name of the application user to be authenticated and “dbuser” is an Oracle database user.
3. Create a generic or GridLink data source and set the user to the value of dbuser.
4a. To use WLS credentials, create an entry in the credential map that maps the value of wlsuser to the value of dbuser, as described earlier.  
4b. To use database credentials, enable “Use Database Credentials”, as described earlier.
5. Enable Oracle Proxy Authentication, see "Configure Oracle parameters" in Oracle WebLogic Server Administration Console Help.
6. Log on to a WebLogic Server instance using the value of wlsuser or dbuser.
6. Get a connection using getConnection(username, password).  The credentials are based on either the WebLogic user that is mapped to a database user or the database user directly, based on the “use database credentials” setting. 

You can see the current user and proxy user by executing:

“select user, sys_context('USERENV','PROXY_USER') from DUAL".

Note: getConnection fails if “Use Database Credentials” is not enabled and the value of the user/password is not valid for a WebLogic Server user.  Conversely, it fails if “Use Database Credentials” is enabled and the value of the user/password is not valid for a database user.

A proxy session is opened on the connection based on the user each time a connection request is made on the pool. The proxy session is closed when the connection is returned to the pool.  Opening or closing a proxy session has the following impact on JDBC objects.
- Closes any existing statements (including result sets) from the original connection.
- Clears the WebLogic Server statement cache.
- Clears the client identifier, if set.
-The WebLogic Server test statement for a connection is recreated for every proxy session.

These behaviors may impact applications that share a connection across instances and expect some state to be associated with the connection.

Oracle proxy session is also implicitly enabled when use-database-credentials is enabled and getConnection(user, password) is called,starting in WLS Release 10.3.6.  Remember that this only works when using the Oracle thin driver.

To summarize, the definition of oracle-proxy-session is as follows.
- If proxy authentication is enabled and identity based pooling is also enabled, it is an error.
- If a user is specified on getConnection() and identity-based-connection-pooling-enabled is false, then oracle-proxy-session is treated as true implicitly (it can also be explicitly true).
- If a user is specified on getConnection() and identity-based-connection-pooling-enabled is true, then oracle-proxy-session is treated as false.


The official blog for Oracle WebLogic Server fans and followers!

Stay Connected


« December 2015