Thursday Jul 25, 2013

CERN Excited about WebLogic 12c release 12.1.2

CERN is world-renowned for their scientific research, operating the world’s largest and most complex scientific instruments to study the basic constituents of matter - the fundamental particles. I bet many of our readers know that! But what you may not have known is that CERN utilizes WebLogic Server as its strategic platform for all developed software, which means support for Administration, Engineering and the Accelerator Center. At the recent WebLogic Customer Advisory Board, we had the pleasure of having with us Artur Wiecek of CERN's Infrastructure IT Department. Artur and other customers had a chance to hear about the new features of WebLogic 12c, release 12.1.2, and we pulled him aside to give you his thoughts on the ones he’s looking to take advantage at at CERN. Watch the video

To learn more about WebLogic 12c release 12.1.2, please join our launch event on July 31st

Wednesday Jul 24, 2013

New Node Manager Configuration Model

Join the Online Launch Event for WLS 12.1.2 on July 31st.

 In WebLogic Server 12.1.2, we have a great new option for configuring and running the Node Manager: by default, Node Manager is now configured with the domain.

We listened!

In recent months, we’ve heard that customers typically manage Node Manager instances with their domains. That is, they install, upgrade, and run Node Manager instances along with the domains where they host their applications. Also, in most cases, it is the same users who manage Node Manager instances along with related domains. The configuration model for Node Manager in previous releases – sharing Node Manager instances across domains – can be more complicated because different groups in the organization upgrade, patch, and have downtime on different schedules.

We listened, and now the default configuration for Node Manager is a Node Manager instance per domain on each of your Managed Server hosts. 

Per-Domain Node Manager Diagram

With this new configuration option, you can now:

  • Align Node Manager lifecycle management with the domains that it manages and monitors
  • Simplify Node Manager configuration through Configuration Wizard/WLST offline and Pack/Unpack commands
  • Maintain your WebLogic installation directory as a read-only directory (Node Manager configuration is now stored with the domain.)

Configuration Wizard Changes

When you create a domain using either WLST or the Configuration Wizard, Node Manager is configured for use with that domain. When I say “configured for that domain,” I mean that:

  • There is a <domain>/nodemanager folder that includes the Node Manager configuration files (,
  • There is a startNodeManager script in the <domain>/bin directory
  • In the <domain>/security folder, there is a domain-specific and host-specific DemoIdentity.jks keystore used by the Node Manager for OOTB evaluation and demonstrations

In other words – everything you need to run the Node Manager for the domain is set up for you and is encapsulated in the domain! 

Extending Your Domain with Pack/Unpack

We’ve also made it easier to set up Node Manager for the Managed Servers in your domain:

  • When you use the “pack” command with the “-managed=true” option, the configuration framework includes changes to the file in the template file.
  • When you use the “unpack” command on the remote host where you want to run your Managed Servers, the configuration framework sets up the domain, regenerates the file, generates a file (no need to run nmEnroll!), generates a DemoIdentity.jks keystore, and generates a startNodeManager script.

Out of the gate, you are ready to go with a per-domain Node Manager configuration for each host where you will run your Managed Servers.

Give it a try! 

More Information

Don’t forget to register for the WebLogic Server 12.1.2 Online Launch Event on July 31st.

You can also see the following links:

Monday Jul 22, 2013

Part 2 - 12c Database and WLS - Application continuity

I introduced WLS support for 12c database in this blog link and described one of the prerequisites for Application Continuity (AC) at this blog link.

When an error occurs on a connection, it would be nice to be able to keep processing on another connection - that's what AC does.  When on a single node database, getting another connection means that the database and network are still available, e.g., maybe just a network glitch.  However, in the case of a Real Application Cluster (RAC), chances are good that even if you lost a connection on one instance you can get a connection on another instance for the same database.  For this feature to work correctly, it's necessary to ensure that all of the work that you did on the connection before the error is replayed on the new connection - that's why AC is also called replay. To replay the operations, it's necessary to keep a list of the operations that have been done on the connection so they can be done again.  Of course, when you replay the operations, data might have changed since the last time so it's necessary to keep track of the results to make sure that they match.  So AC may fail if replaying the operations fail.

You can read the WLS documentation, an AC white paper, and the Database documentation to get the details. This article gives an overview of this feature (and a few tidbits that the documentation doesn't cover).

To use this feature, you need to use both the 12c driver and a 12c database.  Note that if you use an driver and/or database, it might appear to work but replay will only happen for read-only transactions (the 11g mode is not supported but we don't have a mechanism to prevent this).

You need to configure a database service to run with AC.  To do that, you need to set the new 12c service attributes FAILOVER_TYPE=TRANSACTION and COMMIT_OUTCOME=true on the server side.

There's not much to turning on AC in WLS - you just use the replay driver "oracle.jdbc.replay.OracleDataSourceImpl" instead of "oracle.jdbc.OracleDriver"  when configuring the data source.  There's no programming in the application needed to use it.  Internal to WLS, when you get a connection we call the API to start collecting the operations and when you close a connection we call the API to clear the operation history.

WLS introduced a labeling callback for applications that use the connection labeling feature.  If this callback is registered, it will be called when a new connection is being initialized before replay.  Even if you aren't using labeling, you might still want to be called and there is a new connection initialization callback that is for replay (labeling callback trumps initialization callback if both exist).

It sounds easy and perfect - what's the catch?   I've already mentioned that you need to get rid of references to concrete classes and use the new Oracle interfaces.  For some applications that might be some significant work.  I've mentioned that if replaying the operations fails or is inconsistent, AC fails.  There are a few other operations that turn off AC - see the documentation for details. One of the big ones is that you can't use replay with XA transactions (at least for now).   Selecting from V$instance or sys_context  or other test traces for test instrumentation needs to be in callouts as the values change when replayed.  If you use sysdate or systimestamp, you need to grant keep date time to your user.

We are also tracking two defects - AC doesn't work with Oracle proxy authentication and it doesn't work with the new DRCP feature. 

There's another more complex topic to consider (not currently in the current WLS documentation).  By default, when a local transaction is completed on the connection, replay ends.  You can keep working on the connection but failure from that point on will return an error to the application. This default is based on the service attribute SESSION_STATE_CONSISTENCY with a value of DYNAMIC.  You can set the value to STATIC if your application does not modify non-transactional session state (NTSS) in the transaction.  I'm not sure how many applications fall into this trap but the safe thing is to default to dynamic.  I'll include such a code fragment below.  A related common problem that people run into is forgetting to disable autocommit, which defaults to true, and the first (implicit) commit turns off replay if SESSION_STATE_CONSISTENCY is set to DYNAMIC.

It's important to know how to turn on debugging so that if a particular sequence doesn't replay, you can understand why.  You simply need to run with the debug driver (ojdbc6_g.jar or ojdbc7_g.jar) and run with -Dweblogic.debug.DebugJDBCReplay (or turn this debug category on in the configuration).

AC won't replay everything and you still need to have some application logic to deal with the failures or return them to the end user.  Also, there's some overhead in time and memory to keep the replay data.  Still, it seems like a great feature for a lot of applications where you don't need to change anything but the driver name and you can avoid showing an error to the end user or simplify some recovery logic.

P.S. Confused about NTSS? So was I.  Examples of non-transactional session state that can change at run-time are ALTER SESSION, PL/SQL global variables, SYS_CONTEXT, and temporary table contents.  Here's an example of a PL/SQL global variable.  Imagine a package with the following body:

 current_order number := null;
 current_line number;
 procedure new_order (customer_id number) is
  current_order := order_seq.nextval;
  insert into orders values (current_order, customer_id);
  current_line := 0;
 end new_order;
 procedure new_line (product_id number, count number) is
  current_line := current_line + 1;
  insert into order_lines values (current_order, current_line,product_id, count);
 end new_line;
end order;

and a psuedo-code sequence in WLS like this:
exec "begin order.new_order(:my_customer_id); end;"
exec "begin order.new_line(:my_product_id, :my_count); end;"
<DB server failure and failover>

In this scenario, we won't replay the first transaction, because it's already committed and we'd end up with two orders in the database. But if we don't replay it, we lose the order_id, so the new_line call won't work. So we have to disable replay after the commit.  If you can guarantee that no such scenario exists, you can mark the service session-state as static.

Wednesday Jul 17, 2013

WebLogic Active GridLink - An Oracle and NEC Joint White Paper

Oracle and NEC have been jointly invested on verifying and testing the Active GridLink for RAC solution implemented within Oracle WebLogic Server which gives the rich functionality for Oracle Database Real Application Clusters integration.

To learn about additional enhancements for Oracle WebLogic - Oracle Database integration, please join the Online Launch Event on July 31st.

In this published white paper, we started with a brief introduction to Oracle RAC and an overview of the Oracle RAC features supported in Oracle WebLogic Server 12c. We then focused on details of the effort that have been jointly done with Oracle and NEC with all the detailed testing scenarios and testing results, along with the analysis. The background and overview of NEC’s test of Active GridLink for RAC will be covered in details. The technical details about Runtime Connection Load Balancing, Web Session Affinity, Fast Connection Failover, and how to remove and add the additional RAC node with zero-downtime will be discussed with different use cases.

Check out this Japanese version paper:

English version is coming soon, please check back later!

Gemalto is excited about new Mobile Capabilities in WebLogic 12c release 12.1.2

Gemalto is the leader in making digital interactions secure and easy. It enables its clients to offer trusted and convenient digital services to billions of individuals.

People worldwide increasingly want the freedom to communicate, travel, shop, bank, entertain and work – anytime, anywhere – in ways that are convenient, enjoyable and secure.

Gemalto delivers on their expanding needs for personal mobile services, identity protection, payment security, authenticated online services, cloud computing access, modern transportation, M2M communication, eHealthcare and eGovernment services.

Gemalto’s telecommunication business unit is using multiple versions of WebLogic: 10.3.1, 10.3.2, and 10.3.4. They use WebLogic for their web-based applications, business logic, JMS, and web services. High availability is a significant requirement for Gemalto as they have strict SLA contracts with their customers demanding up to five nines availability during production. So clustering is very important. They are also excited about the new mobile capabilities in WebLogic 12c release 12.1.2. See the short video to hear their story and join the Cloud Application Foundation 12c launch event to learn more about mobile capabilities in WebLogic 12.1.2

Tuesday Jul 16, 2013

Oracle WebLogic Server 12c, Release 12.1.2: And now the Whitepaper

As you read in Will Lyon’s blog from last week, the brand new Oracle WebLogic Server 12.1.2 was made available for download on July 11th!  With this new release customers are able to deliver next generation applications, such as mobile applications, on a mission-critical cloud platform. A big investment went into simplifying operations with native cloud management and finally for the developer audience, the new release offers a modern development platform and integrated tools to increase productivity and accelerate time to market.

The Oracle WebLogic Server 12c, Release 12.1.2 whitepaper contains details of the investment motivations, the focus areas, and use cases. It goes deep into areas like WebLogic Server dynamic clusters for cloud elasticity, existing integration with Oracle RAC and how this release drives even deeper integration with Oracle Database 12c in the areas of Oracle Multi-Tenant, virtualized connection pooling, application continuity, and Global Data Services. The whitepaper discusses features that provide native cloud management with elastic JMS for flexible cloud scaling and the convergence of Oracle WebLogic Server and Oracle Coherence management framework to drive operational efficiency. Maven and WebSocket support across the platform and how developers can leverage these capabilities along with Toplink Data Services for Web and mobile development are highlighted.

Here is an excerpt:

... Oracle WebLogic Server lets customers build an application server infrastructure using standard Java Standard Edition (SE) and Java Enterprise Edition (EE) tools. This industry-leading application server platform provides tremendous deployment flexibility. If you are not yet ready for cloud, it offers a best-of-breed choice for deploying packaged and homegrown applications to run on conventional hardware platforms. When ready you can run your applications in a private cloud on conventional hardware platforms or engineered systems, or have your applications hosted by a third party vendor, such as Oracle Cloud, or mix them together in a hybrid cloud model...


Also, don't forget to join us for the Oracle WebLogic Server 12c, Release 12.1.2 launch event later this month:
July 31st, 10am, Pacific Time

Thursday Jul 11, 2013

Oracle WebLogic Server 12.1.2 is available

Good news for Oracle WebLogic Server users! Oracle WebLogic Server 12.1.2 has been made available for download today.  See the press release for more on what is included in the overall Cloud Application Foundation 12c release.  

Oracle WebLogic Server is the #1 application server in the industry, enabling deployment from conventional systems to cloud environments. This means you can choose to deploy the same applications on standard Windows/Unix platforms, on Oracle Exalogic Elastic Cloud to leverage engineered systems optimizations, or in private or public clouds such as the Oracle Cloud. Oracle WebLogic Server 12.1.2 adds new capabilities that extend its leadership for all of these deployment environments. Here's a brief summary.

First, we've delivered enhancements to make Oracle WebLogic Server 12.1.2 the ideal mission critical cloud platform, with unmatched performance, scalability and availability. New integration features with Oracle Database 12c deliver availability, multi-tenancy and scalability benefits for your on-premise clouds using Oracle's latest database technology. Dynamic cluster support make it easy to create high-availability cluster configurations, and to scale them up or down as performance requirements dictate. Additional optimizations for Exalogic Elastic Cloud Systems deliver new performance benefits - up to 3x performance improvements for JMS applications.   Oracle Coherence 12.1.2 has also been released, with new powerful features that can be leveraged in mission-critical applications.

Second, we've simplified operations by delivering native cloud management - efficient, unified, built-in management capabilities that span the web server, application server and data grid tiers of your cloud infrastructure. Oracle HTTP Server, Oracle WebLogic Server and Oracle Coherence now share a common installation and patching framework for efficient provisioning. The WebLogic Management Framework has been extended to manage Oracle HTTP Server and Managed Coherence Server configurations.  We continue to enhance the manageability of Oracle WebLogic Server itself via management enhancements such as elastic JMS for scaling your JMS deployments.   A new release of Oracle Enterprise Manager 12c Release 3 complements and extends these capabilities.

Third, we continue to invest in providing a modern development platform that enables developers to accelerate delivery of innovative applications. Enhancements to our Maven integration with updated plug-ins, POMs and archetypes make it easier to use build Oracle WebLogic Server, Coherence, TopLink, and Fusion Middleware applications with Maven-based repositories and build systems. We've added support for the WebSocket protocol standard for delivery of rich, interactive applications for standard HTML5 and mobile clients. Oracle TopLink Data Services make it easy to generate RESTful interfaces to enterprise data sources and deliver services for rich client applications built with JavaScript or similar technologies.  Updated releases of Oracle JDeveloper, Oracle Enterprise Pack for Eclipse, and Oracle Application Development Framework are available with new features for productive application development.

Taken together, these new capabilities make Oracle WebLogic Server 12.1.2 a compelling platform for developing and deploying your next generation of applications, and upgrading your existing applications. You'll be hearing more about all of the above capabilities during the coming weeks. Some things you can do today:

1) Register for the upcoming Oracle Cloud Application Foundation Webcast to learn more about WebLogic Server 12.1.2 and the new Cloud Application Foundation 12c release.

2) Download Oracle WebLogic Server 12.1.2 from the Oracle Technology Network for free development and evaluation purposes.  Licensed customers should download the software from the Oracle Software Delivery Cloud site, which offers different licensing terms.

3) Read the Oracle WebLogic Server 12.1.2 whitepaper for an overview of new features and business benefits.

4) Review the product documentation for full details on all the new capabilities.



Using Oracle JDBC Type Interfaces

One of the hot new features in Oracle database 12c is Application Continuity (AC).  The feature basically will detect that a connection has gone bad and substitute a new one under the covers (I'll talk about it more in another article).  To be able to do that, the application is given a connection wrapper instead of the real connection.  Wrappers or dynamic proxies can only be generated for classes based on interfaces.  The Oracle types, like REF and ARRAY, were originally introduced as concrete classes.   In WLS 10.3.6 and/or the JDBC driver, there are new interfaces for the Oracle types that you will need to use to take advantage of this new AC feature.

First, some history so that you understand the needed API changes.  In the early days of WebLogic data source, any references to vendor proprietary methods were handled by hard-coded references.  Keeping up with adding and removing methods was a significant maintenance problem. At the peak of the insanity, we had over one thousand lines of code that referenced Oracle-proprietary methods and the server could not run without an Oracle jar in the classpath (even for DB2-only shops).  In release 8.1 in March 2003, we introduced wrappers for all JDBC objects such that we dynamically generated proxies that implement all public interface methods of, and delegate to, the underlying vendor object.  The hard-coded references and the maintenance nightmare went away and just as importantly, we could provide debugging information, find leaked connections, automatically close objects when the containing object closed, replace connections that fail testing, etc.  The Oracle types were concrete classes so proxies were generated for these classes implementing the WLS vendor interfaces*.  Applications can cast to the WLS vendor interfaces or use getVendorObj to access the underlying driver object. Later, we added an option to unwrap data types, with a corresponding loss of functionality like no debug information.

Although the focus of this article is Oracle types, the dynamic proxies work for any vendor.   For example, you can cast a DB2 connection to use a proprietary method  (("myname").

Starting with Oracle driver, the database team needed wrappers for the new AC feature and introduced new interfaces.  For WebLogic data source users, that's good news - no more unwrapping, the weblogic.jdbc.vendor package is no longer needed, and it's all transparent.  Before you go and change your programs to use the new Oracle proprietary interfaces, the recommended approach is to first see if you can just use standard JDBC API's.  In fact, as part of defining the new interfaces, Oracle proprietary methods were dropped if there was an equivalent standard JDBC API or the method was not considered to add significant value.  This table defines the mapping.  The goal is to get rid of references to the first and second columns and replace them with the third column.

" align="center">Old Oracle types
Deprecated WLS Interface
New interfaces

This is a job for a shell hacker!  Much of it can be automated and the compiler can tell you if you are referencing a method that has gone away - then check if the missing method is in the equivalent jdbc.sql interface (e.g., getARRAY() becomes the JDBC standard getArray()).

You can take a look at a sample program that I wrote to demonstrate all of these new interfaces at (note that this is actually a ".java" program). It covers programming with all of these Oracle types.  While use of Blob and Clob might be popular, Ref and Struct might not be used as much.  The sample program shows how to create, insert, update, and access each type using both standard and extension methods.  Note that you need to use the Oracle proprietary createOracleArray() instead of the standard createArrayOf(). Although the sample program doesn't use the standard createBlob() or createClob(), these are supported for the Oracle driver.

The API's can be reviewed starting with the version of the Javadoc.  A copy is available at .

This is a first step toward using Application Continuity.  But it's also a good move to remove Oracle API's that will eventually go away and use standard JDBC interfaces and new Oracle interfaces.

Tuesday Jul 02, 2013

Part 1 - 12c Database and WLS - Overview

The download of Oracle 12c database became available on June 25, 2013.  There are some big new features in 12c database and WebLogic Server will take advantage of them. Immediately, we will support using 12c database and drivers with WLS 10.3.6 and 12.1.1.  With WLS 12.1.2, available July 11, 2013 (join the Online Launch Event on July 31st), additional functionality is supported (those rows in the table below with all "No" values will get a "Yes).  The following table maps the Oracle 12c Database features supported with various combinations of currently available WLS releases, 11g and 12c Drivers, and 11g and 12c Databases.

Feature WebLogic Server 10.3.6/12.1.1 with 11g drivers and 11gR2 DB WebLogic Server 10.3.6/12.1.1 with 11g drivers and 12c DB WebLogic Server 10.3.6/12.1.1 with 12c drivers and 11gR2 DB WebLogic Server 10.3.6/12.1.1 with 12c drivers and 12c DB
JDBC replay No No No Yes (Active GridLink only in 10.3.6, add generic in 12.1.1)
Multi Tenant Database No Yes (except set container) No Yes (except set container)
Dynamic switching between Tenants No No No No
Database Resident Connection pooling (DRCP) No No No No
Oracle Notification Service (ONS) auto configuration No No No No
Global Database Services (GDS) No Yes (Active GridLink only) No Yes (Active GridLink only)
JDBC 4.1 (using ojdbc7.jar files & JDK 7) No No Yes Yes

 The My Oracle Support (MOS) document covering this is "WebLogic Server 12.1.1 and 10.3.6 Support for Oracle 12c Database [ID 1564509.1]" at the link

The following documents are also key references:
12c Oracle Database Developer Guide
12c Oracle Database Administrator's Guide .

I plan to write some related blog articles not to duplicate existing product documentation but to introduce the features, provide some examples, and tie together some information to make it easier to understand.

How do you get started with 12c?  The easiest way is to point your data source at a 12c database.  The only change on the WLS side is to update the URL in your data source (assuming that you are not just upgrading your database).  You can continue to use the driver jar files that shipped with WLS 10.3.6 or 12.1.1.  You shouldn't see any changes in your application.  You can take advantage of enhancements on the database side that don't affect the mid-tier.  On the WLS side, you can take advantage of using Global Data Service or connecting to a tenant in a multi-tenant database transparently.

If you want to use the 12c client jar files, it's a bit of work because they aren't shipped with WLS and you can't just drop in ojdbc6.jar as in the old days.  You need to use a matched set of jar files and they need to come before existing jar files in the CLASSPATH.  The MOS article is written from the standpoint that you need to get the jar files directly - download almost 1G and install over 600M footprint to get 15 jar files.  Assuming that you have the database installed and you can get access to the installation (or ask the DBA), you need to copy the 15 jar files to each machine with a WLS installation and get them in your CLASSPATH.  You can play with setting the PRE_CLASSPATH but the more practical approach may be to just update WL_HOME/common/bin/ directly.  There's a change in the transaction completion behavior (read the MOS) so if you think you might run into that, you will want to set -Doracle.jdbc.autoCommitSpecCompliant=false.  Also if you are running with Active GridLink, you must set -Doracle.ucp.PreWLS1212Compatible=true (how's that for telling you that this is fixed in WLS 12.1.2). 

Once you get the configuration out of the way, you can start using the new ojdbc7.jar in place of the ojdbc6.jar to get the new JDBC 4.1 API's.  You can also start using Application Continuity.  This feature is also known as JDBC Replay because when a connection fails you get a new one with all JDBC operations up to the failure point automatically replayed.  As you might expect, there are some limitations but it's an interesting feature.

 Obviously I'm going to focus on the 12c database features that we can leverage in WLS data source.  You will need to read other sources or the product documentation to get all of the new features.

PS Since this was written, WLS 12.1.2 is now available.  See blog entry Oracle WebLogic Server 12.1.2 is available for more information. There is an Online Launch Event on July 31st.

Thursday Jun 20, 2013

JDBC in Java Mission Control

In case you haven't tried them before, JRockit Flight Recorder (JFR) and JRockit Mission Control (JMC) are useful tools to analyze what's happening in the Java Virtual Machine (JVM) and your running application. Basically, you make a record of events and information in a defined time period and then analyze the generated jfr file with JMC.  WebLogic Server 10.3.3 added support in the diagnostics component to work tightly with JMC. Part of that support is analysis of what is happening on the JDBC front and that's what I'm going to focus on here.  If you want some background, read about JFR at  and JMC at

The command name to get started is jrmc.If the JVM browser is not visible, click Window -> Show View -> JVM Browser. Right click your WebLogic Server JVM in the list and select Start Console.  It should look something like the following screen shot.

Now you need to record a sample for analysis.  Right click the JVM connection again and select Start Recording.  Select a timer period like one minute and click start.  For this article, I ran a demo program OTRADE that was used at Oracle Open World 2012.  It has a simple load generator that simulates updating stock values for Oracle stock.  The variety of SQL statements is limited but enough to demonstrate how JFR works.  A window opens to the JFR analysis when the time period is complete.  I'm going to focus the analysis on JDBC events. Select the Events button on the left.  Select Window -> Show View -> Event Types. Unselect Java Application, expand WebLogic, and select JDBC (minimize the Event Type window).  This is what the overview tab looks like with just the JDBC events showing.

The Log tab allows you to drill into individual records.

The Trace tab allows you to see the stack trace where the SQL call was made.  This can be useful if there is a misbehaving SQL statement and you want to know who is calling it.

There are a lot of other interesting areas for analysis that I won't dive into.  You might want to look at Code -> Hot methods to see where time is being spent or Memory -> object statistics to see the big memory users.

It's possible to load some "unsupported" plug-ins. I'd recommend the one for WebLogic.  Select Help, Install Plugins, Expand, WebLogic Tab Pack, Next, Finish, and Install All.  You will need to restart jrmc and select the WebLogic button on the left.

Then select the Database tab.

The default WLDF diagnostic volume is Low.  You can reset it from the administrative console by selecting Environment -> Servers, clicking on the name of your server, selecting Configuration -> General, and changing the Diagnostic Volume value to Medium or High.

If you are aware of what's happening on the JVM front, you will realize that the last version of JRockit is JDK6.  That's fine for WLS 10.3.6, which supports the server running on JDK6 or JDK7, but 12.1.1 moves to JDK7, which is based on HotSpot.  The early versions of  JDK7 on HotSpot don't have the ability to run JFR but that's coming back in Oracle JDK7 with updates greater than or equal to 7u4.  I tested using the latest publicly available update at this time, 7u25, with WLS 10.3.6.  You need to have a Java Advanced or Java Suite license (included in WebLogic Server EE and WebLogic Suite license). You can download it from "All Java SE Downloads on MOS [ID 1439822.1]". Of course, all of the "JRockit" names now become "Java".  JRMC is now JMC (JFR remains the same).  The JFR component is now gated behind switches and needs to be actively turned on.  Also, HotSpot still needs to set PermGen (that isn't fixed until JDK8).  So your command line will be something like

java -Xmx1024m -XX:MaxPermSize=350m -XX:+UnlockCommercialFeatures -XX:+FlightRecorder   weblogic.Server

This article is a bit long but I've only scratched the surface on how to use this tool to analyze your WebLogic data source usage among the many other WLS components.  I realize the pictures in this format of the blog appear small - use whatever mechanism your browser has to zoom in on them - the detail is available.

P.S. The JFR recording data is included in WebLogic Diagnostic Framework (WLDF) Diagnostic Images, and is also included in DFW incident reports. Those mechanisms can be used to capture JFR recording data as well.  See Using WLDF with Oracle JRockit Flight Recorder and Configuring and Capturing Diagnostic Images.

Friday Jun 07, 2013

Using try-with-resources with JDBC objects

There is a new language construct that is introduced in JDK7 that applies to JDBC.  It’s called try-with-resource.  There is a tutorial at
It allows you to automatically close an object when it is no longer needed.  Theoretically, the object must extend java.lang.AutoCloseable.  In JDK7, that list includes java.sql.CallableStatement, Connection, PreparedStatement, Statement, ResultSet, and *RowSet.  The following code

Statement stmt = null;
try {
  stmt = con.createStatement();
} catch (Exception ignore) {
} finally {
  if (stmt != null) stmt.close()
becomes a shorter version using the new syntax
try (Statement stmt = con.createStatement()) {
} catch (Exception ignore) {

Here’s what a larger example looks like that I embedded in a WebLogic servlet for testing.  Note that there are two resources in the first try-with-resource, separated by a semi-colon.

private String doit() {
  String table2 = "test222";
  String dropsql = "drop table " + table2 + "";
  String createsql = "create table " + table2 + " ( col1 int, col2 int )";
  String insertsql = "insert into " + table2 + " values (1,2)";
  String selectsql = "select col1, col2 from " + table2 + "";
  try {
    ds = getDS();
  } catch(Exception e) {
    return("failed to get datasource");
  try (Connection conn = ds.getConnection();
    Statement stmt = conn.createStatement()) {
    try {
   } catch (Exception ignore) {} // ignore if table not dropped
   try (ResultSet rs = stmt.executeQuery(selectsql)) {;
   } catch (Exception e2) {
 } catch(Exception e) {
 return "DONE";

I’m not sure whether or not I like the programming paradigm – you can decide that for yourself.

How do I know that the connection, statement, and result set get closed?  I can turn on JDBCSQL debugging in the server and look at the server log output (this is a good trick to use to see what is happening under the covers).  Here is a stripped down version of the output.

 Connection@1 CreateStatement() 
 Connection@1 CreateStatement returns StatementWrapper@2 
 StatementWrapper@2 execute(drop table test222) 
 StatementWrapper@2 execute(drop table test222) throws java.sql.SQLSyntaxErrorException: table or view does not exist
 StatementWrapper@2 execute(create table test222 ( col1 int, col2 int )) 
 StatementWrapper@2 execute returns false 
 StatementWrapper@2 execute(insert into test222 values (1,2)) 
 StatementWrapper@2 execute returns false 
 StatementWrapper@2 executeQuery(select col1, col2 from test222) 
 StatementWrapper@2 executeQuery returns ResultSetImpl@3 
 ResultSetImpl@3 next() 
 ResultSetImpl@3 next returns true 
 ResultSetImpl@3 close() 
 ResultSetImpl@3 close returns 
 StatementWrapper@2 close() 
 StatementWrapper@2 close returns 
 Connection@1 close() 
 Connection@1 close returns

You might be thinking that there are not a lot of JDBC 4.1 compliant drivers out there and that’s true (the Oracle thin driver will have an ojdbc7.jar in version 12c, when it is released).  However, an object like java.sql.Statement implements AutoClosable so when running on JDK7 the driver Statement also is assumed to implement AutoClosable.  The object just needs to have a close() method so all of these JDBC objects have met the criteria since the initial specification.  WebLogic Server started supporting JDK7 in version 10.3.6.  Testing with all drivers that are shipped with WebLogic server, plus some drivers from various vendors, show that this JDK7 language feature works just fine with pre-JDBC 4.1 drivers. 

Note that this trick doesn't work for new JDK7 methods - you will get an AbstractMethodError.

So if you like this programming paradigm, start using it now with WebLogic Server 10.3.6 or later.

Friday May 17, 2013

Troubleshooting Tools Part 3 - Heap Dumps

A Heap Dump is a snapshot of the Java process heap memory at a given time.  It's a useful tool to troubleshoot memory leaks and other memory related issues.  Heap dumps are to memory what thread dumps are to java threads.

There are different ways to dump the java heap used by a Java application such as WebLogic Server.  Heap dumps can be triggered automatically by JVMs such as HotSpot or JRockit when an OutOfMemory event occurs.


java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\myApp\hprof-dumps myApp  

This command will generate a heap dump file in the defined heap dump directory when an OOM is thrown.  

The file will be in HPROF binary format (*.hprof).  If no path is defined the file will be created in the directory from which the java process was launched.  The format of the file with JRockit is jrockit_pid,hprof, pid being the process ID of this JVM instance.  With HotSpot the format of the file will be java_pidpid.hprof.


java.lang.OutOfMemoryError: Java heap space

Dumping heap to D:\java_pid4148.hprof ...

Heap dump file created [24921731 bytes in 1.400 secs]

For WebLogic Server these properties, HeapDumpOnOutOfMemoryError and HeapDumpPath (optional), can be added directly to the startup script that contain the Java command that launches WLS.  On Windows the script (with default name startWebLogic.cmd) lives in the domain bin directory and the parameters can be directly added to the existing JAVA_OPTIONS.

With a JRockit VM process the following command can also be used to generate a heap dump.


jrcmd pid hprofdump filename=name_of_dump_file

However if you are using this command against a non JRockit process the following error will come up: Command failed in target VM

You can also manually generate a heap dump with tools such as jmap or VisualVM.


jmap -heap:live,format=b,file=filename pid

This command will generate a heap dump file in binary format and will only count live objects (the live option is optional). Live objects are used by applications and generally reside in the Young Generation part of the heap.

As illustrated in the screenshot below, generating a heap dump with VisualVM is very simple.  VisualVM is shipped as part of the HotSpot JVM  (/bin/jvisualvm.exe).  However, VisualVM can connect to java processes that started with other JVM such as JRockit.

Once generated, the heap dump will be parsed as shown below:

VisualVM enables you to browse heap dumps as shown above with the Classes view where you can easily identify the number and percentage of instances referred by each class.

jhat can also be used to analyze heap dump files.


jhat dumpfile

Reading from dumpfile...
Dump file created Fri May 17 15:25:41 PDT 2013
Snapshot read, resolving...
Resolving 580390 objects...
Chasing references, expect 116 dots.............................................
Eliminating duplicate references................................................
Snapshot resolved.
Started HTTP server on port 7000
Server is ready.

Then you can open http://localhost:7000/ to navigate through the content of the dump file to identify objects with high number of occurrences.

Finally, as shown below, the OpenSource Eclipse MAT offers a very rich GUI that includes histograms and leak hunter actions, overview and leak suspect reports and much more.

Previous posts on troubleshooting tools:

Troubleshooting Tools Part 1 - VisualVM

Troubleshooting Tools Part 2 - jstack

Tuesday May 14, 2013

WebLogic Server on Oracle Database Appliance - How to configure a WebLogic cluster

How to configure a WebLogic cluster on Oracle Database Appliance?

It's easy with the support of Appliance Manager, WebLogic configuration for Oracle Database Appliance!

Follow the wizard driven process, you can deploy and set up a WebLogic cluster just in hours!

  1. Pick the WebLogic version. In our 2.5 release, you can choose either WebLogic 12c (12.1.1) or 11g (10.3.6).
  2. Choose the size of cluster, you can set up a 2, 4 or 8 node cluster.
  3. Input your networking information.
  4. Input load balancer networking information.
  5. Input the data source configuration information as optional.
  6. Done. Ready to deploy!

It's simple, easy and straightforward.


Oracle Traffic Director - download and check out new cool features in

As Oracle's strategic layer-7 software load balancer product, Oracle Traffic Direct is fast, reliable, secure, easy-to-use and scalable; that you can deploy as the reliable entry point for all TCP, HTTP and HTTPS traffic to application servers and web servers in your network.

The latest release Oracle Traffic Director is available for ExaLogic and Database Appliance! For download and details please visit the Traffic Director OTN website.

It this release, we have introduced some major new functionality and improvements.

Web application firewall. Oracle Traffic Director supports web application firewalls. A web application firewall (WAF) is a filter or server plugin that applies a set of rules, called rule sets, to an HTTP request. Using a web application firewall, users can inspect traffic and deny requests to protect back-end applications from CSRF vulnerabilities and common attacks such as cross-site scripting.

WebSocket Connections. Oracle Traffic Director handles WebSocket connections by default. WebSocket connections are long-lived and allow support for live content, games in real-time, video chatting, and so on.

Support for LDAP/T3 Load Balancing. Oracle Traffic Director now supports basic LDAP/T3 load balancing at layer 7, where requests are handled as generic TCP connections for traffic tunneling. It works in full-NAT mode.

Please download and try it out. For more information, check out the data sheet and the documentation:

Wednesday Apr 24, 2013

Getting Connections with Active GridLink

As a follow-up to an earlier blog, someone asked "Can you please tell us more about gravitationShrinkFrequencySeconds and DRCP ?"   DRCP support isn’t shipping yet (wait until the next version of the Database and of WLS) so I can’t talk about it.

I think that people understand how connections are allocated in generic data sources and multi data sources (MDS).   Let me talk generally about how connections are allocated in Active GridLink (AGL). With the former (generic and MDS), connection allocation is pretty much dependent on WLS.  With AGL, it is also very dependent on the database configuration and runtime statistics.  The runtime information is provided to WLS via Oracle Notification Service (ONS) in the form of up and down events and Runtime Load Balancing events.

Connections are added to the pool initially based on the configured initial capacity. Connect time load balancing based on the listener(s) is used to spread the connections across the instances in the RAC cluster. For that to work correctly, you must either specify a Single Client Access Name (SCAN) address or use LOAD_BALANCE=ON for multiple non-SCAN addresses.  If you have multiple addresses and you forget to use LOAD_BALANCE=on, then all of the connections will end up on the first RAC instance – not what you want.  Connection load balancing is intended to give out connections based on load on the available RAC instances.  It’s not perfect.  If you have two instances and they are evenly loaded, there should be approximately 50% of the connections on each instance (but don’t enter a bug if they split up 49/51).  There a general problem with load balancing that the statistics can lag reality.  If you have big swings in demand on the two instances, then you can end up creating a connection on the more heavily loaded instance.

If you go to get a connection from the pool, runtime load balancing is used to pick the instance.  That’s again based on the load on the various instances and the information is updated by default every 30 seconds.  If a connection is available on the desired instance, then you have a hit on the pool and you get the available connection.

If you go to get a connection from the pool and one doesn’t exist on the desired instance, then you have a miss.  In this case, a connection is added to the pool on demand based on connection load balancing.  That is, if you go to reserve a connection and there isn’t one available in the pool, then a new one is created and it is done using connection load balancing as described above.  There’s an assumption here that the connect time and runtime load balancing match up.

If you are running within an XA transaction, then that takes priority in determining the connection that you get.  There are restrictions on XA support within a RAC cluster such that all processing for an XA transaction branch must take place on one instance in the cluster (the restrictions are complicated so the one-instance limitation is the only safe approach).  Additionally, performance is significantly better using a single instance so you want that to happen anyway.  The first time that you get a connection, it’s via the mechanisms described above (runtime load balancing or connection load balancing).  After that, any additional requests for connections within the same XA transaction will get a connection from the same instance.  If you can’t get a connection with “XA affinity,” the application will get an exception.

Similar to XA affinity, if you are running within a Web session, you will get affinity for multiple connections created by the same session.  Unlike XA affinity, “Session Affinity” is not mandatory.  If you can’t get a connection to the same instance, it will go to another instance (but there will be a performance penalty).

When you take down a RAC instance, it generates a “planned down event”.  Any unused connections for that instance are released immediately and connections in use are released when returned to the pool.  If a RAC instance fails, it generates an “unplanned down event.”  In that case, all connections for that instance are destroyed immediately.

When a RAC instance becomes available, either a new instance or an instance is restarted, it generates an “up event.”  When that occurs, connections are proactively created on the new instance.

The pool can get filled up with connections to the “wrong” instance (heavily loaded) if load changes over time.  To help that situation, we release unused connections based on runtime load balancing information.  When gravitation shrinking occurs, one unused connection is destroyed on a heavily loaded instance.  The default period for gravitation shrinking is 30 seconds but you can control it by setting the system property “weblogic.jdbc.gravitationShrinkFrequencySeconds” to an integer value.  We don’t actively create a connection at this point.  We wait until there is demand for a new connection and then the connection load balancing will kick in.

Finally, normal shrinking happens if not disabled.  When this occurs, half of the unused connections down to minimum capacity are destroyed.  The algorithm currently just takes the first set of connections on the list without regard to load balancing (it’s random with respect to instances).  The default period is 900 seconds and you can configure this using ShrinkFrequencySeconds.

There are possible improvements that could be made with respect to pool misses, and gravitational and normal shrinking.  And the database team is working on improving the load balancing done on their end. Still, it works pretty well with instances dynamically coming and going and the load changing over time.


The official blog for Oracle WebLogic Server fans and followers!

Stay Connected


« April 2014