Monday Mar 09, 2015

PSU, Patching and Classpath Problems on Weblogic Server

When applying a new PSU on a Weblogic Server, there are some facts that you must keep in mind
- First of all, never install a PSU over another one. As can be seen on the MOS Note 1573509.2: “Each PSU will conflict with any prior PSU in the series. To install a subsequent PSU, any existing PSU must first be uninstalled.”.
- And what happens if I am unable to uninstall a PSU? That issue is addressed by the MOS Note 1349322.1.
- It can also happen that when attempting to apply a patch, a conflict message like the following is thrown: "Patch A is mutually exclusive and cannot coexist with patch(es): B", and when trying to remove patch B, Smart Update fails with a message "Patch not installed: B". Such situations are described on MOS Note 1970064.1.
- Avoid having different PSU levels on the Weblogic, even if you have multiple domains or clustered across different physical machines.

And why could the above facts be related to the Classpath?
As can be seen on MOS Note 1509703.1, there could be situations where, after applying a WebLogic Server 10.3.6 PSU, a managed server fails to start when the classpath is provided and started from the admin console. A critical BEA-000362 message is thrown:
<BEA-000362> <Server failed. Reason: [Management:141266]Parsing Failure in config.xml: failed to find method MethodName{methodName='setCacheInAppDirectory', paramTypes=[boolean]} on class>
This happens because in the PSUs for WLS 10.3.6 (, every time an application deployment is done, it adds a <cache-in-app-directory> element into the config.xml file for that application. To parse this new element, the classes for the PSU must be loaded rather than the original classes for application deployment. So specifying WL_HOME/server/lib/weblogic_sp.jar;WL_HOME/server/lib/weblogic.jar in the classpath of Server-start may cause the problem. There is no need to set these in the classpath of Server-start since they will come in from the system classpath. The weblogic_patch.jar must precede weblogic_sp.jar and weblogic.jar -- this ensures that the classes in the patch are loaded rather than the unpatched classes.
The already mentioned MOS Note 1509703.1 contains an additional procedure for deployments after applying a PSU.

Same jars/classes included multiple times on a Classpath
Also, note that sometimes same jars/classes may appear multiple times on a classpath (this happens mainly because the command lines are modified as any other component of the WLS architecture as time goes by with new versions of Oracle products and customer's apps). The JVM searches for them according to the specified order, and it would be correct in general terms. However, it will depend on the implementation of the classloader. But there are some potential problems, for example:
- When loading classes within a web framework the deployed jar/war/ear/sar files may be checked before the official classpath.
- And what would happen if two different versions of a same jar are invoked?

Wednesday Dec 17, 2014

Checking the Oracle JDBC Driver Version on a Weblogic Server

The JDBC driver is typically located at the location WL_HOME/server/lib of the installation directory. The file is ojdbc7.jar or ojdbc6.jar (for new versions of WLS), or ojdbc14.jar (for older versions of WLS).

- One way to check the JDBC driver version is to open the ojdbc jar file and go inside the META-INF folder, and then open the "MANIFEST.MF" file. The version can be seen next to "Specification-Version".

- Another way is to run the command below on the location mentioned previously:
   java -jar ojdbc6.jar -getversion

Note that you must use the JDBC JAR file intended for the version of the JDK that you are running. For example, "ojdbc5.jar" is intended for use with JDK 1.5. So if you run JDK 1.5 with the JDBC driver JAR file "ojdbc6.jar" then a "java.lang.UnsupportedClassVersionError: Bad version number in .class file" error message will be thrown when performing this check.

Thursday Jun 26, 2014

Data source in suspended state: BEA-001156 error because maximum number of sessions was exceeded in the database (ORA-00018)

Recently, I worked a Service Request where a data source was in suspended state. In the log files it could be seen a BEA-001156 error message, and the stack trace (obviously shortened in this example) contained something like the following:
<BEA-001156> <Stack trace associated with message 001129 follows:
java.sql.SQLException: ORA-00018: maximum number of sessions exceeded
at oracle.jdbc.driver.T4CTTIoer.processError(
at oracle.jdbc.driver.T4CTTIoer.processError(
at oracle.jdbc.driver.T4CTTIoer.processError(
at oracle.jdbc.driver.T4CTTIoauthenticate.processError(
at oracle.jdbc.driver.T4CTTIfun.receive(
at oracle.jdbc.driver.T4CTTIfun.doRPC(
at oracle.jdbc.driver.T4CTTIoauthenticate.doOSESSKEY(
at oracle.jdbc.driver.T4CConnection.logon(
at oracle.jdbc.driver.PhysicalConnection.(
at oracle.jdbc.driver.T4CConnection.(
at oracle.jdbc.driver.T4CDriverExtension.getConnection(
at oracle.jdbc.driver.OracleDriver.connect(
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(
at oracle.jdbc.xa.client.OracleXADataSource.getPooledConnection(
at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(
at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(

Seeing at the error message at the top, it is clearly a session handling problem at database level. Note that, depending on how your application is designed/programmed, recursive sessions can be created and sometimes it could be hard to track all of them, even more in periods of high load.

When this type of issue occur, the most common solution is to increase the SESSIONS parameter of the init.ora configuration file.

It is usually recommended to preserve 50% of the SESSIONS value for recursive sessions.

Wednesday Mar 26, 2014

Independent Clusters

Some days ago, I was asked if it's possible to have an Oracle Service Bus (OSB) Domain domain with 2 independent clusters. The answer is no. In a domain you can have one and only one OSB cluster. An OSB cluster can co-exist with other clusters, like a WLS-only clusters, SOA Suite cluster, etc., but not with another OSB cluster. As a matter of fact, in this case, the observed behavior was that all services were assigned to the first created OSB cluster, and the newer one was not responding the URL calls. Works as designed.

However, for a WLS domain, is it possible to have 2 independent clusters? In this case, the answer is yes. Here you can check what types of objects can be clustered and what cannot be clustered.

The following types of objects can be clustered in a WebLogic Server deployment:
  • Servlets
  • JSPs
  • EJBs
  • Remote Method Invocation (RMI) objects
  • Java Messaging Service (JMS) destinations
The following APIs and internal services cannot be clustered in WebLogic Server:
  • File services including file shares
  • Time service

Wednesday Dec 12, 2012

Managed servers getting down regularly by Node Manager. WAD?

Recently I have been working on a service request where several instances were running, and several technologies were being used, including SOA, BAM, BPEL and others.

At a first glance, this may seem to be a Node Manager problem. But on this situation, the problem was actually at JMS - Persistent Store level. Node Manager can automatically restart Managed Servers that have the "failed" health state, or have shut down unexpectedly due to a system crash or reboot. As a matter of fact, from the provided log files it was clear that the instance was becoming unhealthy because of a Persistent Store problem.

So finally, the problem here was not with Node Manager as it was working as designed, and the restart was being caused by the Persistent Store. After this Persistent Store problem was fixed, everything went fine.

This particular issue that I worked was on an Exalogic machine, but note that this may happen on any hardware running Weblogic.

Friday May 18, 2012

CONNECTION_REFUSED messages on load balancing in Weblogic with OHS or Apache

In the last months I have had to work on some issues related to load balancing. It is very important to understand how the layers interact between them and where specific settings must be done.

Some people gets upset with the fact that OHS/Apache do load balancing even to servers that are shutdown and may be losing transactions.

This document provides very good tips about how many Production critical issues can be resolved just by setting the appropriate values for some parameters.

Personally, I think that the DynamicServerList parameter (which is in fact the first one mentioned on the document linked above) is particularly important to understand. As can be seen at this documentation from Oracle:
In a clustered environment, a plug-in may dispatch requests to an unavailable WebLogic Server instance because the DynamicServerList is not current in all plug-in processes.
DynamicServerList=ON works with a single Apache server (httpd daemon process), but for more than one server, such as StartServers=5, the dynamic server list will not be updated across all httpd instances until they have all tried to contact a WebLogic Server instance. This is because they are separate processes. This delay in updating the dynamic server list could allow an Apache httpd process to contact a server that another httpd process has marked as dead. Only after such an attempt will the server list will be updated within the proxy. One possible solution if this is undesirable is to set the DynamicServerList to OFF.
In a non-clustered environment, a plug-in may lose the stickiness of a session created after restarting WebLogic Server instances, because some plug-in processes do not have the new JVMID of those restarted servers, and treat them as unknown JVMIDs.
To avoid these issues, upgrade to Apache 2.0.x and configure Apache to use the multi-threaded and single-process model, mpm_worker_module.

Also, this Oracle documentation provides inportant information about "Failover, Cookies, and HTTP Sessions", and "Tuning to Reduce Connection_Refused Errors".

As can be seen at this Apache document, the MaxRequestsPerChild directive sets the limit on the number of requests that an individual child server will handle during its life.

Note that mod_proxy and related modules implement a proxy/gateway for Apache HTTP Server, supporting a number of popular protocols as well as several different load balancing algorithms. Third-party modules can add support for additional protocols and load balancing algorithms.

On Oracle Forums I also found a very interesting thread:
The error which you are getting is a common which can be fixed by increasing the "AcceptBackLog" value by 25% until error disappears from weblogic console (Path: Servers => => Configuration tab=> Tuning sub-tab.) and setting the value to ON for "KeepAlive" in the httpd.conf which should take care of your issue.
Topic: Tuning Connection Backlog Buffering
Search for "KeepAliveEnabled":
Also here is a link which would be helpful to understand some common issue which occurs when using a plug-in and there are solutions:

May transactions be affected because of this?
Certainly yes, but it depends on how your application is developed. A good practice would be to create a bunch of transactions and track them to check if some are missed or not. This Transaction and redelivery in JMS article may be helpful.

Wednesday Dec 14, 2011

WebLogic Server 12c

WebLogic Server 12c has been released!

It is now certified for the full Java EE 6 platform specification, which enables higher developer productivity with standards-based, modern APIs, including Servlet 3.0, JAX-RS 1.1, Java Server Faces 2.1, EJB 3.1, Context and Dependency Injection for Java, and many others.

It also adds support for reading and using GlassFish's web deployment descriptors. This is useful for providing specific GlassFish behavioral settings and mappings for resources and security to WebLogic Server.

Monday Aug 08, 2011

Error " Address already in use: JVM_Bind" when running Node Manager in Windows

After a brand new install of WebLogic Server, when accessing node manager the following error is thrown:
GRAVE: Fatal error in node manager server Address already in use: JVM_Bind at Method) at at at Source) at at at at Source) at Source) at weblogic.nodemanager.server.SSLListener.init( at weblogic.nodemanager.server.NMServer.start( at weblogic.nodemanager.server.NMServer.main( at weblogic.NodeManager.main(

When WebLogic is installed in Windows, the NodeManager is set as a Windows service which is automatically started up and runs on port 5556 (default).

The node manager is already started. It can be shut down in the windows services, and to avoid that it is started up automatically along with Windows, the startup type can be changed.

Monday Jul 25, 2011

WLST on web

My colleague Antonio De Juan, who is a partner of mine in the Chilean office of Oracle, has created a really innovative application named wlstonweb, which allows to have interactive WLST Sessions and WLST Script Management both via a web browser.

The tool can be accessed here.

About the tool Creator: Antonio De Juan
Antonio has been part of the Middleware - Application Server Team, supporting WebLogic Server for almost two years and he has recently joined to Coherence Support as second product too.

Tuesday Jul 19, 2011

Using WebLogic Server to Simply Store Static Content

A couple of weeks ago I saw a forum thread which I found very interesting. A guy was asking if it's possible or not to use WebLogic as a Web Server, in a similar way than when an index.html file is placed on the 'www' directory of Apache. So I answered him.

Actually you can use WebLogic as a web server, but unlike Apache where you place a html file to the www directory and you can see it immediately, in WebLogic you have to deploy it to a domain.

1. The html files (or any other static content like pdf files) must be in a directory of your choice (i.e. C:\testhtml).

2. On that directory create a WEB-INF directory (from the example of step 1 it would be C:\testhtml\WEB-INF).

3. In the WEB-INF directory create a file called web.xml with the following content:
<?xml version="1.0" encoding="UTF-8"?>
"-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN"

4. Login to the WebLogic Console to deploy the application (i.e. http://[your_server]:7001/console )

5. Click on Deployments.

6. Click Install and set the path to the directory of step 3 (if the directory only contains the html file and the WEB-INF subdirectory you will see no files to select, but the Next button will be enabled anyway).

7. Leave default "Install this deployment as an application" and click Next.

8. Select the servers you wish to deploy this.

9. Accept the defaults and click Finish.

10. Activate Changes if the message appears.

11. You should now be able to see the application started in the deployments screen.

12. You can now access your static content on the WebLogic Server port via the following URL: http://[your_server]:7001/testhtml/[your_static_file]

Thursday May 19, 2011

Beware the JVM arguments!

In the last weeks I have had a bunch of Service Requests where a JVM argument ("-D" or "-X") is set as startup setting of a server on the WebLogic Server Administration Console, then the server is started up through the console (running a shell script) and the specified settings do not take effect.

This is an expected behavior because, according to the WebLogic lifecycle, the arguments set on the administration console (web) work only when the managed server is started up through the console. Arguments set on the shell script work for any way of start up.

So, keep this in mind:
- Set the JVM arguments on the startup script to make sure they will take effect
- Or if you set those parameters through the WebLogic Server Administration Console, then you should start the managed server using the console too instead of the, startManagedWebLogic.cmd or any other script

Principal Technical Support Engineer in the Engineered (Systems) Enterprise Support Team - EEST.
Former member of the Coherence and Java Technologies Support Teams.


« October 2015