Tuesday May 06, 2014

How to delete a JDBC Store?

If you see the message "Skipping unexpected null message encountered on store boot", you will need to recreate your JDBC store, because it got corrupted.

You will see a message like this in the weblogic server log:
<Warning> <Messaging> <BEA-282013> <Skipping unexpected null message encountered on store boot, [ sequence=4862 queue=12 name=MyModule!MyQueue state=0 deliveryCount=2 ]>

Steps to fix the issue:
1. Make sure managed server is not running

2.  Delete the corrupted JDBC store because messages are corrupted.

See Managing JDBC Store Tables


If you have provided the prefix MY when you created your JDBC store, your table is named MYWLStore, otherwise, it is just named WLStore

To drop the table, you can use the option below or simply drop it using a database tool:

$ echo "drop MYWLStore;" > drop.ddl
$ cd <YourDomain>/bin
$. ./setDomainEnv.sh
  **Notice the . before ./
$ java utils.Schema jdbc:weblogic:oracle:DEMO weblogic.jdbc.oci.Driver -u user1 -p foobar \
  -verbose drop.ddl

3. Start managed server. The store table will be recreated when managed server is started


Monday Apr 21, 2014

How to monitor JMS Queue counts using JConsole ?

The most common way to monitor your JMS resources is by using weblogic console, e.g.:


You can also use WLST


If you like JConsole, here is the trick:

1. Set your environment
   (Linux) . ./setDomainEnv.sh
   (Windows) setDomainEnv.cmd

2. Invoke jconsole using the arguments below    

jconsole -J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote -J-Dcom.sun.tools.jconsole.mbeans.keyPropertyList=Location,type,Type,j2eeType,name,Name -debug

3. See messages counts as follows:

Related to the JMSServer
JMSServerRuntime > <MyJMSServer> > <MyWebLogicServer>  > Attributes > MessagesCurrentCount

Related to the a JMS Queue:
JMSDestinationRuntime > <MyQueueJNDIName>  >MyWebLogicServer > MyJMSServer > Attributes


Monday Apr 07, 2014

How to debug a WebLogic Server Windows Service?

Debugging WebLogic Server Windows Services can be a hard time. Sometimes it is difficult to collect debug information and in other cases, you can't even start it.

So here is my super hero "beasvc -debug"!

You can try :
beasvc -debug "Windows Service Name"

That will really show you the real error when you are not even able to start your windows service.



You can read more information about installing weblogic server as windows service in this My Oracle Support(MOS) documents(*):

How to Install MS Windows Services For FMW 11g WebLogic Domain Admin and Managed Servers (Doc ID 1060058.1)

Other useful documents that are useful when you need to collect debug information when you are using a Windows Service
How Do I Get A Java Heap Dump When Running Weblogic As A Windows Service? (Hotspot JVM) (Doc ID 1264826.1)

Enabling stdout Logging and Generating Thread Dumps When Running WebLogic Server as a Windows service (Doc ID 858746.1)

How to connect to a remote JRockit Java Virtual Machine using JRockit Mission Control (Doc ID 1075329.1)

How to Monitor using JRockit Mission Control a WebLogic Server instance on top of JRockit JVM running as a Windows service (Doc ID 1329897.1)

(*) Links will work if you are logged to MOS.

Thursday Mar 20, 2014

Why do I need to collect several thread dumps?

I used to work with samurai to read thread dumps


But I don't miss him any more after I met thread logic :).

You can read more and download about Threadlogic here:


It also provide excellet guidelines on what is happening in the environment.

But, again Why do I need to collect several thread dumps?

One question for you, Do you understand a video when you see just one screenshot? probably no, even if you have one picture from 'The End'.

Even if you don't see the movie, if you see several screenshots, you can get a better idea of what happened at some time.

Lets say that the most important part of the film for you was the end, if you have several pictures from that time, you will have a very good idea of how the movie finished.

The same is for weblogic server. If we see five thread dumps 10 or 20 seconds apart from each other when one issue occurs, we can have a very good idea of what methods were being executed when one issue occurs.
We can see what weblogic server is waiting for. This helps to identify slowness in subsystems like databases, web services or bugs in weblogic server or other products that were being executed at the time of the issue.


Wednesday Mar 12, 2014

I am configuring a weblogic messaging bridge. What is a member's JNDIName?

As per oracle documentation:


"Messaging bridge can send to and receive from distributed destinations. Oracle recommends the following configurations:

  •  If the source is distributed destination, the bridge is pinned to one of the members when it connects to the destination. It stays connected only to that member until it reconnects. This means that the bridge will not receive messages from the other members of the distributed destination. Therefore, the best practice is to configure one bridge for each member of a distributed destinations using the member's JNDIName.

  •  If the target is a distributed destination, the best practice is to send to the distributed destination using the distributed destination's JNDIName and disable server affinity.
This allows the distributed destination to load balance incoming messages."

When you configure Uniform Distributed Queue, you can see in weblogic console:

Services > Messaging > JMS Modules > <My JMS Module> > <My Uniform DistributedQueue>  > Configuration > Members > Member Name

You will see here all member's JNDIName for each member of your Uniform Distributed Queue.



Wednesday Feb 26, 2014

My new Active Directory Provider is not Working!

When you create LDAP providers, an easy way to verify they are working fine is by verifying in WebLogic console, you can see the users listed.

After added Active Directory Provider, Users and Groups are listed in Admin Console:

Security Realms -> My Realm -> Users and Groups.

When attempting to login to an application that is using the users, the login is being denied, and you might not see any clues in weblogic server logs.

If you enable Atn debug, the following is observed in server log:

<Debug> <SecurityAtn> <MyDomain> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)' for workmanager: consoleapp@null@consoleWorkManager> <<WLS Kernel>> <> <593625378f0917fe:-23dcaa48:143ea3e7180:-8000-0000000000000400> <1391205135889> <BEA-000000> <weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.authenticate authenticate failed for user MyUser>

This can occur when  default authenticator is selected as REQUIRED  by default. So the login process is denied by the default authenticator due to it is not aware of users in Active Directory.

So, to fix the issue

1. Go to Admin Console > Security Realms > <Your Realm> >Providers.
2. Make Active Directory provider is in the top of the list and set Control Flag SUFFICIENT.
3. Make default authenticator Control Flag is set to OPTIONAL.

You can read more in this My Oracle Support document:

How to Configure Active Directory as the LDAP Provider for WebLogic Server (Doc ID 1299072.1)


Friday Feb 14, 2014

How to install weblogic Administration Console after it is uninstalled?

As per security purposes, you might need to uninstall your WebLogic Administration Console, or perhaps by mistake.
I would say that this is a very uncommon scenario, but one of my customer's faced it recently.

So,  How do I uninstall WebLogic Administration Console?

1. To uninstall WebLogic Administration Console, execute uninstall script, and select Administration Console to be uninstall it.


For Windows Environments


For Linux/Unix Environments


2. Hit Next

You have Administration Console Uninstalled it.

Admin Server will start normally, but you won't have access to admin console e.g.: http://localhost:7001/console

What if you just did it by mistake and you need to access it again?

So, How do I install it again ?


1. Use the installer you used to re install the components uninstalled, e.g:

java -d64 -jar wls1036_generic.jar

2. Start Admin Server

3. Access Administration Console at http://localhost:7001/console


Tuesday Feb 04, 2014

Do I need to run WebLogic Server in a certified platform?

Well, the answer is Yes. But, Why is it important to be in a certified platform?

We have faced several issues which customers in a non certified platforms where after a long time and huge
 troubleshooting efforts we concluded that the issue was fixed after moving to a tested configuration.

WebLogic Server might run in any other non certified platform like any flavor of Linux, but we can not assure that it will work correctly.

So, What if you have a problem when running a non certified configuration?  If you don't find a known issue in the net or in My Oracle Support, try to reproducing the issue in a certified configuration. That will save a lot of time because as per out experience, the most probably cause of the issue is that the configuration where you are running WebLogic Server is causing the issue.

Eventually, if the issue still occurs, that will help to focus on the real issue and not in the noise added by the environment used to run WebLogic server.

It is also recommended to be in the latest JDK, that will help you to include in your environment the latest security fixes and patches for known issues already faced by other customers.

Another thing to consider is that you are upgrading your WebLogic Server. Then, you will probably need to upgrade your JDK as well. Always check certification matrix

Here you can find the supported configurations of Fusion Middleware products:


If you are in 12.1.1 or 12.1.2, make sure you pick the correct link. WebLogic Server 12.1.2 is very different from 12.1.1.

If you are in 11g, you will need to pick the document depending on your configuration.

You can see more information here:


Friday Dec 27, 2013

How to Connect to WebLogic DataSource from a Java Client?

Some time ago, one of my customers had issues connecting from a client to a WebLogic Datasource.
I provided the sample code to try the functionality, here are the steps:

1. Save the code below as DataSourceClient.java in /myfolder

import java.sql.*;
import javax.sql.DataSource;
import javax.naming.Context;
import javax.naming.InitialContext;
import java.util.*;
import java.io.*;
import oracle.jdbc.OracleConnection;

public class DataSourceClient {
  public static void main(String [] args) {
    OracleConnection conn=null;
    javax.sql.DataSource ds=null;
    Hashtable env = new Hashtable();
    env.put( Context.INITIAL_CONTEXT_FACTORY, "weblogic.jndi.WLInitialContextFactory" );
    env.put(Context.PROVIDER_URL, "t3://localhost:7001");

      Context context=new InitialContext( env );
      //you will need to have create a Data Source with JNDI name testDS
      ds=(javax.sql.DataSource) context.lookup ("testDS");
      conn=(OracleConnection) ds.getConnection();
      java.util.Properties prop = new java.util.Properties();
      System.out.println("Connection object details : "+conn);
    }catch(Exception ex){
      //handle the exception

2. Modify PROVIDER_URL if needed. The value is set to t3://localhost:7001
3. Add to WebLogic Server JAVA_OPTIONS -Dweblogic.jdbc.remoteEnabled=true
4. Start WebLogic Server
5. Open a UNIX console. In the same console run:
    i. cd <domain>/bin
    ii. . ./setDomainEnv.sh (Notice the '.' before './setDomainEnv.sh')
    iii. CLASSPATH="/myfolder:${CLASSPATH}"
    iv   cd /myfolder
    v.   javac DataSourceClient.java
    vi.  java DataSourceClient

If you succeeed, you will see something like this:

Connection object details : weblogic.jdbc.rmi.SerialConnection_weblogic_jdbc_rmi_internal_ConnectionImpl_weblogic_jdbc_wrapper_JTAConnection_weblogic_jdbc_wrapper_XAConnection_oracle_jdbc_driver_Logic

This means, you sucessfully connected to your WLS Datasource.

Do you need further help?

Please check WebLogic Documentation:

Tuesday Dec 17, 2013

How to review a FileStore?

You might be interested in reviewing what is inside your FileStore, below is shown a good way to start with it.

First of all, we need to create the environment:
1. Create a Persistent  Store (FileStore) FileStore-0
2. Create JMS server target to manage server MS1, persistent Store FileStore-0
3. Create a JMS Module target to  MS1
4. Create a Connection factory JNDI name: CF  (default targeting)
5. Create a Queue JNDI name Queue-0  (subdeployment target to the JMS server)
6. Restart Managed Server

Then, lets add some messages to your queue:
1. Add QueueSend to classpath
   set CLASSPATH=C:\jms\queue send transaction;%CLASSPATH%
2. <wls_domain>/bin/. ./setDonmainEnv.sh
3. java QueueSendTX t3://localhost:7003 QueuePS1 ConnectionFactoryPS1

Finally, lets dump the FileStore to a xml file, make sure your MS1 is stopped and then execute:

1. java weblogic.store.Admin
2. openfile -store FileStore-0
3. dump -store FileStore-0 -out MyStore.xml -conn -deep
4. close -store FileStore-0
5. quit

Below you can see,
QueueSendTX.java: source code of QueueSendTX.java.
beforeConsumption.xml: is the dump when I added 9 messages to my Queue.
consumed.xml: is the dump after messages were removed from the queue.



Tuesday Nov 26, 2013

How to enable Weblogic Server debugs using WebLogic Console?

This is a very useful topic for those that would like to have a first insight on why an issue is occuring.

1. The easiest way to enable debug flags in WebLogic Server is to use weblogic console. This is done in:

Environment > Servers > MyServer > Debug > weblogic

Then, enable the level of debug you need, e.g.: security, servlet, jdbc.

This change does not require weblogic Server Restart.

2. You need Make sure the severity is set to debug in weblogic console:

Environment > Servers > MyServer > Logging >Advanced > Minimum severity to log: Debug

This change requires server restart.


Monday Nov 11, 2013

How to show pending messages using WLST?

Here are the steps:

1. . ./setDomainEnv.sh
2. java weblogic.WLST
3. connect('weblogic','welcome1','t3://localhost:7001')
4. domainRuntime()
5. cd('ServerRuntimes/MS1/JMSRuntime/MS1.jms/JMSServers/JMSServer1/Destinations/JMSModule1!Queue1')
6. cursor1=cmo.getMessages('true',9999999,10)
7. msgs = cmo.getNext(cursor1, 10)
                  ** This step gets 10 messages, you can call again cmo.getNext(cursor1, 10) to get the next 10 msgs
8. print(msgs)

My assumption, is that you had created:
a. Managed Server MS1.
b. JMS Server JMSServer1.
c. Module called JMSModule1.
d. Inside of JMSModule1, a Queue called Queue1.

If you read my previous post:

How to get Messages Pending Count from a Queue using WLST?

You can see that both are very similar.  Sometimes it is difficult to get a WLST Script sample, but you can use ls() function to know about other functionalities you don't have a sample code.

***Until step 5, nothing new comparing to my previous post.
5. cd('ServerRuntimes/MS1/JMSRuntime/MS1.jms/JMSServers/JMSServer1/Destinations/JMSModule1!Queue1')
6. ls()

You will see, MessagesPendingCount, getMessages along a lot of other functionalities available in this Queue.

e.g, you can see:
-r-x   getMessages                                  String : String(selector),Integer(timeout),Integer(state)

Here you can check the complete MBean Reference:
See JMSDestinationRuntimeMBean.


Wednesday Oct 30, 2013

How to get Messages Pending Count from a Queue using WLST?

WLST is a scripting Language that helps to achieve similar functionality as the ones you have in WebLogic console, but in a command line fashion.

You can develop your WLST Scripts using Eclipse OEPE, read more here:

Finally, here is an example to get Messages Pending Count using WLST:

 . ./setDomainEnv.sh
 java weblogic.WLST
 jms= getMBean ('ServerRuntimes/MyManagedServer/JMSRuntime/MyManagedServer.jms/JMSServers/MyJMSServer/Destinations/MyModule!MyQueue')


WLST documentation:

Monday Oct 21, 2013

java.util.zip.ZipException: Error opening file When Deploying an Application to Weblogic Server

The latest weeks we had a hard time trying to solve a deployment issue.

* WebLogic Server 10.3.6
* Target: WLS Cluster

<21-10-2013 05:29:40 PM CLST> <Error> <Console> <BEA-240003> <Console encountered the following error weblogic.management.DeploymentException:
        at weblogic.servlet.internal.WarDeploymentFactory.findOrCreateComponentMBeans(WarDeploymentFactory.java:69)
        at weblogic.application.internal.MBeanFactoryImpl.findOrCreateComponentMBeans(MBeanFactoryImpl.java:48)
        at weblogic.application.internal.MBeanFactoryImpl.createComponentMBeans(MBeanFactoryImpl.java:110)
        at weblogic.application.internal.MBeanFactoryImpl.initializeMBeans(MBeanFactoryImpl.java:76)
        at weblogic.management.deploy.internal.MBeanConverter.createApplicationMBean(MBeanConverter.java:89)
        at weblogic.management.deploy.internal.MBeanConverter.createApplicationForAppDeployment(MBeanConverter.java:67)
        at weblogic.management.deploy.internal.MBeanConverter.setupNew81MBean(MBeanConverter.java:315)
        at weblogic.deploy.internal.targetserver.operations.ActivateOperation.compatibilityProcessor(ActivateOperation.java:81)
        at weblogic.deploy.internal.targetserver.operations.AbstractOperation.setupPrepare(AbstractOperation.java:295)
        at weblogic.deploy.internal.targetserver.operations.ActivateOperation.doPrepare(ActivateOperation.java:97)
        at weblogic.deploy.internal.targetserver.operations.AbstractOperation.prepare(AbstractOperation.java:217)
        at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentPrepare(DeploymentManager.java:747)
        at weblogic.deploy.internal.targetserver.DeploymentManager.prepareDeploymentList(DeploymentManager.java:1216)
        at weblogic.deploy.internal.targetserver.DeploymentManager.handlePrepare(DeploymentManager.java:250)
        at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.prepare(DeploymentServiceDispatcher.java:159)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:13)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:46)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: java.util.zip.ZipException: Error opening file - C:\Oracle\Middleware\user_projects\domains\MyDomain\servers\MyServer\stage\myapp\myapp.war Message - error in opening zip file
        at weblogic.servlet.utils.WarUtils.existsInWar(WarUtils.java:87)
        at weblogic.servlet.utils.WarUtils.isWebServices(WarUtils.java:76)
        at weblogic.servlet.internal.WarDeploymentFactory.findOrCreateComponentMBeans(WarDeploymentFactory.java:61)

So the first idea you have with that error is that the war file is corrupted or has incorrect privileges.        

We tried:
1. Unzipping the  war file, the file was perfect.
2. Checking the size, same size as in other environments.
3. Checking the ownership of the file, same as in other environments.
4. Checking the permissions of the file, same as other applications.

Then we accepted the file was fine, so we tried enabling some deployment debugs, but no clues.

We also tried:

1. Delete all contents of <MyDomain/servers/<MyServer>/tmp> a and <MyDomain/servers/<MyServer>/cache> folders, the issue persisted.
2. When renaming the application the deployment was sucessful
3. When targeting to the Admin Server, deployment was also working.
4. Using 'Copy this application onto every target for me' didn't help either.

Finally, my friend 'Test Case' solved the issue again.

I saw this name in the config.xml


So, it turned out that customer had created a DataSource with the same name as the application 'myapp' in the above example.

By deleting the datasource and created another exact DataSource with a different name the issue was solved.

At this point, Do you know Why 'java.util.zip.ZipException: Error opening file' was occurring?

Because all names is WebLogic Server need to be unique.


"Assigning Names to WebLogic Server Resources
Make sure that each configurable resource in your WebLogic Server environment has a unique name. Each, domain, server, machine, cluster, JDBC data source, virtual host, or other resource must have a unique name."


Monday Oct 14, 2013

Troubleshooting java.sql.SQLRecoverableException: IO Error: Invalid Packet Lenght

Some time ago, one of my customers had the following intermittent issue in WebLogic Server:

java.sql.SQLRecoverableException: IO Error: Invalid Packet Lenght
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:875)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1153)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1275)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3620)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
at jdbc.JDBCTester.main(JDBCTester.java:31)
Caused by: oracle.net.ns.NetException: Invalid Packet Lenght
at oracle.net.ns.Packet.processHeader(Packet.java:372)
at oracle.net.ns.Packet.receive(Packet.java:304)
at oracle.net.ns.DataPacket.receive(DataPacket.java:104)

I tried enabling jdbc debugs but no clues on what was happening.

I decided to see if this was happening outside of WLS.

My customer tested using sqlplus but the issue was not occurring.

Then, I decided to check if the issue occurred with any Java application connected to the database.

My customer run my test program about 1000 times and then voila!, the issue was reproduced outside of WebLogic Server.

It turned out that there were network issues with this box that were fixed by customer's network team.

You can use these types of testers under similar issues you might experience to try to isolate the component/layer where the issue occurs.


Here is my JDBC tester.

JDBC Tester

i. Save the code below as TestJDBC.java
The successful case will print
finished DB testing

package jdbc;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.Statement;
import java.sql.DriverManager;
import javax.sql.DataSource;
class TestJDBC {
public static void main(String[] args) {
try {
String url = "jdbc:oracle:thin:@//localhost:1521/orcl";
Connection conn = DriverManager.getConnection(url, "javauser", "javauser");
PreparedStatement ps = null;
ResultSet rs = null;
String sid;
String sSQL = "select count(1) from dual";
try {
ps = conn.prepareStatement(sSQL);
rs = ps.executeQuery();

} catch (Exception e) {
} finally {
try {
} catch (Exception rse) {
try {
} catch (Exception sse) {
try {
} catch (Exception cse) {
System.out.println("finished DB testing");

} catch (Exception e) {
System.err.println("Got an exception! ");
ii. mkdir jdbc
iii. mv TestJDBC.java jdbc
iv. cp WL_HOME/server/lib/ojdbc6.jar  jdbc
vi. Modify jdbc/TestJDBC.java according to your settings
                     String url = "jdbc:oracle:thin:@//localhost:1521/orcl";

Connection conn = DriverManager.getConnection(url, "javauser", "javauser");
vii. execute:
. ./setDomainEnv.sh (this file is locate at <domain>/bin folder. Notice the . before ./)
viii. CLASSPATH="${CLASSPATH}:.:/jdbc"
ix. export CLASSPATH
x. cd jdbc
xi. javac TestJDBC.java
xii. cd ..
xiii. java jdbc.TestJDBC


My name is Luz Mestre. I work as Principal Technical Support Engineer at Oracle Support. I'll post here the most interesting challenges I have in my daily work. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.


« November 2015