Inter-cluster Data Replication

Coherence clustering technology makes a lot of sense for customers to be able to scale applications horizontally and reliably with very fast predictable performance.  This is of course easy to do within one data center on a fast network.  One of the common challenges that customers have is around High Availability and Disaster Recovery and keeping data synchronized across data centers.  Normally Coherence is optimized to use UDP unicast or multicast, but what happens if the network is unreliable and/or has high latency which is common when networking multiple data centers?  The answer is Coherence TCP Extend, which addresses these challenges by using TCP and the result is that multiple Coherence clusters can communicate together.

Multiple data centers are not the only reason you need to keep multiple clusters in sync.  I recently came across a use case where it made sense to have separate Coherence clusters on different physical machines.  The JVM processes for each machine have a single digit millisecond SLA for round trip time per request, so they are very sensitive to any outside events and it was a requirement to isolate the processes as much as possible.  In this scenario one physical machine was Active and one was Passive for fail-over.  During load-testing on the Active machine we found that bringing up cluster members on the Passive machine in the middle of the test impacted performance when both machines were part of the same cluster.  Using a single Coherence cluster per machine ensures that the impact of cluster membership events are isolated to that machine only.

Leveraging the Coherence Incubator Push Replication pattern, multiple Coherence clusters can keep data in sync whether they are on the same network subnet with low-latency or hundreds of miles apart in separate data centers.  Let's take a look at the simplest example, which is Active - Passive scenario.  In this situation we'll use Push Replication to make sure that Cache Entry operations (insert/update/delete) in the Active cache are replicated in the Passive cache.

Run the Example

To run this example called ActivePassiveExample - which is included in the src distribution of the Push Replication Pattern, I used the following software:

• Coherence 3.5.2 for Java,
• Coherence Common 1.4.0
• Command Pattern 2.4.0
• Messaging Pattern 2.4.0
• Push Replication Pattern 2.4.0
• Apache Ant 1.7.1
• JMX RI 1.2.1 - this is optional, simply do not issue the "-http" on either of the jmx ant targets which will prevent the JMX HTTP Adapter from binding
• My build.xml and other example code

The Ant Approach

One of my colleagues Randy Stafford introduced me using an ant build.xml file to organize different types of Coherence cluster processes.  I found it to be a much simpler and cleaner approach than what I had been doing (multiple shell scripts or multiple Eclipse launch configurations).  This way it is simple to centralize the shared configuration to a set of properties that are used by all of the ant targets and each process can easily override one of the properties with a -DpropertyName=propertyValue at the command line.  Take a look at the build.xml file and let me know what you think.

Step by Step

• Set the build.properties paths according to your environment
• Set up shells with ANT_HOME and JAVA_HOME and the PATH, setAntEnv.txt is an example
• Run ant from those shells with the appropriate targets as described in build.xml.  Minimum for this example:
  ant compile (you can reuse this shell)
  ant run_active_cache_server
  ant run_passive_cache_server
  ant run_active_publisher
• Most likely you will also want to run the JMX and Console processes to see the values.
  ant run_active_jmx
  ant run_passive_jmx
  ant run_passive_console

Guided Tour

Once the JMX processes are running for Active and Passive, you can use jconsole to locally bind to the two MBeanConnector process.

To verify that the passive cache received all the updates you can execute the run_passive_console target and execute two commands to see the contents.

• cache passive-cache
• list

Notice that at the bottom of my output that the entry values are the last ten number leading up to 10000, which is what we expect looking at the sample publishing code.

     [java]   ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.5, OldestMemberId=1}
     [java]   InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
     [java]   )
cache passive-cache
     [java] 2009-10-22 17:07:31.797/10.344 Oracle Coherence GE 3.5.1/461p2 <Info> (thread=Main Thread, member=3): Loaded cache configuration from "jar:file:/C:/Oracle/coherence-v3.5.1b461/coherence/lib/coherence.jar!/coherence-cache-config.
xml"
     [java] 2009-10-22 17:07:32.156/10.703 Oracle Coherence GE 3.5.1/461p2 <D5> (thread=DistributedCache, member=3): Service DistributedCache joined the cluster with senior service member 1
     [java] 2009-10-22 17:07:32.172/10.719 Oracle Coherence GE 3.5.1/461p2 <D5> (thread=DistributedCache, member=3): Service DistributedCache: received ServiceConfigSync containing 259 entries
     [java] <distributed-scheme>
     [java]   <scheme-name>example-distributed</scheme-name>
     [java]   <service-name>DistributedCache</service-name>
     [java]   <backing-map-scheme>
     [java]     <local-scheme>
     [java]       <scheme-ref>example-binary-backing-map</scheme-ref>
     [java]     </local-scheme>
     [java]   </backing-map-scheme>
     [java]   <autostart>true</autostart>
     [java] </distributed-scheme>
     [java] Map (?):
list
     [java] 2 = 9992
     [java] 3 = 9993
     [java] 1 = 9991
     [java] 0 = 10000
     [java] 5 = 9995
     [java] 4 = 9994
     [java] 6 = 9996
     [java] 8 = 9998
     [java] 9 = 9999
     [java] 7 = 9997
     [java] Map (passive-cache):

Let's look at the Active cluster processes.  You can see that there are caches defined for many of the Incubator patterns.  Below you can see that the publishing-active cache should have a size of 10 after running the run_active_publisher ant target.

Consider the role of some of the other caches:
DistributedCacheForCommandPattern - stores command contexts and co-located commands

DistributedCacheForMessages - stores messages (in this case - entries that need to be pushed to the other cluster)

DistributedCacheForSubscriptions - maintains durable subscribers for the publishers

You can learn more about these caches by looking at the Coherence Incubator Wiki and reviewing the cache-config xml files that correspond to the Incubator project they are named for.

Looking at the MBean for the PublishingService, you can find operations to suspend, resume, and drain messages that have yet to be published.

So if you suspend() the PublishingService, any changes to the publishing-active cache will be queued up as messages in the DistributedCacheForMessages coherence.messagingpattern.messages cache.  You can try this by executing the suspend() operation then the run_active_publisher target again.  You should then see the messaging cache full of the operations waiting to be replicated to the passive-cache.

Now if you execute the resume() operation on the PublisherService, the messages will replicate to the passive-cache.  Alternatively you could execute the drain() operation and all of the messages waiting to be replicated would be deleted and the passive-cache would never receive those updates.

On the Passive cluster, you can see the TCP Extend working by looking at the ConnectionManager and the Connection MBeans.  The passive process is listening in this case on port 20000 and has 1 active connection.

Conclusion

This should give you a quick idea of the capabilities of the Push Replication Pattern and some of the JMX enabled capabilities in the Coherence Incubator projects.  This is the most basic example and there are also samples for Hub-and-Spoke, Active-Active, and Federated patterns which are more complex.

I have also posted a Deployment Plan Sample on http://samplecode.oracle.com.  The sample illustrates how to change a web.xml context parameter value from “development” to “qualityassurance” to “production” all without modifying the EAR file at all.  The presentation and sample should help you become more familiar with the capabilities.

UPDATED 9/9/2009 – Also there is a complete Oracle By Example – showing how to create and use a Deployment Plan step-by-step.

The new OTN URL for legacy BEA releases is:
http://www.oracle.com/technology/documentation/bea_doc_index.html
The new OTN URL for legacy WLS releases is:
http://www.oracle.com/technology/documentation/weblogic_server.html

My Documentation Tips

In case you forget or misplace these URL’s it’s helpful to know how to navigate there from scratch.  Let me give you a short tour that will at least point you on how to find the older WebLogic Server release documentation.  The documentation for the 11g release of middleware (powered by WebLogic 10.3.1) has a new look and feel compared to edocs.   You can always find the documentation by starting at OTN http://www.oracle.com/technology/.

• There is a drop-down menu that will take you to the middleware documentation. http://www.oracle.com/technology/documentation/middleware.html
• If you click through to the 11g html library you’ll land at a page like this: http://download.oracle.com/docs/cd/E12839_01/index.htm
• There is a link in the left column to drill down into WebLogic Server which takes you here: http://download.oracle.com/docs/cd/E12839_01/wls.htm
• On the bottom right-hand side of the WLS page you’ll see the links to the older releases.

Personally the way I like to use the documentation is via google search.

I have a keyboard shortcut that allows me to type “wls securing web applications” which then automatically opens a the google search for:

• site:http://download.oracle.com/docs/cd/E12839_01/ securing web applications

Check out the results, it works pretty well.

http://www.google.com/search?&ie=utf-8&oe=utf-8&rls=org.mozilla:en-US:official&client=firefox-a&q=site%3Ahttp%3A%2F%2Fdownload.oracle.com%2Fdocs%2Fcd%2FE12839_01%2F+securing+web+applications

11gR1 Release

It is a very exciting time to be involved with Oracle Fusion Middleware.  First, there are many new features and capabilities in the recently released 11gR1 that I’m learning about in training this week.  My focus is mostly on WebLogic Server and surrounding technologies categorized in the Application Grid space and the SOA products.  Beyond the best-of-breed qualities in the stand-alone middleware products, there is an ever-increasing amount of synergy between Oracle’s other products, both middleware and elsewhere.  The number of products certified and making use of WebLogic Server is incredible.  This has high value for customers in the consolidation of skill-sets and infrastructure.  I am personally very impressed with the amount of innovation that has occurred in a little over a year since Oracle acquired BEA Systems, and I’m excited about what is coming in future releases.  My blog folder where I store ideas for future blog entries just keeps getting more and more backlogged as I come across things worth investigating and sharing.

Deployment Plan JSR-88 Example in WebLogic Server

The Forum Question

One of the ways I gauge the increasing interest in WebLogic Server is the increasing number of posts in the WebLogic Server - General forum on OTN.  Today was an especially busy day and someone inquired on a fairly straight-forward problem:

Can I change the context root of a web application without modifying the archive?

How can you do it?

Fortunately if you need to make minor configuration adjustments in application deployment descriptors (such as web.xml, weblogic.xml, application.xml, or weblogic-application.xml), but do not need to change any application code you can use a Deployment Plan (also known as JSR-88) to do this in WebLogic Server deployments.  I’m not going to go into great detail here, but the essential idea is that an xml file, usually called Plan.xml that is located outside the application archive, can be used to override or set undeclared values in the deployment descriptors that are embedded in an application archive.  See the image below.

Why would you ever need this?

There are many reasons why you may not want to modify an application archive, one of which is testing.  For example, if you have successfully completed testing for a particular version an application, it is desirable to keep the application archive unmodified between environments so you can have increased confidence that the application will behave the same in multiple environments as it is promoted.  Another reason might be portability.  You could have one generic J2EE or JEE application archive without proprietary deployment descriptors and put all of those proprietary deployment descriptor values outside of that archive.  I know that WebLogic Portal code used this feature at one time to change the verbosity of a log level that was set with a Servlet initialization paramter, so you could also use this to change configuration values at runtime provided those are in a supported deployment descriptor.

Several other WebLogic bloggers have addressed Deployment Plans, so before I write about how I addressed this particular forum question, just know that you can get a general overview and more detailed examples from the following posts:

• http://m-button.blogspot.com/2008/08/how-to-use-deployment-plan.html
• http://biemond.blogspot.com/2009/04/using-weblogic-deployment-plan-to.html
• In French – but with some screencasts - http://blog.xebia.fr/2008/04/17/les-plans-de-deploiement-weblogic/ and translated to English with Google http://bit.ly/SbQPx

A Simple Example

There is a context-root element in a web application module’s weblogic.xml file that specifies the context root of the application.  When building a web application with Oracle Enterprise Pack for Eclipse, the value is automatically set to the Web Application name.  Check out the WEB-INF/weblogic.xml from my Web Application named PlanWEB.

<?xml version="1.0" encoding="UTF-8"?>
<wls:weblogic-web-app xmlns:wls="http://xmlns.oracle.com/weblogic/weblogic-web-app" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd http://xmlns.oracle.com/weblogic/weblogic-web-app http://xmlns.oracle.com/weblogic/weblogic-web-app/1.0/weblogic-web-app.xsd">
    <wls:weblogic-version>10.3.1</wls:weblogic-version>
    <wls:context-root>PlanWEB</wls:context-root>
</wls:weblogic-web-app>

I have confirmed that my application is deployed and listening at the /PlanWEB context root on my server.  How you deploy it, weblogic.Deployer, WLST, or the console is up to you.  Now lets say I want to change the context root to /FooWEB at runtime, even though the weblogic.xml file in the application archive will have PlanWEB listed in the deployment descriptor.

Option 1 – The Console – The easy GUI driven way

Browse the deployment, specifically to the web module of the context root you want to change.  Here’s a shot of my Deployments page after I have expanded the EAR to see it’s modules.

Home –> Deployments –> PlanEAR

Click on PlanWEB, then click the configuration tab and look at the bottom of that page.  You’ll see the current context root, but not editable (depending on your console preferences it may already be editable).

• Lock and edit the console if you have not already and put in a new value like “FooWEB” and click the save button.
• The next page should prompt you for the location where the console should write the plan.xml file on the file system.  Choose whatever location makes sense, but it probably makes sense to put it somewhere in the domain directory structure or some other standard or convention you come up with.
• Now activate the session.  The context root should now be /FooWEB and the configuration tab of the console show now also reflect that.

Here is my plan.xml that the WebLogic Server Console generated for me.

<?xml version='1.0' encoding='UTF-8'?>
<deployment-plan xmlns="http://xmlns.oracle.com/weblogic/deployment-plan" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/deployment-plan http://xmlns.oracle.com/weblogic/deployment-plan/1.0/deployment-plan.xsd">
  <application-name>foo</application-name>
  <variable-definition>
    <variable>
      <name>WeblogicWebApp_ContextRoots_12483492503430</name>
      <value>"FooWEB"</value>
    </variable>
  </variable-definition>
  <module-override>
    <module-name>PlanEAR.ear</module-name>
    <module-type>ear</module-type>
    <module-descriptor external="false">
      <root-element>weblogic-application</root-element>
      <uri>META-INF/weblogic-application.xml</uri>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>application</root-element>
      <uri>META-INF/application.xml</uri>
    </module-descriptor>
    <module-descriptor external="true">
      <root-element>wldf-resource</root-element>
      <uri>META-INF/weblogic-diagnostics.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>PlanWEB.war</module-name>
    <module-type>war</module-type>
    <module-descriptor external="false">
      <root-element>weblogic-web-app</root-element>
      <uri>WEB-INF/weblogic.xml</uri>
      <variable-assignment>
        <name>WeblogicWebApp_ContextRoots_12483492503430</name>
        <xpath>/weblogic-web-app/context-root</xpath>
        <operation>replace</operation>
      </variable-assignment>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>web-app</root-element>
      <uri>WEB-INF/web.xml</uri>
    </module-descriptor>
  </module-override>
  <config-root>C:\temp\foo\plan</config-root>
</deployment-plan>

Option 2 – weblogic.PlanGenerator – a more manual approach that could be used for automation

Here are the steps for creating the deployment plan yourself with weblogic.PlanGenerator.

• Export the EAR file to a directory – my enterprise archive name is PlanEAR.ear
• Go to your domain's bin dir with a shell and call setDomainEnv
• Call a command like this from the dir where the EAR is with the configured shell:

java weblogic.PlanGenerator -all -plan Plan.xml PlanEAR.ear

• That should generate a Plan.xml file in that directory with a bunch of stubs for values you could change at runtime in the Plan.xml file.  
• Change the value of the variable for the context root.  Note that the variable name is auto-generated, but it still yields a clue by the variable name prefix.

Original:

<variable>
<name>WeblogicWebApp_ContextRoots_12482790756401</name>
<value xsi:nil="true"></value>
</variable>

New:

<variable>
<name>WeblogicWebApp_ContextRoots_12482790756401</name>
<value>FooWEB</value>
</variable>

• Note that I did not use quotes for FooWEB whereas the console approach did use them.  Either way seemed to work just fine.
• In order to override the context root we need to do a replace operation in the variable assignment since the element already exists in the archive. Edit Plan.xml again, this time modifying the relevant variable assignment section of the Plan.xml.  Note, if the weblogic.xml did not already have the context-root explicitly listed as an xml element, then the original section could have been used.

Original:

<variable-assignment> 
<name>WeblogicWebApp_ContextRoots_12482790756401</name> 
<xpath>/weblogic-web-app/context-root</xpath>
</variable-assignment>

New:

<variable-assignment> 
<name>WeblogicWebApp_ContextRoots_12482790756401</name> 
<xpath>/weblogic-web-app/context-root</xpath>
<operation>replace</operation> 
</variable-assignment>

Now we can use the weblogic.Deployer command for a redeployment with this new deployment plan Plan.xml and the EAR.  We also could have used WLST.

java weblogic.Deployer -adminurl t3://localhost:7001 -user weblogic -password welcome1 -redeploy -name PlanEAR -source PlanEAR.ear -targets AdminServer -plan Plan.xml

Special thanks to fellow bloggers Max, Edwin, and Benoit for their entries related to this topic.  Cheers.

Cancelling a task

One of the things to note about the JobScheduler is that you cannot call certain CommonJ API calls, particularly around cancelling tasks.  The way to do that is via the JobSchedulerRuntimeMBean, exposed with console functionality or via JMX.  Obviously the console is great for on-demand access, but what if you want to script things to make automated decisions?

Here is some sample output from a WLST session that shows me browsing to the JobSchedulerRuntimeMBean, finding an active task and cancelling it.

wls:/offline> connect('weblogic','weblogic','t3://localhost:7011')
Connecting to t3://localhost:7011 with userid weblogic ...
Successfully connected to managed Server 'managedServer1' that belongs to domain 'wldf_domain'.
Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.
wls:/wldf_domain/serverConfig> domainRuntime()
DomainRuntime MBeanServer is not enabled on a Managed Server.
wls:/wldf_domain/serverConfig> ls()
dr--   AdminConsole
dr--   AppDeployments
dr--   BridgeDestinations
dr--   Clusters
dr--   CustomResources
dr--   DeploymentConfiguration
dr--   Deployments
dr--   EmbeddedLDAP
dr--   ErrorHandlings
dr--   FileStores
dr--   InternalAppDeployments
dr--   InternalLibraries
dr--   JDBCDataSourceFactories
dr--   JDBCStores
dr--   JDBCSystemResources
dr--   JMSBridgeDestinations
dr--   JMSInteropModules
dr--   JMSServers
dr--   JMSSystemResources
dr--   JMX
dr--   JTA
dr--   JoltConnectionPools
dr--   Libraries
dr--   Log
dr--   LogFilters
dr--   Machines
dr--   MailSessions
dr--   MessagingBridges
dr--   MigratableTargets
dr--   RemoteSAFContexts
dr--   SAFAgents
dr--   SNMPAgent
dr--   SNMPAgentDeployments
dr--   Security
dr--   SecurityConfiguration
dr--   SelfTuning
dr--   Servers
dr--   ShutdownClasses
dr--   SingletonServices
dr--   StartupClasses
dr--   SystemResources
dr--   Targets
dr--   VirtualHosts
dr--   WLDFSystemResources
dr--   WLECConnectionPools
dr--   WSReliableDeliveryPolicies
dr--   WTCServers
dr--   WebAppContainer
dr--   WebserviceSecurities
dr--   XMLEntityCaches
dr--   XMLRegistries
-r--   AdminServerName                              AdminServer
-r--   AdministrationMBeanAuditingEnabled           false
-r--   AdministrationPort                           9002
-r--   AdministrationPortEnabled                    false
-r--   AdministrationProtocol                       t3s
-r--   ArchiveConfigurationCount                    0
-r--   ClusterConstraintsEnabled                    false
-r--   ConfigBackupEnabled                          false
-r--   ConfigurationAuditType                       none
-r--   ConfigurationVersion                         10.3.0.0
-r--   ConsoleContextPath                           console
-r--   ConsoleEnabled                               true
-r--   ConsoleExtensionDirectory                    console-ext
-r--   DomainVersion                                10.3.0.0
-r--   GuardianEnabled                              false
-r--   InternalAppsDeployOnDemandEnabled            true
-r--   LastModificationTime                         0
-r--   Name                                         wldf_domain
-r--   Notes                                        null
-r--   Parent                                       null
-r--   ProductionModeEnabled                        false
-r--   RootDirectory                                C:\Oracle\wls10gR3\user_projects\domains\wldf_domain
-r--   Type                                         Domain
-r-x   freezeCurrentValue                           Void : String(attributeName)
-r-x   isSet                                        Boolean : String(propertyName)
-r-x   unSet                                        Void : String(propertyName)
wls:/wldf_domain/serverConfig> serverRuntime()
wls:/wldf_domain/serverRuntime> ls()
dr--   ApplicationRuntimes
dr--   AsyncReplicationRuntime
dr--   ClusterRuntime
dr--   ConnectorServiceRuntime
dr--   DefaultExecuteQueueRuntime
dr--   EntityCacheCumulativeRuntime
dr--   EntityCacheCurrentStateRuntime
dr--   EntityCacheHistoricalRuntime
dr--   ExecuteQueueRuntimes
dr--   JDBCServiceRuntime
dr--   JMSRuntime
dr--   JTARuntime
dr--   JVMRuntime
dr--   JoltRuntime
dr--   LibraryRuntimes
dr--   LogBroadcasterRuntime
dr--   LogRuntime
dr--   MANAsyncReplicationRuntime
dr--   MANReplicationRuntime
dr--   MailSessionRuntimes
dr--   MaxThreadsConstraintRuntimes
dr--   MinThreadsConstraintRuntimes
dr--   PathServiceRuntime
dr--   PersistentStoreRuntimes
dr--   RequestClassRuntimes
dr--   SAFRuntime
dr--   SNMPAgentRuntime
dr--   ServerChannelRuntimes
dr--   ServerSecurityRuntime
dr--   ServerServices
dr--   SingleSignOnServicesRuntime
dr--   ThreadPoolRuntime
dr--   TimerRuntime
dr--   WANReplicationRuntime
dr--   WLDFRuntime
dr--   WTCRuntime
dr--   WebServerRuntimes
dr--   WorkManagerRuntimes
-r--   ActivationTime                               1237559025844
-r--   AdminServer                                  false
-r--   AdminServerHost                              192.168.2.1
-r--   AdminServerListenPort                        7001
-r--   AdminServerListenPortSecure                  false
-r--   AdministrationPort                           9002
-r--   AdministrationPortEnabled                    false
-r--   AdministrationURL                            t3://192.168.2.1:7011
-r--   CurrentDirectory                             C:\Oracle\wls10gR3\user_projects\domains\wldf_domain\.
-r--   CurrentMachine
-r--   DefaultExecuteQueueRuntime                   null
-r--   DefaultURL                                   t3://192.168.2.1:7011
-r--   EntityCacheCumulativeRuntime                 null
-r--   EntityCacheCurrentStateRuntime               null
-r--   EntityCacheHistoricalRuntime                 null
-r--   HealthState                                  Component:ServerRuntime,State:HEALTH_OK,MBean:managedServer1,ReasonCode:[]
-r--   JoltRuntime                                  null
-r--   ListenAddress                                JBAYER-LAP/192.168.2.1
-r--   ListenPort                                   7011
-r--   ListenPortEnabled                            true
-r--   MANAsyncReplicationRuntime                   null
-r--   MANReplicationRuntime                        null
-r--   Name                                         managedServer1
-r--   OpenSocketsCurrentCount                      3
-r--   Parent                                       null
-r--   PathServiceRuntime                           null
-r--   RestartRequired                              false
-r--   RestartsTotalCount                           0
-r--   SSLListenAddress                             null
-r--   SSLListenPort                                7002
-r--   SSLListenPortEnabled                         false
-r--   ServerClasspath                              C:\Oracle\wls10gR3\JROCKI~1\jre\bin\jrockit\jrockit1.6.0.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\bin\jrockit\jmapi.jar;C:\Oracle\wls10gR3\JROCKI~1\
3\JROCKI~1\jre\bin\jrockit\rmp.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\bin\jrockit\latency.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\lib\resources.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\lib\rt.jar;C:\Oracle\w
acle\wls10gR3\JROCKI~1\jre\lib\jsse.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\lib\jce.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\lib\charsets.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\classes;C:\JDeveloper\mywork\J
ener.jar;C:\Oracle\wls10gR3\patch_wlw1030\profiles\default\sys_manifest_classpath\weblogic_patch.jar;C:\Oracle\wls10gR3\patch_wls1030\profiles\default\sys_manifest_classpath\weblogic_patch.jar;C:
ult\sys_manifest_classpath\weblogic_patch.jar;C:\Oracle\wls10gR3\patch_cie670\profiles\default\sys_manifest_classpath\weblogic_patch.jar;C:\Oracle\wls10gR3\patch_cie660\profiles\default\sys_manif
s10gR3\patch_alsb1030\profiles\default\sys_manifest_classpath\weblogic_patch.jar;C:\Oracle\wls10gR3\JROCKI~1\lib\tools.jar;C:\Oracle\wls10gR3\WLSERV~1.3\server\lib\weblogic_sp.jar;C:\Oracle\wls10
cle\wls10gR3\modules\features\weblogic.server.modules_10.3.0.0.jar;C:\Oracle\wls10gR3\WLSERV~1.3\server\lib\webservices.jar;C:\Oracle\wls10gR3\modules\ORGAPA~1.5/lib/ant-all.jar;C:\Oracle\wls10gR
\Oracle\wls10gR3\jdeveloper\modules\features\adf.share_11.1.1.jar;;C:\Oracle\wls10gR3\WLSERV~1.3\common\eval\pointbase\lib\pbclient57.jar;C:\Oracle\wls10gR3\WLSERV~1.3\server\lib\xqrl.jar;;;C:\Or
sysext_manifest_classpath\weblogic_ext_patch.jar
-r--   SocketsOpenedTotalCount                      3
-r--   State                                        RUNNING
-r--   StateVal                                     2
-r--   Type                                         ServerRuntime
-r--   WANReplicationRuntime                        null
-r--   WLECConnectionServiceRuntime                 null
-r--   WeblogicVersion                              WebLogic Server Temporary Patch for CR380042 Thu Sep 11 13:33:40 PDT 2008
WebLogic Server Temporary Patch for 7372756 Fri Sep 12 17:05:44 EDT 2008
WebLogic Server Temporary Patch for CR380913 Wed Oct 15 13:24:22 PDT 2008
WebLogic Server Temporary Patch for CR381739 Tue Oct 21 14:06:14 IST 2008
WebLogic Server Temporary Patch for CR381056 Mon Oct 06 10:48:50 EDT 2008
WebLogic Server Temporary Patch for CR374413, CR378680 Tue Sep 02 09:55:36 PDT 2008
WebLogic Server Temporary Patch for CR378102 Wed Sep 10 23:28:48 PDT 2008
WebLogic Server Temporary Patch for CR378741 Tue Sep 09 13:08:51 PDT 2008
WebLogic Server Temporary Patch for CR378781, CR380313 Fri Sep 19 13:34:16 PDT 2008
WebLogic Server 10.3  Fri Jul 25 16:30:05 EDT 2008 1137967
-r-x   addRequestClassRuntime                       Boolean : WebLogicMBean(weblogic.management.runtime.RequestClassRuntimeMBean)
-r-x   forceShutdown                                Void :
-r-x   forceSuspend                                 Void :
-r-x   getServerChannel                             java.net.InetSocketAddress : String(protocol)
-r-x   getURL                                       String : String(protocol)
-r-x   preDeregister                                Void :
-r-x   restartSSLChannels                           Void :
-r-x   resume                                       Void :
-r-x   shutdown                                     Void :
-r-x   shutdown                                     Void : Integer(timeout),Boolean(ignoreSessions)
-r-x   start                                        Void :
-r-x   suspend                                      Void :
-r-x   suspend                                      Void : Integer(timeout),Boolean(ignoreSessions)
wls:/wldf_domain/serverRuntime> cd('ClusterRuntime')
wls:/wldf_domain/serverRuntime/ClusterRuntime> ls()
dr--   myCluster
wls:/wldf_domain/serverRuntime/ClusterRuntime> cd('myCluster')
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster> ls()
dr--   JobSchedulerRuntime
dr--   ServerMigrationRuntime
dr--   UnicastMessaging
-r--   ActiveSingletonServices                      java.lang.String[TimerMaster]
-r--   AliveServerCount                             1
-r--   CurrentMachine                               null
-r--   CurrentSecondaryServer
-r--   DetailedSecondariesDistribution              null
-r--   ForeignFragmentsDroppedCount                 0
-r--   FragmentsReceivedCount                       581
-r--   FragmentsSentCount                           581
-r--   HealthState                                  Component:null,State:HEALTH_OK,MBean:null,ReasonCode:[]
-r--   MulticastMessagesLostCount                   0
-r--   Name                                         myCluster
-r--   PrimaryCount                                 0
-r--   ResendRequestsCount                          0
-r--   SecondaryCount                               0
-r--   SecondaryDistributionNames                   null
-r--   SecondaryServerDetails
-r--   ServerNames                                  java.lang.String[managedServer1]
-r--   Type                                         ClusterRuntime
-r--   UnicastMessaging                             null
-r-x   preDeregister                                Void :
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster> cd('JobSchedulerRuntime')
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime> ls()
dr--   JobSchedulerRuntime
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime> cd('JobSchedulerRuntime')
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime> ls()
dr--   ExecutedJobs
-r--   Name                                         JobSchedulerRuntime
-r--   Type                                         JobSchedulerRuntime
-r-x   getJob                                       WebLogicMBean : String(id)
-r-x   preDeregister                                Void :
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime> cd('ExecutedJobs')
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime/ExecutedJobs> ls()
dr--   JobRuntime-managedServer1_1237564053328
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime/ExecutedJobs> cd('JobRuntime-managedServer1_1237564053328')
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime/ExecutedJobs/JobRuntime-managedServer1_1237564053328> ls()
-r--   Description                                  jamesbayer.client.SystemOutTimerListener@2ed941
-r--   ID                                           managedServer1_1237564053328
-r--   LastLocalExecutionTime                       1237564580328
-r--   LocalExecutionCount                          18
-r--   Name                                         JobRuntime-managedServer1_1237564053328
-r--   Period                                       30000
-r--   State                                        Running
-r--   Timeout                                      6719
-r--   TimerListener                                jamesbayer.client.SystemOutTimerListener@333c5d
-r--   Type                                         JobRuntime
-r-x   cancel                                       Void :
-r-x   preDeregister                                Void :
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime/ExecutedJobs/JobRuntime-managedServer1_1237564053328> ls()
-r--   Description                                  jamesbayer.client.SystemOutTimerListener@2ed941
-r--   ID                                           managedServer1_1237564053328
-r--   LastLocalExecutionTime                       1237564850547
-r--   LocalExecutionCount                          27
-r--   Name                                         JobRuntime-managedServer1_1237564053328
-r--   Period                                       30000
-r--   State                                        Running
-r--   Timeout                                      18812
-r--   TimerListener                                jamesbayer.client.SystemOutTimerListener@349b7f
-r--   Type                                         JobRuntime
-r-x   cancel                                       Void :
-r-x   preDeregister                                Void :
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime/ExecutedJobs/JobRuntime-managedServer1_1237564053328> cmo.cancel()
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime/ExecutedJobs/JobRuntime-managedServer1_1237564053328> ls()
-r--   Description                                  jamesbayer.client.SystemOutTimerListener@2ed941
-r--   ID                                           managedServer1_1237564053328
-r--   LastLocalExecutionTime                       1237564850547
-r--   LocalExecutionCount                          27
-r--   Name                                         JobRuntime-managedServer1_1237564053328
-r--   Period                                       30000
-r--   State                                        Cancelled
-r--   Timeout                                      -1
-r--   TimerListener                                null
-r--   Type                                         JobRuntime
-r-x   cancel                                       Void :
-r-x   preDeregister                                Void :

Automating JobScheduler status checking

One of my customers asked me about how they could periodically check if a task was running in the cluster, and if it is not then send someone an email, etc.  Basically, the sky is the limit with scripting.  A more advanced use of this would be to connect this to alerts in WebLogic Diagnsotic Framework or SNMP traps in Oracle Enterprise Manager, but this is a start.  Here is a python script that you can invoke with WLST that will do several things:

• Connect to the Admin Server
• Get a list of all managed servers in the cluster
• Check each managed server for a particular running timer task
• If no running tasks are found exit with a non-zero exit code, otherwise exit a zero exit code

isJobRunning.py

user='weblogic'
pw='weblogic'
adminServer='localhost'
adminPort='7001'
clusterName='myCluster'
timerRunning=false
timerClass='jamesbayer.client.SystemOutTimerListener'
## Standard Connection
try:
  connect(user,pw,'t3://'+adminServer+':'+adminPort)
except:
  print 'Error connecting to Admin Server'
## Get the list of managed servers in the cluster
try:
  domainConfig()
  cd('Clusters/'+clusterName)
  managedServers=cmo.getServers()
  print 'Found ' + `len(managedServers)` + ' managed servers'
except:
  print 'Error navigating DomainConfig MBean tree for the list of managed servers'
## Loop through managed servers and look for the job
domainRuntime()
for managedServer in managedServers:
  try:
    managedServerName=managedServer.getName()
    print 'Trying ' + managedServerName
    cd('/ServerRuntimes/'+managedServerName)
    if cmo.getState() == 'RUNNING':
      cd( '/ServerRuntimes/'+managedServerName+'/ClusterRuntime/'+clusterName+'/JobSchedulerRuntime/JobSchedulerRuntime')
      try:
        jobs=cmo.getExecutedJobs()
        for job in jobs:
          if job.getDescription().find(timerClass) > -1 and job.getState() == 'Running':
            timerRunning=true
            print timerClass + ' is running on ' +  managedServerName
      except:
        print 'Error checking ' + managedServerName + ' ExecutedJobs'
    else:
      print managedServerName + ' is not RUNNING'
  except:
    print 'Error checking JobScheduler on ' + managedServerName
  if timerRunning:
    break
disconnect()
if timerRunning:
  print 'The Job is Running'
  exit(exitcode=0)
else:
  print 'The Job is NOT Running'
  exit(exitcode=1)

Script output

C:\Oracle\wls10gR3\user_projects\domains\wldf_domain>bin\setDomainEnv.cmd
C:\Oracle\wls10gR3\user_projects\domains\wldf_domain>java weblogic.WLST C:\MyData\Accounts\PepsiAmericas\JobScheduler\isJobRunning.py
Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
Connecting to t3://localhost:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'wldf_domain'.
Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.
Location changed to serverRuntime tree. This is a read-only tree with DomainMBean as the root.
For more help, use help(domainConfig)
Found 1 managed servers
Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root.
For more help, use help(domainRuntime)
Trying managedServer1
Disconnected from weblogic server: AdminServer
The Job is Running
Exiting WebLogic Scripting Tool.
C:\Oracle\wls10gR3\user_projects\domains\wldf_domain>

Background

Oracle (then BEA Systems) and IBM collaborated on a common API for Timers and Work Managers (think multi-threading) that would be portable across application servers.  They aptly named this API CommonJ.  This API has been supported in WebLogic Server since 9.x and apparently in WebSphere since 6.x.  When combined with WebLogic clustering support, the CommonJ Timer API is known as the Job Scheduler.  It requires some configuration as described in the docs, but essentially the way it works under the covers is by serializing the class that will execute the task and some related metadata to a database.  Using the leasing feature of WebLogic Server, exactly one cluster member will be responsible for JobScheduler (singleton service) which executes the Timer Tasks.  The server responsible for this service will check the database every 30 seconds to see if there are new tasks to execute.

Instructions for the System Out Example Timer

These instructions assume starting from scratch with nothing but a basic WebLogic Server installation and a database like Oracle XE.  I run all of this easily on my 2+ yr old laptop (I do have 3+ gig of ram).  The documentation is more detailed now, so definitely refer to that as well.

1. Create a new domain with 1 Admin Server and at least one Managed Server that will be a part of a cluster.  Refer to the Oracle By Example page for WLS if you need help with this, but you should be able to do all of this from the Configuration Wizard.  I named my cluster “myCluster” and my managed servers “managedServer1” and “managedServer2”.
2. Set up the database – Basically run the DDL for your DB, which will create a table called WEBLOGIC_TIMERS. If you don’t set up database often, here is my simple POC sequence with Oracle XE.  In production, I would recommend Oracle RAC with a Multi Data Source, but you can POC this with an XE database or a single database node if you can live with a single point of failure.  Open the XE database web console and create a new database user, of course I use weblogic/weblogic for the user name password and give them admin rights.  Start sqlplus and login as that user.  Navigate to the directory that contains the DDL you want to run and use the @ syntax to execute the DDL like this.  You can of course ignore the messages about the comments not being valid commands:
3. Keep the sql prompt open, we’ll use it again in a minute.  Notice that the script is in the oracle\920 directory, that simply means for 9.2+ of Oracle DB, it works totally fine for me on XE 10g.
4. Configure the data source – Create a new data source for this and target it at all the servers in your domain (or at least the ones in your cluster) and specify it as the cluster’s JobScheduler Data Source.  I recommend that you test your data source when you are creating it.
5. Set up leasing – You can use consensus or database leasing, but since we already using the DB for the Job Scheduler, just do database leasing which creates a table called ACTIVE (at least in the recent releases of WLS, it used to be something like WEBLOGIC_ACTIVE in earlier releases).  This means running the appropriate WL_HOME/server/db/dbname/leasing.ddl as well as configuring the data source, which can be the same one you used before, and configuring the cluster to use that data source for leasing.  Make sure you note database based leasing requires a highly-available database.  If the connection goes away, the managed servers will fail.  For this reason you can use multi data sources with Oracle RAC to get an even more HA architecture, but Oracle XE works fine for a POC.  I got consensus leasing to work at one time in the past using WLS 9.2, but if I remember correctly I had to install Node Manager and configure “machines” for my servers for it to work.  So DB-based leasing is what I’m using in this example because it’s simpler and I have not had issues with it before.
6. Write some code – There is a sample in the docs showing how to look up the job scheduler and schedule a task.  I’ll take care of that for you, here is a WAR (Servlet 2.5 based) that has code to register the task via a servlet and a JAR file that contains the class that implements the TimerListener interface.  The source code is at the way bottom of this post.
7. Deploy the WAR file, target it at the cluster, and make sure the application is started.
8. Edit your server classpath to contain the timer implementation class which is in the JAR referenced earlier.  You might be asking yourself, why the heck does the timer listener class need to be placed on the system classpath?!!  Well if you think about how the classloaders work, the singleton service running the tasks does not have access to the classes from the JEE module that registered the timer.  It is running as a WebLogic system thread.  This class will be deserialized from the database, potentially on another server.  So I recommend not doing anything too sophisticated in the timer listener.  If you have lots of logic with multiple classes, just consider having the timer listener drop a message on a distributed JMS queue, which can have an MDB from another application that does all of your heavy lifting.  Back to putting the TimerListener implementation on the classpath, I accomplished this by added the following line to my domain’s setDomainEnv.cmd script.  There is also the option to use an EJB Timer, which apparently does not have the system classpath requirement, but I haven’t tried that before.

   set PRE_CLASSPATH=C:\JDeveloper\mywork\JobSchedulerApp\TimerListener\deploy\TimerListener.jar
   

9. Optional – Enable verbose logging for the related sub-systems, I find this helpful when troubleshooting. 

   set JAVA_OPTIONS=%JAVA_OPTIONS% -Dweblogic.debug.DebugSingletonServices=true -Dweblogic.JobScheduler=true

   
   
10. Restart your Admin Server and Managed Server.  Make sure you see the JAR in the classpath settings in the log file. 
11. If your enabled the additional logging, you should see that leasing and job scheduler are creating log entries like the following in the managed server log file (not in standard out!). 

####<Apr 8, 2009 3:10:01 PM CDT> <Debug> <SingletonServices> <jbayer-lap> <managedServer1> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1239221401078> <BEA-000000> <SingletonMonitor: Checking Failed Leases> 

####<Apr 8, 2009 3:10:11 PM CDT> <Debug> <SingletonServices> <jbayer-lap> <managedServer1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1239221411078> <BEA-000000> <SingletonMonitor: Now checking lease statuses.> 

####<Apr 8, 2009 3:10:11 PM CDT> <Debug> <SingletonServices> <jbayer-lap> <managedServer1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1239221411078> <BEA-000000> <SingletonMonitor: Checking for registered Migratable Targets and Singleton Services without a lease> 

####<Apr 8, 2009 3:10:11 PM CDT> <Debug> <SingletonServices> <jbayer-lap> <managedServer1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1239221411078> <BEA-000000> <SingletonMonitor: TimerMaster - Found an owner - -7090657857495109585/managedServer1> 

####<Apr 8, 2009 3:10:11 PM CDT> <Debug> <SingletonServices> <jbayer-lap> <managedServer1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1239221411078> <BEA-000000> <SingletonMonitor: Checking existant, but expired leases> 

####<Apr 8, 2009 3:10:11 PM CDT> <Debug> <SingletonServices> <jbayer-lap> <managedServer1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1239221411078> <BEA-000000> <SingletonMonitor: Checking Failed Leases> 

####<Apr 8, 2009 3:10:21 PM CDT> <Debug> <SingletonServices> <jbayer-lap> <managedServer1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1239221421078> <BEA-000000> <SingletonMonitor: Now checking lease statuses.> 

####<Apr 8, 2009 3:10:21 PM CDT> <Debug> <SingletonServices> <jbayer-lap> <managedServer1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1239221421078> <BEA-000000> <SingletonMonitor: Checking for registered Migratable Targets and Singleton Services without a lease> 

####<Apr 8, 2009 3:10:21 PM CDT> <Debug> <SingletonServices> <jbayer-lap> <managedServer1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1239221421078> <BEA-000000> <SingletonMonitor: TimerMaster - Found an owner - -7090657857495109585/managedServer1> 

####<Apr 8, 2009 3:10:21 PM CDT> <Debug> <SingletonServices> <jbayer-lap> <managedServer1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1239221421078> <BEA-000000> <SingletonMonitor: Checking existant, but expired leases> 

####<Apr 8, 2009 3:10:21 PM CDT> <Debug> <SingletonServices> <jbayer-lap> <managedServer1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1239221421078> <BEA-000000> <SingletonMonitor: Checking Failed Leases> 

12. Register the timer by hitting the URL or invoking the web service of one of the managed servers.  In my case that is http://localhost:7011/TimerWeb/timerservlet.  Note that each time you invoke the URL successfully a new timer listener will be registered with the cluster, so you probably only want to do this once.
13. Notice the messages in standard out if this is working correctly.

timerExpired() called at 4/8/09 3:12 PM

timerExpired() called at 4/8/09 3:13 PM

14. Shut down the managed server that has the messages printing to standard out.  You should see the messages show up on another managed server in the cluster in standard out.
15. Now shut down all of the managed servers and restart at least one.
16. You should see the timer task start printing to standard out when one of the managed servers in the cluster is available.

Summary

So you should have a working timer example in WebLogic Server that survives failures.  The debug messages in the managed server log files can be helpful in trouble-shooting in case anything goes wrong.  A couple other things to note, you cannot cancel timer tasks via CommonJ API calls when running in the JobScheduler.  You have to use the JobRuntimeMBean for that.  I’ll show an example of using this bean with WLST in another entry.  Here is the source code for the servlet and the timer listener classes.  Hopefully this will help you utilize this powerful capability built-in to WebLogic.

package view.jamesbayer;
import commonj.timers.TimerManager;
import jamesbayer.client.SystemOutTimerListener;
import java.io.IOException;
import java.io.PrintWriter;
import javax.naming.InitialContext;
import javax.servlet.*;
import javax.servlet.http.*;
public class TimerServlet extends HttpServlet {
    public void init(ServletConfig config) throws ServletException {
        super.init(config);
    }
    public void service(HttpServletRequest request,
                        HttpServletResponse response) throws ServletException,
                                                             IOException {
        PrintWriter out = response.getWriter();
        out.println("<html>");
        out.println("<head><title>TimerServlet</title></head>");
        try {
            System.out.println("service() entering try block to intialize the timer from JNDI");
            InitialContext ic = new InitialContext();
            TimerManager jobScheduler =
                (TimerManager)ic.lookup("weblogic.JobScheduler");
            System.out.println("jobScheduler reference " + jobScheduler);
            commonj.timers.TimerListener timerListener =
                new SystemOutTimerListener();
            System.out.println("timerListener reference " + timerListener);
            jobScheduler.schedule(timerListener, 0, 30 * 1000);
            //execute this job every 30 seconds
            System.out.println("service() started the timer");
            out.write("Started the timer - status:");
        } catch (Throwable t) {
            System.out.println("service() error initializing timer manager with JNDI name weblogic.JobScheduler " +                               t);
        }
        out.println("</body></html>");
        out.close();
    }
}

package jamesbayer.client;
import java.io.Serializable;
import java.text.SimpleDateFormat;
import java.util.Date;
import commonj.timers.Timer;
import commonj.timers.TimerListener;
public class SystemOutTimerListener implements Serializable, TimerListener 
{
	private static final long serialVersionUID = 8313912206357147939L;
	public void timerExpired(Timer timer) 
	{
		SimpleDateFormat sdf = new SimpleDateFormat();
		System.out.println( "timerExpired() called at " + sdf.format( new Date() ) );
	}
}

I came across a surprising interoperability issue the other day using JConsole to browse WLS 10gR3 JMX MBeans.  Read on to see how to I got around it.  One of my customers wanted to monitor some WLS services running in a cluster recently, so in doing a little research I decided to pull up JConsole to see what the JMX Beans looked like.  Of course WLST works well for this, but I didn’t quite know where the MBean was located so I wanted something visual that I could navigate in a tree fashion without typing commands.

In order to bring up JConsole, you need to follow some basic steps in the documentation to enable RMI on the server you want to connect to: http://edocs.bea.com/wls/docs103/jmxinst/accesscust.html#wp1107229  Be sure to set the default RMI user and password and you will also need to restart the servers after you perform this configuration so it takes effect.

So normally you could just launch JConsole from the JDK’s bin directory and plug-in your connection info.  In my case I am using JDK 6 from the WLS Example server which is running on port 7001.  This is the URI I put into JConsole:

service:jmx:iiop:///jndi/iiop://localhost:7001/weblogic.management.mbeanservers.runtime

My RMI default user and password is “weblogic”

Well, to my surprise I got a JConsole Output window that kept scrolling with errors.  Here is the beginning of the stack:

com.sun.corba.se.impl.encoding.CDRInputStream_1_0_read_value
WARNING: "IOP00810257: (MARSHAL) Could not find class"
 at com.sun.corba.se.imple.logging.ORBUtilSystemException.couldNotFindClass(ORBUtilSystemException:7756)
 ....

I looked around a bit on Google and found that since JDK 5 there has been a bug in the IIOP RMI stack in the JDK.  Thankfully there is a work-around that involves creating a simple poor-man’s proxy.  So I compiled and ran the RMIIIOPProxy.java file from that link (I removed the package name for brevity) and tried again to connect again while the proxy was running.  This time it worked.

C:\temp>java RMIIIOPProxy 7011 weblogic.management.mbeanservers.runtime localhost 7001 weblogic.management.mbeanservers.runtime
Connection established with south server at: service:jmx:iiop:///jndi/iiop://localhost:7001/weblogic.management.mbeanservers.runtime
Proxy server listening at: service:jmx:rmi:///jndi/rmi://localhost:7011/weblogic.management.mbeanservers.runtime
Strike <Enter> to exit

So now there is a simple proxy that listens on port 7011 which then connects to port 7001 on my behalf and gets around the bug.  I simply take the JMX URI from the proxy console output and plug that back in to JConsole, this time with a successful connection.  Notice that this URI does not include IIOP.  Now I get browse the MBeans, see the attributes, and execute operations like I expect.  I also tried this with JDK 5’s JConsole and the same issue was present and solved by the work-around.

• I like using the off-line installation from Sourceforge for reasons that will probably become apparent below, but there is also a Web Start version on the project home page.
• I found an old comment that I had written that claimed using the same JVM as the WLS server with Hermes helps, so I modify the %HERMES_INSTALL%\bin\hermes.bat file to use JRockit from WLS 10gR3:

SET JAVA_HOME=C:\Oracle\wls10gR3\jrockit_160_05
set PATH=C:\Oracle\wls10gR3\jrockit_160_05\bin

• Launch Hermes and go to Options->Preferences to add the weblogic.jar to the providers tab, this is nice because presumably I could use multiple WLS versions (I haven’t tried that)

• Configure a new Session in the GUI.

• Then right click on the session and select “Discover” and you should have all the destinations available to browse.
• The Session gets saved in hermes-config.xml (C:\.hermes\hermes-config.xml on my machine) and the relevant snippet looks like this:
  

      <factory classpathId="weblogic10.3">
  
          <provider className="hermes.JNDIConnectionFactory">
  
              <properties>
  
                  <property name="initialContextFactory" value="weblogic.jndi.WLInitialContextFactory"/>
  
                  <property name="providerURL" value="t3://localhost:7001"/>
  
                  <property name="securityCredentials" value="weblogic"/>
  
                  <property name="binding" value="javax/jms/QueueConnectionFactory"/>
  
                  <property name="securityPrincipal" value="weblogic"/>
  
              </properties>
  
          </provider>
  
          <connection clientID="" connectionPerThread="false">
  
              <session audit="false" id="examplesQCF" reconnects="0" transacted="true" useConsumerForQueueBrowse="false"/>
  
          </connection>
  
          <destination domain="1" name="weblogic/wsee/DefaultQueue"/>
  
          <destination domain="1" name="weblogic/examples/ejb30/ExampleQueue"/>
  
          <destination domain="2" name="weblogic/examples/jms/exampleTopic"/>
  
          <destination domain="1" name="weblogic/examples/jms/exampleQueue"/>
  
          <destination domain="1" name="jms/MULTIDATASOURCE_MDB_QUEUE"/>
  
          <destination domain="2" name="quotes"/>
  
          <destination domain="2" name="stockTopic"/>
  
          <extension className="hermes.ext.weblogic.WebLogicJMSAdminFactory">
  
              <properties/>
  
          </extension>
  
      </factory>
  
  

  
  

Now you should be ready to browse.  You could also take the alternative route of binding to the Initial Context of the server, which then makes it easy to create multiple Sessions from the Connection Factory objects.  However, I had trouble doing this without editing the hermes-config.xml due to what might be a fluke in the GUI for changing the class-loader provider from System to the WebLogic10.3 one that has the weblogic.jar.  If you have trouble with this you can either edit the hermes-config.xml file to have the correct provider or just add the weblogic.jar to the classpath (so System inherits it).  Good luck with Hermes.

Prior to joining BEA over 3 years ago, I never used WebLogic Server and had to come up to speed very quickly.  I have assembled my list of favorite resources for acquiring WebLogic knowledge.  Now that Oracle has declared WebLogic Server as the strategic application server platform I expect that many Oracle employees, customers and partners will be in a similar situation as I was when I joined BEA.  The good news is that there are many excellent ways to come up to speed in many different media formats.

The Basics – Terminology and Concepts

Step zero is to understand some of the concepts and terminology that are specific to WebLogic Server.  There is a very short and easy presentation that does this that has been posted on slideshare.  If you need the very high-level, then start here.  Of course you can get all of this from the documentation in a more verbose way; however I have not found a better concise introduction than this presentation.

Baby Steps – Oracle By Example

Oracle has a very nice concept called “Oracle By Example” where they explain in detail with screenshots how to perform specific tasks.  Now there is a complete Oracle By Example section on WebLogic Server with lessons for Installing and Configuring Oracle WebLogic Server Instance , Create a Basic Cluster, Using the WebLogic Server Administration Console and many others.  You have to crawl before you walk and run, so be sure to check these out as they are great for getting started.

Samples and Examples Ship With the Product

It surprises me how often people do not know that WebLogic Server has example source code and an sample domain that can be installed with the product.  In the past this was part of the default installation, but I believe with 10gR3 that you now have to specify the option to install the samples as part of a custom installation.  If you did the default installation you should be able to add the examples by running the installer again and selecting the custom installation to add only the examples to an existing installation.  You can see detail about the samples by opening the file <WEBLOGIC_INSTALLATION>\wlserver_10.3\samples\server\wls_samples_overview.html

Check out the complete index of the 10gR3 examples on the image below and see how many specific examples are included.  For example, there web services examples for several of the WS-* specifications, in some cases mixed and matched together.  A simple rule of thumb is to always check the examples whenever you are getting started with a feature for the first time or you need a refresher on a feature you have not used in a while.  Many of the examples also can be used as a template to build on for your scripts and code.

The examples instructions also include instructions on how to start the Examples Server also known as the samples domain, so be sure to check those instructions out as most examples will get deployed to the samples domain located at <WEBLOGIC_INSTALLATION>\wlserver_10.3\samples\domains\wl_server

Required Reading - Documentation

Although it is not as fun as watching a screencast or as quick as a topic tailored presentation, you absolutely must become familiar with the standard documentation when you get in to any level of detail on WebLogic features.  Here are a couple non-obvious items from the docs table of contents that I have circled on the left that you should note.

What’s New – Provides a breakdown by subsystem of what has changed between releases, sometimes there are new features that do not get featured on the front-page, but that are very nice to know about.

Known and Resolved Issues – These can come in very handy when you trying to discover if an issue you’ve run into is currently known about and potentially resolved.  It has the specific bug number (known as a CR in WebLogic speak) that can come in handy when interfacing with support.

Supported Configurations – I cannot stress enough how important it is that you review the documentation for the specific OS / Chipset / JVM combination that you are using in various environments.  When using proprietary OS’s like HP-UX or AIX for example, only very specific JVM versions are supported and there may be specific patches required for the OS, the JVM, and WebLogic Server.  I have seen issues come up many times in the field when these pre-tested and certified versions are not adhered to strictly.  Take a look at this page for AIX 5.3 for example which gives precise version information and special installation instructions and patch information for this combination.

Documentation Search Tip - One short-cut that I use often when searching the documentation is the google “site:” syntax.  So for example, to constrain my search the WebLogic Server 10gR3 documentation for the words “deployment descriptor” you can specify the path to restrict the results to.  I type this in the google search bar: 

site:http://edocs.bea.com/wls/docs103/ deployment descriptor

In fact, I have another short-cut that simplifies this further, I use a tool called SlickRun where I can create my own short-cuts by keyword.  I have created a “wls” keyword so I can type the following command into a prompt that is always keyboard accessible:

wls deployment descriptor

and SlickRun will open my browser, put in the google site syntax for me, and substitue the “deployment descriptor” phrase to be my search terms.  But I digress…

The final thing I will mention about the documentation is that it is a living set of documents.  If you find a gap that needs to be filled, an area that needs improvement or more detail, or a mistake then please participate in making the documentation better by sending feedback via the link at the bottom of each page or emailing it to docsupport at bea dot com.

Oracle Technology Network (OTN)

Many of the resources mentioned here are part of the Oracle Technology Network which has tons of information.  Here are some of my favorites for WebLogic:

WebLogic Server Product Page – This is the center of the universe for WebLogic.  Do not miss the recorded demos (a list is below).

WebLogic Server Product Blog – Many great posts by the product team.

RSS feed for all WebLogic related articles and news

Forums

There are many product specific forums on the Oracle Technology Network that are monitored by experts to answer informed questions.  In WebLogic Server’s case there are even sub-forums for specific sub-systems of WebLogic Server that are often monitored by developers and experts in those subject areas.  In addition to the Generic WebLogic Server forum that I like to read, there are currently forums specific to JDBC, Security, Web Services, etc.  This is not a comprehensive list and they are adding more all the time, so check the Application Server category to see if there is a specific WebLogic forum for your question.  One of the most important things to remember is that the detailed, well-researched, and well-explained questions are most likely to get a helpful response.  If you are lazy and have not used some of the resources above or if you do not provide enough detail so that someone with no context can figure out what you are asking, then you end up wasting people’s time.  There is also a read-only archive of the former BEA dev2dev forums that is especially helpful for researching questions on older releases.

I really like using the RSS feeds from the forums to find out about new postings in my RSS Reader.  You can also do other handy things like watch a specific thread so you receive emails about any updates to that thread only instead of for the entire forum.

Oracle provides a formal support process for their customers with defined Service Level Agreements and the forums are not a substitute for that.  They are a community-based volunteer mechanism to help both the questioners and the responders further their knowledge.  So if you use the forums to post questions, give back and help answer some of the questions too.  Even if you only start by redirecting people to previous answers or a better forum to post their question in, you can help build the community and learn yourself.

Support

Whenever an issue or question comes up that has urgency, I urge my customers to open a support case.  I use this simple support quick reference guide with my customers that has all contact information and describes how to open a support case, escalate an issue, etc on only one page.  It’s easy to print out and post somewhere where management, developers and administrators can see it.  WebLogic support is still handled by the http://support.bea.com website as of this post, but over time it will migrate to the My Oracle Support website formerly known as Metalink.

Support Patterns - The WebLogic support patterns are a fantastic way to try self-service support on common issues.  These are used by the support engineers and are common enough to be published for anyone to use.  They are broken down by topic area and I often find myself pointing customers to this information.  For example, if you need to troubleshoot multicast issues in a cluster there is a page dedicated to that topic.

Oracle University

Formal training courses from Oracle University and certifications are a great way to build knowledge.  WebLogic courses are currently listed under the BEA section.

Summary

I’m sure that I have omitted some excellent resources by accident.  There are published books on these topics and many WebLogic blogs out there, not all of which are hosted on blogs.oracle.com.

Once you participate in the community awhile you make connections with experts in many subject areas.  The relationships that I have built up over time with the product team and specialists are helpful, but it did not happen over night.  So my advice is to participate in the community and take advantage of the many resources out there.  Leave a comment if you have any favorites resources that I left out.

dojox.cometd.connectionTypes.unregister("long-polling-json-encoded");

Now that I ruined the punch-line, read-on if you're interested in how the Bayeux protocol handshake happens and how I trouble-shot this.  When the client initializes there is a handshake message exchanged with the server so they can agree on which method of communicating to use.  Apparently WLS 10.3 as it ships does not like a new supported connection type that is available in Dojo 1.2.0.

The client normally uses code like this to initialize a connection:

<!-- Import Dojo -->
<script type="text/javascript" src="dojo-release-1.2.0/dojo/dojo.js"
    djConfig="parseOnLoad:true, isDebug:true"></script>

<script type="text/javascript">

    dojo.require("dojox.cometd");

    dojox.cometd.init("cometd");

</script>

Using Firebug and a simple URL decoder we can see that it sends the following with a POST to the server:

message=[{"version":"1.0","minimumVersion":"0.9","channel":"/meta /handshake","id":"0","supportedConnectionTypes":["long-polling","long-polling-json-encoded ","callback-polling"]}]

The response looks like this:

[{"channel": "/meta/handshake", "successful": false, "error": "402:long-polling-json-encoded:Invalid
 supportedConnectionTypes attribute.", "supportedConnectionTypes": ["long-polling","callback-polling"
], "version": "1.0", "minimumVersion": "0.1", "advice": {"reconnect": "none", "interval": 500, "multiple-clients"
: false}}]

Uh oh, notice how there is an error in the handshake indicating that WLS does not like the long-polling-json-encoded connection type?  So after posting in the Dojox support forum, I got a response suggesting specifying a property as an argument to the init() call so as not to send the extra connection type.  That did not work for me, but it put me on the path of using Firebug, the js source, and the Dojo API docs.  I discovered that the connection types are stored in an dojo.AdapterRegistry and that there is an unregister function for the connection types.  That's it.  So now my client code looks like this with the new unregister invocation:

<!-- Import Dojo -->
<script type="text/javascript" src="dojo-release-1.2.0/dojo/dojo.js"
    djConfig="parseOnLoad:true, isDebug:true"></script>

<script type="text/javascript">

    dojo.require("dojox.cometd");

    dojox.cometd.connectionTypes.unregister("long-polling-json-encoded");

    dojox.cometd.init("cometd");

</script>

Which results in a POST like this:

message=[{"version":"1.0","minimumVersion":"0.9","channel":"/meta /handshake","id":"0","supportedConnectionTypes":["long-polling","callback-polling "]}]

And a successful response like this:

[{"channel": "/meta/handshake", "version": "1.0", "supportedConnectionTypes": ["long-polling","callback-polling"
], "clientId": "NNR2BOeRyCVKjLN", "successful": true, "minimumVersion": "0.1", "authSuccessful": true
, "advice": {"interval": 500, "multiple-clients": false}}]

Which causes the Dojo cometd framework to send another request specifying that it prefers long-polling:

[{"channel":"/meta/connect","clientId":"NNR2BOeRyCVKjLN","connectionType":"long-polling","id":"1"}]

And a confirmation response that long-polling should be initiated:

[{"channel": "/meta/connect", "successful": true, "clientId": "NNR2BOeRyCVKjLN", "error": "", "timestamp"
: "2008-10-22 01:59:27", "connectionId": "null", "advice": {"interval": 500, "multiple-clients": false
}}]

Which results in an immediate call to the server:

[{"channel":"/meta/connect","connectionType":"long-polling","clientId":"lMRDFU5fBxnE5kg","id":"3"}]

The response either times out on the server after about 80 seconds or returns if there is an event that the client has subscribed to sooner.  Then the process simply repeats.  Here is the response after a timeout.

[{"channel": "/meta/connect", "successful": true, "clientId": "lMRDFU5fBxnE5kg", "error": "", "timestamp"
: "2008-10-22 02:15:57", "connectionId": "null", "advice": {"reconnect": "retry", "interval": 500, "multiple-clients"
: false}}]

This causes the connect message to be sent again.  Firebug shows me the spinning request (see the last POST) indicating that it is waiting for the server response, and I know that it's working.  And the world is right again.  A helpful diagram showing how some of the various delivery methods work for this is on Carol McDonald's blog.