Using WLST with the WebLogic Server Job Scheduler (James Bayer's Blog)

I recently wrote about a simple example using the WebLogic Server Job Scheduler. This describes how to cancel the jobs and get visibility into job execution status.

Cancelling a task

One of the things to note about the JobScheduler is that you cannot call certain CommonJ API calls, particularly around cancelling tasks. The way to do that is via the JobSchedulerRuntimeMBean, exposed with console functionality or via JMX. Obviously the console is great for on-demand access, but what if you want to script things to make automated decisions?

Here is some sample output from a WLST session that shows me browsing to the JobSchedulerRuntimeMBean, finding an active task and cancelling it.

wls:/offline> connect('weblogic','weblogic','t3://localhost:7011')
Connecting to t3://localhost:7011 with userid weblogic ...
Successfully connected to managed Server 'managedServer1' that belongs to domain 'wldf_domain'.
Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.
wls:/wldf_domain/serverConfig> domainRuntime()
DomainRuntime MBeanServer is not enabled on a Managed Server.
wls:/wldf_domain/serverConfig> ls()
dr--   AdminConsole
dr--   AppDeployments
dr--   BridgeDestinations
dr--   Clusters
dr--   CustomResources
dr--   DeploymentConfiguration
dr--   Deployments
dr--   EmbeddedLDAP
dr--   ErrorHandlings
dr--   FileStores
dr--   InternalAppDeployments
dr--   InternalLibraries
dr--   JDBCDataSourceFactories
dr--   JDBCStores
dr--   JDBCSystemResources
dr--   JMSBridgeDestinations
dr--   JMSInteropModules
dr--   JMSServers
dr--   JMSSystemResources
dr--   JMX
dr--   JTA
dr--   JoltConnectionPools
dr--   Libraries
dr--   Log
dr--   LogFilters
dr--   Machines
dr--   MailSessions
dr--   MessagingBridges
dr--   MigratableTargets
dr--   RemoteSAFContexts
dr--   SAFAgents
dr--   SNMPAgent
dr--   SNMPAgentDeployments
dr--   Security
dr--   SecurityConfiguration
dr--   SelfTuning
dr--   Servers
dr--   ShutdownClasses
dr--   SingletonServices
dr--   StartupClasses
dr--   SystemResources
dr--   Targets
dr--   VirtualHosts
dr--   WLDFSystemResources
dr--   WLECConnectionPools
dr--   WSReliableDeliveryPolicies
dr--   WTCServers
dr--   WebAppContainer
dr--   WebserviceSecurities
dr--   XMLEntityCaches
dr--   XMLRegistries
-r--   AdminServerName                              AdminServer
-r--   AdministrationMBeanAuditingEnabled           false
-r--   AdministrationPort                           9002
-r--   AdministrationPortEnabled                    false
-r--   AdministrationProtocol                       t3s
-r--   ArchiveConfigurationCount                    0
-r--   ClusterConstraintsEnabled                    false
-r--   ConfigBackupEnabled                          false
-r--   ConfigurationAuditType                       none
-r--   ConfigurationVersion                         10.3.0.0
-r--   ConsoleContextPath                           console
-r--   ConsoleEnabled                               true
-r--   ConsoleExtensionDirectory                    console-ext
-r--   DomainVersion                                10.3.0.0
-r--   GuardianEnabled                              false
-r--   InternalAppsDeployOnDemandEnabled            true
-r--   LastModificationTime                         0
-r--   Name                                         wldf_domain
-r--   Notes                                        null
-r--   Parent                                       null
-r--   ProductionModeEnabled                        false
-r--   RootDirectory                                C:\Oracle\wls10gR3\user_projects\domains\wldf_domain
-r--   Type                                         Domain
-r-x   freezeCurrentValue                           Void : String(attributeName)
-r-x   isSet                                        Boolean : String(propertyName)
-r-x   unSet                                        Void : String(propertyName)
wls:/wldf_domain/serverConfig> serverRuntime()
wls:/wldf_domain/serverRuntime> ls()
dr--   ApplicationRuntimes
dr--   AsyncReplicationRuntime
dr--   ClusterRuntime
dr--   ConnectorServiceRuntime
dr--   DefaultExecuteQueueRuntime
dr--   EntityCacheCumulativeRuntime
dr--   EntityCacheCurrentStateRuntime
dr--   EntityCacheHistoricalRuntime
dr--   ExecuteQueueRuntimes
dr--   JDBCServiceRuntime
dr--   JMSRuntime
dr--   JTARuntime
dr--   JVMRuntime
dr--   JoltRuntime
dr--   LibraryRuntimes
dr--   LogBroadcasterRuntime
dr--   LogRuntime
dr--   MANAsyncReplicationRuntime
dr--   MANReplicationRuntime
dr--   MailSessionRuntimes
dr--   MaxThreadsConstraintRuntimes
dr--   MinThreadsConstraintRuntimes
dr--   PathServiceRuntime
dr--   PersistentStoreRuntimes
dr--   RequestClassRuntimes
dr--   SAFRuntime
dr--   SNMPAgentRuntime
dr--   ServerChannelRuntimes
dr--   ServerSecurityRuntime
dr--   ServerServices
dr--   SingleSignOnServicesRuntime
dr--   ThreadPoolRuntime
dr--   TimerRuntime
dr--   WANReplicationRuntime
dr--   WLDFRuntime
dr--   WTCRuntime
dr--   WebServerRuntimes
dr--   WorkManagerRuntimes
-r--   ActivationTime                               1237559025844
-r--   AdminServer                                  false
-r--   AdminServerHost                              192.168.2.1
-r--   AdminServerListenPort                        7001
-r--   AdminServerListenPortSecure                  false
-r--   AdministrationPort                           9002
-r--   AdministrationPortEnabled                    false
-r--   AdministrationURL                            t3://192.168.2.1:7011
-r--   CurrentDirectory                             C:\Oracle\wls10gR3\user_projects\domains\wldf_domain\.
-r--   CurrentMachine
-r--   DefaultExecuteQueueRuntime                   null
-r--   DefaultURL                                   t3://192.168.2.1:7011
-r--   EntityCacheCumulativeRuntime                 null
-r--   EntityCacheCurrentStateRuntime               null
-r--   EntityCacheHistoricalRuntime                 null
-r--   HealthState                                  Component:ServerRuntime,State:HEALTH_OK,MBean:managedServer1,ReasonCode:[]
-r--   JoltRuntime                                  null
-r--   ListenAddress                                JBAYER-LAP/192.168.2.1
-r--   ListenPort                                   7011
-r--   ListenPortEnabled                            true
-r--   MANAsyncReplicationRuntime                   null
-r--   MANReplicationRuntime                        null
-r--   Name                                         managedServer1
-r--   OpenSocketsCurrentCount                      3
-r--   Parent                                       null
-r--   PathServiceRuntime                           null
-r--   RestartRequired                              false
-r--   RestartsTotalCount                           0
-r--   SSLListenAddress                             null
-r--   SSLListenPort                                7002
-r--   SSLListenPortEnabled                         false
-r--   ServerClasspath                              C:\Oracle\wls10gR3\JROCKI~1\jre\bin\jrockit\jrockit1.6.0.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\bin\jrockit\jmapi.jar;C:\Oracle\wls10gR3\JROCKI~1\
3\JROCKI~1\jre\bin\jrockit\rmp.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\bin\jrockit\latency.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\lib\resources.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\lib\rt.jar;C:\Oracle\w
acle\wls10gR3\JROCKI~1\jre\lib\jsse.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\lib\jce.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\lib\charsets.jar;C:\Oracle\wls10gR3\JROCKI~1\jre\classes;C:\JDeveloper\mywork\J
ener.jar;C:\Oracle\wls10gR3\patch_wlw1030\profiles\default\sys_manifest_classpath\weblogic_patch.jar;C:\Oracle\wls10gR3\patch_wls1030\profiles\default\sys_manifest_classpath\weblogic_patch.jar;C:
ult\sys_manifest_classpath\weblogic_patch.jar;C:\Oracle\wls10gR3\patch_cie670\profiles\default\sys_manifest_classpath\weblogic_patch.jar;C:\Oracle\wls10gR3\patch_cie660\profiles\default\sys_manif
s10gR3\patch_alsb1030\profiles\default\sys_manifest_classpath\weblogic_patch.jar;C:\Oracle\wls10gR3\JROCKI~1\lib\tools.jar;C:\Oracle\wls10gR3\WLSERV~1.3\server\lib\weblogic_sp.jar;C:\Oracle\wls10
cle\wls10gR3\modules\features\weblogic.server.modules_10.3.0.0.jar;C:\Oracle\wls10gR3\WLSERV~1.3\server\lib\webservices.jar;C:\Oracle\wls10gR3\modules\ORGAPA~1.5/lib/ant-all.jar;C:\Oracle\wls10gR
\Oracle\wls10gR3\jdeveloper\modules\features\adf.share_11.1.1.jar;;C:\Oracle\wls10gR3\WLSERV~1.3\common\eval\pointbase\lib\pbclient57.jar;C:\Oracle\wls10gR3\WLSERV~1.3\server\lib\xqrl.jar;;;C:\Or
sysext_manifest_classpath\weblogic_ext_patch.jar
-r--   SocketsOpenedTotalCount                      3
-r--   State                                        RUNNING
-r--   StateVal                                     2
-r--   Type                                         ServerRuntime
-r--   WANReplicationRuntime                        null
-r--   WLECConnectionServiceRuntime                 null
-r--   WeblogicVersion                              WebLogic Server Temporary Patch for CR380042 Thu Sep 11 13:33:40 PDT 2008
WebLogic Server Temporary Patch for 7372756 Fri Sep 12 17:05:44 EDT 2008
WebLogic Server Temporary Patch for CR380913 Wed Oct 15 13:24:22 PDT 2008
WebLogic Server Temporary Patch for CR381739 Tue Oct 21 14:06:14 IST 2008
WebLogic Server Temporary Patch for CR381056 Mon Oct 06 10:48:50 EDT 2008
WebLogic Server Temporary Patch for CR374413, CR378680 Tue Sep 02 09:55:36 PDT 2008
WebLogic Server Temporary Patch for CR378102 Wed Sep 10 23:28:48 PDT 2008
WebLogic Server Temporary Patch for CR378741 Tue Sep 09 13:08:51 PDT 2008
WebLogic Server Temporary Patch for CR378781, CR380313 Fri Sep 19 13:34:16 PDT 2008
WebLogic Server 10.3  Fri Jul 25 16:30:05 EDT 2008 1137967
-r-x   addRequestClassRuntime                       Boolean : WebLogicMBean(weblogic.management.runtime.RequestClassRuntimeMBean)
-r-x   forceShutdown                                Void :
-r-x   forceSuspend                                 Void :
-r-x   getServerChannel                             java.net.InetSocketAddress : String(protocol)
-r-x   getURL                                       String : String(protocol)
-r-x   preDeregister                                Void :
-r-x   restartSSLChannels                           Void :
-r-x   resume                                       Void :
-r-x   shutdown                                     Void :
-r-x   shutdown                                     Void : Integer(timeout),Boolean(ignoreSessions)
-r-x   start                                        Void :
-r-x   suspend                                      Void :
-r-x   suspend                                      Void : Integer(timeout),Boolean(ignoreSessions)
wls:/wldf_domain/serverRuntime> cd('ClusterRuntime')
wls:/wldf_domain/serverRuntime/ClusterRuntime> ls()
dr--   myCluster
wls:/wldf_domain/serverRuntime/ClusterRuntime> cd('myCluster')
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster> ls()
dr--   JobSchedulerRuntime
dr--   ServerMigrationRuntime
dr--   UnicastMessaging
-r--   ActiveSingletonServices                      java.lang.String[TimerMaster]
-r--   AliveServerCount                             1
-r--   CurrentMachine                               null
-r--   CurrentSecondaryServer
-r--   DetailedSecondariesDistribution              null
-r--   ForeignFragmentsDroppedCount                 0
-r--   FragmentsReceivedCount                       581
-r--   FragmentsSentCount                           581
-r--   HealthState                                  Component:null,State:HEALTH_OK,MBean:null,ReasonCode:[]
-r--   MulticastMessagesLostCount                   0
-r--   Name                                         myCluster
-r--   PrimaryCount                                 0
-r--   ResendRequestsCount                          0
-r--   SecondaryCount                               0
-r--   SecondaryDistributionNames                   null
-r--   SecondaryServerDetails
-r--   ServerNames                                  java.lang.String[managedServer1]
-r--   Type                                         ClusterRuntime
-r--   UnicastMessaging                             null
-r-x   preDeregister                                Void :
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster> cd('JobSchedulerRuntime')
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime> ls()
dr--   JobSchedulerRuntime
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime> cd('JobSchedulerRuntime')
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime> ls()
dr--   ExecutedJobs
-r--   Name                                         JobSchedulerRuntime
-r--   Type                                         JobSchedulerRuntime
-r-x   getJob                                       WebLogicMBean : String(id)
-r-x   preDeregister                                Void :
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime> cd('ExecutedJobs')
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime/ExecutedJobs> ls()
dr--   JobRuntime-managedServer1_1237564053328
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime/ExecutedJobs> cd('JobRuntime-managedServer1_1237564053328')
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime/ExecutedJobs/JobRuntime-managedServer1_1237564053328> ls()
-r--   Description                                  jamesbayer.client.SystemOutTimerListener@2ed941
-r--   ID                                           managedServer1_1237564053328
-r--   LastLocalExecutionTime                       1237564580328
-r--   LocalExecutionCount                          18
-r--   Name                                         JobRuntime-managedServer1_1237564053328
-r--   Period                                       30000
-r--   State                                        Running
-r--   Timeout                                      6719
-r--   TimerListener                                jamesbayer.client.SystemOutTimerListener@333c5d
-r--   Type                                         JobRuntime
-r-x   cancel                                       Void :
-r-x   preDeregister                                Void :
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime/ExecutedJobs/JobRuntime-managedServer1_1237564053328> ls()
-r--   Description                                  jamesbayer.client.SystemOutTimerListener@2ed941
-r--   ID                                           managedServer1_1237564053328
-r--   LastLocalExecutionTime                       1237564850547
-r--   LocalExecutionCount                          27
-r--   Name                                         JobRuntime-managedServer1_1237564053328
-r--   Period                                       30000
-r--   State                                        Running
-r--   Timeout                                      18812
-r--   TimerListener                                jamesbayer.client.SystemOutTimerListener@349b7f
-r--   Type                                         JobRuntime
-r-x   cancel                                       Void :
-r-x   preDeregister                                Void :
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime/ExecutedJobs/JobRuntime-managedServer1_1237564053328> cmo.cancel()
wls:/wldf_domain/serverRuntime/ClusterRuntime/myCluster/JobSchedulerRuntime/JobSchedulerRuntime/ExecutedJobs/JobRuntime-managedServer1_1237564053328> ls()
-r--   Description                                  jamesbayer.client.SystemOutTimerListener@2ed941
-r--   ID                                           managedServer1_1237564053328
-r--   LastLocalExecutionTime                       1237564850547
-r--   LocalExecutionCount                          27
-r--   Name                                         JobRuntime-managedServer1_1237564053328
-r--   Period                                       30000
-r--   State                                        Cancelled
-r--   Timeout                                      -1
-r--   TimerListener                                null
-r--   Type                                         JobRuntime
-r-x   cancel                                       Void :
-r-x   preDeregister                                Void :

Automating JobScheduler status checking

One of my customers asked me about how they could periodically check if a task was running in the cluster, and if it is not then send someone an email, etc. Basically, the sky is the limit with scripting. A more advanced use of this would be to connect this to alerts in WebLogic Diagnsotic Framework or SNMP traps in Oracle Enterprise Manager, but this is a start. Here is a python script that you can invoke with WLST that will do several things:

Connect to the Admin Server

Get a list of all managed servers in the cluster

Check each managed server for a particular running timer task

If no running tasks are found exit with a non-zero exit code, otherwise exit a zero exit code

isJobRunning.py

user='weblogic'
pw='weblogic'
adminServer='localhost'
adminPort='7001'
clusterName='myCluster'
timerRunning=false
timerClass='jamesbayer.client.SystemOutTimerListener'
## Standard Connection
try:
  connect(user,pw,'t3://'+adminServer+':'+adminPort)
except:
  print 'Error connecting to Admin Server'
## Get the list of managed servers in the cluster
try:
  domainConfig()
  cd('Clusters/'+clusterName)
  managedServers=cmo.getServers()
  print 'Found ' + `len(managedServers)` + ' managed servers'
except:
  print 'Error navigating DomainConfig MBean tree for the list of managed servers'
## Loop through managed servers and look for the job
domainRuntime()
for managedServer in managedServers:
  try:
    managedServerName=managedServer.getName()
    print 'Trying ' + managedServerName
    cd('/ServerRuntimes/'+managedServerName)
    if cmo.getState() == 'RUNNING':
      cd( '/ServerRuntimes/'+managedServerName+'/ClusterRuntime/'+clusterName+'/JobSchedulerRuntime/JobSchedulerRuntime')
      try:
        jobs=cmo.getExecutedJobs()
        for job in jobs:
          if job.getDescription().find(timerClass) > -1 and job.getState() == 'Running':
            timerRunning=true
            print timerClass + ' is running on ' +  managedServerName
      except:
        print 'Error checking ' + managedServerName + ' ExecutedJobs'
    else:
      print managedServerName + ' is not RUNNING'
  except:
    print 'Error checking JobScheduler on ' + managedServerName
  if timerRunning:
    break
disconnect()
if timerRunning:
  print 'The Job is Running'
  exit(exitcode=0)
else:
  print 'The Job is NOT Running'
  exit(exitcode=1)

Script output

C:\Oracle\wls10gR3\user_projects\domains\wldf_domain>bin\setDomainEnv.cmd
C:\Oracle\wls10gR3\user_projects\domains\wldf_domain>java weblogic.WLST C:\MyData\Accounts\PepsiAmericas\JobScheduler\isJobRunning.py
Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
Connecting to t3://localhost:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'wldf_domain'.
Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.
Location changed to serverRuntime tree. This is a read-only tree with DomainMBean as the root.
For more help, use help(domainConfig)
Found 1 managed servers
Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root.
For more help, use help(domainRuntime)
Trying managedServer1
Disconnected from weblogic server: AdminServer
The Job is Running
Exiting WebLogic Scripting Tool.
C:\Oracle\wls10gR3\user_projects\domains\wldf_domain>

Comments (3)

Prashant Virpariya:

Hello James,

Thanks for blogging on couple of exremely useful features of WLS (scheduling timers with JobScheduler using CommonJ API and Cancelling them using WLST). I am in process of evaluating different options to implement timers in our Java EE based product and I already did some research around these APIs using sample implementation. Thats when I found your blog and wanted to share few things with you.

- I tried implementing EJB 3.0 timer services with WLS 10 (thats my first preference, BTW, since we want our product to be as independant of app server specific APIs as we can) but that seems to have issues in clustered environment.
- Thats when I turned to CommonJ APIs and through my tests, I saw that they are perfectly cluster aware (what you mentioned in your first blog of JobScheduler).
- However, in our product, we would want to cancel scheduled timers programatically based on some events. And CommonJ specifications do not have good set of APIs to cancel those schdueled timers programatically (Except the cancel() method called on Timer object created while scheduling that timer). In other words, there is no mechanism wherein I can cancel a particular timer based on some identifier. Do you agree? Please let me know if you think otherwise.
- Another concern that I have is at any given point of time, we'll have more than 50000 active timers and at any given day, based on series of business events, we would want to cancel hundred thousands of timers (berore they are fired even once). I am suspecious as to how well JobScheduling APIs will scale even if we found some way to cancel those timers. Can you please let me know if you have seen these APIs being used at such a scale?

I understand this being relatively new standard, we might not have many use cases. But this is just in case you came across one.

Thanks again for your time and effort.. Keep up the good work!!

Thanks,
Prashant.

Posted by Prashant Virpariya | April 10, 2009 8:08 AM

Posted on April 10, 2009 08:08

James Bayer:

Prashant,

Some answers below:

JB: I have not tried the EJB 3.0 timer approach at all and suggest you try and work with support to get through these issues as it should be completely supported.

- Thats when I turned to CommonJ APIs and through my tests, I saw that they are perfectly cluster aware (what you mentioned in your first blog of JobScheduler).
- However, in our product, we would want to cancel scheduled timers programatically based on some events. And CommonJ specifications do not have good set of APIs to cancel those schdueled timers programatically (Except the cancel() method called on Timer object created while scheduling that timer). In other words, there is no mechanism wherein I can cancel a particular timer based on some identifier. Do you agree? Please let me know if you think otherwise.

JB: First, timers in the JobScheduler cannot be cancelled via the API as is documented here:
http://e-docs.bea.com/wls/docs103/commonj/commonj.html#wp1061781
There is something that I should have mentioned in my entry but forgot to, before I knew about the MBean interface I figured out that you could simply delete the row from the weblogic_timers table and that would also work to cancel - kind of like kill -9 :). If you can figure out the how to identify the row, then you could go directly at the table.

- Another concern that I have is at any given point of time, we'll have more than 50000 active timers and at any given day, based on series of business events, we would want to cancel hundred thousands of timers (berore they are fired even once). I am suspecious as to how well JobScheduling APIs will scale even if we found some way to cancel those timers. Can you please let me know if you have seen these APIs being used at such a scale?

JB: Wow that's a lot of timers! Consider this point from the docs, which should show how could cancel the tasks if you keep track of the ID's someplace:
Call JobSchedulerRuntimeMBean.getJob(id) with the ID of a scheduled job. To get the ID, call the JobScheduler.schedule() method to return a Timer object, then use the Timer’s toString() method to return the ID.
If you consider how this is designed, every 30 secs the singleton service will run a query on that weblogic_timers table and find out which tasks should be executed on the singleton service's server, so I am fairly sure that there is no implicit load-balancing. I would also strongly consider making sure that you look at the query to make sure that table is indexed properly so that query performs well. If there is heavy-lifting in these timer events you should consider the something like the JMS / MDB mechanism I mentioned which should help with load distribution. I'm unaware of anyone using the timers on that kind of scale, my customers that used it were using it with much lighter volumes. You may want to consider posting in the WLS EJB forum or the WLS General forum on OTN to see if you can get other responses.

Thanks,

James

Posted by James Bayer | April 10, 2009 8:57 AM

Posted on April 10, 2009 08:57

Lubosi:

Hi James

Thanks for blogging these tasks, i've found them to be very useful.

Posted by Lubosi | October 23, 2009 1:15 AM

Posted on October 23, 2009 01:15

James Bayer's Blog