Are My Servers Running?

Sometimes, it is necessary to know whether one or more servers within a WebLogic Server domain are not running, and perform appropriate actions. For example, one may want to execute custom scripts if one or more servers within the domain are not in the RUNNING state. The WebLogic Diagnostic Framework (WLDF), which is part of the WebLogic Server makes it easy to send notifications when such conditions are detected. When configured rules match runtime conditions, notifications associated with the rules are sent. WLDF can send notifications in various forms, such as emails, JMX notifications, JMS message, SNMP traps and also cause a Diagnostic Image to be created, which consists of important state of the entire server.

In this blog, I will describe how a watch-notification can be configured to send a JMX notification when one of the servers in the domain is not running. I will also show a simple JMX notification listener program which can process WLDF JMX notifications. In this case, it will simply print the contents of the notification. However, it can be easily adapted to do other things such as running a WLST script.

For simplicity, we will have only two servers in the domain, myserver (admin-server) and MS1 (managed-server). We will configure WLDF on the admin-server to send a JMX notification when the state of the managed-server MS1 (actually, any server within the domain) is not RUNNING.

Configuring WLDF watch/notifications

We will use WebLogic Server Administration Console to configure WLDF. It can also be done with WLST. Console provides a nice feature to Create Repeatable Configuration Scripts, which can be handy for building such scripts. Configuration steps for this case consist of:


  1. Create a Diagnostic Module.

  2. Enable periodic metrics collection in the Diagnostic module.

  3. Create a JMX notification type.

  4. Create a watch to detect inactive servers

  5. Associate the created JMX notification with the configured watch

  6. Target the Diagnostic module to the admin server

We will go over these steps and see how to configure WLDF using Console for this task. Log into Console and acquire Edit Lock so we can add new configuration to WebLogic Server.

Create a Diagnostic Module.

From the left navigation tree, open the Diagnostics node and click on Diagnostic Modules. Console will show a table of existing Diagnostic modules within the domain. Click the New button to create a new diagnostic module. Call it myWLDF. Click Save to create the module. At this point, we have an empty myWLDF diagnostic module. We will proceed to configure it.

Enable periodic metrics collection in the Diagnostic module.

Click on the myWLDF module link in the table of Diagnostics modules. Click on Collected Metrics sub-tab under Configuration tab. Check the Enabled checkbox and set the Sampling Period to 10000 (10 seconds). Click Save.

Create a JMX notification type.

Configuring a watch/notification has two aspects. The first aspect is a watch rule which specifies the condition that WLDF will check. The second aspect is the set of notifications that will be sent when the rule condition is met. Console provides configuration assistants to make the configuration task easier. To create a notification type:


  • Click Watches and Notifications sub-tab under Configuration tab.

  • On the Notifications sub-tab, click New in the Notifications table.

  • Select "JMX Notification" for the notification type from the dropdown and click Next.

  • Give a name to the notification type (myJMX)

  • check the Enable Notification checkbox and click OK to create the notification type.

Create a watch to detect inactive servers

Now, we will create the watch rule. We will create a watch rule based on runtime mbean data. Specifically, we will use the State attribute on the ServerLifeCycleRuntimeMBean mbeans. For each server within the domain, there is a ServerLifeCycleRuntimeMBean mbean in the domain runtime mbean server. The State attribute reflects the current state of the server. We will configure a rule which will fire when at least one server in the domain is not in the RUNNING state.


  • Click on the Watches sub-tab. Click New in the Watches table.

  • Set Watch Name to WatchServerStatus. Select Collected Metrics for Watch Type, check Enabled checkbox and click Next.

Now we come to the interesting part, which is configuration of the watch rule expression itself. You can hand-edit the watch expression by clicking the Edit button and entering the watch expression yourself. However, that will require some familiarity with the watch
expression syntax. For now, we will use the rule-expression builder.


  • Click on Add Expression button.

  • Select DomainRuntime for the MBean Server Location and click Next.

  • From the drop-down list for MBean Type, select weblogic.management.runtime.ServerLifeCycleRuntimeMBean and click Next.

  • Leave the Instance list empty and click Next.

  • Select State attribute from the list of available attributes.

  • Select "!=" (Not Equals) as comparison operator.

  • Set RUNNING as the value for the comparison and click Finish.

  • Click Finish to complete the rule-expression.

Note that the ServerLifeCycleRuntimeMBean is only registered in the Domain runtime mbean server. So, we selected DomainRuntime for the MBean Server Location. Also, since we want to receive notification for any server which is not running, we left the instance list empty and not restrict it to specific instances.

The rule-expression builder would create a rule expression:

(${DomainRuntime//[weblogic.management.runtime.ServerLifeCycleRuntimeMBean]//State} != 'RUNNING')

Associate notification and configure alarm type


  • Click on the WatchServerStatus watch from the Watches table.

  • Click on the Notifications tab

  • Choose the myJMX notification from the chooser and click Save

  • Click on the Alarms tab and select "Don't use an alarm". Click Save.

With this, we associated the created JMX notification with the watch. With "Don't use an alarm", we disabled the watch-alarms for now, which will cause a notification to be sent in every cycle when the rule evaluates to true. Later, you may want to use other options such as "Automatic Alarm Reset" so notifications will be suppressed for a configured interval to avoid a flood of notifications.

Finishing the configuration

Do not forget to target the created Diagnostic module to the admin-server (myserver). It will take effect only when the module is targeted to the server. Another thing to note is that, WLDF allows at most one module to be targeted to a server. So, if you have another diagnostic module targeted to the server, you will need to un-target it first. Ofcourse, in that case, you may add the above configuration to the pre-existing module instead of creating a new one.

Lastly, activate the changes so it will take effect.

Ok, so we have configured WLDF watch/notifications. What does it mean? Once above configuration is activated, WLDF will periodically evaluate configured watch rules The period is specified by the Sampling Period. If a watch rule evaluates to true, its associated notifications
are sent. As per the configuration above, WLDF will periodically (every 10 seconds) check if the State attribute of any of the ServerLifeCycleRuntimeMBean mbeans is anything other than RUNNING. If so, it will send a JMX notification.

Receiving Notifications

WLDF sends a JMX notification on a specific WLDF runtime mbean, whose ObjectName is of the form:

com.bea:Name=DiagnosticsJMXNotificationSource,ServerRuntime=$SERVER,
Type=WLDFWatchJMXNotificationRuntime,WLDFRuntime=WLDFRuntime,
WLDFWatchNotificationRuntime=WatchNotification

where $SERVER is the name of the WebLogic Server instance. For our case (myserver), it is:

com.bea:Name=DiagnosticsJMXNotificationSource,ServerRuntime=myserver,
Type=WLDFWatchJMXNotificationRuntime,WLDFRuntime=WLDFRuntime,
WLDFWatchNotificationRuntime=WatchNotification

By registering for JMX notifications on this mbean, a client program can listen to generated notifications.

JMXWatchNotificationListener.java
is a simple notification listener for WLDF JMX notifications. It simply prints the contents of received notification, but can be easily adapted to perform other actions.

Run the program and kill the managed server MS1. The client program displays notifications as shown below, when the managed server MS1 exits the RUNNING state.

$ java JMXWatchNotificationListener -server myserver -host localhost -port 7001 -user weblogic -password welcome1

URL=service:jmx:t3://localhost:7001/jndi/weblogic.management.mbeanservers.runtime
Adding notification handler for: com.bea:Name=DiagnosticsJMXNotificationSource,ServerRuntime=myserver,
Type=WLDFWatchJMXNotificationRuntime,WLDFRuntime=WLDFRuntime,
WLDFWatchNotificationRuntime=WatchNotification
===============================================
Notification name: myjmx called. Count= 1.
Watch severity: Notice
Watch time: Oct 12, 2009 4:45:48 PM EDT
Watch ServerName: myserver
Watch RuleType: Harvester
Watch Rule: (${DomainRuntime//[weblogic.management.runtime.ServerLifeCycleRuntimeMBean]//State} != 'RUNNING'
)
Watch Name: WatchServerStatus
Watch DomainName: mydomain
Watch AlarmType: None
Watch AlarmResetPeriod: 60000
Watch Message: WatchName: WatchServerStatus WatchSeverityLevel: Notice
Watch Data: com.bea:Name=MS1,Type=ServerLifeCycleRuntime//State = SHUTDOWN com.bea:Name=myserver,Type=ServerLifeCycleRun
time//State = RUNNING
===============================================

As mentioned above, other forms of notifications can be configured as well. For example, the article "Monitoring Performance Using the WebLogic Diagnostics Framework" shows how to configure SNMP traps with the watch rules.


Comments:

In the step Select DomainRuntime for the MBean Server Location and click Next, I receive a big Java error. If I select ServerRuntime, I don't receive the error. Any advice?

Posted by Brion Maciel on May 05, 2010 at 05:00 AM PDT #

What is the error you see? Also, what is the WebLogic server version? Regards, /Raj

Posted by Anonymous on May 05, 2010 at 05:24 AM PDT #

This issue was found and fixed in WLS 10.3.2 Meanwhile, as a workaround, you can try repeating the steps after it fails the first time. If that still fails, please try the following: 1. Configure a watch rule with ServerRuntime 2. Then try steps to configure the watch rule using DomainRuntime Regards, /Raj

Posted by Anonymous on May 06, 2010 at 06:14 AM PDT #

Even i saw the error when trying to create watch bu using DomainRuntime,if i use serverruntime everything works fine. Here is the error stack trace: java.lang.NullPointerException at weblogic.diagnostics.harvester.internal.MetricArchiver.getKnownHarvestableTypes(MetricArchiver.java:973) at weblogic.diagnostics.harvester.internal.HarvesterRuntimeMBeanImpl.getKnownHarvestableTypes(HarvesterRuntimeMBeanImpl.java:390) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at

Posted by guest on May 06, 2010 at 07:30 AM PDT #

Hi, I have followed the way you explained and also used the code for JMXWatchNotificationListener.java. But I'm not getting any notification. It says "Adding notification handler for....." and keep on waiting. Didn't throw any error or exception. Only change I did is I have used serviceURL = new JMXServiceURL("t3", hostName, port, JNDI + DomainRuntimeServiceMBean.MBEANSERVER_JNDI_NAME); as The WLDF MBean ObjectName oname = new ObjectName( "com.bea:Location=" + serverName + ",Name=" + JMXWatchNotification.GLOBAL_JMX_NOTIFICATION_PRODUCER_NAME + ",ServerRuntime=" + serverName + ",Type=WLDFWatchJMXNotificationRuntime," + "WLDFRuntime=WLDFRuntime," + "WLDFWatchNotificationRuntime=WatchNotification" ); was throwing NoInstanceFound Exception. My WLS is 10.3.3 I'm using Eclipse to run this program. Could you please guide me what could be the issue. Thanks, Regards

Posted by Jyotisekhar on June 29, 2010 at 04:53 AM PDT #

What was the ObjectName used when you encountered InstanceNotFoundException? The sample JMXWatchNotificationListener program uses myserver as the name of the admin server by default. (sample program takes -server option). Was correct server name being used while building the objectname? Regards

Posted by Anonymous on July 02, 2010 at 02:51 AM PDT #

I agree with your thoughts here and I really love your blog! I've bookmarked it so that I can come back & read more in the future.

Posted by Gene Ristow on February 03, 2011 at 06:08 PM PST #

I know this if off topic but I'm looking into starting my own blog and was curious what all is needed to get set up? I'm assuming having a blog like yours would cost a pretty penny? I'm not very web savvy so I'm not 100% certain. Any tips or advice would be greatly appreciated. Cheers

Posted by Sports Picks on April 14, 2011 at 01:59 AM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

The official blog for Oracle WebLogic Server fans and followers!

Stay Connected

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
5
6
7
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today