JMX Based Monitoring - Part Four - Business App Server Monitoring

In the last blog entry I talked about the Oracle Utilities Application Framework V4 feature for monitoring and managing aspects of the Web Application Server using JMX.

In this blog entry I am going to discuss a similar new feature that allows JMX to be used for management and monitoring the Oracle Utilities business application server component. This feature is primarily focussed on performance tracking of the product.

In the first release of Oracle Utilities Customer Care And Billing (V1.x I am talking about), we used to use Oracle Tuxedo as part of the architecture. In Oracle Utilities Application Framework V2.0 and above, we removed Tuxedo from the architecture. One of the features that some customers used within Tuxedo was the performance tracking ability. The idea was that you enabled performance logging on the individual Tuxedo servers and then used a utility named txrpt to produce a performance report. This report would list every service called, the number of times it was called and the average response time. When I worked a performance consultant, I used this report to identify badly performing services and also gauge the overall performance characteristics of a site.

When Tuxedo was removed from the architecture this information was also lost. While you can get some information from access.log and some Mbeans supplied by the Web Application Server it was not at the same granularity as txrpt or as useful.

I am happy to say we have not only reintroduced this facility in Oracle Utilities Application Framework but it is now accessible via JMX and also we have added more detail into the performance tracking. Most of this new design was working with customers around the world to make sure we introduced a new feature that not only satisfied their performance tracking needs but allowed for finer grained performance analysis.

As with the Web Application Server, the Business Application Server JMX monitoring is enabled by specifying a JMX port number in RMI Port number for JMX Business and initial credentials in the JMX Enablement System User ID and JMX Enablement System Password configuration options. These options are available using the configureEnv[.sh] -a utility. These credentials are shared across the Web Application Server and Business Application Server for authorization purposes.

Once this is information is supplied a number of configuration files are built (by the initialSetup[.sh] utility) to configure the facility:


  • spl.properties - contains the JMX URL, the security configuration and the mbeans that are enabled. For example, on my demonstration machine:

spl.runtime.management.rmi.port=6750
spl.runtime.management.connector.url.default=service:jmx:rmi:///jndi/rmi://localhost:6750/oracle/ouaf/ejbAppConnector
jmx.remote.x.password.file=scripts/ouaf.jmx.password.file
jmx.remote.x.access.file=scripts/ouaf.jmx.access.file
ouaf.jmx.com.splwg.ejb.service.management.PerformanceStatistics=enabled


  • ouaf.jmx.* files - contain the userid and password. The default configuration uses the JMX default configuration. You can use additional security features by altering the spl.properties file manually or using a custom template. For more security options see JMX Security for more details.

Once it has been configured and the changes reflected in the product using the initialSetup[.sh] utility the JMX facility can be used. For illustrative purposes I will use jconsole but any JSR160 complaint browser or client can be used (with the appropriate configuration).

Once you start jconsole (ensure that splenviron[.sh] is executed prior to execution to set the environment variables or for remote connection, ensure java is in your path and jconsole.jar in your classpath) you specify the URL in the spl.runtime.management.connnector.url.default entry. For example:

Example Business App JMX

You are then able to track performance of the product using the PerformanceStatistics Mbean. The attributes of the PerformanceStatistics Mbean are counts of each object type. This is where this facility differs from txrpt. The information that is collected includes the following:


  • The Service Type is captured so you can filter the results in terms of the type of service. For maintenance type services you can even see the transaction type (ADD, CHANGE etc) so you can see the performance of updates against read transactions.

  • The Minimum and Maximum are also collected to give you an idea of the spread of performance.

  • The last call is recorded. The date, time and user of the last call are recorded to give you an idea of the timeliness of the data.

  • The Mbean maintains a set of counters per Service Type to give you a summary of the types of transactions being executed. This gives you an overall picture of the types of transactions and volumes at your site.

There are a number of interesting operations that can also be performed:


  • reset - This resets the statistics back to zero. This is an important operation. For example, txrpt is restricted to collecting statistics per hour, which is ok for most people. But what if you wanted to be more granular? This operation allows to set the collection period to anything you wish. The statistics collected will represent values since the last restart or last reset.

  • completeExecutionDump - This is the operation that produces a CSV in memory to allow extraction of the data. All the statistics are extracted (see the Server Administration Guide for a full list). This can be then loaded into a database, a tool or simply into your favourite spreadsheet for analysis.

Here is an extract of an execution dump from my demonstration environment to give you an idea of the format:

ServiceName, ServiceType, MinTime, MaxTime, Avg Time, # of Calls, Latest Time, Latest Date, Latest User
...
CFLZLOUL, EXECUTE_LIST, 15.0, 64.0, 22.2, 10, 16.0, 2009-12-16::11-25-36-932, ASHORTEN
CILBBLLP, READ, 106.0, 1184.0, 466.3333333333333, 6, 106.0, 2009-12-16::11-39-01-645, BOBAMA
CILBBLLP, DELETE, 70.0, 146.0, 108.0, 2, 70.0, 2009-12-15::12-53-58-280, BPAYS
CILBBLLP, ADD, 860.0, 4903.0, 2243.5, 8, 860.0, 2009-12-16::17-54-23-862, LELLISON
CILBBLLP, CHANGE, 112.0, 3410.0, 815.1666666666666, 12, 112.0, 2009-12-16::11-40-01-103, ASHORTEN
CILBCBAL, EXECUTE_LIST, 8.0, 84.0, 26.0, 22, 23.0, 2009-12-16::17-54-01-643, LJACKMAN
InitializeUserInfoService, READ_SYSTEM, 49.0, 962.0, 70.83777777777777, 450, 63.0, 2010-02-25::11-21-21-667, ASHORTEN
InitializeUserService, READ_SYSTEM, 130.0, 2835.0, 234.85777777777778, 450, 216.0, 2010-02-25::11-21-21-446, ASHORTEN
MenuLoginService, READ_SYSTEM, 530.0, 1186.0, 703.3333333333334, 9, 530.0, 2009-12-16::16-39-31-172, ASHORTEN
NavigationOptionDescriptionService, READ_SYSTEM, 2.0, 7.0, 4.0, 8, 2.0, 2009-12-21::09-46-46-892, ASHORTEN
...

There are other operations and attributes available. Refer to the Server Administration Guide provided with your product to understand the full et of operations and attributes.

This is one of the many features I am proud that we implemented as it allows flexible monitoring of the performance of the product.

Comments:

Hi Anthony, are these changes going to be backported to any other versions? al

Posted by alwyn on June 15, 2010 at 07:09 AM EST #

Great stuff ici.

Posted by Myra Bergold on July 25, 2010 at 09:39 PM EST #

voluminousstately entry you've sit on

Posted by Bill Burnett on November 08, 2010 at 09:30 PM EST #

It's quite a moment as long as here's a new twist.

Posted by Evalyn Neider on November 19, 2010 at 03:04 AM EST #

Sweet post bro. I LOVE your sense of writing.

Posted by Wilfredo Jappa on November 19, 2010 at 08:49 AM EST #

very good \o/

Posted by Sigarayi birakmak on November 21, 2010 at 11:11 AM EST #

Only as you do know yourself can your brain serve you as a sharp and efficient tool. Know your own failings, passions, and prejudices so you can separate them from what you see.

Posted by Lucien Sternberger on November 27, 2010 at 04:58 AM EST #

I am really happy after I discovered this webpage!

Posted by Raphael Woolstenhulme on November 30, 2010 at 10:59 PM EST #

I'm sure there's a logical explanation for and I'm sucker for punishment.

Posted by Leslie Quickle on December 02, 2010 at 04:43 PM EST #

Definition - Sarchasm: The gulf between the author of sarcastic wit and the person who doesnt get it.

Posted by Orland on December 19, 2010 at 08:13 AM EST #

You have done a marvellous job by exploring this subject with such an honesty and depth. Thanks for sharing it with us! These weren’t low class individuals.

Posted by Vincenza Hoock on December 27, 2010 at 11:33 AM EST #

Great tips! I will try to use them. Why not hold some sort of contest for your readers?

Posted by Cabinet Maker Brisbane on December 30, 2010 at 09:44 PM EST #

I'll gear this review to 2 types of people: current Zune owners who are considering an upgrade, and people trying to decide between a Zune and an iPod. (There are other players worth considering out there, like the Sony Walkman X, but I hope this gives you enough info to make an informed decision of the Zune vs players other than the iPod line as well.)

Posted by yesilcam erotik on December 31, 2010 at 03:00 AM EST #

Very interesting and amusing subject. I read with great pleasure. I remember speaking to the professor of the class about how much I enjoyed the course and how fascinating it was to me, and he confessed that out of all of the classes he taught, abnormal psychology was his favorite as well.

Posted by Teena Hentges on December 31, 2010 at 07:33 AM EST #

i had to go through it again since i could not grab the entire escence of the post in one go.That was a very articulate and logical point you make there.I appreciate you letting me know your thinking on this.

Posted by wedding party on January 05, 2011 at 06:56 AM EST #

You have done a marvellous job by exploring this subject with such an honesty and depth. Thanks for sharing it with us! His emphasis on hope appealed to the idealistic age group, and they showed up in droves on election day.

Posted by Connie Steelmon on January 17, 2011 at 10:58 AM EST #

I'd like to realise all these sick comment shapers do an elevated chore than you. Keep it up ;).

Posted by sidebar gadgets on January 17, 2011 at 06:59 PM EST #

Dang i thought your blog was killer, gave me a car load of information, i never knew, thanks blogger.

Posted by Abbie Abarca on February 04, 2011 at 01:23 AM EST #

Straight to the point, i love it. Dont let anyone stop us bloggers.

Posted by Newton Amor on February 04, 2011 at 11:19 AM EST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Anthony Shorten
Hi, I am Anthony Shorten, I am the Principal Product Manager for the Oracle Utilities Application Framework. I have been working for over 20+ years in the IT Business and am the author of many a technical whitepaper, manual and training material. I am one of the product managers working on strategy and designs for the next generation of the technology used for the Utilities and Tax markets. This blog is provided to announce new features, document tips and techniques and also outline features of the Oracle Utilities Application Framework based products. These products include Oracle Utilities Customer Care and Billing, Oracle Utilities Meter Data Management, Oracle Utilities Mobile Workforce Management and Oracle Enterprise Taxation and Policy Management. I am the product manager for the Management Pack for these products.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
9
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today