Proactive Monitoring on an Apps 11i Web Tier Server

Editor Oct 20 Update:  Corrected LogFormat syntax in Example #1.  Thanks, Petra!
Dec 4 Update:  Convinced blogging software to display missing backslash in LogFormat syntax.


When delivering a talk on Middle Tier performance to Apps DBAs at an UK OUG event (members can download the presentation).  I asked the audience how many were proactively monitoring their access_log.   Not many people put their hands up.

This was a little surprising to me, as there is a wealth of information in the access_log that can be mined to give information about user behaviour, possible user or system problems, as well as having the potential to give valuable performance information.  

Here are just three examples where you can quickly answer important questions about your system with very little effort:

1.  How long does it take 9iAS to serve requests?

The main performance information is the time taken to serve a request.  This is not included in the access_log by default (I have raised enhancement request 5349693 to include this data), but can easily be added by adding the following line to the httpd.conf file via an AutoConfig customization (see Metalink Note 270519.1):

      LogFormat "%h %l %u %t \"%r\" %>s %b %T"  

This line can be added anywhere in the file, but would sensibly be placed after the existing LogFormat entries. This directive adds an extra column into the access_log to show the time in seconds it took from receiving a request to sending back a response.  Valid log formats are described in the Apache documentation

2.  Am I getting any server errors?

The access_log includes the HTTP status code of the request.   They are listed in full in RFC 2616 but the codes we would be immediately concerned about are any in the 400 or 500 range. 

For example:

  • Status 500 (internal server error) may typically be seen for a JServ request and often means the JVM has some kind of problem or has died.  

    For example:  This entry may indicate that the JServ JVM is not responding to any requests:

    192.168.1.10 - - [21/Jun/2006:13:25:30 +0100] "POST /oa_servlet/actions/processApplicantSearch HTTP/1.1" 500 0

  • Status 403 (forbidden) could typically be seen for oprocmgr requests and often means there is a misconfiguration that needs to be resolved. 

    For example:  This entry in access_log may indicate a problem with system configuration (oprocmgr.conf):
myserver - - [21/Jun/2006:13:25:30 +0100] "GET /oprocmgr-service?cmd=Register&index=0&modName=JServ
&grpName=OACoreGroup&port=16000 HTTP/1.1" 403 226


3. Are users having problems accessing pages?

The status code of 200 means the request was successful, however a dash ( - ) for the "Bytes sent" column, normally means that the request hit the Apache timeout.  You would also see a time taken to serve request above 300 seconds as the default Apache timeout is 5 minutes.   

If you see this situation occurring regularly then your users are either navigating away from the browser page before it has rendered, or are likely to be getting a "white screen of death" in their browser window where it will appear to hang.

In this situation you need to identify why the requests are not being processed in good time, which is a large subject in itself.

Identifying these issues

Hopefully these examples will inspire you to want to analyse your access_log, but I hear you ask, "Where will I get the time?"   

Luckily the access_log is a simple text file, so if you do not have commercial monitoring software you can use some "quick and dirty" scripts to report just the exceptions you are interesting in seeing.   For example I would often use the following to scan access_logs for problems:

## Start of script
##
## Check for HTTP statuses in 400 or 500 range for JServ
## or PLSQL requests only
##
awk ' $9>=400 && $9<=599 { print $0 }' access_log* | grep -e
"servlet" -e "\/pls\/" | grep -v .gif
##
## Check for requests taking more than 30 seconds to be returned
##
awk ' $11>30 {print $0} ' access_log*
##
## This one is not an exception report, you need to manually check
## Look for when the JVMs are restarting
##
grep "GET /oprocmgr-service?cmd=Register" access_log*
##
## End of script

[Editor:  Due to formatting restrictions on this blog, if you're cutting-and-pasting this script, you must manually join the line above ending with grep -e with the following line starting with "servlet" into a single unbroken line.]

Conclusion

Proactive monitoring of the access_log will help you to :-

  • Baseline your system performance
  • Identify user usage patterns
  • Highlight possible system or user problems
  • Identify areas with possible performance issues
  • Verify user reported problems

Comments:

hi mike,
i was wondareing is there a way to know which ip address are connect to forms on my apps 11i. i need to keep a track of some forms.

thanks
fadi

Posted by Fadi Hasweh on August 28, 2006 at 01:10 AM PDT #

Hello Fadi

Not an easy answer to that question, maybe this is something I can try to address in a future discussion. It really depends on your exact business requirements, network architecture and whether you are using socket/forms servlet

Having said that, enabling form level auditing may satisfy the business requirement or a combination of v$session and "lsof" at operating system level may provide a crude solution

Mike

Posted by Mike Shaw on September 10, 2006 at 11:05 PM PDT #

Hi Steve, if you want to continue a unix command over to the next line.. you can simply say \ at the end of the top line and continue with the remainder of the command in the next line.. there should not be any space after the \ for command continuation...

e.g.

$ awk '{print $1, $2, ......}' | xargs grep \
-i log_format | wc -l

just a tip.

Posted by gaurav verma on January 21, 2007 at 05:35 AM PST #

Hi Mike,

Can we setup for forms server log files also ?
We have performance issue when users entering data or navigating betweek different forms.
appreciated your help.

regards,
Phani.k

Posted by Phani K on February 22, 2007 at 12:24 PM PST #

The simple answer is yes, but the answer you really want is quite long winded

Rather than try to give an abridged version as a response to your comment here, I would rather create a new article to cover the topic in the detail it deserves.

I should be able to do this soon, so hope this response won't be too disappointing

regards

Mike

Posted by Mike Shaw on February 22, 2007 at 07:13 PM PST #

Mike,

We are trying to get # of web hits/day, # of users logged in, etc. I found if I use the Oracle ICX.ICX_SESSIONS table to answer these questions, I tend to get less hits than reality because if users use the same brower login apps multiple times, it has only one record in the ICX.ICX_SESSIONS table (Oracle probably counts as the same cookie session)?

What is the more actuate way to answer these questions?

Thanks in advance for your time and help.
Jennifer.

Posted by Jennifer Chen on April 10, 2007 at 07:28 AM PDT #

We are experiencing "white screen of death" issue that has been mentioned on this blog. When we have this issue, simple jsp's also don't execute, until we bounce apache and jserv.

What is the best way to troubleshoot this issue ? Is it a problem with Apache or JVM ? We are under the impression that the JVM is busy and is not responding to the user requests on time resulting in white screen.

Any help will be greatly appreciated as our production application is getting impacted.

Posted by Ram on May 07, 2008 at 02:20 PM PDT #

Ram,I'm sorry to hear that you're experiencing that with your environment.&nbsp; I'd recommend logging a formal Service Request via Metalink to get one of our techstack specialists to help narrow things down to a set of possible root causes.&nbsp; Feel free to drop me an email with the SR number when it's been logged.Regards,Steven&nbsp;

Posted by Steven Chan on May 08, 2008 at 01:44 AM PDT #

In the access_log is there a way to show user_name or user_id,

httpd.conf LogFormat contains %u but shows - in access_log, I want a way to track username logged from which IP

Sally

Posted by guest on August 03, 2011 at 07:12 PM PDT #

Hello Sally,

You're not getting username from Apache perspective as this would be if logged in using BASIC authentication (for example)

Depending on exactly what you are after, I would imagine that Page Access Tracking may give you what you are after.

Refer

Page Access Tracking in Oracle Applications 11i (Doc ID 278881.1)
or
Page Access Tracking in Oracle Applications Release 12 (Doc ID 402116.1)

for more details

Hope this helps

Mike

Posted by Mike Shaw on August 07, 2011 at 11:40 PM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
4
5
6
7
8
9
10
11
12
13
14
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today