Tuesday Nov 20, 2012

RDA and Fusion Middleware Diagnostic Framework Integration

Further to my last blog entry "FMw Diagnostic Framework : Automatic Capture of Diagnostic Data Upon First Failure!" I have spent some time exploring how Remote Diagnostic Agent (RDA) integrates with Diagnostic Framework.

Remote Diagnostic Agent, by default, collects the information about the last 10 incidents which have been captured in the Automatic Diagnostic Repository (ADR). This information can be located via RDA's Start Page Menu system. See screenshot below.

Screenshot - Viewing Diagnostic Framework Incident Information in a RDA Package


Note: In the next release of RDA - version 4.30 - the Diagnostic Repository menu label will have it's own position in the weblogic managed server sub menu hierarchy rather than be a child menu item of the logs menu

Diagnostic Framework is also capable of launching RDA engine and including the output in an incident package. This is achieved via the command "IPS GENERATE PACKAGE". The RDA output is written to

DOMAIN_HOME/servers/<server name>/adr/diag/ofm/<domain name>/<server name>/
incpkg/pkg_[number]/seq_[number]/rda

The RDA collected files are best viewed (as shown in the screenshot above) by opening the RDA Start Page - "DFW__start.htm" - from this directory in a browser. If you do not have a browser available on the host machine, zip the contents of the rda directory and transfer and extract to a machine which has a browser. 

The explanation of the integration goes a little deeper. If you want to know more, read My Oracle Support document:

Understanding RDA and FMW 11g Diagnostic Framework Integration [Document 1503644.1]

Friday Nov 02, 2012

FMw Diagnostic Framework : Automatic Capture of Diagnostic Data Upon First Failure!

Introduction

There is nothing more frustrating than a problem that "cannot be reproduced". Logs, configuration files have been analysed but there just isn't enough information to establish the root cause. The issue maybe closed, but you are left with the feeling that the problem will raise its ugly head again in the future. Trouble is, to resolve such issues you need to capture diagnostic data at the exact time the incident occurs. Step forward Fusion Middleware Diagnostic Framework! 

Diagnostic Framework monitors WebLogic Managed Servers and delivers "Automatic capture of diagnostic data upon first failure". To quote from

Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
Chapter 13 Diagnosing Problems

"When a critical error occurs ... the Diagnostic Framework automatically collects diagnostics, such as thread dumps, DMS metric dumps, and WebLogic Diagnostics Framework (WLDF) server image dumps ... The data is stored in a file-based repository and is accessible with command-line utilities."

In other words the data collected upon first failure - especially the thread and image dumps - provides a snapshot of the system as or immediately after the problem occurs. The table below shows the type of WebLogic Server issues which fall into the scope of Diagnostic Framework

How to Configure Diagnostic Framework?

Depending on your Fusion Middleware product choice you may not need to do anything! Diagnostic Framework is automatically installed, configured and initiated for any WebLogic Domain which has the Oracle Java Required Files (JRF) template applied. This template is applied by default whenever you configure WebLogic Managed Servers for products such as

  • Portal / Forms / Reports / Discoverer
  • Identity Management ( OID , OAM , OIM etc)
  • WebCenter
  • SOA

Check your WebLogic Domain directory structure. If you have an "adr" sub directory under

DOMAIN_HOME/servers/<servername>/

then JRF template has been applied and Diagnostic Framework will be in play.

Should the "adr" sub directory not exist, review the advice given in My Oracle Support article

How to Apply FMW ( EM ) Control and JRF to a WebLogic Domain and Managed Servers [ID 947043.1]

If you are working with a standalone WebLogic Server solution and applying Oracle JRF is not acceptable, consider using WLDF - WebLogic Diagnostic Framework. (Fusion Middleware Diagnostic Framework makes use of WLDF under the covers.) Couple of useful links about WLDF are listed below

How to Get Started With Diagnostic Framework

To be frank, the Fusion Middleware Administrator's Guide is the best place to start your learning

Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
Chapter 13 Diagnosing Problems

A lot of reading here,  but if you are in hurry and just want to get the right information to Oracle Support to help resolve your issue, check out the next section below.

How to Upload Diagnostic Framework Incident Data to Oracle Support

Some Background Information

There are three interfaces to the Repository:

  1. Enterprise Manager Cloud Control (Support Workbench)
  2. WLST (Command Line)
  3. ADRCI (Command Line)

The Enterprise Manager Cloud Control does provide a nice GUI interface to search, view and package diagnostic framework incidents. However, this software is not to be confused with Fusion Middleware (EM) Control. Cloud Control (formerly known as Grid Control) is part of the Enterprise Manager media package. EM Cloud Control has it's own install and configuration story. Therefore, for the benefit of those yet to install and play with Cloud Control, I am going to describe how to use the command line tools.

Ideally, you would only need to one command line interface, but currently I suggest using both - mainly due to the fact that ADRCI SHOW INCIDENTS does not reveal the description behind the Diagnostic Framework error code.

Instructions:

Note:

WLST and ADRCI are case sensitive when it comes to handling parameter values. If you make a mistake, expect an unfriendly syntax error message.

1) Find the incident

Note:

The managed server which you are troubleshooting must be up and running. If the managed server is down, ensure the domain's Admin Server is accessible. If you cannot connect to the Admin Server or the Managed Server the example WLST commands will not work.

a) Launch WLST 

Note: Use the WLST which resides in the "oracle_common" directory (not WL_HOME/common/bin) otherwise you will get a syntax error like the one below

Traceback (innermost last):
  File "<console>", line 1, in ?
NameError: listIncidents

MW_HOME/oracle_common/common/bin/wlst.sh

b) Connect to the managed server or the admin server e.g.

wls:/offline> connect('weblogic','welcome1','t3://localhost:7020')

c) Run the command

wls:/mydomain/serverConfig> listIncidents()

This will list the incidents for the server to which you have connected. If you have connected to the Admin Server and want to list the incidents for a managed server within the domain, use the command

wls:/mydomain/serverConfig> listIncidents(adrHome='diag\ofm\mydomain\mymanagedserver'
,server='mymanagedserver')

Example output

Incident Id     Problem Key              Incident Time
        1       DFW-99998 [java.lang.NullPointerException]
[oracle.error.simulator.ErrorSimulator.createNullPointerException][errorWebApp_1-0-0-0]        
Fri Nov 02 10:38:46 GMT 2012

 The piece highlighted in bold is the description you do not see when using the ADRCI 'SHOW INCIDENT' command.

Make a note of the incident id. You are ready to move to step 2

2. Package the incident

a) Set up the environment - example commands below are for Unix

cd <DOMAIN_HOME>/bin
. ./setDomainEnv.sh

LD_LIBRARY_PATH=$WL_HOME/server/adr; export LD_LIBRARY_PATH

If you want ADRCI to run a Remote Diagnostic Agent collection (recommended) at generate package time, point ORACLE_HOME at oracle_common

ORACLE_HOME=$MW_HOME/oracle_common; export ORACLE_HOME

To prevent ADRCI from running RDA at generate package time, point ORACLE_HOME at WL_HOME/server/adr directory. 

ORACLE_HOME=$WL_HOME/server/adr; export ORACLE_HOME

b) Launch adrci

$WL_HOME/server/adr/adrci

c) Set BASE and HOMEPATH

adrci> SET BASE /oracle/middleware/user_projects/domains/
mydomain/servers/mymanagedserver/adr
adrci> SET HOMEPATH diag/ofm/mydomain/mymanagedserver

d)  Optionally run SHOW INCIDENTS e.g.

adrci> SHOW INCIDENTS -MODE DETAIL
ADR Home = /oracle/middleware/user_projects/domains/mydomain/
servers/mymanagedserver/adr/diag/ofm/mydomain/mymanagedserver:
*************************************************************************

**********************************************************
INCIDENT INFO RECORD 1
**********************************************************
   INCIDENT_ID                   1
   STATUS                        ready
   CREATE_TIME                   2012-11-02 10:38:46.468000 +00:00
   PROBLEM_ID                    1
   CLOSE_TIME                    <NULL>
   FLOOD_CONTROLLED              none
   ERROR_FACILITY                DFW
   ERROR_NUMBER                  99998
   ERROR_ARG1                    <NULL>
   ERROR_ARG2                    <NULL>
   ERROR_ARG3                    <NULL>
   ERROR_ARG4                    <NULL>
   ERROR_ARG5                    <NULL>
   ERROR_ARG6                    <NULL>
   ERROR_ARG7                    <NULL>
   ERROR_ARG8                    <NULL>
   ERROR_ARG9                    <NULL>
   ERROR_ARG10                   <NULL>
   ERROR_ARG11                   <NULL>
   ERROR_ARG12                   <NULL>
   SIGNALLING_COMPONENT          <NULL>
   SIGNALLING_SUBCOMPONENT       <NULL>
   SUSPECT_COMPONENT             <NULL>
   SUSPECT_SUBCOMPONENT          <NULL>
   ECID                          5162744c6a2eea5e:155ff445:13ac0aae7cb:-8000-000
0000000000325
   IMPACTS                       0
1 rows fetched

e)  Create a logical package

IPS CREATE PACKAGE INCIDENT incident_number

e.g.

adrci> IPS CREATE PACKAGE INCIDENT 1
Created package 1 based on incident id 1, correlation level typical

f) Generate the package

IPS GENERATE PACKAGE package_number IN path

e.g.

adrci> IPS GENERATE PACKAGE 1 IN /tmp
Generated package 1 in file /tmp/DFW99998j_20121102113633_COM_1.zip, mode complete

Note:

If the generate package command hangs, ADRCI may be experiencing an issue when running RDA. To avoid such trouble, exit ADRCI and point the ORACLE_HOME environment variable at WL_HOME/server/adr

3) Upload the package zip to Oracle Support via your Service Request

a) Log into My Oracle Support and locate your Service Request

b) Click on "Add Attachments


c) And upload the zip file


Monday Sep 10, 2012

Resolve SRs Faster Using RDA - Find the Right Profile

Introduction

Remote Diagnostic Agent (RDA) is an excellent command-line data collection tool that can aid troubleshooting / problem solving. The tool covers the majority of Oracle's vast product range, and its data collection capability is comprehensive. RDA collects data about

  • the operating system and environment, including
    • environment variable, kernel settings
    • network
    • o/s performance
    • o/s patches
    • and much more
  • the Oracle Products installed, including
    • patches
    • logs and debug
    • metrics
    • configuration
    • and much more

In effect, RDA can obtain a snapshot of an Oracle Product and its environment. Oracle Support encourages the use of RDA because it greatly reduces service request resolution time by minimizing the number of requests from Oracle Support for more information. RDA is designed to be as unobtrusive as possible; it does not modify systems in any way. It collects useful data for Oracle Support only and a security filter is provided if required.

RDA can be downloaded via this My Oracle Support document

Remote Diagnostic Agent (RDA) 4 - Getting Started [ID 314422.1]

Find and Use the Right RDA Profile

One problem of any tool / utility, which covers a large range of products, is knowing how to target it against only the products you wish to troubleshoot. RDA does not have a GUI. Nor does RDA have an intelligent mechanism for detecting and automatically collecting data only for those Oracle products installed. Instead, you have to tell RDA what to do.

There is a mind boggling large number of RDA data collection modules which you can configure RDA to use. It is easier, however, to setup RDA to use a "Profile". A profile consists of a list of data collection modules and predefined settings. As such profiles can be used to diagnose a problem with a particular product or combination of products.

How to run RDA with a profile?

( <rda> represents the command you selected to run RDA (for example, rda.pl, rda.cmd, rda.sh, and perl rda.pl).)

1. Use the embedded spreadsheet to find the RDA profile which is appropriate for your problem / chosen Oracle Fusion Middleware products.

2. Use the following command to perform the setup

<rda> -S -p <profile_name> 

3. Run the data collection

<rda>

Run the data collection. If you want to perform setup and run in one go, then use a command such as the following:

<rda> -vnSCRP -p <profile name>

For more information, refer to:

Remote Diagnostic Agent (RDA) 4 - Profile Manual Pages [ID 391983.1]

Additional Hints / Tips:

1. Be careful! Profile names are case sensitive.

2. When profiles are not used, RDA considers all existing modules by default. For example, if you have downloaded RDA for the first time and run the command

<rda> -S

you will see prompts for every RDA collection module many of which will be of no interest to you. Also, you may, in your haste to work through all the questions, forget to say "Yes" to the collection of data that is pertinent to your particular problem or product. Profiles avoid such tedium and help ensure the right data is collected at the first time of asking.

3. RDA has a security filter which removes potentially sensitive data such as

  • Domain names
  • Group names
  • Host names
  • IPv4 and IPv6 addresses
  • LDAP domain components
  • Network masks
  • User names

You can switch on at RDA setup by appending "-Security" to the profile value

<rda> -vnSCRP -p <profile name>-Security

e.g.

<rda> -vnSCRP -p FM11g_WlsWebTier-Security

If you already have a setup.cfg, you can enable the security filter by running the command

<rda> -X Filter enable 

About

This is the blog of the Oracle Fusion Middleware Proactive Support Delivery Team. Here we will provide information about our activities, publications, product related information and more. Feedback welcome.

Follow OracleMWSupport on Twitter

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today