Thursday Mar 13, 2014

New Java Collections in Remote Diagnostic Agent (RDA) 8.03

What is RDA?

Not familiar with RDA (Remote Diagnostic Agent)? Learn more, take a look at :

Resolve Problems Faster! Use Remote Diagnostic Agent - Fusion Middleware and WebLogic Server [ID 1498376.1

RDA 8.03

RDA 8.03 was released on March 11th 2014. The download links can be found in the support document referenced above. Changes / fixes / enhancements in this version can be found in "RDA Release Notes [ID 414970.1]". In particular, there are new data collection profiles for:

  • Glassfish Server
  • Business Intelligence Applications
  • Oracle Identity Manager

And here are some other Fusion Middleware product highlights:

     Product  Enhancement
     WebLogic
    • Included Oracle Java Virtual Machine-related information
    • MBean configuration values collected even if WLST connection to Admin Server fails.
    Oracle Internet Directory
    • Added replication information
    • Added WebLogic process information
    • Added DIP profile configuration information
    SOA Suite
    • Extended MBean collections (via WLST)
    • Support finding of DB details from multiple data sources
    • Included version information
    Oracle Service Bus
    • Extended MBean collections (via WLST)

    New JVM Diagnostic Collections - WebLogic Server and Glassfish

    A significant addition to the WebLogic and Glassfish data collection set is the capture of JVM related information. RDA 8.03 does the following:

    • harvests JVM information from the log files
    • makes best efforts to find and collect garbage collection and JVM crash logs
    • identifies the JVM pid and leverages os (unix only) and JDK tools to capture diagnostic information such as heap summary, process map, address space and native thread stack.

    Screenshots of JVM Related Info collections in RDA

    Currently these new java collections are performed by RDA if run against a result set configuration created using the WebLogicServer or GlassFishServer profiles - example:

    Create a WebLogicServer profile result set configuration. In this example, we will use the '-s' to explicitly name the result set configuration rather than leave RDA to create the result set configuration using the default name - output.cfg

    rda.sh -S -s mywlscollection -p WebLogicServer

    Run the collection:

    rda.sh -s mywlscollection

    Instructions to view the output will be provided in the standard out after the collection is complete.

    Monday Feb 24, 2014

    RDA Can Now Be Upgraded Via OPatch (FMW 11g and higher)

    What is RDA?

    Not familiar with RDA (Remote Diagnostic Agent)? Learn more, take a look at :

    Resolve Problems Faster! Use Remote Diagnostic Agent - Fusion Middleware and WebLogic Server [ID 1498376.1

    About RDA via OPatch

    Until now RDA has been available via a zip package. The task of install and/or upgrade of RDA is therefore quick and simple. Delivery of RDA via a zip package, however, has one significant drawback - the version of RDA extracted is not recorded in the Oracle Inventory.

    Fusion Middleware products, such as SOA or WebCenter, do lay down a version of Remote Diagnostic Agent as part of their install. You may see an "rda" directory in the following locations:

    • Fusion Middleware 11g:
      • MW_HOME/ORACLE_HOME/rda
      • MW_HOME/oracle_common/rda
    • Fusion Middleware 12c:
      • ORACLE_HOME/oracle_common/rda

    RDA bundled with the Fusion Middleware media, falls behind the latest version. For example, Fusion Middleware 11g (11.1.1.8 and lower) ship RDA version 4 rather than version 8. Furthermore, the RDA installed does NOT include the RDA compiled engine - required if RDA is unable to find or access Perl.

    Therefore, replacing the old version of RDA with the latest is strongly encouraged. Trouble is, while a manual upgrade of RDA (via zip extraction) works fine, the version change is not recorded in the Oracle Inventory. Now we have a solution. RDA is available in OPatch patch format for Fusion Middleware 11g and 12c solutions. Any patch applied via OPatch is recorded in the inventory. In addition,  you can take advantage of the OPatch rollback feature.

    Download  

    Find the download link for the RDA in OPatch format in the My Oracle Support document:

    Resolve Problems Faster! Use Remote Diagnostic Agent - Fusion Middleware and WebLogic Server [ID 1498376.1

    After download, carefully follow the instructions in the readme.  

    Known Issues

    1. Unix only:

    If you have previously upgraded RDA using the standard RDA zip package (e.g. RDA 4.28 to 8.01),  a subsequent of upgrade of RDA via OPatch will fail with permission errors - here's just one example:

    Copy Action: Destination File "/u01/app/oracle/product/fmw/oracle_common/rda/mesg/en/STB.txt" is not writeable.
    'oracle.rda, 11.1.1.7.0': Cannot copy file from 'STB.txt' to '/u01/app/oracle/product/fmw/oracle_common/rda/mesg/en/STB.txt'

    To work-around, before running 'opatch apply', manually change the file permissions in the RDA home e.g.

    cd $MW_HOME/oracle_common
    chmod -R 660 rda


    2. If RDA's compiled engine has been enabled, OPatch will throw a pre-requisite check error like this one

    RDA compiled engine in use
    Command invocation returned Error... '', Return Code = 83
    Execution of PRE script failed, with return value = 83
    [ Error during Pre-script processing Phase]. Detail: OPatch failed: ApplySession failed in system modification phase... '

    The work-around is to revert the parameters in the RDA_HOME/engine/rda.cfg to their default values and then run 'opatch apply' again. Default values in rda.cfg are:

    RDA_ENG=""
    RDA_EXE="rda.exe"
    D_RDA=".." 

    3. And finally, RDA in OPatch format can only be applied against "rda" which has previously been installed by an Oracle Fusion Middleware product. OPatch looks for evidence of a previous RDA install in the Oracle Inventory. If your Oracle Inventory has no record of RDA, then download the RDA zip package and install / upgrade manually.  

    Thursday Dec 19, 2013

    RDA not working due to Perl being unavailable or inaccessible?

    What is RDA?

    Not familiar with RDA (Remote Diagnostic Agent)? Learn more, take a look at :

    • Resolve Problems Faster! Use Remote Diagnostic Agent - Fusion Middleware [ID 1498376.1]

    The Perl Problem 

    RDA is written in perl. If RDA cannot find Perl 5.005 or above, you may see one of the following messages:

    Error: ORACLE_HOME is not set
    Please set your ORACLE_HOME.

    or 

    Error: Perl not found in the PATH or in known folder locations.
    Although the default RDA engine requires Perl, a compiled version without Perl requirements is available.
    Please download the platform-specific RDA engine from My Oracle Support and place it within the top folder of your RDA installation.

    To work around you can configure RDA to use a compiled binary engine. The compiled engine is available for most platforms and is shipped as part of the RDA / OCM bundle. Find the engine binary in "RDA_HOME/engine".

    Good news! RDA 8.02 auto-enables the compiled engine

    The recently released RDA 8.02 automatically enables the compiled engine should RDA determines Perl is not available, accessible or valid. If you are reading this blog, having experienced the messages above or other perl errors, the likelihood is that you are using an older version of RDA. Upgrade to 8.02 now to benefit from this enhancement and others!

    Earlier versions of RDA require the compiled engine to be manually enabled if RDA is not able to use Perl.  

    Reference: 
    How to Manually Enable the Remote Diagnostic Agent ( RDA ) Compiled Engine .. [ID 1603021.1]

    Using RDA 8.02 and the perl problem described above still reproduces? Check the following:

    1. Does "RDA_HOME/engine" contain a compiled engine binary (rda_<platform> e.g rda_linux) and a rda.cfg.

    • You may have downloaded RDA standalone rather than the RDA / OCM Bundle. Or, you are working on a platform for which no compiled engine is available.

    2. Does the os user have read / write privileges to:

    • the directory from which they are launching the RDA command script
    • the RDA_HOME/engine/rda.cfg 

    3. Are you able to manually enable the compiled engine as per the instructions given in  [ID 1603021.1]

    Monday Oct 07, 2013

    RDA 8.01 - Now A Better Experience for WebLogic Administrators

    What is RDA?

    Not familiar with RDA (Remote Diagnostic Agent)? Well I won't use this blog entry to provide a lesson, but if you want to learn more, take a look at these two support articles:

    • Get Proactive with Fusion Middleware : Resolve Problems Faster! Use Remote Diagnostic Agent [ID 1498376.1]
    • How to Run Remote Diagnostic Agent ( RDA ) Against Oracle WebLogic Server ( WLS ) [ID 1262157.1]

    Too many questions in set up

    "I have to answer too many questions at set up time!". The first of two criticisms leveled at a RDA collection which targeted a WebLogic Server based technology stack. Well I pleased to say that the number of questions of has been reduced.

    If, for example, you set up RDA using the "WebLogicServer" profile, you will encounter a set up dialogue similar to the one copied / pasted from a Weblogic 12c solution, on Unix, below.

    Tip: Work through the set up questions a little quicker by configuring your environment variables to point at your Middleware Home and location of your WebLogic domain prior to running RDA set up. Doing so, allows RDA to find default values for certain questions, which means you can just hit return and move on.

    # Setting the environment
    
    . /home/oracle/app/FMW12c_1212/config/user_projects/domains/my_domain/bin/setDomainEnv.sh
    ORACLE_HOME=/home/oracle/app/FMW12c_1212/oh; export ORACLE_HOME
    
    # Running RDA set up 
    
    cd /home/oracle/software/rda
    ./rda.sh -S -p WebLogicServer
     
    # Set up dialogue
    Enter the network domain name for this server
    > uk.acme.com
    
    Enter the Oracle home to be used for data analysis
    Hit "Return" to accept the default (/home/oracle/app/FMW12c_1212/oh)
    > 
    
    Enter the location of the directory where the domains to analyze are located
    (For example in UNIX, <BEA Home>/user_projects/domains or 
    <Middleware Home>/user_projects/domains)
    Hit "Return" to accept the default
    (/home/oracle/app/FMW12c_1212/config/user_projects/domains)
    > 
    For a successful WLS connection, ensure that the domain Admin Server is up 
    and running.
    
    Data Collection Type:
      1  Perform offline collection
      2  Perform online collection (using WLS connection)
    
    Enter the item number
    Hit "Return" to accept the default (2)
    > 
    
    Select a Oracle WebLogic Server domain home target list:
      1  Domain my_domain (in
         /home/oracle/app/FMW12c_1212/config/user_projects/domains)
    
      A  All Oracle WebLogic Server domain home targets
      D  Define a new Oracle WebLogic Server domain home target
      E  Edit an Oracle WebLogic Server domain home target
      N  No Oracle WebLogic Server domain home targets
     
    Enter your selection, as a comma-separated list of item numbers or ranges
    Hit "Return" to accept the default (1)
    > 
    
    Server list for 'my_domain' domain:
      1  AdminServer
      2  ms3
    Enter your selection, as a comma-separated list of item numbers or ranges
    Hit "Return" to accept the default (1-2)
    > 
    Are custom scripts used to start Oracle WebLogic servers (Y/N)?
    Hit "Return" to accept the default (N)
    >
    
    # The next prompt concerns OCM. You are not required to configure OCM. 
    # If time is short, just answer 'N'
    
    RDA benefits from Oracle Configuration Manager by improving its 
    data collection. Oracle Configuration Manager (OCM) collects the 
    configuration information and uploads it to the OCM Repository 
    in connected mode. When OCM is configured in disconnected mode, it 
    adds the OCM payload to the RDA report packages in future RDA runs.
    
    Pending Configurations in /home/oracle/app/FMW12c_1212/oh:
      MH_T0001  Middleware home /home/oracle/app/FMW12c_1212/oh
    
    Hit 'Return' to continue
    
    Do you want RDA to configure OCM now (Y/N)?
    Hit "Return" to accept the default (Y)
    > N
    
    
    # After answering the prompts set up will be complete. 
    # Now just launch the command line script (no parameters) 
    # to run the collection
    
    ./rda.sh 

    "Cannot see the wood for the trees" in the output

    This is the second of the criticisms. RDA does collects many files and metrics - this is one its strengths. Yet I think it is fair to say that the presentation of WebLogic Server data in the RDA Start Menu (generated as part of the output) lacked some order and structure. For example, server log files were somewhat buried, and similarly, unless you knew to scroll down to WLST collections, you could not easily ascertain the basic contents of a targeted Weblogic domain. In RDA 8.01, the sub menus for the WebLogic Domain and Servers have been re-ordered to address this concern. Review the screenshots below.

    Figure 1 : WebLogic Domain Sub Menu


    Note: In the Main Index, RDA will only list managed servers which have been targeted for collection. In this example, only AdminServer and ms3 were targeted for collection. Yet we can see from the Domain MBeans that the domain contains other managed servers and a cluster. 

    Figure 2 : MBean Drill Down Example - Admin Server State

    Figure 3: WebLogic Server Sub Menu


    To take advantage of these changes download the latest version of RDA. The download links can be found in

    Get Proactive with Fusion Middleware : Resolve Problems Faster! Use Remote Diagnostic Agent [ID 1498376.1]

    Tuesday Nov 20, 2012

    RDA and Fusion Middleware Diagnostic Framework Integration

    Further to my last blog entry "FMw Diagnostic Framework : Automatic Capture of Diagnostic Data Upon First Failure!" I have spent some time exploring how Remote Diagnostic Agent (RDA) integrates with Diagnostic Framework.

    Remote Diagnostic Agent, by default, collects the information about the last 10 incidents which have been captured in the Automatic Diagnostic Repository (ADR). This information can be located via RDA's Start Page Menu system. See screenshot below.

    Screenshot - Viewing Diagnostic Framework Incident Information in a RDA Package


    Note: In the next release of RDA - version 4.30 - the Diagnostic Repository menu label will have it's own position in the weblogic managed server sub menu hierarchy rather than be a child menu item of the logs menu

    Diagnostic Framework is also capable of launching RDA engine and including the output in an incident package. This is achieved via the command "IPS GENERATE PACKAGE". The RDA output is written to

    DOMAIN_HOME/servers/<server name>/adr/diag/ofm/<domain name>/<server name>/
    incpkg/pkg_[number]/seq_[number]/rda
    

    The RDA collected files are best viewed (as shown in the screenshot above) by opening the RDA Start Page - "DFW__start.htm" - from this directory in a browser. If you do not have a browser available on the host machine, zip the contents of the rda directory and transfer and extract to a machine which has a browser. 

    The explanation of the integration goes a little deeper. If you want to know more, read My Oracle Support document:

    Understanding RDA and FMW 11g Diagnostic Framework Integration [Document 1503644.1]

    Friday Nov 02, 2012

    FMw Diagnostic Framework : Automatic Capture of Diagnostic Data Upon First Failure!

    Introduction

    There is nothing more frustrating than a problem that "cannot be reproduced". Logs, configuration files have been analysed but there just isn't enough information to establish the root cause. The issue maybe closed, but you are left with the feeling that the problem will raise its ugly head again in the future. Trouble is, to resolve such issues you need to capture diagnostic data at the exact time the incident occurs. Step forward Fusion Middleware Diagnostic Framework! 

    Diagnostic Framework monitors WebLogic Managed Servers and delivers "Automatic capture of diagnostic data upon first failure". To quote from

    Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
    Chapter 13 Diagnosing Problems

    "When a critical error occurs ... the Diagnostic Framework automatically collects diagnostics, such as thread dumps, DMS metric dumps, and WebLogic Diagnostics Framework (WLDF) server image dumps ... The data is stored in a file-based repository and is accessible with command-line utilities."

    In other words the data collected upon first failure - especially the thread and image dumps - provides a snapshot of the system as or immediately after the problem occurs. The table below shows the type of WebLogic Server issues which fall into the scope of Diagnostic Framework

    How to Configure Diagnostic Framework?

    Depending on your Fusion Middleware product choice you may not need to do anything! Diagnostic Framework is automatically installed, configured and initiated for any WebLogic Domain which has the Oracle Java Required Files (JRF) template applied. This template is applied by default whenever you configure WebLogic Managed Servers for products such as

    • Portal / Forms / Reports / Discoverer
    • Identity Management ( OID , OAM , OIM etc)
    • WebCenter
    • SOA

    Check your WebLogic Domain directory structure. If you have an "adr" sub directory under

    DOMAIN_HOME/servers/<servername>/

    then JRF template has been applied and Diagnostic Framework will be in play.

    Should the "adr" sub directory not exist, review the advice given in My Oracle Support article

    How to Apply FMW ( EM ) Control and JRF to a WebLogic Domain and Managed Servers [ID 947043.1]

    If you are working with a standalone WebLogic Server solution and applying Oracle JRF is not acceptable, consider using WLDF - WebLogic Diagnostic Framework. (Fusion Middleware Diagnostic Framework makes use of WLDF under the covers.) Couple of useful links about WLDF are listed below

    How to Get Started With Diagnostic Framework

    To be frank, the Fusion Middleware Administrator's Guide is the best place to start your learning

    Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
    Chapter 13 Diagnosing Problems

    A lot of reading here,  but if you are in hurry and just want to get the right information to Oracle Support to help resolve your issue, check out the next section below.

    How to Upload Diagnostic Framework Incident Data to Oracle Support

    Some Background Information

    There are three interfaces to the Repository:

    1. Enterprise Manager Cloud Control (Support Workbench)
    2. WLST (Command Line)
    3. ADRCI (Command Line)

    The Enterprise Manager Cloud Control does provide a nice GUI interface to search, view and package diagnostic framework incidents. However, this software is not to be confused with Fusion Middleware (EM) Control. Cloud Control (formerly known as Grid Control) is part of the Enterprise Manager media package. EM Cloud Control has it's own install and configuration story. Therefore, for the benefit of those yet to install and play with Cloud Control, I am going to describe how to use the command line tools.

    Ideally, you would only need to one command line interface, but currently I suggest using both - mainly due to the fact that ADRCI SHOW INCIDENTS does not reveal the description behind the Diagnostic Framework error code.

    Instructions:

    Note:

    WLST and ADRCI are case sensitive when it comes to handling parameter values. If you make a mistake, expect an unfriendly syntax error message.

    1) Find the incident

    Note:

    The managed server which you are troubleshooting must be up and running. If the managed server is down, ensure the domain's Admin Server is accessible. If you cannot connect to the Admin Server or the Managed Server the example WLST commands will not work.

    a) Launch WLST 

    Note: Use the WLST which resides in the "oracle_common" directory (not WL_HOME/common/bin) otherwise you will get a syntax error like the one below

    Traceback (innermost last):
      File "<console>", line 1, in ?
    NameError: listIncidents

    MW_HOME/oracle_common/common/bin/wlst.sh

    b) Connect to the managed server or the admin server e.g.

    wls:/offline> connect('weblogic','welcome1','t3://localhost:7020')

    c) Run the command

    wls:/mydomain/serverConfig> listIncidents()

    This will list the incidents for the server to which you have connected. If you have connected to the Admin Server and want to list the incidents for a managed server within the domain, use the command

    wls:/mydomain/serverConfig> listIncidents(adrHome='diag\ofm\mydomain\mymanagedserver'
    ,server='mymanagedserver')

    Example output

    Incident Id     Problem Key              Incident Time
            1       DFW-99998 [java.lang.NullPointerException]
    [oracle.error.simulator.ErrorSimulator.createNullPointerException][errorWebApp_1-0-0-0]        
    Fri Nov 02 10:38:46 GMT 2012

     The piece highlighted in bold is the description you do not see when using the ADRCI 'SHOW INCIDENT' command.

    Make a note of the incident id. You are ready to move to step 2

    2. Package the incident

    a) Set up the environment - example commands below are for Unix

    cd <DOMAIN_HOME>/bin
    . ./setDomainEnv.sh
    

    LD_LIBRARY_PATH=$WL_HOME/server/adr; export LD_LIBRARY_PATH

    If you want ADRCI to run a Remote Diagnostic Agent collection (recommended) at generate package time, point ORACLE_HOME at oracle_common

    ORACLE_HOME=$MW_HOME/oracle_common; export ORACLE_HOME

    To prevent ADRCI from running RDA at generate package time, point ORACLE_HOME at WL_HOME/server/adr directory. 

    ORACLE_HOME=$WL_HOME/server/adr; export ORACLE_HOME

    b) Launch adrci

    $WL_HOME/server/adr/adrci

    c) Set BASE and HOMEPATH

    adrci> SET BASE /oracle/middleware/user_projects/domains/
    mydomain/servers/mymanagedserver/adr
    
    adrci> SET HOMEPATH diag/ofm/mydomain/mymanagedserver

    d)  Optionally run SHOW INCIDENTS e.g.

    adrci> SHOW INCIDENTS -MODE DETAIL
    ADR Home = /oracle/middleware/user_projects/domains/mydomain/
    servers/mymanagedserver/adr/diag/ofm/mydomain/mymanagedserver:
    *************************************************************************

    **********************************************************
    INCIDENT INFO RECORD 1
    **********************************************************
       INCIDENT_ID                   1
       STATUS                        ready
       CREATE_TIME                   2012-11-02 10:38:46.468000 +00:00
       PROBLEM_ID                    1
       CLOSE_TIME                    <NULL>
       FLOOD_CONTROLLED              none
       ERROR_FACILITY                DFW
       ERROR_NUMBER                  99998
       ERROR_ARG1                    <NULL>
       ERROR_ARG2                    <NULL>
       ERROR_ARG3                    <NULL>
       ERROR_ARG4                    <NULL>
       ERROR_ARG5                    <NULL>
       ERROR_ARG6                    <NULL>
       ERROR_ARG7                    <NULL>
       ERROR_ARG8                    <NULL>
       ERROR_ARG9                    <NULL>
       ERROR_ARG10                   <NULL>
       ERROR_ARG11                   <NULL>
       ERROR_ARG12                   <NULL>
       SIGNALLING_COMPONENT          <NULL>
       SIGNALLING_SUBCOMPONENT       <NULL>
       SUSPECT_COMPONENT             <NULL>
       SUSPECT_SUBCOMPONENT          <NULL>
       ECID                          5162744c6a2eea5e:155ff445:13ac0aae7cb:-8000-000
    0000000000325
       IMPACTS                       0
    1 rows fetched

    e)  Create a logical package

    IPS CREATE PACKAGE INCIDENT incident_number

    e.g.

    adrci> IPS CREATE PACKAGE INCIDENT 1
    Created package 1 based on incident id 1, correlation level typical

    f) Generate the package

    IPS GENERATE PACKAGE package_number IN path

    e.g.

    adrci> IPS GENERATE PACKAGE 1 IN /tmp
    Generated package 1 in file /tmp/DFW99998j_20121102113633_COM_1.zip, mode complete
    

    Note:

    If the generate package command hangs, ADRCI may be experiencing an issue when running RDA. To avoid such trouble, exit ADRCI and point the ORACLE_HOME environment variable at WL_HOME/server/adr

    3) Upload the package zip to Oracle Support via your Service Request

    a) Log into My Oracle Support and locate your Service Request

    b) Click on "Add Attachments


    c) And upload the zip file


    Wednesday Oct 31, 2012

    How to Use RDA to Generate WLS Thread Dumps At Specified Intervals?

    Introduction

    There are many ways to generate a thread dump of a WebLogic Managed Server. For example, take a look at:

    Taking Thread Dumps - [an excellent blog post on the Middleware Magic site]

    or 

    Different ways to take thread dumps in WebLogic Server (Document 1098691.1)

    There is another method - use Remote Diagnostic Agent! The solution described below is not documented, but it is relatively straightforward to execute. One advantage of using RDA to collect the thread dumps is RDA will also collect configuration, log files, network, system, performance information at the same time.

    Instructions

    1. Not familiar with Remote Diagnostic Agent? Take a look at my previous blog "Resolve SRs Faster Using RDA - Find the Right Profile"

    2. Choose a profile, which includes the WebLogic Server data collection modules (for example the profile "WebLogicServer"). At RDA setup time you should see the prompt below:

    -------------------------------------------------------------------------------
    S301WLS: Collects Oracle WebLogic Server Information
    -------------------------------------------------------------------------------
    Enter the location of the directory where the domains to analyze are located
    (For example in UNIX, <BEA Home>/user_projects/domains or <Middleware
    Home>/user_projects/domains)
    Hit 'Return' to accept the default
    (/oracle/11AS/Middleware/user_projects/domains)
    > 
    
    For a successful WLS connection, ensure that the domain Admin Server is up and
    running.
    
    Data Collection Type:
      1  Collect for a single server (offline mode)
      2  Collect for a single server (using WLS connection)
      3  Collect for multiple servers (using WLS connection)
    
    Enter the item number
    Hit 'Return' to accept the default (1)
    > 2
    

    Choose option 2 or 3.

    Note: Collect for a single server or multiple servers using WLS connection means that RDA will attempt to connect to execute online WLST commands against the targeted server(s). The thread dumps are collected using the WLST function - "threadDumps()". If WLST cannot connect to the managed server, RDA will proceed to collect other data and ignore the request to collect thread dumps. If in the final output you see no Thread Dump menu item, then it's likely that the managed server is in a state which prevents new connections to it. If faced with this scenario, you would have to employ alternative methods for collecting thread dumps.

    3. The RDA setup will create a setup.cfg file in the RDA_HOME directory. Open this file in an editor. You will find the following parameters which govern the number of thread dumps and thread dump interval.

    #N.Number of thread dumps to capture
    WREQ_THREAD_DUMP=10
    #N.Thread dump interval
    WREQ_THREAD_DUMP_INTERVAL=5000

    The example lines above show the default settings. In other words, RDA will collect 10 thread dumps at 5000 millisecond (5 second) intervals. You may want to change this to something like:

    #N.Number of thread dumps to capture
    WREQ_THREAD_DUMP=10
    #N.Thread dump interval
    WREQ_THREAD_DUMP_INTERVAL=30000

    However, bear in mind, that such change will increase the total amount of time it takes for RDA to complete its run.

    4. Once you are happy with the setup.cfg, run RDA. RDA will collect, render, generate and package all files in the output directory.

    5. For ease of viewing, open up the RDA Start html file - "xxxx__start.htm". The thread dumps can be found under the WLST Collections for the target managed server(s). See screenshots below

    Screenshot 1:RDA Start Page - Main Index

    Screenshot of RDA Start Page

    Screenshot 2: Managed Server Sub Index

    Screenshot of Managed Server Sub Index

    Screenshot 3: WLST Collections

    Screenshot of WLST Collections - pointing out Thread Dump menu option

    Screenshot 4: Thread Dump Page - List of dump file links

    Screenshot of Thread Dump Page - list of dump file links

    Screenshot 5: Thread Dump Dat File Link

    Screenshot of what you see when you click a dump file link

    Additional Comment:

    A) You can view the thread dump files within the RDA Start Page framework, but most likely you will want to download the dat files for in-depth analysis via thread dump analysis tools such as:

    If you are new to thread dump analysis - take a look at this recorded Support Advisor Webcast 

    Oracle WebLogic Server: Diagnosing Performance Issues through Java Thread Dumps
    [Slidedeck from webcast in PDF format]

    B) I have logged a couple of enhancement requests for the RDA Development Team to consider:

    1. Add timestamp to dump file links, dat filename and at the top of the body of the dat file
    2. Package the individual thread dumps in a zip so all dump files can be conveniently downloaded in one go.

    Monday Sep 10, 2012

    Resolve SRs Faster Using RDA - Find the Right Profile

    Introduction

    Remote Diagnostic Agent (RDA) is an excellent command-line data collection tool that can aid troubleshooting / problem solving. The tool covers the majority of Oracle's vast product range, and its data collection capability is comprehensive. RDA collects data about

    • the operating system and environment, including
      • environment variable, kernel settings
      • network
      • o/s performance
      • o/s patches
      • and much more
    • the Oracle Products installed, including
      • patches
      • logs and debug
      • metrics
      • configuration
      • and much more

    In effect, RDA can obtain a snapshot of an Oracle Product and its environment. Oracle Support encourages the use of RDA because it greatly reduces service request resolution time by minimizing the number of requests from Oracle Support for more information. RDA is designed to be as unobtrusive as possible; it does not modify systems in any way. It collects useful data for Oracle Support only and a security filter is provided if required.

    RDA can be downloaded via this My Oracle Support document

    Remote Diagnostic Agent (RDA) 4 - Getting Started [ID 314422.1]

    Find and Use the Right RDA Profile

    One problem of any tool / utility, which covers a large range of products, is knowing how to target it against only the products you wish to troubleshoot. RDA does not have a GUI. Nor does RDA have an intelligent mechanism for detecting and automatically collecting data only for those Oracle products installed. Instead, you have to tell RDA what to do.

    There is a mind boggling large number of RDA data collection modules which you can configure RDA to use. It is easier, however, to setup RDA to use a "Profile". A profile consists of a list of data collection modules and predefined settings. As such profiles can be used to diagnose a problem with a particular product or combination of products.

    How to run RDA with a profile?

    ( <rda> represents the command you selected to run RDA (for example, rda.pl, rda.cmd, rda.sh, and perl rda.pl).)

    1. Use the embedded spreadsheet to find the RDA profile which is appropriate for your problem / chosen Oracle Fusion Middleware products.

    2. Use the following command to perform the setup

    <rda> -S -p <profile_name> 

    3. Run the data collection

    <rda>

    Run the data collection. If you want to perform setup and run in one go, then use a command such as the following:

    <rda> -vnSCRP -p <profile name>

    For more information, refer to:

    Remote Diagnostic Agent (RDA) 4 - Profile Manual Pages [ID 391983.1]

    Additional Hints / Tips:

    1. Be careful! Profile names are case sensitive.

    2. When profiles are not used, RDA considers all existing modules by default. For example, if you have downloaded RDA for the first time and run the command

    <rda> -S

    you will see prompts for every RDA collection module many of which will be of no interest to you. Also, you may, in your haste to work through all the questions, forget to say "Yes" to the collection of data that is pertinent to your particular problem or product. Profiles avoid such tedium and help ensure the right data is collected at the first time of asking.

    3. RDA has a security filter which removes potentially sensitive data such as

    • Domain names
    • Group names
    • Host names
    • IPv4 and IPv6 addresses
    • LDAP domain components
    • Network masks
    • User names

    You can switch on at RDA setup by appending "-Security" to the profile value

    <rda> -vnSCRP -p <profile name>-Security

    e.g.

    <rda> -vnSCRP -p FM11g_WlsWebTier-Security

    If you already have a setup.cfg, you can enable the security filter by running the command

    <rda> -X Filter enable 

    Wednesday Jun 20, 2012

    How to Update Remote Diagnostic Agent (RDA) to Latest Version?

    Remote Diagnostic Agent (RDA) 4.28 was released on 12th June. Full details can be found in this My Oracle Support document

    RDA 4 Release Notes [ID 414970.1]

    From a Fusion Middleware Core Component, Install and Administration perspective this latest release does not offer any significant new features or changes. However, despite the lack of Fusion Middleware specific new features in version 4.28, Remote Diagnostic Agent still comes as highly recommended. It is incredibly useful problem solving / troubleshooting aid. Support engineers dealing with Service Requests often request RDA output as it collects just about everything you might need to get a view of the state and configuration of the host operating system, network setup and Fusion Middleware components. To find out more take a look at

    • Running RDA Against Oracle Fusion Middleware 11g [ID 853437.1]
    • Getting Started With Remote Diagnostic Agent: Case Study - Oracle WebLogic Server (Video) [ID 1262157.1]

    Note: While the latter document looks at RDA from the perspective of WebLogic Server, much of the advice given in the videos can be applied to other Fusion Middleware products.

    Ok, let's get back on track with the topic suggested by the title. If you are already familiar with Remote Diagnostic Agent you may ask the question - 'How do I keep my RDA at the latest version?'

    The answer is in "Running RDA Against Oracle Fusion Middleware 11g [ID 853437.1]". To quote:

    There are two methods:
    1. Upgrade RDA via OCM (Oracle Configuration Manager) 
    Refer to the advice given in:
    Remote Diagnostic Agent (RDA) Upgrade README [ID 1309034.1] 
    OR
    2. Manually download and upgrade to the latest version. 
    To quote from
    Remote Diagnostic Agent (RDA) 4 - FAQ [ID 330363.1]
    +++++++++++++++++++++++++++++++++++++++++++++++++++++++
    How do I upgrade my RDA 4.x installation from the prior release?
    The most simplest and reliable way to upgrade your RDA installation is 
    delete or move your old installation  to a new location. Then install the 
    new release into the location you had the prior release installed. 
    
    If you want to reuse you old setup.cfg file, you can place the older version 
    into the new <rda> directory and it will try to upgrade your setup.cfg 
    to the new features.
    
    A second approach is to install the latest RDA into another directory, 
    then if needed copy the old setup.cfg file to the new RDA directory. 
    When the new RDA is run for the first time, it will try to upgrade 
    your setup.cfg to the new features.
    +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 
    

    The upgrade method via Oracle Configuration Manager is nice because it allows RDA to be auto updated whenever a new release of RDA is made available (which roughly speaking is every 3 months). However, it does require you to install and configure Oracle Configuration Manager in addition to RDA. A quick guide to Fusion Middleware 11g and OCM can be found in this support document.

    Configuring OCM in Oracle Fusion Middleware 11g? A Quick and Easy Guide [ID 1096871.1]

    About

    This is the blog of the Oracle Fusion Middleware Proactive Support Delivery Team. Here we will provide information about our activities, publications, product related information and more. Feedback welcome.

    Follow OracleMWSupport on Twitter

    Search

    Archives
    « April 2014
    SunMonTueWedThuFriSat
      
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
       
           
    Today