Monday Feb 04, 2013

Fusion Middleware Performance Problems? Take Advantage of DMS

Introduction

Why is my application running slowly?
How can I access configuration data and related metrics?
I need to a tool to help me tune my Fusion Middleware solution?

If you are asking these types of questions then take some time out to find out more about DMS. 

What is DMS?

Yes, another acronym .. sorry ... translation : "Dynamic Monitoring Service." It is a service which enables Oracle Fusion Middleware components to provide data regarding the component's performance, state and ongoing behavior. DMS measures and reports metrics, trace events and system performance and provides a context correlation service for these components.

Benefits include

  • Comprehensive set of performance metrics
  • Multiple tools for easier viewing and analysis of the metrics - including:
    • Fusion Middleware (EM) Control
    • DMS Spy Servlet
    • WLST
    • Remote Diagnostic Agent
    • Diagnostic Framework
  • Event Tracing
    • configure live tracing with no restarts
    • trace issues within a given time period or collect specific data at a specific time for a specific set of criteria

Oracle Fusion Middleware Performance and Tuning Guide 11g
Chapter 6 Oracle Dynamic Monitoring Service Performance Tuning

is the place to find out more.

Important to Note:

The majority of Oracle Fusion Middleware 11g WebLogic based products incorporate DMS in their product code. As such there is no special set up required. DMS works out of the box. For WebLogic based products Diagnostic Framework is provided via JRF (Java Required Files). If a WebLogic Managed Server does not have JRF applied, DFW will NOT be available. Domains created with standalone WebLogic Server will not, by default, have JRF applied. Refer to:

Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
19 Scaling Your Environment
19.3.1 Applying Oracle JRF Template to a Managed Server or Cluster

If you have a login to My Oracle Support, also check out the knowledge article:

How to Apply FMW ( EM ) Control and JRF to a WebLogic Domain and Managed Servers [ID 947043.1]

DMS and Fusion Middleware (EM) Control

As stated above there are many ways to integrate with DMS. In this blog post, I am going to focus on Fusion Middleware (EM) Control as I think it tackles the task rather well. In Fusion Middleware Control, the DMS metrics for a component (e.g. a WebLogic managed server) are accessed via the Performance Summary option.

Screenshot : Performance Summary Menu Option

The Performance Summary will show a default selection of metrics for the component - some displayed in chart form, others listed in tables. However, within a matter of minutes you can change the metric selection and configure a metric view which suits your purpose or analysis.

For example; the screenshot below shows charts which plots, by way of comparison, the CPU and Memory usage of two WebLogic Managed Servers residing in the same domain.

Screenshot: WebLogic Managed Server - CPU and Memory Comparison


How is this done? Well, first click the button "Show Metric Pallette". 

Screenshot : Show Metric Pallette

The metric pallette allows you to pick and choose from a wide range of metrics. It is worth exploring. In this example, CPU and Memory Usage under the "JVM Metrics" has been chosen for WC_Spaces1.

Screenshot : WC_Spaces1

Then we scrolled down and selected the same metrics for WC_Spaces2 which is found under "Related Targets". To plot metrics on the same chart, the trick is to drag the metric from the pallette onto the existing chart which is showing just the one value. It's a bit difficult to show this in a screenshot :-) Just give it a go.

Screenshot: WC_Spaces2

You can also play around with the metric time period by enabling the Slider, or by specifying default time selection periods (15 minutes, 30 minutes, 1 hour), or by specifying your own start and end date / times.

Screenshot: Metric Time Period

And a final tip. Try using the Overlay menu option to build a chart which shows a data comparison - for example another Oracle WebLogic Server or Yesterday's data. (You will only see the latter option if you have been logged into Fusion Middleware (EM) Control for over 24 hours.)

Screenshot: Overlay

I hope this gives a flavor of what Dynamic Monitoring Service has to offer and that it can help you resolve performance related issues.

Friday Jan 11, 2013

Guide to Finding Fusion Middleware "End of Support" Information

When does support end for my Fusion Middleware version? Such a simple question and frequently asked. But there is more to the answer than you might expect. Furthermore, finding the full detail can be challenging.

In this blog entry, I will describe the navigation path through the relevant documents - all of which need to be read not only to obtain the "end of support" dates, but to also gain an understanding of Oracle Lifetime Support and Error Correction (bug fixing) policies.

The starting point is the Oracle Lifetime Support page

http://www.oracle.com/us/support/lifetime-support/index.html

This page lays out the Oracle Lifetime Support terms, definitions and benefits. Once these are understood, go to the product specific Lifetime Support documents listed on the page. The direct link to the Fusion Middleware document is:

http://www.oracle.com/us/support/library/lifetime-support-middleware-069163.pdf

This is a weighty tome (60 + pages) which contains the actual "end of support" dates. The challenge is to pick out the dates for the different Middleware products which form your chosen technology stack / solution. On page 8, there is a table labelled - "Fusion Middleware Releases". While some dates are listed in this table, for 10.1.3, 11g and 12c you are referred to the footnotes e.g.

Screenshot of rows in Fusion Middleware Release Table on Page 8

To quote the footnote

"... For specific dates refer to the respective sections in this document. To find a section, refer to the Table of Contents."

Therefore, be prepared to jump to different sections to find the dates which are specific to the products you have chosen for your Middleware solution.

There is one other very important footnote on this page, to quote:

"For more-detailed information on bug fix and patch release policies, please refer to the “Error Correction Support Policy” on My Oracle Support ."

This statement leads you down the path to find "Error Correction Support" dates for given patch set version numbers. Establishing the "Error Correction Support" dates for your solution is key because it will inform your plans for patching / maintenance. 

For Fusion Middleware, you can cut to the chase and go straight to:

Error Correction Support Dates for Oracle Fusion Middleware 11g (11.1.1/11.1.2) [ID 1290894.1]

Thursday Dec 13, 2012

How to Run Apache Commands From Oracle HTTP Server 11g Home

Every now and then you come across a problem when there is nothing in the "troubleshooting manual" which can help you. Instead you need to think outside the box. This happened to me two or three years back. Oracle HTTP Server (OHS) 11g did not start. The error reported back by OPMN was generic and gave no clue, and worse the HTTP Server error log was empty, and remained so even after I had increased the OPMN and HTTP Server log levels. After checking configuration files, operating system resources, etc I was still no nearer the solution.

And then the light bulb moment!

OHS is based on Apache - what happens if I attempt to start HTTP Server using the native apache command. Trouble was the OHS 11g solution has its binaries and configuration files in separate "home" directories

  • ORACLE_HOME contains the binaries
  • ORACLE_INSTANCE contains the configuration files

How to set the environment so that native apache commands run without error? Eventually, with help from a colleague, the knowledge article

How to Start Oracle HTTP Server 11g Without Using opmnctl [ID 946532.1]

was born!

To be honest, I cannot remember the exact cause and solution to that OHS problem two or three years ago. But, I do remember that an attempt to start HTTP Server using the native apache command threw back an error to the console which led me to discover the culprit was some unusual filesystem fault.

The other day, I was asked to review and publish a new knowledge article which described how to use the apache command to dump a list of static and shared loaded modules. This got me thinking that it was time
[ID 946532.1] was given an update. The result

How To Run Native Apache Commands in an Oracle HTTP Server 11g Environment [
ID 946532.1]

Highlights:

  • Title change
  • Improved environment setting scripts
    • Interactive, should be no need to manually edit the scripts (although readers are welcome to do so)
    • Automatically dump out some diagnostic information
  • Inclusion of some links to other troubleshooting collateral

To view the knowledge article you need a My Oracle Support login. For convenience, you can obtain the scripts via the links below.

MS Windows:

Wrapper cmd script - calls main cmd script [After download, remove the ".txt" file extension]
Main cmd script - sets OHS 11g environment to run Apache commands [After download, remove the ".txt" file extension]

Unix:

Shell script - sets OHS 11g environment to run Apache commands on Unix

Please note: I cannot guarantee that the scripts held in the blog repository will be maintained. Any enhancements or faults will applied to the scripts attached to the knowledge article.

Lastly, to find out more about native apache commands, refer to the Apache Documentation

apachectl - Apache HTTP Server Control Interface
[http://httpd.apache.org/docs/2.2/programs/apachectl.html]

httpd - Apache Hypertext Transfer Protocol Server
[http://httpd.apache.org/docs/2.2/programs/httpd.html]

Tuesday Nov 20, 2012

RDA and Fusion Middleware Diagnostic Framework Integration

Further to my last blog entry "FMw Diagnostic Framework : Automatic Capture of Diagnostic Data Upon First Failure!" I have spent some time exploring how Remote Diagnostic Agent (RDA) integrates with Diagnostic Framework.

Remote Diagnostic Agent, by default, collects the information about the last 10 incidents which have been captured in the Automatic Diagnostic Repository (ADR). This information can be located via RDA's Start Page Menu system. See screenshot below.

Screenshot - Viewing Diagnostic Framework Incident Information in a RDA Package


Note: In the next release of RDA - version 4.30 - the Diagnostic Repository menu label will have it's own position in the weblogic managed server sub menu hierarchy rather than be a child menu item of the logs menu

Diagnostic Framework is also capable of launching RDA engine and including the output in an incident package. This is achieved via the command "IPS GENERATE PACKAGE". The RDA output is written to

DOMAIN_HOME/servers/<server name>/adr/diag/ofm/<domain name>/<server name>/
incpkg/pkg_[number]/seq_[number]/rda

The RDA collected files are best viewed (as shown in the screenshot above) by opening the RDA Start Page - "DFW__start.htm" - from this directory in a browser. If you do not have a browser available on the host machine, zip the contents of the rda directory and transfer and extract to a machine which has a browser. 

The explanation of the integration goes a little deeper. If you want to know more, read My Oracle Support document:

Understanding RDA and FMW 11g Diagnostic Framework Integration [Document 1503644.1]

Friday Nov 02, 2012

FMw Diagnostic Framework : Automatic Capture of Diagnostic Data Upon First Failure!

Introduction

There is nothing more frustrating than a problem that "cannot be reproduced". Logs, configuration files have been analysed but there just isn't enough information to establish the root cause. The issue maybe closed, but you are left with the feeling that the problem will raise its ugly head again in the future. Trouble is, to resolve such issues you need to capture diagnostic data at the exact time the incident occurs. Step forward Fusion Middleware Diagnostic Framework! 

Diagnostic Framework monitors WebLogic Managed Servers and delivers "Automatic capture of diagnostic data upon first failure". To quote from

Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
Chapter 13 Diagnosing Problems

"When a critical error occurs ... the Diagnostic Framework automatically collects diagnostics, such as thread dumps, DMS metric dumps, and WebLogic Diagnostics Framework (WLDF) server image dumps ... The data is stored in a file-based repository and is accessible with command-line utilities."

In other words the data collected upon first failure - especially the thread and image dumps - provides a snapshot of the system as or immediately after the problem occurs. The table below shows the type of WebLogic Server issues which fall into the scope of Diagnostic Framework

How to Configure Diagnostic Framework?

Depending on your Fusion Middleware product choice you may not need to do anything! Diagnostic Framework is automatically installed, configured and initiated for any WebLogic Domain which has the Oracle Java Required Files (JRF) template applied. This template is applied by default whenever you configure WebLogic Managed Servers for products such as

  • Portal / Forms / Reports / Discoverer
  • Identity Management ( OID , OAM , OIM etc)
  • WebCenter
  • SOA

Check your WebLogic Domain directory structure. If you have an "adr" sub directory under

DOMAIN_HOME/servers/<servername>/

then JRF template has been applied and Diagnostic Framework will be in play.

Should the "adr" sub directory not exist, review the advice given in My Oracle Support article

How to Apply FMW ( EM ) Control and JRF to a WebLogic Domain and Managed Servers [ID 947043.1]

If you are working with a standalone WebLogic Server solution and applying Oracle JRF is not acceptable, consider using WLDF - WebLogic Diagnostic Framework. (Fusion Middleware Diagnostic Framework makes use of WLDF under the covers.) Couple of useful links about WLDF are listed below

How to Get Started With Diagnostic Framework

To be frank, the Fusion Middleware Administrator's Guide is the best place to start your learning

Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
Chapter 13 Diagnosing Problems

A lot of reading here,  but if you are in hurry and just want to get the right information to Oracle Support to help resolve your issue, check out the next section below.

How to Upload Diagnostic Framework Incident Data to Oracle Support

Some Background Information

There are three interfaces to the Repository:

  1. Enterprise Manager Cloud Control (Support Workbench)
  2. WLST (Command Line)
  3. ADRCI (Command Line)

The Enterprise Manager Cloud Control does provide a nice GUI interface to search, view and package diagnostic framework incidents. However, this software is not to be confused with Fusion Middleware (EM) Control. Cloud Control (formerly known as Grid Control) is part of the Enterprise Manager media package. EM Cloud Control has it's own install and configuration story. Therefore, for the benefit of those yet to install and play with Cloud Control, I am going to describe how to use the command line tools.

Ideally, you would only need to one command line interface, but currently I suggest using both - mainly due to the fact that ADRCI SHOW INCIDENTS does not reveal the description behind the Diagnostic Framework error code.

Instructions:

Note:

WLST and ADRCI are case sensitive when it comes to handling parameter values. If you make a mistake, expect an unfriendly syntax error message.

1) Find the incident

Note:

The managed server which you are troubleshooting must be up and running. If the managed server is down, ensure the domain's Admin Server is accessible. If you cannot connect to the Admin Server or the Managed Server the example WLST commands will not work.

a) Launch WLST 

Note: Use the WLST which resides in the "oracle_common" directory (not WL_HOME/common/bin) otherwise you will get a syntax error like the one below

Traceback (innermost last):
  File "<console>", line 1, in ?
NameError: listIncidents

MW_HOME/oracle_common/common/bin/wlst.sh

b) Connect to the managed server or the admin server e.g.

wls:/offline> connect('weblogic','welcome1','t3://localhost:7020')

c) Run the command

wls:/mydomain/serverConfig> listIncidents()

This will list the incidents for the server to which you have connected. If you have connected to the Admin Server and want to list the incidents for a managed server within the domain, use the command

wls:/mydomain/serverConfig> listIncidents(adrHome='diag\ofm\mydomain\mymanagedserver'
,server='mymanagedserver')

Example output

Incident Id     Problem Key              Incident Time
        1       DFW-99998 [java.lang.NullPointerException]
[oracle.error.simulator.ErrorSimulator.createNullPointerException][errorWebApp_1-0-0-0]        
Fri Nov 02 10:38:46 GMT 2012

 The piece highlighted in bold is the description you do not see when using the ADRCI 'SHOW INCIDENT' command.

Make a note of the incident id. You are ready to move to step 2

2. Package the incident

a) Set up the environment - example commands below are for Unix

cd <DOMAIN_HOME>/bin
. ./setDomainEnv.sh

LD_LIBRARY_PATH=$WL_HOME/server/adr; export LD_LIBRARY_PATH

If you want ADRCI to run a Remote Diagnostic Agent collection (recommended) at generate package time, point ORACLE_HOME at oracle_common

ORACLE_HOME=$MW_HOME/oracle_common; export ORACLE_HOME

To prevent ADRCI from running RDA at generate package time, point ORACLE_HOME at WL_HOME/server/adr directory. 

ORACLE_HOME=$WL_HOME/server/adr; export ORACLE_HOME

b) Launch adrci

$WL_HOME/server/adr/adrci

c) Set BASE and HOMEPATH

adrci> SET BASE /oracle/middleware/user_projects/domains/
mydomain/servers/mymanagedserver/adr
adrci> SET HOMEPATH diag/ofm/mydomain/mymanagedserver

d)  Optionally run SHOW INCIDENTS e.g.

adrci> SHOW INCIDENTS -MODE DETAIL
ADR Home = /oracle/middleware/user_projects/domains/mydomain/
servers/mymanagedserver/adr/diag/ofm/mydomain/mymanagedserver:
*************************************************************************

**********************************************************
INCIDENT INFO RECORD 1
**********************************************************
   INCIDENT_ID                   1
   STATUS                        ready
   CREATE_TIME                   2012-11-02 10:38:46.468000 +00:00
   PROBLEM_ID                    1
   CLOSE_TIME                    <NULL>
   FLOOD_CONTROLLED              none
   ERROR_FACILITY                DFW
   ERROR_NUMBER                  99998
   ERROR_ARG1                    <NULL>
   ERROR_ARG2                    <NULL>
   ERROR_ARG3                    <NULL>
   ERROR_ARG4                    <NULL>
   ERROR_ARG5                    <NULL>
   ERROR_ARG6                    <NULL>
   ERROR_ARG7                    <NULL>
   ERROR_ARG8                    <NULL>
   ERROR_ARG9                    <NULL>
   ERROR_ARG10                   <NULL>
   ERROR_ARG11                   <NULL>
   ERROR_ARG12                   <NULL>
   SIGNALLING_COMPONENT          <NULL>
   SIGNALLING_SUBCOMPONENT       <NULL>
   SUSPECT_COMPONENT             <NULL>
   SUSPECT_SUBCOMPONENT          <NULL>
   ECID                          5162744c6a2eea5e:155ff445:13ac0aae7cb:-8000-000
0000000000325
   IMPACTS                       0
1 rows fetched

e)  Create a logical package

IPS CREATE PACKAGE INCIDENT incident_number

e.g.

adrci> IPS CREATE PACKAGE INCIDENT 1
Created package 1 based on incident id 1, correlation level typical

f) Generate the package

IPS GENERATE PACKAGE package_number IN path

e.g.

adrci> IPS GENERATE PACKAGE 1 IN /tmp
Generated package 1 in file /tmp/DFW99998j_20121102113633_COM_1.zip, mode complete

Note:

If the generate package command hangs, ADRCI may be experiencing an issue when running RDA. To avoid such trouble, exit ADRCI and point the ORACLE_HOME environment variable at WL_HOME/server/adr

3) Upload the package zip to Oracle Support via your Service Request

a) Log into My Oracle Support and locate your Service Request

b) Click on "Add Attachments


c) And upload the zip file


Wednesday Oct 31, 2012

How to Use RDA to Generate WLS Thread Dumps At Specified Intervals?

Introduction

There are many ways to generate a thread dump of a WebLogic Managed Server. For example, take a look at:

Taking Thread Dumps - [an excellent blog post on the Middleware Magic site]

or 

Different ways to take thread dumps in WebLogic Server (Document 1098691.1)

There is another method - use Remote Diagnostic Agent! The solution described below is not documented, but it is relatively straightforward to execute. One advantage of using RDA to collect the thread dumps is RDA will also collect configuration, log files, network, system, performance information at the same time.

Instructions

1. Not familiar with Remote Diagnostic Agent? Take a look at my previous blog "Resolve SRs Faster Using RDA - Find the Right Profile"

2. Choose a profile, which includes the WebLogic Server data collection modules (for example the profile "WebLogicServer"). At RDA setup time you should see the prompt below:

-------------------------------------------------------------------------------
S301WLS: Collects Oracle WebLogic Server Information
-------------------------------------------------------------------------------
Enter the location of the directory where the domains to analyze are located
(For example in UNIX, <BEA Home>/user_projects/domains or <Middleware
Home>/user_projects/domains)
Hit 'Return' to accept the default
(/oracle/11AS/Middleware/user_projects/domains)
> 

For a successful WLS connection, ensure that the domain Admin Server is up and
running.

Data Collection Type:
  1  Collect for a single server (offline mode)
  2  Collect for a single server (using WLS connection)
  3  Collect for multiple servers (using WLS connection)

Enter the item number
Hit 'Return' to accept the default (1)
> 2

Choose option 2 or 3.

Note: Collect for a single server or multiple servers using WLS connection means that RDA will attempt to connect to execute online WLST commands against the targeted server(s). The thread dumps are collected using the WLST function - "threadDumps()". If WLST cannot connect to the managed server, RDA will proceed to collect other data and ignore the request to collect thread dumps. If in the final output you see no Thread Dump menu item, then it's likely that the managed server is in a state which prevents new connections to it. If faced with this scenario, you would have to employ alternative methods for collecting thread dumps.

3. The RDA setup will create a setup.cfg file in the RDA_HOME directory. Open this file in an editor. You will find the following parameters which govern the number of thread dumps and thread dump interval.

#N.Number of thread dumps to capture
WREQ_THREAD_DUMP=10
#N.Thread dump interval
WREQ_THREAD_DUMP_INTERVAL=5000

The example lines above show the default settings. In other words, RDA will collect 10 thread dumps at 5000 millisecond (5 second) intervals. You may want to change this to something like:

#N.Number of thread dumps to capture
WREQ_THREAD_DUMP=10
#N.Thread dump interval
WREQ_THREAD_DUMP_INTERVAL=30000

However, bear in mind, that such change will increase the total amount of time it takes for RDA to complete its run.

4. Once you are happy with the setup.cfg, run RDA. RDA will collect, render, generate and package all files in the output directory.

5. For ease of viewing, open up the RDA Start html file - "xxxx__start.htm". The thread dumps can be found under the WLST Collections for the target managed server(s). See screenshots below

Screenshot 1:RDA Start Page - Main Index

Screenshot of RDA Start Page

Screenshot 2: Managed Server Sub Index

Screenshot of Managed Server Sub Index

Screenshot 3: WLST Collections

Screenshot of WLST Collections - pointing out Thread Dump menu option

Screenshot 4: Thread Dump Page - List of dump file links

Screenshot of Thread Dump Page - list of dump file links

Screenshot 5: Thread Dump Dat File Link

Screenshot of what you see when you click a dump file link

Additional Comment:

A) You can view the thread dump files within the RDA Start Page framework, but most likely you will want to download the dat files for in-depth analysis via thread dump analysis tools such as:

If you are new to thread dump analysis - take a look at this recorded Support Advisor Webcast 

Oracle WebLogic Server: Diagnosing Performance Issues through Java Thread Dumps
[Slidedeck from webcast in PDF format]

B) I have logged a couple of enhancement requests for the RDA Development Team to consider:

  1. Add timestamp to dump file links, dat filename and at the top of the body of the dat file
  2. Package the individual thread dumps in a zip so all dump files can be conveniently downloaded in one go.

Wednesday Oct 17, 2012

Critical Patch Update For Oracle Fusion Middleware - CPU October 2012

The latest Critical Patch Update (CPU) has been released for Oracle products. Start your reading here

Critical Patch Updates, Security Alerts and Third Party Bulletin 

This is the home page containing links to all "Critical Patch Updates" released to date, along with sections detailing 

  • Security Alerts 
  • Third Party Bulletin
  • Public Vulnerabilities Fixed
  • Policies
  • Reporting Security Vulnerabilities 

On this page you will find the link to the

Oracle Critical Patch Update Advisory - October 2012

The advisory lists the support documents that cover the patch availability for all Oracle products.

From an Oracle Fusion Middleware perspective, you can cut to the chase by using the links below which take you to the appropriate sections in

Patch Set Update and Critical Patch Update October 2012 Availability Document [ID 1477727.1]

  1. Oracle Fusion Middleware 11g Release 2 
  2. Oracle Fusion Middleware 11g Release 1
  3. Oracle Application Server 10g Release 3
The #anchor links above should work in Firefox and IE provided you have already logged into My Oracle Support within the same browser session. For some reason, Chrome always takes you to the top of the document :-/

Tip:

Error Correction Support for Oracle Identity Management 10g, version 10.1.4.x ended in December 2011. For this reason, there is no section which is specific to this version. However, Error Correction Support remains in place, until end of this year, for the Oracle Identity Management 10.1.4.x components

    • Single Sign On (SSO)
    • Delegated Administration Services (OIDDAS)

provided you are using them as part of a Single Sign-On solution (OID 11g + SSO / OIDDAS 10.1.4.3) for a Portal / Forms / Reports and Discoverer 11.1.1.x architecture.   

As such there are security related patches available for Fusion Middleware Single Sign On. You will find the patch numbers listed in the sections for 11.1.1.4, 11.1.1.5 and 11.1.1.6

And finally, if you are hit any unexpected errors when applying the CPU patches, check out the known issues documented in these two support documents.

Critical Patch Update October 2012 Oracle Fusion Middleware Known Issues (Doc ID 1455408.1)

Critical Patch Update October 2012 Database Known Issues (Doc ID 1477865.1)

Monday Sep 10, 2012

Resolve SRs Faster Using RDA - Find the Right Profile

Introduction

Remote Diagnostic Agent (RDA) is an excellent command-line data collection tool that can aid troubleshooting / problem solving. The tool covers the majority of Oracle's vast product range, and its data collection capability is comprehensive. RDA collects data about

  • the operating system and environment, including
    • environment variable, kernel settings
    • network
    • o/s performance
    • o/s patches
    • and much more
  • the Oracle Products installed, including
    • patches
    • logs and debug
    • metrics
    • configuration
    • and much more

In effect, RDA can obtain a snapshot of an Oracle Product and its environment. Oracle Support encourages the use of RDA because it greatly reduces service request resolution time by minimizing the number of requests from Oracle Support for more information. RDA is designed to be as unobtrusive as possible; it does not modify systems in any way. It collects useful data for Oracle Support only and a security filter is provided if required.

RDA can be downloaded via this My Oracle Support document

Remote Diagnostic Agent (RDA) 4 - Getting Started [ID 314422.1]

Find and Use the Right RDA Profile

One problem of any tool / utility, which covers a large range of products, is knowing how to target it against only the products you wish to troubleshoot. RDA does not have a GUI. Nor does RDA have an intelligent mechanism for detecting and automatically collecting data only for those Oracle products installed. Instead, you have to tell RDA what to do.

There is a mind boggling large number of RDA data collection modules which you can configure RDA to use. It is easier, however, to setup RDA to use a "Profile". A profile consists of a list of data collection modules and predefined settings. As such profiles can be used to diagnose a problem with a particular product or combination of products.

How to run RDA with a profile?

( <rda> represents the command you selected to run RDA (for example, rda.pl, rda.cmd, rda.sh, and perl rda.pl).)

1. Use the embedded spreadsheet to find the RDA profile which is appropriate for your problem / chosen Oracle Fusion Middleware products.

2. Use the following command to perform the setup

<rda> -S -p <profile_name> 

3. Run the data collection

<rda>

Run the data collection. If you want to perform setup and run in one go, then use a command such as the following:

<rda> -vnSCRP -p <profile name>

For more information, refer to:

Remote Diagnostic Agent (RDA) 4 - Profile Manual Pages [ID 391983.1]

Additional Hints / Tips:

1. Be careful! Profile names are case sensitive.

2. When profiles are not used, RDA considers all existing modules by default. For example, if you have downloaded RDA for the first time and run the command

<rda> -S

you will see prompts for every RDA collection module many of which will be of no interest to you. Also, you may, in your haste to work through all the questions, forget to say "Yes" to the collection of data that is pertinent to your particular problem or product. Profiles avoid such tedium and help ensure the right data is collected at the first time of asking.

3. RDA has a security filter which removes potentially sensitive data such as

  • Domain names
  • Group names
  • Host names
  • IPv4 and IPv6 addresses
  • LDAP domain components
  • Network masks
  • User names

You can switch on at RDA setup by appending "-Security" to the profile value

<rda> -vnSCRP -p <profile name>-Security

e.g.

<rda> -vnSCRP -p FM11g_WlsWebTier-Security

If you already have a setup.cfg, you can enable the security filter by running the command

<rda> -X Filter enable 

Monday Aug 13, 2012

Find Oracle FMw 11g Other Certification Information - Client/Browser, 3rd Party, Product Interoperability

Here is the fourth and final viewlet in a series of viewlets which discuss how to find Oracle Fusion Middleware certification information. The viewlet below explores the process of searching for Oracle Fusion Middleware 11g Client, 3rd Party and Interoperability certification information. The objective is to help you find the answers to questions such as

  • Which versions of WebLogic Server are compatible with the latest FMw 11g patchset?
  • Which browsers are certified e.g. Is Forms / Reports 11g certified to work with IE 9?
  • Is it supported to integrate FMW 11g products with Websphere?
  • Does FMW 11g support IPv6?

[Note - the viewlets have no sound and use the swf (FlashPlayer) format. It will take approximately 10 to 15 minutes to work through the show.

If you are unable to use Flash Player, click here to grab a PDF version of the presentation.]

If you have not done so already, check out the previous viewlets in the series

Friday Aug 03, 2012

How to Find Oracle Application Server 10g Certification Information?

Here is the third in a series of viewlets which discuss how to find Oracle Fusion Middleware certification information. The viewlet below covers Oracle Application Server 10g - with hints / tips relevant to searching for certification information related to versions

  • Oracle Application Server 10g Release 3 (10.1.3) - including SOA Suite
  • Oracle Application Server 10g Release 2 (10.1.2)
  • Oracle Identity Management 10g Release 3 (10.1.4)

[Note - the viewlets have no sound and use the swf (FlashPlayer) format. It will take approximately 10 to 15 minutes to work through the show.

If you are unable to use Flash Player, click here to grab a PDF version of the presentation.]

If you have not done so already, check out the other viewlets in the series

Friday Jul 27, 2012

How to Find Oracle FMw Certification Information? - An Introduction

Every month we receive a significant number of Oracle Fusion Middleware certification enquiries. The answers to most questions are in the public domain, but they are not always easy to find. Below we present the first in a series of viewlets which provides hints / tips for finding answers to questions such as

  • Where is the Oracle Fusion Middleware certification information located e.g versions 10g and 11g ?
  • Is the platform / os I am targeting for my install certified?
  • Which versions of the Oracle Database can I use with my Fusion Middleware choice of products?
  • Which JDK version can I use?
  • Where can I find Oracle or 3rd Party product interoperability / compatibility information?
  • Which versions of the browser does Fusion Middleware support?

To get the ball rolling we start with a short introduction. Further viewlets in this series are in pipeline - watch this space!

[Note - the viewlets have no sound and use the swf (FlashPlayer) format. It should take no more than 5 to 10 minutes to work through the show.

If you are unable to use Flash Player, click here to grab a PDF version of the presentation]

If you have not done so already, check out the other viewlets in the series

Wednesday Jul 18, 2012

Critical Patch Update For Oracle Fusion Middleware - CPU July 2012

If you have not noticed already, the latest Critical Patch Update has been released for Oracle Products. You should start your reading here

Critical Patch Updates, Security Alerts and Third Party Bulletin 

This is the home page containing links to all the Critical Patch Updates released along with sections detailing 

  • Security Alerts 
  • Third Party Bulletin
  • Public Vulnerabilities Fixed
  • Policies
  • Reporting Security Vulnerabilities 

On this page you will find the link to the

Oracle Critical Patch Update Advisory - July 2012

The advisory lists the support documents that cover the patch availability for all Oracle Products.

From a Oracle Fusion Middleware perspective, you can cut to the chase by using the links below which take you to the appropriate sections in

Patch Set Update and Critical Patch Update July 2012 Availability Document [ID 1455387.1]

  1. Oracle Fusion Middleware 11g Release 2 
  2. Oracle Fusion Middleware 11g Release 1
  3. Oracle Application Server 10g Release 3
The #anchor links above should work in Firefox and IE provided you have already logged into My Oracle Support within the same browser session. For some reason, Chrome always takes you to the top of the document :-/

Tip:

Error Correction Support for Oracle Identity Management 10g, version 10.1.4.x ended in December 2011. For this reason, there is no section which is specific to this version. However, Error Correction Support remains in place, until end of this year, for the Oracle Identity Management 10.1.4.x components

    • Single Sign On (SSO)
    • Delegated Administration Services (OIDDAS)

provided you are using them as part of a Single Sign-On solution (OID 11g + SSO / OIDDAS 10.1.4.3) for a Portal / Forms / Reports and Discoverer 11.1.1.x architecture.   

As such there are two security related patches available for SSO / OIDDAS 10.1.4 - one related to OC4J, released in  January 2012, and another related to Oracle HTTP Server, released in October 2011. You may have applied these patches already .. but it is worthwhile double checking. You will find the patch numbers listed in the sections for 11.1.1.5, 11.1.1.6 

And finally, if you are hit any unexpected errors when applying the CPU patches, check out the known issues documented in these two support documents.

Critical Patch Update July 2012 Oracle Fusion Middleware Known Issues (Doc ID 1455408.1)

Critical Patch Update July 2012 Database Known Issues (Doc ID 1452634.1)

Monday Jun 11, 2012

New Information Center - Reviewing Security For FMW 11g

Announcing ...

Information Center: Reviewing Security For Oracle Fusion Middleware 11g [ID 1458051.2] has been published. 

Screenshot of ID 1458051.2

Screenshot of FMW 11g Security IC

What is an Information Center?

Information Centers use widgets to aggregate knowledge content, such as support documents, product documentation, support community threads, which is pertinent to a given task or intent. Widgets either contain static lists or better still some widgets are dynamic. A dynamic widget uses a query criteria to present a list of support documents relevant to the title / subject matter of the widget. The content of a dynamic widget is refreshed automatically every 24 hours.

Once you are in an Information Center, you can use the left hand menu to navigate to other Tasks / Intent Information Centers (e.g "Install and Configure", "Patch", "Troubleshoot", "Upgrade" which are available for the chosen product.

Are Information Centers easy to find?

You can go straight to the new "Reviewing Security" Information Center by using the hyperlink given above. There are, however, two other methods which make Information Centers easier to find.

  • Browse Knowledge
  • Refine Your Search

Browse Knowledge

The "Browse Knowledge" is currently found in the "Knowledge" Tab Page in My Oracle Support. As illustrated by the screenshots below, you can find Information Centers by choosing a product (e.g "Oracle Fusion Middleware"), a version and an action / intent. If an Information Center exists for your selection the "Advisor Found" button is enabled. Clicking on this button will take you straight to the desired Information Center.

Screenshot - Browse Knowledge 1

Using Browse Knowledge - Screenshot 1

Screenshot - Browse Knowledge 2

Using Browse Knowledge - Screenshot 2

Screenshot - Browse Knowledge 3

Using Browse Knowledge - Screenshot 3

Refine Your Search

Refine your search is a dialogue which is triggered by certain keywords that you may enter into the Global Search field in the top right hand corner of My Oracle Support. The "Refine Your Search" works in a similar manner to "Browse Knowledge". Choose your product and version. The appropriate Task / Intent should already be selected for you. Thereafter, click the Go button.

Screenshot - Refine Your Search 1

Using Refine Your Search - Screenshot 1

Screenshot - Refine Your Search 2

Using Refine Your Search - Screenshot 2

Screenshot - Refine Your Search 3

Using Refine Your Search - Screenshot 3


Monday May 21, 2012

Which JDK is my FMW 11g WebLogic Domain Configured to Use?

Introduction

If you are the administrator who ran the Fusion Middleware Installer or Configuration Wizard, the question in the title may seem a little daft. But, in fact the advice below may become handy should you face one of the following situations:

  • You have been asked to look after a Middleware Home which was previously built by someone else
  • You are configuring a new FMW 11g AS Instance, and unexpectedly hitting errors or hanging at the WebLogic Domain Creation time. For example, see:

    FMW 11g "IDM" or "Portal/Forms/Reports/Discoverer" Configuration Wizards on 64bit Platforms Hang at 0% 'Creating Domain'  [ID 865462.1]

    This support document describes what happens if you inadvertently attempt to configure an AS Instance with a mixture of 64bit (Oracle Home Binaries - ORACLE_HOME) and 32bit JDK (WebLogic Binary Home - WL_HOME)

There are two parts to the question:

  1. What version of JDK will the Configuration Wizard, by default, configure a new WebLogic Domain to use?
  2. Which JDK and version is being used by an existing WebLogic Domain?

JDK Used at WebLogic Domain Creation Time

The answer is in the default WebLogic environment script:

Unix

WL_HOME/common/bin/commEnv.sh

MS Windows

WL_HOME/common/bin/commEnv.cmd

(e.g WL_HOME is typically - MW_HOME/wlserver_10.3/common/bin/commEnv.sh)

Look for lines like the ones below (examples are from a WLS install on a Unix machine)

#JAVA_USE_64BIT, true if JVM uses 64 bit operations
JAVA_USE_64BIT=true
if [ -z "${JAVA_HOME}" -o -z "${JAVA_VENDOR}" ]; <---- This line means if 
JAVA_HOME OR JAVA_VENDOR is NOT set, so if either of them are empty 
the settings in the "then" part of the statement are applied. 
If JAVA_VENDOR and JAVA_HOME are both set then the if statement is skipped
then
      # Set up JAVA HOME
      JAVA_HOME="/oracle/SOAWC/jdk1.6.0_24"
      # Set up JAVA VENDOR, possible values are
      #Oracle, HP, IBM, Sun ...
      JAVA_VENDOR=Sun
      # PRODUCTION_MODE, default to the development mode
      PRODUCTION_MODE=""
fi

Important to Note: It is possible to set JAVA_HOME, JAVA_VENDOR environment variables prior to launching the Installer or Configuration Wizard. Therefore you cannot always assume that commEnv.sh is the source of these environment variable values. In the same session from which you launch the Installer or Configuration Wizard double check whether JAVA_HOME and JAVA_VENDOR have values e.g.

Unix:

echo $JAVA_HOME
echo $JAVA_VENDOR

MS Windows:

set $JAVA_HOME
set $JAVA_VENDOR 

If either of these environment variables are not set then you know that the values will be derived from the commEnv.sh.

The JAVA_HOME and JAVA_VENDOR values do not tell you whether the JDK is 32bit or 64bit. To find out this detail you need to run a switch to the bin directory where the JDK resides e.g

cd /oracle/SOAWC/jdk1.6.0_24/bin 

and then run the java version command. To quote from:

How to tell if Weblogic Server is running in 32-bit or 64-bit mode [ID 1066808.1]

"To identify if you have a 32-bit or 64-bit JDK installed you can run"

java -version

which will give you more information.

By running

java -d64 -version

you can test if the 64-bit mode is supported by your JVM. You will get an error message if this is a 32-bit only JVM. 32/64-bit-hybrid JVMs exist for some platforms. In this case you would have to include the -d64 flag to tell the JVM to run in 64-bit mode."

Which JDK is my existing WebLogic Domain using?

Typically, the JDK used by a WebLogic Domain will be the same as specified by commEnv.sh. However, you must check the domain's environment variable setting file:

Unix:

DOMAIN_HOME/bin/setDomainEnv.sh

MS Windows:

DOMAIN_HOME/bin/setDomainEnv.cmd

(e.g A DOMAIN_HOME might look something like MW_HOME/user_projects/domains/myDomain/)

Couple of examples (taken from installs on a Unix machine)

A Portal / Forms / Reports and/or Discoverer WebLogic Domain; the setDomainEnv has these lines

BEA_JAVA_HOME="/oracle/app/product/oracle/FMW11gR1PS1/jrockit_160_14_R27.6.5-32"
export BEA_JAVA_HOME

SUN_JAVA_HOME="/oracle/app/product/oracle/FMW11gR1PS1/jdk160_14_R27.6.5-32"
export SUN_JAVA_HOME

    if [ "${JAVA_VENDOR}" = "Oracle" ] ; then
        JAVA_HOME="${BEA_JAVA_HOME}"
        export JAVA_HOME
    else
    if [ "${JAVA_VENDOR}" = "Sun" ] ; then
        JAVA_HOME="${SUN_JAVA_HOME}"
        export JAVA_HOME
    else
        JAVA_VENDOR="Oracle"
        export JAVA_VENDOR
        JAVA_HOME="/oracle/app/product/oracle/FMW11gR1PS1/jrockit_160_14_R27.6.5-32"
        export JAVA_HOME
    fi
    fi

A FMW 11g SOA Suite, the setDomainEnv has these lines

    if [ "${JAVA_VENDOR}" = "Oracle" ] ; then
            JAVA_HOME="${BEA_JAVA_HOME}"
            export JAVA_HOME
    else
            if [ "${JAVA_VENDOR}" = "Sun" ] ; then
                    JAVA_HOME="${SUN_JAVA_HOME}"
                    export JAVA_HOME
            else
                    JAVA_VENDOR="Sun"
                    export JAVA_VENDOR
                    JAVA_HOME="/oracle/SOAWC/jdk1.6.0_24"
                    export JAVA_HOME
            fi
    fi

The JAVA_VENDOR environment variable is not normally set prior to running setDomainEnv.sh, in which case the JAVA_VENDOR will default to the "else" part of the statement (highlighted in Red)

The commEnv (.sh or .cmd) is invoked later on in the setDomainEnv script. In the commEnv script the if statement

if [ -z "${JAVA_HOME}" -o -z "${JAVA_VENDOR}" ]; 

is bypassed because both JAVA_VENDOR and JAVA_HOME have been set.

Again, the setDomainEnv.sh does not tell you whether the specified JAVA_HOME is 32bit or 64bit. The only way to find this information is to run the "java -version" as previously described.

Additional Useful Reference:

How To Change Type of JDK (Sun / JRockit) for FMW 11g Domain (ID 1058804.1)

Note - however I have added a comment to this support document as I believe the advice does not take into account the possibility of JAVA_VENDOR and JAVA_HOME being set before commEnv is launched.

Friday Apr 27, 2012

Getting Proactive with Oracle Health Checks

The My Oracle Support support document describing the Oracle Health Checks catalogue has been given a make-over. Complements to those responsible for the changes because they have done rather a nice job. The document now allows you to view relevant Video, support documents and communities inline by exploring the drop down lists.

Screenshot

Screenshot of Support Document 868955.1

It is a worth a visit and bookmark, go to:

Get Proactive - Oracle Health Checks [ID 868955.1]

As the material in this document indicates, you must install and configure Oracle Configuration Manager (OCM) to leverage the Health Check functionality in My Oracle Support. If you have not installed and configured OCM against an Oracle Fusion Middleware 10g or 11g home, be aware there is an initial learning curve. Set aside some time to read / work through the following support document:

  • Using OCM (Oracle Configuration Manager) With Core Oracle Fusion Middleware Components [ID 1060723.1]

If you are pushed for time, try using one of these "guides"

  • Configuring OCM in Oracle Fusion Middleware 11g? A Quick and Easy Guide [ID 1096871.1]
  • Configuring OCM in Oracle Application Server 10g? A Quick and Easy Guide [ID 1096780.1]

One final remark. While information collected by OCM can be useful to Oracle Support Engineers working Oracle Fusion Middleware Service Requests, the ultimate tool for data collection to aid troubleshooting is Remote Diagnostic Agent (RDA). Therefore, it is not uncommon for Support Engineers to request RDA outputs to be uploaded via the Service Request interface to expedite SR resolution, even if OCM collected data is also exposed in the SR.

This is not to say that one should be used instead of the other, OCM and RDA service different needs and purpose. OCM allows you to leverage proactive functionality in My Oracle Support such as Health Checks, Patch Recommendations and Plans. Whereas, RDA is a "just in time" troubleshooting / problem solving tool. RDA data which is collected and uploaded via a SR is deleted from the Repository at Oracle after SR closure. Therefore RDA data cannot be leveraged by the proactive functionality in My Oracle Support.

For more information about RDA take a look at:

  • Running RDA Against Oracle Fusion Middleware 11g [ID 853437.1]
  • Running RDA and Health Checks for Oracle Application Server 10g Environments [ID 413558.1]

About

This is the blog of the Oracle Fusion Middleware Proactive Support Delivery Team. Here we will provide information about our activities, publications, product related information and more. Feedback welcome.

Follow OracleMWSupport on Twitter

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today