Monday Oct 07, 2013

RDA 8.01 - Now A Better Experience for WebLogic Administrators

What is RDA?

Not familiar with RDA (Remote Diagnostic Agent)? Well I won't use this blog entry to provide a lesson, but if you want to learn more, take a look at these two support articles:

  • Get Proactive with Fusion Middleware : Resolve Problems Faster! Use Remote Diagnostic Agent [ID 1498376.1]
  • How to Run Remote Diagnostic Agent ( RDA ) Against Oracle WebLogic Server ( WLS ) [ID 1262157.1]

Too many questions in set up

"I have to answer too many questions at set up time!". The first of two criticisms leveled at a RDA collection which targeted a WebLogic Server based technology stack. Well I pleased to say that the number of questions of has been reduced.

If, for example, you set up RDA using the "WebLogicServer" profile, you will encounter a set up dialogue similar to the one copied / pasted from a Weblogic 12c solution, on Unix, below.

Tip: Work through the set up questions a little quicker by configuring your environment variables to point at your Middleware Home and location of your WebLogic domain prior to running RDA set up. Doing so, allows RDA to find default values for certain questions, which means you can just hit return and move on.

# Setting the environment

. /home/oracle/app/FMW12c_1212/config/user_projects/domains/my_domain/bin/setDomainEnv.sh
ORACLE_HOME=/home/oracle/app/FMW12c_1212/oh; export ORACLE_HOME
# Running RDA set up 
cd /home/oracle/software/rda
./rda.sh -S -p WebLogicServer
 
# Set up dialogue
Enter the network domain name for this server
> uk.acme.com

Enter the Oracle home to be used for data analysis
Hit "Return" to accept the default (/home/oracle/app/FMW12c_1212/oh)
> 
Enter the location of the directory where the domains to analyze are located
(For example in UNIX, <BEA Home>/user_projects/domains or 
<Middleware Home>/user_projects/domains)
Hit "Return" to accept the default
(/home/oracle/app/FMW12c_1212/config/user_projects/domains)
> 
For a successful WLS connection, ensure that the domain Admin Server is up 
and running.

Data Collection Type:
  1  Perform offline collection
  2  Perform online collection (using WLS connection)

Enter the item number
Hit "Return" to accept the default (2)
> 

Select a Oracle WebLogic Server domain home target list:
  1  Domain my_domain (in
     /home/oracle/app/FMW12c_1212/config/user_projects/domains)

  A  All Oracle WebLogic Server domain home targets
  D  Define a new Oracle WebLogic Server domain home target
  E  Edit an Oracle WebLogic Server domain home target
  N  No Oracle WebLogic Server domain home targets
 
Enter your selection, as a comma-separated list of item numbers or ranges
Hit "Return" to accept the default (1)
> 
Server list for 'my_domain' domain:
  1  AdminServer
  2  ms3
Enter your selection, as a comma-separated list of item numbers or ranges
Hit "Return" to accept the default (1-2)
> 
Are custom scripts used to start Oracle WebLogic servers (Y/N)?
Hit "Return" to accept the default (N)
>

# The next prompt concerns OCM. You are not required to configure OCM. 
# If time is short, just answer 'N'

RDA benefits from Oracle Configuration Manager by improving its 
data collection. Oracle Configuration Manager (OCM) collects the 
configuration information and uploads it to the OCM Repository 
in connected mode. When OCM is configured in disconnected mode, it 
adds the OCM payload to the RDA report packages in future RDA runs.

Pending Configurations in /home/oracle/app/FMW12c_1212/oh:
  MH_T0001  Middleware home /home/oracle/app/FMW12c_1212/oh

Hit 'Return' to continue

Do you want RDA to configure OCM now (Y/N)?
Hit "Return" to accept the default (Y)
> N

# After answering the prompts set up will be complete. 
# Now just launch the command line script (no parameters) 
# to run the collection

./rda.sh 

"Cannot see the wood for the trees" in the output

This is the second of the criticisms. RDA does collects many files and metrics - this is one its strengths. Yet I think it is fair to say that the presentation of WebLogic Server data in the RDA Start Menu (generated as part of the output) lacked some order and structure. For example, server log files were somewhat buried, and similarly, unless you knew to scroll down to WLST collections, you could not easily ascertain the basic contents of a targeted Weblogic domain. In RDA 8.01, the sub menus for the WebLogic Domain and Servers have been re-ordered to address this concern. Review the screenshots below.

Figure 1 : WebLogic Domain Sub Menu


Note: In the Main Index, RDA will only list managed servers which have been targeted for collection. In this example, only AdminServer and ms3 were targeted for collection. Yet we can see from the Domain MBeans that the domain contains other managed servers and a cluster. 

Figure 2 : MBean Drill Down Example - Admin Server State

Figure 3: WebLogic Server Sub Menu


To take advantage of these changes download the latest version of RDA. The download links can be found in

Get Proactive with Fusion Middleware : Resolve Problems Faster! Use Remote Diagnostic Agent [ID 1498376.1]

Monday Sep 16, 2013

OOW 2013 - Sessions Focused on Support Services and Oracle Fusion Middleware


Only a few days to go and Oracle OpenWorld 2013 will be upon us! Oracle Support and Services will be at Oracle OpenWorld in full force this year with 500 Services experts, 40+ sessions, a Support Stars Bar, a My Oracle Support Community Meet up, and demos of powerful new support tools. Learn best practices for maintaining Oracle Fusion Middleware, how to utilize Oracle Fusion Middleware products in a private cloud environment, tips for running Oracle Fusion Middleware in critical environments, and more. Go here a listing of Fusion Middleware specific sessions - for example:

Conference Sessions

Best Practices for a Private Cloud with Oracle Fusion Middleware

Running Oracle Fusion Middleware in Critical Environments: Tips from SWIFT

Best Practices for Maintaining Oracle Fusion Middleware

Demo

Migration Tools and Resources for Oracle Fusion Middleware

And more

There is also a My Oracle Support Meet Up Event" on Monday, Sep 23 and do come visit the Support Stars Bar - Monday 23rd through to Wednesday 25th

To all going, have an enjoyable OpenWorld 2013!

Friday Sep 06, 2013

Read all about it! Middleware Support News

The September 2013 edition of Oracle Fusion Middleware Support News is now available - featuring:

  • Introducing Fusion Middleware 12c (12.1.2) Infrastructure
  • New to SOA 11g Adapters? Here's an overview
  • WebCenter 11.1.1.8 - Center of Mobile Engagement for Business
  • Get an Identity Management education in bitesize chunks!
  • + all the latest proactive patch and patch set release news

Since the last time I blogged about the support news (June 2012), this My Oracle Support knowledge article has undergone a face lift. As well as looking a little more groovy, you can collapse / expand news articles to make for easier reading. Here's a screenshot.


Find the Support News by navigating to the document below:

Oracle Fusion Middleware Support News : Current Edition - Volume 9 : September 2013 (Document 1347075.1)

Enjoy! 

Thursday Jul 18, 2013

Critical Patch Update for Oracle Fusion Middleware - CPU July 2013

The latest Critical Patch Update (CPU) has been released for Oracle products. Start your reading here

Critical Patch Updates, Security Alerts and Third Party Bulletin 

This is the home page containing links to all "Critical Patch Updates" released to date, along with sections detailing 

  • Security Alerts 
  • Third Party Bulletin
  • Public Vulnerabilities Fixed
  • Policies
  • Reporting Security Vulnerabilities

On this page you will find the link to the

Oracle Critical Patch Update Advisory - July 2013

The advisory lists the support documents that cover the patch availability for all Oracle products.

For Oracle Fusion Middleware, go to:

Patch Set Update and Critical Patch Update July 2013 Availability Document [ID 1548709.1]

Highlights - WebLogic Server Patch Set Updates (PSU)

  • WebLogic Server - 12.1.1.0.5
  • WebLogic Server - 10.3.6.0.5

July 2013 marks the end of CPU/PSU patches for Weblogic Server 10.3.5. Looking forward, we will see the final CPU patches for Weblogic Server 9.2.3.0 and 9.2.4.0 in October 2013

If you are hit any unexpected errors when applying the CPU patches, check out the known issues documented in these two support documents.

  • Critical Patch Update July 2013 Oracle Fusion Middleware Known Issues  [ID 1548690.1]
  • Critical Patch Update July 2013 Database Known Issues [ID 1546428.1]

And lastly, for an informal summary of what the Critical Patch Update fixes, check out the blog posts by "Oracle Software Security Assurance" team

July 2013 Critical Patch Update Released

Wednesday Apr 17, 2013

Critical Patch Update for Oracle Fusion Middleware - CPU April 2013

The latest Critical Patch Update (CPU) has been released for Oracle products. Start your reading here

Critical Patch Updates, Security Alerts and Third Party Bulletin 

This is the home page containing links to all "Critical Patch Updates" released to date, along with sections detailing 

  • Security Alerts 
  • Third Party Bulletin
  • Public Vulnerabilities Fixed
  • Policies
  • Reporting Security Vulnerabilities 

On this page you will find the link to the

Oracle Critical Patch Update Advisory - April 2013

The advisory lists the support documents that cover the patch availability for all Oracle products.

For Oracle Fusion Middleware, go to:

Patch Set Update and Critical Patch Update April 2013 Availability Document [ID 1525152.1]

If you are hit any unexpected errors when applying the CPU patches, check out the known issues documented in these two support documents.

  • Critical Patch Update April 2013 Oracle Fusion Middleware Known Issues  [ID 1525130.1]
  • Critical Patch Update April 2013 Database Known Issues [ID 1523844.1]
And lastly, for an informal summary of what the Critical Patch Update fixes, check out the blog posts by "Oracle Software Security Assurance" team

April 2013 Critical Patch Update Released

Wednesday Apr 03, 2013

One Stop Shop For Your Middleware Product Certification Questions

Introduction

A certification is a combination of Oracle and third-party products and operating systems that Oracle has tested and should work together.

In more detail, product certifications cover:

  • Platform/OS
  • Client, Browser and Java (JRE)
  • Database, Java (JDK)
  • Web Server Plug-Ins
  • Third Party Adapters and Interoperability
  • Network - IPv6
  • Oracle Virtual Assembly Builder

This month we are excited to announce two initiatives designed to help you quickly find the answers to your product certification questions:

"Get Proactive" My Oracle Support Knowledge Article

Review and bookmark this knowledge article

Get Proactive with Fusion Middleware : Find Product Certifications [ID 1532687.1]

The article contains:

  • Viewlets which provides hints and tips for finding product certification data
  • Product Breakdown to drill down to the location of product specific certifications.
  • FAQs
  • Information and links to related resources
Screenshot 1: ID 1532687.1

Screenshot of the new Get Proactive Certification Note

And a New Support Community!

Still have questions / issues. No need to create a Service Request. Instead post your Middleware product certification question to the Middleware Certification support community! A dedicated team of Oracle Support Engineers are monitoring the community and will respond to your enquiry in timely manner.

Screenshot 2: Middleware Certification Support Community

Snapshot of Cert Community

Note: A valid My Oracle Support license / login is required to access the knowledge article and support community.

Monday Feb 04, 2013

Fusion Middleware Performance Problems? Take Advantage of DMS

Introduction

Why is my application running slowly?
How can I access configuration data and related metrics?
I need to a tool to help me tune my Fusion Middleware solution?

If you are asking these types of questions then take some time out to find out more about DMS. 

What is DMS?

Yes, another acronym .. sorry ... translation : "Dynamic Monitoring Service." It is a service which enables Oracle Fusion Middleware components to provide data regarding the component's performance, state and ongoing behavior. DMS measures and reports metrics, trace events and system performance and provides a context correlation service for these components.

Benefits include

  • Comprehensive set of performance metrics
  • Multiple tools for easier viewing and analysis of the metrics - including:
    • Fusion Middleware (EM) Control
    • DMS Spy Servlet
    • WLST
    • Remote Diagnostic Agent
    • Diagnostic Framework
  • Event Tracing
    • configure live tracing with no restarts
    • trace issues within a given time period or collect specific data at a specific time for a specific set of criteria

Oracle Fusion Middleware Performance and Tuning Guide 11g
Chapter 6 Oracle Dynamic Monitoring Service Performance Tuning

is the place to find out more.

Important to Note:

The majority of Oracle Fusion Middleware 11g WebLogic based products incorporate DMS in their product code. As such there is no special set up required. DMS works out of the box. For WebLogic based products Diagnostic Framework is provided via JRF (Java Required Files). If a WebLogic Managed Server does not have JRF applied, DFW will NOT be available. Domains created with standalone WebLogic Server will not, by default, have JRF applied. Refer to:

Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
19 Scaling Your Environment
19.3.1 Applying Oracle JRF Template to a Managed Server or Cluster

If you have a login to My Oracle Support, also check out the knowledge article:

How to Apply FMW ( EM ) Control and JRF to a WebLogic Domain and Managed Servers [ID 947043.1]

DMS and Fusion Middleware (EM) Control

As stated above there are many ways to integrate with DMS. In this blog post, I am going to focus on Fusion Middleware (EM) Control as I think it tackles the task rather well. In Fusion Middleware Control, the DMS metrics for a component (e.g. a WebLogic managed server) are accessed via the Performance Summary option.

Screenshot : Performance Summary Menu Option

The Performance Summary will show a default selection of metrics for the component - some displayed in chart form, others listed in tables. However, within a matter of minutes you can change the metric selection and configure a metric view which suits your purpose or analysis.

For example; the screenshot below shows charts which plots, by way of comparison, the CPU and Memory usage of two WebLogic Managed Servers residing in the same domain.

Screenshot: WebLogic Managed Server - CPU and Memory Comparison


How is this done? Well, first click the button "Show Metric Pallette". 

Screenshot : Show Metric Pallette

The metric pallette allows you to pick and choose from a wide range of metrics. It is worth exploring. In this example, CPU and Memory Usage under the "JVM Metrics" has been chosen for WC_Spaces1.

Screenshot : WC_Spaces1

Then we scrolled down and selected the same metrics for WC_Spaces2 which is found under "Related Targets". To plot metrics on the same chart, the trick is to drag the metric from the pallette onto the existing chart which is showing just the one value. It's a bit difficult to show this in a screenshot :-) Just give it a go.

Screenshot: WC_Spaces2

You can also play around with the metric time period by enabling the Slider, or by specifying default time selection periods (15 minutes, 30 minutes, 1 hour), or by specifying your own start and end date / times.

Screenshot: Metric Time Period

And a final tip. Try using the Overlay menu option to build a chart which shows a data comparison - for example another Oracle WebLogic Server or Yesterday's data. (You will only see the latter option if you have been logged into Fusion Middleware (EM) Control for over 24 hours.)

Screenshot: Overlay

I hope this gives a flavor of what Dynamic Monitoring Service has to offer and that it can help you resolve performance related issues.

Friday Jan 11, 2013

Guide to Finding Fusion Middleware "End of Support" Information

When does support end for my Fusion Middleware version? Such a simple question and frequently asked. But there is more to the answer than you might expect. Furthermore, finding the full detail can be challenging.

In this blog entry, I will describe the navigation path through the relevant documents - all of which need to be read not only to obtain the "end of support" dates, but to also gain an understanding of Oracle Lifetime Support and Error Correction (bug fixing) policies.

The starting point is the Oracle Lifetime Support page

http://www.oracle.com/us/support/lifetime-support/index.html

This page lays out the Oracle Lifetime Support terms, definitions and benefits. Once these are understood, go to the product specific Lifetime Support documents listed on the page. The direct link to the Fusion Middleware document is:

http://www.oracle.com/us/support/library/lifetime-support-middleware-069163.pdf

This is a weighty tome (60 + pages) which contains the actual "end of support" dates. The challenge is to pick out the dates for the different Middleware products which form your chosen technology stack / solution. On page 8, there is a table labelled - "Fusion Middleware Releases". While some dates are listed in this table, for 10.1.3, 11g and 12c you are referred to the footnotes e.g.

Screenshot of rows in Fusion Middleware Release Table on Page 8

To quote the footnote

"... For specific dates refer to the respective sections in this document. To find a section, refer to the Table of Contents."

Therefore, be prepared to jump to different sections to find the dates which are specific to the products you have chosen for your Middleware solution.

There is one other very important footnote on this page, to quote:

"For more-detailed information on bug fix and patch release policies, please refer to the “Error Correction Support Policy” on My Oracle Support ."

This statement leads you down the path to find "Error Correction Support" dates for given patch set version numbers. Establishing the "Error Correction Support" dates for your solution is key because it will inform your plans for patching / maintenance. 

For Fusion Middleware, you can cut to the chase and go straight to:

Error Correction Support Dates for Oracle Fusion Middleware 11g (11.1.1/11.1.2) [ID 1290894.1]

Thursday Dec 13, 2012

How to Run Apache Commands From Oracle HTTP Server 11g Home

Every now and then you come across a problem when there is nothing in the "troubleshooting manual" which can help you. Instead you need to think outside the box. This happened to me two or three years back. Oracle HTTP Server (OHS) 11g did not start. The error reported back by OPMN was generic and gave no clue, and worse the HTTP Server error log was empty, and remained so even after I had increased the OPMN and HTTP Server log levels. After checking configuration files, operating system resources, etc I was still no nearer the solution.

And then the light bulb moment!

OHS is based on Apache - what happens if I attempt to start HTTP Server using the native apache command. Trouble was the OHS 11g solution has its binaries and configuration files in separate "home" directories

  • ORACLE_HOME contains the binaries
  • ORACLE_INSTANCE contains the configuration files

How to set the environment so that native apache commands run without error? Eventually, with help from a colleague, the knowledge article

How to Start Oracle HTTP Server 11g Without Using opmnctl [ID 946532.1]

was born!

To be honest, I cannot remember the exact cause and solution to that OHS problem two or three years ago. But, I do remember that an attempt to start HTTP Server using the native apache command threw back an error to the console which led me to discover the culprit was some unusual filesystem fault.

The other day, I was asked to review and publish a new knowledge article which described how to use the apache command to dump a list of static and shared loaded modules. This got me thinking that it was time
[ID 946532.1] was given an update. The result

How To Run Native Apache Commands in an Oracle HTTP Server 11g Environment [
ID 946532.1]

Highlights:

  • Title change
  • Improved environment setting scripts
    • Interactive, should be no need to manually edit the scripts (although readers are welcome to do so)
    • Automatically dump out some diagnostic information
  • Inclusion of some links to other troubleshooting collateral

To view the knowledge article you need a My Oracle Support login. For convenience, you can obtain the scripts via the links below.

MS Windows:

Wrapper cmd script - calls main cmd script [After download, remove the ".txt" file extension]
Main cmd script - sets OHS 11g environment to run Apache commands [After download, remove the ".txt" file extension]

Unix:

Shell script - sets OHS 11g environment to run Apache commands on Unix

Please note: I cannot guarantee that the scripts held in the blog repository will be maintained. Any enhancements or faults will applied to the scripts attached to the knowledge article.

Lastly, to find out more about native apache commands, refer to the Apache Documentation

apachectl - Apache HTTP Server Control Interface
[http://httpd.apache.org/docs/2.2/programs/apachectl.html]

httpd - Apache Hypertext Transfer Protocol Server
[http://httpd.apache.org/docs/2.2/programs/httpd.html]

Tuesday Nov 20, 2012

RDA and Fusion Middleware Diagnostic Framework Integration

Further to my last blog entry "FMw Diagnostic Framework : Automatic Capture of Diagnostic Data Upon First Failure!" I have spent some time exploring how Remote Diagnostic Agent (RDA) integrates with Diagnostic Framework.

Remote Diagnostic Agent, by default, collects the information about the last 10 incidents which have been captured in the Automatic Diagnostic Repository (ADR). This information can be located via RDA's Start Page Menu system. See screenshot below.

Screenshot - Viewing Diagnostic Framework Incident Information in a RDA Package


Note: In the next release of RDA - version 4.30 - the Diagnostic Repository menu label will have it's own position in the weblogic managed server sub menu hierarchy rather than be a child menu item of the logs menu

Diagnostic Framework is also capable of launching RDA engine and including the output in an incident package. This is achieved via the command "IPS GENERATE PACKAGE". The RDA output is written to

DOMAIN_HOME/servers/<server name>/adr/diag/ofm/<domain name>/<server name>/
incpkg/pkg_[number]/seq_[number]/rda

The RDA collected files are best viewed (as shown in the screenshot above) by opening the RDA Start Page - "DFW__start.htm" - from this directory in a browser. If you do not have a browser available on the host machine, zip the contents of the rda directory and transfer and extract to a machine which has a browser. 

The explanation of the integration goes a little deeper. If you want to know more, read My Oracle Support document:

Understanding RDA and FMW 11g Diagnostic Framework Integration [Document 1503644.1]

Friday Nov 02, 2012

FMw Diagnostic Framework : Automatic Capture of Diagnostic Data Upon First Failure!

Introduction

There is nothing more frustrating than a problem that "cannot be reproduced". Logs, configuration files have been analysed but there just isn't enough information to establish the root cause. The issue maybe closed, but you are left with the feeling that the problem will raise its ugly head again in the future. Trouble is, to resolve such issues you need to capture diagnostic data at the exact time the incident occurs. Step forward Fusion Middleware Diagnostic Framework! 

Diagnostic Framework monitors WebLogic Managed Servers and delivers "Automatic capture of diagnostic data upon first failure". To quote from

Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
Chapter 13 Diagnosing Problems

"When a critical error occurs ... the Diagnostic Framework automatically collects diagnostics, such as thread dumps, DMS metric dumps, and WebLogic Diagnostics Framework (WLDF) server image dumps ... The data is stored in a file-based repository and is accessible with command-line utilities."

In other words the data collected upon first failure - especially the thread and image dumps - provides a snapshot of the system as or immediately after the problem occurs. The table below shows the type of WebLogic Server issues which fall into the scope of Diagnostic Framework

How to Configure Diagnostic Framework?

Depending on your Fusion Middleware product choice you may not need to do anything! Diagnostic Framework is automatically installed, configured and initiated for any WebLogic Domain which has the Oracle Java Required Files (JRF) template applied. This template is applied by default whenever you configure WebLogic Managed Servers for products such as

  • Portal / Forms / Reports / Discoverer
  • Identity Management ( OID , OAM , OIM etc)
  • WebCenter
  • SOA

Check your WebLogic Domain directory structure. If you have an "adr" sub directory under

DOMAIN_HOME/servers/<servername>/

then JRF template has been applied and Diagnostic Framework will be in play.

Should the "adr" sub directory not exist, review the advice given in My Oracle Support article

How to Apply FMW ( EM ) Control and JRF to a WebLogic Domain and Managed Servers [ID 947043.1]

If you are working with a standalone WebLogic Server solution and applying Oracle JRF is not acceptable, consider using WLDF - WebLogic Diagnostic Framework. (Fusion Middleware Diagnostic Framework makes use of WLDF under the covers.) Couple of useful links about WLDF are listed below

How to Get Started With Diagnostic Framework

To be frank, the Fusion Middleware Administrator's Guide is the best place to start your learning

Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
Chapter 13 Diagnosing Problems

A lot of reading here,  but if you are in hurry and just want to get the right information to Oracle Support to help resolve your issue, check out the next section below.

How to Upload Diagnostic Framework Incident Data to Oracle Support

Some Background Information

There are three interfaces to the Repository:

  1. Enterprise Manager Cloud Control (Support Workbench)
  2. WLST (Command Line)
  3. ADRCI (Command Line)

The Enterprise Manager Cloud Control does provide a nice GUI interface to search, view and package diagnostic framework incidents. However, this software is not to be confused with Fusion Middleware (EM) Control. Cloud Control (formerly known as Grid Control) is part of the Enterprise Manager media package. EM Cloud Control has it's own install and configuration story. Therefore, for the benefit of those yet to install and play with Cloud Control, I am going to describe how to use the command line tools.

Ideally, you would only need to one command line interface, but currently I suggest using both - mainly due to the fact that ADRCI SHOW INCIDENTS does not reveal the description behind the Diagnostic Framework error code.

Instructions:

Note:

WLST and ADRCI are case sensitive when it comes to handling parameter values. If you make a mistake, expect an unfriendly syntax error message.

1) Find the incident

Note:

The managed server which you are troubleshooting must be up and running. If the managed server is down, ensure the domain's Admin Server is accessible. If you cannot connect to the Admin Server or the Managed Server the example WLST commands will not work.

a) Launch WLST 

Note: Use the WLST which resides in the "oracle_common" directory (not WL_HOME/common/bin) otherwise you will get a syntax error like the one below

Traceback (innermost last):
  File "<console>", line 1, in ?
NameError: listIncidents

MW_HOME/oracle_common/common/bin/wlst.sh

b) Connect to the managed server or the admin server e.g.

wls:/offline> connect('weblogic','welcome1','t3://localhost:7020')

c) Run the command

wls:/mydomain/serverConfig> listIncidents()

This will list the incidents for the server to which you have connected. If you have connected to the Admin Server and want to list the incidents for a managed server within the domain, use the command

wls:/mydomain/serverConfig> listIncidents(adrHome='diag\ofm\mydomain\mymanagedserver'
,server='mymanagedserver')

Example output

Incident Id     Problem Key              Incident Time
        1       DFW-99998 [java.lang.NullPointerException]
[oracle.error.simulator.ErrorSimulator.createNullPointerException][errorWebApp_1-0-0-0]        
Fri Nov 02 10:38:46 GMT 2012

 The piece highlighted in bold is the description you do not see when using the ADRCI 'SHOW INCIDENT' command.

Make a note of the incident id. You are ready to move to step 2

2. Package the incident

a) Set up the environment - example commands below are for Unix

cd <DOMAIN_HOME>/bin
. ./setDomainEnv.sh

LD_LIBRARY_PATH=$WL_HOME/server/adr; export LD_LIBRARY_PATH

If you want ADRCI to run a Remote Diagnostic Agent collection (recommended) at generate package time, point ORACLE_HOME at oracle_common

ORACLE_HOME=$MW_HOME/oracle_common; export ORACLE_HOME

To prevent ADRCI from running RDA at generate package time, point ORACLE_HOME at WL_HOME/server/adr directory. 

ORACLE_HOME=$WL_HOME/server/adr; export ORACLE_HOME

b) Launch adrci

$WL_HOME/server/adr/adrci

c) Set BASE and HOMEPATH

adrci> SET BASE /oracle/middleware/user_projects/domains/
mydomain/servers/mymanagedserver/adr
adrci> SET HOMEPATH diag/ofm/mydomain/mymanagedserver

d)  Optionally run SHOW INCIDENTS e.g.

adrci> SHOW INCIDENTS -MODE DETAIL
ADR Home = /oracle/middleware/user_projects/domains/mydomain/
servers/mymanagedserver/adr/diag/ofm/mydomain/mymanagedserver:
*************************************************************************

**********************************************************
INCIDENT INFO RECORD 1
**********************************************************
   INCIDENT_ID                   1
   STATUS                        ready
   CREATE_TIME                   2012-11-02 10:38:46.468000 +00:00
   PROBLEM_ID                    1
   CLOSE_TIME                    <NULL>
   FLOOD_CONTROLLED              none
   ERROR_FACILITY                DFW
   ERROR_NUMBER                  99998
   ERROR_ARG1                    <NULL>
   ERROR_ARG2                    <NULL>
   ERROR_ARG3                    <NULL>
   ERROR_ARG4                    <NULL>
   ERROR_ARG5                    <NULL>
   ERROR_ARG6                    <NULL>
   ERROR_ARG7                    <NULL>
   ERROR_ARG8                    <NULL>
   ERROR_ARG9                    <NULL>
   ERROR_ARG10                   <NULL>
   ERROR_ARG11                   <NULL>
   ERROR_ARG12                   <NULL>
   SIGNALLING_COMPONENT          <NULL>
   SIGNALLING_SUBCOMPONENT       <NULL>
   SUSPECT_COMPONENT             <NULL>
   SUSPECT_SUBCOMPONENT          <NULL>
   ECID                          5162744c6a2eea5e:155ff445:13ac0aae7cb:-8000-000
0000000000325
   IMPACTS                       0
1 rows fetched

e)  Create a logical package

IPS CREATE PACKAGE INCIDENT incident_number

e.g.

adrci> IPS CREATE PACKAGE INCIDENT 1
Created package 1 based on incident id 1, correlation level typical

f) Generate the package

IPS GENERATE PACKAGE package_number IN path

e.g.

adrci> IPS GENERATE PACKAGE 1 IN /tmp
Generated package 1 in file /tmp/DFW99998j_20121102113633_COM_1.zip, mode complete

Note:

If the generate package command hangs, ADRCI may be experiencing an issue when running RDA. To avoid such trouble, exit ADRCI and point the ORACLE_HOME environment variable at WL_HOME/server/adr

3) Upload the package zip to Oracle Support via your Service Request

a) Log into My Oracle Support and locate your Service Request

b) Click on "Add Attachments


c) And upload the zip file


Wednesday Oct 31, 2012

How to Use RDA to Generate WLS Thread Dumps At Specified Intervals?

Introduction

There are many ways to generate a thread dump of a WebLogic Managed Server. For example, take a look at:

Taking Thread Dumps - [an excellent blog post on the Middleware Magic site]

or 

Different ways to take thread dumps in WebLogic Server (Document 1098691.1)

There is another method - use Remote Diagnostic Agent! The solution described below is not documented, but it is relatively straightforward to execute. One advantage of using RDA to collect the thread dumps is RDA will also collect configuration, log files, network, system, performance information at the same time.

Instructions

1. Not familiar with Remote Diagnostic Agent? Take a look at my previous blog "Resolve SRs Faster Using RDA - Find the Right Profile"

2. Choose a profile, which includes the WebLogic Server data collection modules (for example the profile "WebLogicServer"). At RDA setup time you should see the prompt below:

-------------------------------------------------------------------------------
S301WLS: Collects Oracle WebLogic Server Information
-------------------------------------------------------------------------------
Enter the location of the directory where the domains to analyze are located
(For example in UNIX, <BEA Home>/user_projects/domains or <Middleware
Home>/user_projects/domains)
Hit 'Return' to accept the default
(/oracle/11AS/Middleware/user_projects/domains)
> 

For a successful WLS connection, ensure that the domain Admin Server is up and
running.

Data Collection Type:
  1  Collect for a single server (offline mode)
  2  Collect for a single server (using WLS connection)
  3  Collect for multiple servers (using WLS connection)

Enter the item number
Hit 'Return' to accept the default (1)
> 2

Choose option 2 or 3.

Note: Collect for a single server or multiple servers using WLS connection means that RDA will attempt to connect to execute online WLST commands against the targeted server(s). The thread dumps are collected using the WLST function - "threadDumps()". If WLST cannot connect to the managed server, RDA will proceed to collect other data and ignore the request to collect thread dumps. If in the final output you see no Thread Dump menu item, then it's likely that the managed server is in a state which prevents new connections to it. If faced with this scenario, you would have to employ alternative methods for collecting thread dumps.

3. The RDA setup will create a setup.cfg file in the RDA_HOME directory. Open this file in an editor. You will find the following parameters which govern the number of thread dumps and thread dump interval.

#N.Number of thread dumps to capture
WREQ_THREAD_DUMP=10
#N.Thread dump interval
WREQ_THREAD_DUMP_INTERVAL=5000

The example lines above show the default settings. In other words, RDA will collect 10 thread dumps at 5000 millisecond (5 second) intervals. You may want to change this to something like:

#N.Number of thread dumps to capture
WREQ_THREAD_DUMP=10
#N.Thread dump interval
WREQ_THREAD_DUMP_INTERVAL=30000

However, bear in mind, that such change will increase the total amount of time it takes for RDA to complete its run.

4. Once you are happy with the setup.cfg, run RDA. RDA will collect, render, generate and package all files in the output directory.

5. For ease of viewing, open up the RDA Start html file - "xxxx__start.htm". The thread dumps can be found under the WLST Collections for the target managed server(s). See screenshots below

Screenshot 1:RDA Start Page - Main Index

Screenshot of RDA Start Page

Screenshot 2: Managed Server Sub Index

Screenshot of Managed Server Sub Index

Screenshot 3: WLST Collections

Screenshot of WLST Collections - pointing out Thread Dump menu option

Screenshot 4: Thread Dump Page - List of dump file links

Screenshot of Thread Dump Page - list of dump file links

Screenshot 5: Thread Dump Dat File Link

Screenshot of what you see when you click a dump file link

Additional Comment:

A) You can view the thread dump files within the RDA Start Page framework, but most likely you will want to download the dat files for in-depth analysis via thread dump analysis tools such as:

If you are new to thread dump analysis - take a look at this recorded Support Advisor Webcast 

Oracle WebLogic Server: Diagnosing Performance Issues through Java Thread Dumps
[Slidedeck from webcast in PDF format]

B) I have logged a couple of enhancement requests for the RDA Development Team to consider:

  1. Add timestamp to dump file links, dat filename and at the top of the body of the dat file
  2. Package the individual thread dumps in a zip so all dump files can be conveniently downloaded in one go.

Wednesday Oct 17, 2012

Critical Patch Update For Oracle Fusion Middleware - CPU October 2012

The latest Critical Patch Update (CPU) has been released for Oracle products. Start your reading here

Critical Patch Updates, Security Alerts and Third Party Bulletin 

This is the home page containing links to all "Critical Patch Updates" released to date, along with sections detailing 

  • Security Alerts 
  • Third Party Bulletin
  • Public Vulnerabilities Fixed
  • Policies
  • Reporting Security Vulnerabilities 

On this page you will find the link to the

Oracle Critical Patch Update Advisory - October 2012

The advisory lists the support documents that cover the patch availability for all Oracle products.

From an Oracle Fusion Middleware perspective, you can cut to the chase by using the links below which take you to the appropriate sections in

Patch Set Update and Critical Patch Update October 2012 Availability Document [ID 1477727.1]

  1. Oracle Fusion Middleware 11g Release 2 
  2. Oracle Fusion Middleware 11g Release 1
  3. Oracle Application Server 10g Release 3
The #anchor links above should work in Firefox and IE provided you have already logged into My Oracle Support within the same browser session. For some reason, Chrome always takes you to the top of the document :-/

Tip:

Error Correction Support for Oracle Identity Management 10g, version 10.1.4.x ended in December 2011. For this reason, there is no section which is specific to this version. However, Error Correction Support remains in place, until end of this year, for the Oracle Identity Management 10.1.4.x components

    • Single Sign On (SSO)
    • Delegated Administration Services (OIDDAS)

provided you are using them as part of a Single Sign-On solution (OID 11g + SSO / OIDDAS 10.1.4.3) for a Portal / Forms / Reports and Discoverer 11.1.1.x architecture.   

As such there are security related patches available for Fusion Middleware Single Sign On. You will find the patch numbers listed in the sections for 11.1.1.4, 11.1.1.5 and 11.1.1.6

And finally, if you are hit any unexpected errors when applying the CPU patches, check out the known issues documented in these two support documents.

Critical Patch Update October 2012 Oracle Fusion Middleware Known Issues (Doc ID 1455408.1)

Critical Patch Update October 2012 Database Known Issues (Doc ID 1477865.1)

Monday Sep 10, 2012

Resolve SRs Faster Using RDA - Find the Right Profile

Introduction

Remote Diagnostic Agent (RDA) is an excellent command-line data collection tool that can aid troubleshooting / problem solving. The tool covers the majority of Oracle's vast product range, and its data collection capability is comprehensive. RDA collects data about

  • the operating system and environment, including
    • environment variable, kernel settings
    • network
    • o/s performance
    • o/s patches
    • and much more
  • the Oracle Products installed, including
    • patches
    • logs and debug
    • metrics
    • configuration
    • and much more

In effect, RDA can obtain a snapshot of an Oracle Product and its environment. Oracle Support encourages the use of RDA because it greatly reduces service request resolution time by minimizing the number of requests from Oracle Support for more information. RDA is designed to be as unobtrusive as possible; it does not modify systems in any way. It collects useful data for Oracle Support only and a security filter is provided if required.

RDA can be downloaded via this My Oracle Support document

Remote Diagnostic Agent (RDA) 4 - Getting Started [ID 314422.1]

Find and Use the Right RDA Profile

One problem of any tool / utility, which covers a large range of products, is knowing how to target it against only the products you wish to troubleshoot. RDA does not have a GUI. Nor does RDA have an intelligent mechanism for detecting and automatically collecting data only for those Oracle products installed. Instead, you have to tell RDA what to do.

There is a mind boggling large number of RDA data collection modules which you can configure RDA to use. It is easier, however, to setup RDA to use a "Profile". A profile consists of a list of data collection modules and predefined settings. As such profiles can be used to diagnose a problem with a particular product or combination of products.

How to run RDA with a profile?

( <rda> represents the command you selected to run RDA (for example, rda.pl, rda.cmd, rda.sh, and perl rda.pl).)

1. Use the embedded spreadsheet to find the RDA profile which is appropriate for your problem / chosen Oracle Fusion Middleware products.

2. Use the following command to perform the setup

<rda> -S -p <profile_name> 

3. Run the data collection

<rda>

Run the data collection. If you want to perform setup and run in one go, then use a command such as the following:

<rda> -vnSCRP -p <profile name>

For more information, refer to:

Remote Diagnostic Agent (RDA) 4 - Profile Manual Pages [ID 391983.1]

Additional Hints / Tips:

1. Be careful! Profile names are case sensitive.

2. When profiles are not used, RDA considers all existing modules by default. For example, if you have downloaded RDA for the first time and run the command

<rda> -S

you will see prompts for every RDA collection module many of which will be of no interest to you. Also, you may, in your haste to work through all the questions, forget to say "Yes" to the collection of data that is pertinent to your particular problem or product. Profiles avoid such tedium and help ensure the right data is collected at the first time of asking.

3. RDA has a security filter which removes potentially sensitive data such as

  • Domain names
  • Group names
  • Host names
  • IPv4 and IPv6 addresses
  • LDAP domain components
  • Network masks
  • User names

You can switch on at RDA setup by appending "-Security" to the profile value

<rda> -vnSCRP -p <profile name>-Security

e.g.

<rda> -vnSCRP -p FM11g_WlsWebTier-Security

If you already have a setup.cfg, you can enable the security filter by running the command

<rda> -X Filter enable 

Monday Aug 13, 2012

Find Oracle FMw 11g Other Certification Information - Client/Browser, 3rd Party, Product Interoperability

Here is the fourth and final viewlet in a series of viewlets which discuss how to find Oracle Fusion Middleware certification information. The viewlet below explores the process of searching for Oracle Fusion Middleware 11g Client, 3rd Party and Interoperability certification information. The objective is to help you find the answers to questions such as

  • Which versions of WebLogic Server are compatible with the latest FMw 11g patchset?
  • Which browsers are certified e.g. Is Forms / Reports 11g certified to work with IE 9?
  • Is it supported to integrate FMW 11g products with Websphere?
  • Does FMW 11g support IPv6?

[Note - the viewlets have no sound and use the swf (FlashPlayer) format. It will take approximately 10 to 15 minutes to work through the show.

If you are unable to use Flash Player, click here to grab a PDF version of the presentation.]

If you have not done so already, check out the previous viewlets in the series

About

This is the blog of the Oracle Fusion Middleware Proactive Support Delivery Team. Here we will provide information about our activities, publications, product related information and more. Feedback welcome.

Follow OracleMWSupport on Twitter

Search

Archives
« April 2015
SunMonTueWedThuFriSat
   
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
  
       
Today