« March 20, 2006 | Main | May 16, 2006 »
If on 9i run the following on one instance 2-3 times - 1 minute apart:
------------------------------------------------------------------------------
SQL> oradebug setmypid
SQL> oradebug unlimit
SQL> oradebug -g all hanganalyze 3
SQL> oradebug -g all dump systemstate 266
#!/usr/bin/ksh
#
# Sample 10g CRS resource status query script
#
# Description:
# - Returns formatted version of crs_stat -t, in tabular
# format, with the complete rsc names and filtering keywords
# - The argument, $RSC_KEY, is optional and if passed to the script, will
# limit the output to HA resources whose names match $RSC_KEY.
# Requirements:
# - $ORA_CRS_HOME should be set in your environment
RSC_KEY=$1
QSTAT=-u
#AWK=/usr/xpg4/bin/awk # if not available use /usr/bin/awk
AWK=/usr/bin/awk
# Table header:echo ""
$AWK
'BEGIN {printf "%-45s %-10s %-18s\n", "HA Resource", "Target", "State";
printf "%-45s %-10s %-18s\n", "-----------", "------", "-----";}'
# Table body:
$ORA_CRS_HOME/bin/crs_stat $QSTAT | $AWK
'BEGIN { FS="="; state = 0; }
$1~/NAME/ && $2~/'$RSC_KEY'/ {appname = $2; state=1};
state == 0 {next;}
$1~/TARGET/ && state == 1 {apptarget = $2; state=2;}
$1~/STATE/ && state == 2 {appstate = $2; state=3;}
state == 3 {printf "%-45s %-10s %-18s\n", appname, apptarget, appstate; state=0;}'
Oracle Support provides several tools for RAC diagnostics.
One of them is racdiag.sql available through metalink.
Note: 135714.1 Script to Collect RAC Diagnostic Information (racdiag.sql)
This script is broken up into different SQL statements that can be used
individually. Each SQL statement adds information to help in debugging an
RAC hang/severe performance scenerio.
The script provide the following information:
RAC Step-By-Step Installation on IBM RS/6000 see Note 199457.1
.
RAC Step-By-Step Installation on LINUX see Note 184821.1
.
RAC Step-By-Step Installation on COMPAQ OPEN VMS see Note 180012.1
.
RAC Step-By-Step Installation on SUN CLUSTER V3 see Note 175465.1
.
RAC Step-By-Step Installation on WINDOWS 2000 or NT see Note 178882.1
.
RAC Step-By-Step Installation on HP TRU64 UNIX CLUSTER see Note 175480.1
.
RAC Step-By-Step Installation on HP-UX see Note 182177.1
.
RDA is a set of command line diagnostic scripts that are executed by an engine written in the Perl programming language. RDA is used to gather detailed information about an Oracle environment; the data gathered is in turn used to aid in problem diagnosis. The output is also useful for seeing the overall system configuration.
Oracle Support encourages the use of RDA because it provides a comprehensive picture of the customer's environment. This can greatly reduce service request resolution time by minimizing the number of requests from Oracle Customer Support for more information. RDA is designed to be as unobtrusive as possible; it does not modify systems in any way, it only collects useful data for Oracle Customer Support.
This guide provides users with an overview of RDA, download instructions, and general steps on to how to execute RDA within a Unix, Windows, and MacOS X.
Is your configuration certified? The certification matrix can be found using Note 184875.1.
Before opening any RAC issues, customers should review the certification matrix. To eliminate any problems on unsupported configurations and any certification inquiries, the certification matrix for RAC/OPS should be checked before opening an issue to verify a certified database, OS, and cluster are being used. Please note that the OS vendor certifies all hardware unless otherwise noted.
No Please use a certified configuration.
Have you installed all required OS patches?
Often problems are discovered that turn out to be operating system issues that are resolved in the latest OS patch. Since Real Application Clusters gets installed on top of the OS, we should verify that the OS version is up to date to ensure that RAC is running in a stable operating environment.
Patch information is available in the Release Notes at http://otn.oracle.com/documentation/content.html.
1. Click on the appropriate platform for your OS.
2. Choose the latest cd pack for your version.
3. Click on the release notes for your platform.
4. Verify the system requirements.
No Please apply all mandatory OS patches.
Please choose the category that best describes your issue:
Concepts and Design - RAC architecture, features, implementation considerations, and HA questions and issues
Setup and Configuration - OS configuration, installation, relinking, database creation, and DBCA questions and issues
SRVCTL and GSD Issues - Questions and issues when using SRVCTO and/or GSD to adminster a RAC database.
Database Administration - ORA-29702 Errors, Administrative questions and issues for existing clustered databases
Internal Errors - Troubleshooting ORA-600 or ORA-7445 errors
Backup and Recovery - RMAN and manual backup and recovery with raw and clustered filesystems
Performance Issues and Hangs - Diagnosing and resolving performance and/or hang issues
Upgrade and Migration - Performing software and database upgrades, patch installations, and migration with a clustered database
The answers to most Concepts and Design questions can be found within the documentation.
The RAC documentation is available via Note 188135.1.
Please select the category that best describes your issue:
Introduction to Real Application Clusters - What is RAC? Learn More About RAC Architecture and New Features
Implementation Considerations - Things to consider when implementing and maintaining RAC databases
High Availability and RAC - High Availability features and considerations with RAC databases.
Please select the category that best describes your issue:
OS Configuration OS specific Cluster Information and Setup
Installation Step-By-Step Installation instructions
Relinking How to resolve Relinking issues
Database Creation Manual database creation troubleshooting
Using DBCA Using DBCA, common issues and errors
Turn on/off RAC Option How to disable or enable RAC
Please select the category that best describes your issue:
PRK Errors - Receiving PRK% Errors When Using SRVCTL or GSD
Usage and Commands - Common SRVCTL commands
Tracing - How to trace SRVCTL and GSD
Please select the category that best describes your issue:
ORA-29702 Errors Receiving ORA-29702 Errors
Parameter Management How to set and manage parameters in RAC with a text file or SPFILE
Enterprise Manager for RAC How to set up Enterprise Manager to administer RAC
Recreating Control Files - How to recreate a controlfile in RAC
Storage - adding redo, undo, etc Storage management and issues with RAC
Changing Archivelog Mode - How to change archivelog mode in RAC
Please select the category that best describes your issue:
What is an internal error? Information about how and why an internal error is generated
Troubleshooting ORA-600 and ORA-7445 Errors in RAC - You are receiving an internal error in RAC
Please select the category that best describes your issue:
RMAN Backup � Using RMAN to backup a RAC database
RMAN Recovery Using RMAN to recover a RAC database
Manual Backup Backing up a RAC database with cold and hot backups
Manual Recovery Recovering a RAC database from cold and hot backups
Is the problem due to a slow/hung query or is the entire database affected? Try running the following SQL statement in SQL*Plus:
select inst_id, sid, state, event, seconds_in_wait seconds
from gv$session_wait
where seconds_in_wait > 5
and event not in ('rdbms ipc message','smon timer','pmon timer',
'SQL*Net message from client','lock manager wait for remote message',
'ges remote message', 'client message', 'SQL*Net more data from client',
'pipe get', 'Null event', 'PX Idle Wait', 'single-task message',
'PX Deq: Execution Msg', 'KXFQ: kxfqdeq - normal deqeue', 'listen endpoint status',
'slave wait','wakeup time manager');
If the query does not return, you may have a hung database.
If the query returns no rows, you may have database performance issues.
If the query returns rows, you may have hung sessions.
Please select the category that best describes your issue:
Hung Database All queries against the database hang
Database Performance Issues Overall database performance is slower than normal
Hung Sessions One or more sessions in the database are hung
Slow Query A particular query is slower than normal (SQL Tuning)
Please select the category that best describes your issue:
OS Software Upgrades Questions and issues related to OS software upgrades and RAC
Database Upgrades and Migrations - Questions and issues related to upgrading a RAC database
Patch Installation Questions and issues related to installing patches or patchsets on RAC
This page contains all entries posted to Alejandro Vargas' Blog in March 2006. They are listed from oldest to newest.
March 20, 2006 is the previous archive.
May 16, 2006 is the next archive.
Many more can be found on the main index page or by looking through the archives.
