After motherboard change on BDA server eth0 network interface is missing

Recently I had a BDA server that was refusing to come up. Every trial to run a start /SYS failed. We were able to identify this as a motherboard failure, then the motherboard was replaced and we were able to bring up the server using the ILOM. Back on track I've tried to setup the management network, but this failed because the eth0 NIC was missing. [root@bdanode01 ~]# ethtool -i eth0 Cannot get driver information: No such device But I've noticed we had eth1 available,...

Sunday, July 15, 2018 | Read More

How to replace old Exadata storage cells with the new X7-2 storage cells, without downtime

Lately I had to help various customers to replace their old storage cells with the new X7-2 ones. There are huge benefits in doing this, the X7 has 25.5TB of flash, 12 x 10TB disks and 192 GB of DDR4 Memory.                       The question my customers asked the most was: Can we do the migration from our old storage to the new X7 without downtime and without risk? The answer was: YES! For doing this I've prepared and implemented a procedure that cover step by step how to migrate...

Friday, June 15, 2018 | ASM | Read More

How to setup passwordless ssh in Exadata using dcli

Setting passwordless ssh root connection using dcli is fast and simple and will easy later to execute commands on all servers using this utility. In order to do that you should have either: DNS resolution to all Database and Storage nodes OR have them registered in /etc/hosts 1) Create a parameter file that contains all the server names you want to reach via dcli, tipically we have a cell_group for storage cells, a dbs_group for database servers and an all_group for both of...

Sunday, October 5, 2014 | Engineered Systems | Read More

Bash security fix made available for Exadata

Complete information about the security fix availability should be reviewed, before applying the fix, in MOS DOC:  Responses to common Exadata security scan findings (Doc ID 1405320.1) The security fix is available for download from: http://public-yum.oracle.com/repo/OracleLinux/OL5/latest/x86_64/getPackage/bash-3.2-33.el5_11.4.x86_64.rpm The summary installation instructions are as follows: 1) Download getPackage/bash-3.2-33.el5_11.4.x86_64.rpm 2) Copy...

Sunday, October 5, 2014 | Engineered Systems | Read More


Exadata auto disk management is controlled by the parameter _AUTO_MANAGE_EXADATA_DISKS. The default value for this parameter is TRUE. When _AUTO_MANAGE_EXADATA_DISKS is enabled, Exadata automate the following disk operations: If a griddisk becomes unavailable/available, ASM will OFFLINE/ONLINE it. If a physicaldisk fails or its status change to predictive failure, for all griddisks built on it ASM will DROP FORCE the failed ones and DROP the ones with predictive failures.If a...

Wednesday, October 17, 2012 | HA | Read More

Oracle Engineered Systems, Amazing Exalogic

Sometimes I have heard that Exalogic is just a bunch of servers connected using infiniband, something that you can easily build yourself at a lower cost. That comments misses completely 2 things: 1) What is the idea behind an Oracle engineered system, and the back that Oracle provides for them  2) What is Exalogic This amazing 5 minute presentation explains Exalogic potential: Oracle Exalogic Elastic Cloud

Thursday, September 27, 2012 | Read More

RAC11.2.0.2 redundant interconnect and the Cluster Health Monitor

There are 2 interesting new features on RAC The first is the cluster HAIP resource, that makes possible to have up to 4 redundant interconnects that will be automatically managed by the cluster for fail-over and load balancing.  The second one is the Cluster Health Monitor. It was previously available as a utility that you can download and install, now is a resource on the cluster and will start to collect valuable OS statistics from the moment the Cluster...

Monday, January 9, 2012 | RAC | Read More

Configuring FTP on Exadata

Exadata is installed with the minimum set of rpm's required to make it work as a database server. In many cases you will need to install by yourself the rpms required to make available some specific functions, like FTP. Exadata is installed either with Oracle Enterprise Linux or Solaris Express. This instructions match the Linux distribution, and can be used on any RH compatible Linux, not only OEL on Exadata.You can find the rpm's on the Oracle Enterprise Linux...

Monday, July 25, 2011 | Linux | Read More Creating a Standby or a Clone Database using Rman Duplicate From Active Database

There are a few things on that you need to take into account to create a standby database or a clone database from an Active Database using Duplicate database command. Points 2, 9 and 10 of this document contains the details I’ve found I needed to change in order to get the clone or duplicate for standby running smoothly on this release: The steps required to complete the task are: 1) Create a parameter file for the clone or standby 2) add the following 3...

Sunday, July 24, 2011 | Data Guard | Read More

RAC, ASM and Linux Forum, May 18, 2011: EXADATA Production Customers updates

Exadata is changing the world of Database Performance, on this forum we will have two EXADATA Production Customers updates. 75 million Customer Telecom Turkcell will be represented by Ferhat Sengonul, Senior OS Administrator, DBA, and Datawarehouse Project Leader, that led Exadata implementation and scale out to their actual 3 full, 24 database nodes, Exadata racks. Ferhat will present his experience with a very large Data-Warehouse in Exadata, including online high...

Sunday, May 8, 2011 | RAC | Read More

How to Install the Oracle-Validated rpm Using a Local Repository

One of the steps required to install Oracle on Linux is to Install all the Linux packages, rpm's, needed by Oracle, and their dependencies. That is followed by creating the Oracle user account and groups and setup the parameters for the kernel. All of these tasks can be automatized by installing a single rpm that is distributed by Oracle, the Oracle-Validated rpm. On the Oracle Enterprise Linux 5.5 distribution disk it is included the Oracle-Validated rpm and also the ASMLib...

Wednesday, November 24, 2010 | Linux | Read More

Using 11g RMAN Duplicate Command to Create a Physical Standby Database Over the Network

This post contains a quick, step-by-step walk over the procedure of creating a Physical Standby Database using Rman Duplicate command, not using any backup. Setting up a physical standby database is a simple operation when the required infrastructure is ready. We need 2 servers, a network that communicate both of them, and storage connected to the servers that is proportional to the database size + extra space for archive logs and backups. The Oracle Home is installed on both...

Sunday, November 21, 2010 | Read More

Oracle RDBMS Home Install Using Cloning

Using a standard Oracle Home, that is updated to the last patches, as the source to install new Oracle Homes can save a lot of time, compared to installing the same Oracle Home + Patches from scratch. The procedure to clone an Oracle Home is simple and is well documented on a set of My Oracle Support documents that can be found on Document 1154613.1 ordered by release. On this post I'm providing a step by step example of cloning a 11g R2 Home: How to clone a 11g R2 Oracle Home Th...

Wednesday, November 17, 2010 | Read More

Data Guard for Manual Failover, Step by Step

about:blank In this post I'm showing the steps used to implement a manual failover scenario. My customer did not want to enable fast start failover but to leave the decision to failover in case of a major crash to the management team. In the example I'm providing here I did configure flashback database with a one hour retention time so that the OS team can have this time to solve any issues on the primary, if they succeed to solve the problem in this time then the old primary...

Saturday, November 13, 2010 | HA | Read More

How to Calculate TCP Socket Buffer Sizes for Data Guard Environments

The MAA best practices contains an example of how to calculate the optimal TCP socket buffer sizes, that is quite important for very busy Data Guard environments, this document Formula to Calculate TCP Socket Buffer Sizes.pdf contains an example of using the instructions provided on the best practices document. In order to execute the calculation you need to know which is the band with or your network interface, usually will be 1Gb, on my example is a 10Gb network; and the...

Wednesday, May 19, 2010 | Read More

Impact of Truncate or Drop Table When Flashback Database is Enabled

Recently I was working on a VLDB on the implementation of a disaster recovery environment configured with data guard physical standby and fast start failover. One of the questions that come up was about the overhead of truncating and dropping tables. There are daily jobs on the database that truncate extremely large partitions, and as note 565535.1 explain, we knew there is an overhead for these operations. But the information on the note was not clear enough, with...

Wednesday, May 19, 2010 | HA | Read More

Oracle Database 11g Underground Advice for Database Administrators, by April C. Sims

Recently I have received a request to review the book "Oracle Database 11g Underground Advice for Database Administrators" by April C. Sims I was happy to have the opportunity to know some details about the author, she is an active contributor to the Oracle DBA community, through her blog "Oracle High Availability" . The book is a serious and interesting work, I think it provides a good study and reference guide for DBA's that want to understand and implement highly available...

Wednesday, May 19, 2010 | HA | Read More

The Next RAC, ASM and Linux Forum. May 4, 2010 Beit HP Raanana

The next RAC, ASM and Linux forum will take place next week, you are still on time to register : Israel Oracle Users Group RAC,ASM and Linux Forum This time we will have a panel formed by Principal Oracle Advanced Customer Services Engineers and RAC experts Galit Elad and Nickita Chernovski and Senior Oracle Advanced Customer Services Engineers and RAC experts Roy Burstein and Dorit Noga. They will address the subject: 5 years of experience with RAC at Israeli Customers,...

Tuesday, April 27, 2010 | ASM | Read More

Cloning A Database On The Same Server Using Rman Duplicate From Active Database

To clone a database using Rman we used to require an existing Rman backup, on 11g we can clone databases using the "from active" database option. In this case we do not require an existing backup, the active datafiles will be used as the source for the clone. In order to clone with the source database open it must be on archivelog mode. Otherwise we can make the clone mounting the source database, as shown in this example. These are the steps required to complete the clone: Confi...

Saturday, March 20, 2010 | RMAN | Read More

Compression for tables with more than 250 columns

Compression for tables with more than 250 columns Tables with more than 250 columns are not supported to be compressed, this restriction remains in place even on 11g R2. On the 11g R2, Sql Language Reference Manual, page 16-36 we can read: Restrictions on Table Compression * COMPRESS FOR OLTP and COMPRESS BASIC are not supported for tables with more than 255 columns. This is a serious limitation specially for Telecoms where CDR tables can have a number of columns way over 255. The...

Monday, March 1, 2010 | Read More

RAC, ASM and Linux Forum, December 15, 13:30 - 18:00 Beit HP Raanana

It's time for our 2nd, 2009 RAC, ASM and Linux Forum in Tel Aviv! I would like to warmly invite you to our next RAC, ASM and Linux forum to be held at Beit HP in Raanana, on December 15. You can register on the Israel Oracle User Group site. On this forum we will have a 11g R2 Technology Update and 2 very interesting Customer Presentations about minimum downtime VLDB Migration to RAC on Linux and Creating and Managing RPM's for Oracle Silent Install on Linux. Panel on Grid...

Sunday, December 13, 2009 | ASM | Read More

Check IO Scripts

These scripts are very useful to check throughput. The original version can be found on My Oracle Support Note 422414.1 by Luca Canali. set lines 250 pages 50000 alter session set nls_date_format='dd-mm-yyyy hh24:mi'; col Phys_Read_Total_Bps for 999999999999 col Phys_Write_Total_Bps for 999999999999 col Redo_Bytes_per_sec for 999999999999 col Phys_Read_IOPS for 999999999999 col Phys_write_IOPS for 999999999999 col Phys_redo_IOPS for 999999999999 col OS_LOad for 999999999999col...

Thursday, December 10, 2009 | Read More

ASM Hands-On Training, Server Enviroment Setup And Aliases

The server where the labs of the ASM Hands-On Training works is configured with tcsh as the default oracle user shell. A .cshrc file containing several aliases was setup to easy moving around and executing repetitive commands. On this document there is a general explanation of how the environment can be used. Details on this file: Server Enviroment Setup And Aliases

Saturday, September 12, 2009 | Read More

ASM Hands-On Training, Lab 21, ASMCMD Use And Options

On this lab we will review some of the useful set of commands provided by the ASM command line utility. Some of the asmcmd commands display information, these information is based on v$asm views, other commands actually make changes to the structure like mkdir or rm. Details on file: ASMCMD Use And Options

Saturday, September 12, 2009 | ASM | Read More

ASM Hands-On Training, Lab 20, Storage Consolidation With ASM

On this lab we will share our ASM disks with a second server and we will open the sati12 database on it. To do that we copied the vm to another location, without including the ASM disks, that are located on a separate folder, this way the second vm is pointing to the same ASM disks as the first one. Details on file: Storage Consolidation With ASM

Saturday, September 12, 2009 | ASM | Read More

ASM Hands-On Training, Lab18, ASM Metadata and Other Checkups

On this lab we will review various utilities that provide further information for managing ASM. Checksum Mismatch After Storage Crash, AMDU an 11g tool that can be used with 10g as well, Blockdumps, Asmiostats and asmdebug Details on this file: ASM Metadata and Other Checkups

Saturday, September 12, 2009 | ASM | Read More

ASM Hands-On Training, Lab 17, Measuring Throughput

On this lab we will review simple methods for getting input about the the database througput and response time. We will produce the load using swingbench and we will gather AWR snapshots every 10 minutes. After some time we will be able to check througput statistics based on the AWR snapshots information. Details on this Document: Measuring Throughput Another useful script for checking IO: set pages 50000 lines 250 alter session set nls_date_format='dd-mm-yy hh24:mi';spool...

Saturday, September 12, 2009 | Read More

ASM Hands-On Training, Lab 16, Configuring And Running Swingbench And OSWatcher

Whenever implementing new environments it is convenient to run stress tests and to gather OS statistics that will provide valuable diagnostics information. Swingbench provide an easy configurable tool to run oltp or dss like stress tests; that is very useful when the customer does not have any possibility to benchmark the new configuration. Oswatcher will collect statistics from the OS in a cyclic 2-day period by default, which can be extended to whatever period is convenient...

Saturday, September 12, 2009 | Read More

ASM Hands-On Training, Lab 15, Cloning A Disk Group

Sometimes it is required having several clones of the same database running on the same server. That was simple to do when the database was based on File System, and is still simple to do with the help of Rman. But there are very large databases, which are usually cloned using storage tools like bcv, srdf or snapshots that cannot consider Rman as a viable possibility. For these databases there is the possibility to implement the procedure we will test on this lab. Note that the...

Saturday, September 12, 2009 | ASM | Read More

ASM Hands-On Training, Lab 14, 11g Compatibility Parameters and Resilience Test

On this lab we will make a review of the 11g Compatibility Parameters and 3 Resilience Tests The compatibility parameters compatible.asm and compatible.rdbms define the minimum ASM and database versions that will be able to connect to a disk group. These parameters can be advanced only On the resilience probe we will do the following tests · Add ASM disks and bring down the ASM instance on the middle of the operation · Remove ASM disks and bring down the ASM instance on...

Saturday, September 12, 2009 | Read More

ASM Hands-On Training, Lab 13, ASM And Rman: Crash, Restore And Recovery Scenarios

On this Lab we will review the following crash and recovery scenarios, and we will implement some of them: · System tablespace loss · Recover system tablespace · User datafile loss · Recover users tablespace · Online redo loss · Recover from redo loss · Controlfile loss · Recover from controlfile loss · Database loss · Recover from total loss Details on this file ASM and Rman Crash Restore and Recovery Scenarios

Saturday, September 12, 2009 | ASM | Read More

ASM Hands-On Training, Labs 11 and 12 : Restoring A Lost Database From Backup And Cleanup File Based ASM Diskgroups

To work on the 11g part of the Labs we will start restoring an 11g database we have on a backup. The backup consist of an incremental level 0 backupset, a controlfile and spfile backup and the archived logs required to recover the database. The restore is done in 6 steps · Re-create the ASM metada · Restore the spfile · Restore the controlfile · Restore the datafiles · Recover the database · Open resetlogs This is covered on this file: Restoring a Lost Database From Backup On...

Saturday, September 12, 2009 | ASM | Read More

ASM Hands-On Training, Lab 10, Migrating From 10g ASM To 11g

In this Lab we will migrate our 10g Disk Groups to 11g. The first steps of the migration, Installing on it's own Oracle Home and applying patch are ready on this environment, so we can go ahead with the migration process. In this lab we will migrate only ASM, the database migration is a little longer for the time frame we have so we will continue by moving to ASM an 11g database that already exist on this server. Details on this document: Migrating 10g ASM to...

Friday, September 11, 2009 | Read More

ASM Hands-On Training, Lab 9, Migrating Raw And Block Devices To ASMLib

ASMLib is best practice for managing ASM on Linux, it may happen that a database server that was configured a couple of years ago with ASM 10g R1 is still using raw devices, or that a new ASM 10g R2 or even 11g was configured using block devices instead of ASMLib. ASMLib provides global open close call that will improve performance and transparently implement device labeling, an important aspect when managing storage on Linux. This lab explain the procedure to migrate these...

Friday, September 11, 2009 | Read More

ASM Hands-On Training, Lab 8, Configure XDB for ASM Access Through FTP and HTML

The use of ftp and html add flexibility to manage the ASM environment, here a brief description of how to implement this configuration. XDB enable use of FTP and HTML protocols to access and manage files located on ASM disksgroups. Files can be easily browsed or moved in/out from ASM this way. This option is available only when the database is fully available. Details on this document: Configure XDB for ASM Access Through FTP and HTML

Friday, September 11, 2009 | Read More

ASM Hands-On Training, Lab 7, 10g Normal Redundancy, Resilience Test-

For Databases requiring normal redundancy I definitively recommend 11g instead of 10g, because the fast mirror resynchronization new feature that makes possible to reinstate a failed disk without requiring to rebuild it as is the case on 10g. This Hands ON series move on a later lab the database to 11g to test this options. In this lab we still work with 10g.Normal Redundancy on 10g provides protection against disk failures, but it does require for the failed disk to be...

Friday, September 11, 2009 | ASM | Read More

ASM Hands-On Training, Lab 6, Looking into PST Metadata

This lab looks into the ASM disk header section that contain the Partnership and Status Table, the PST. I have found in some limited occasions, specially following storage crashes that result in problems to mount the ASM disk groups, that looking into the PST may help to clarify the extent of the damage and asses if we need to go for a backup or not. The lab can be found on this document: Looking into PST Metadata Note that when reading the Metadata to seee the PST you must...

Friday, September 11, 2009 | RAC | Read More

ASM Hands-On Training, Lab 5, Using RMAN To Migrate a Database Into ASM

One of the ways to migrate a database to ASM is to use the Rman “Backup as Copy” command to create a database copy into ASM storage and then switch the database to the copy. This technique can be used, combined with incremental backups, to move even very large databases into ASM. The first backup may take a long time, one or more incremental backups can be used to update the first backup and close the gap, until a downtime window is obtained to switch the database on filesystem...

Saturday, July 4, 2009 | Read More

ASM Hands-On Training, Lab 4, Install, Configure and Run ORION

Oracle Input Output Numbers, ORION, provides a simple way to check storage performance before the Oracle Database is created, that gives us a clean storage performance report that can be compared with the awr IO statistics once the database is running.When performance problems related to poor performance at the storage level are detected in a late implementation stage the number of variables may make the task to detect the bottleneck difficult, and the time to find a solution...

Friday, July 3, 2009 | Read More

ASM Hands-On Training, Lab 3, Creating The ASM Instance And Managing ASM Disk Groups

On this Lab we create the ASM instance on a single instance environment, and create ASM diskgroups in several different ways using different kind of devices. Note that these exercises provide some degree of confidence on managing devices but are valid only for training. On a production environment we will use mostly a high end SAN or NAS storage, usually connected by fiber channel, and we will provision LUN's for ASM. The lab can be accessed here : Lab3-Create-ASM-Instance-and-M...

Friday, July 3, 2009 | ASM | Read More

ASM Hands-On Training, Lab 2, Install and Configure ASMLib

ASMLib provide an easy way to present storage to ASM on Linux, it takes care of device labeling and improves the performance of open-close calls to the devices. Is Oracle Best Practice for managing disks on Linux. Lab2-Install-and-Configure-ASMLib.pdf The following points are covered on this lab: Install The ASMLib: Cheking The Kernel Version Downloading the Drivers for ASMLib Installing the ASMLib Drivers Configure ASMLib : Create ASM Disks ASMLib Directory Structures ASM...

Friday, July 3, 2009 | Read More

ASM Hands-On Training

Early this year I developed an ASM training that was implemented on Madrid and Tel Aviv. The idea was to have a set of exercises that will provide lots of hands-on experience to the participants, in a few days. On this and the following posts I will present the labs of the training. The training was implemented using the student’s laptops. A Virtual Machine containing a Linux 5 environment setup with Oracle 10g and 11g, ready for implementing the labs, was distributed to the...

Sunday, May 31, 2009 | ASM | Read More

RAC and ASM on Linux Forum Meeting, June 3 at Beit HP in Raanana

On June 3, 2009 we had the 5th conference of the RAC/ASM/Linux Forum, this time at Beit HP, in Raanana, Israel. RAC, ASM and Linux are building blocks of the HP Oracle Database Machine and we wanted to start this meeting with a review of the technology revolution that Exadata is bringing in. Ehood Baratz from HP introduced the concepts and Annie Flint from Oracle RACPack Team, that is already working on implementation projects with customers around Europe, provided a deep...

Thursday, May 7, 2009 | Read More CRS bundle1 released

CRS Bundle Patch I was released for version It is available for Linux x86 and x86-64, AIX 64-bit, HP-UX Itanium and HP-UX RISC 64-bit. Details can be found on metalink Note 810663.1 11.1.0.X CRS Bundle Patch Information Before applying the patch be sure to follow instructions on Note 458485.1 : "How to find whether the one-off Patches will conflict or not?", to check for potential conflicts between patches already applied and the new patches. Apply on test, check and...

Sunday, May 3, 2009 | RAC | Read More

High Level Tracing For Monitoring And Debug With Event 10046

Whenever there is an error condition that do not provide enough information to immediately diagnose the root cause and find a solution, tracing the session that is generating the problem is the best first option to get more information. In many cases I did find on the raw trace the additional information required to get to the root cause of a problem. This method is good also to generate high level SQL traces to be used for SQL tuning. This is a short reference for running...

Saturday, April 25, 2009 | Read More

Oracle Open World Call for Papers

Answering feedback from attendees to the last Open World, the possibility to contribute experiences for the next conference has been opened to the entire Oracle community: Customers, Partners and Employees If you wish to send a paper for consideration log into this site: OOW 2009 Call for Papers. The deadline for submission was extended until April 26.

Monday, April 6, 2009 | Read More

Useful Scripts

- Service level objectives setup - Have the dba speak the same language as the storage admin - Throughput measure of transfer bits across the media throughput cannot be faster than the slowest link of the path from source to destination. Tak einto account that 2gbit HBA does not provide 2Gbytes throughput but 256 Mbytes To convert 2Gbits into Mbytes : 2Gbits = (2* 1024 * 1024 * 1024) / (1024* 1024) / 8bits = 2147483648 / 1048576 = 2048 / 8 = 256Mbytes Always size for...

Sunday, February 8, 2009 | Read More

How To Manually Open The Standby Database When The Primary Is Lost

This post explains how to manually convert the Physical Standby into the Primary database, when the Primary is not available anymore. The procedure is very simple, still is convenient to have it available so that in the stress situation of a production crash we can implement it without problems. Personally I prefer to work harder at the implementation stage and build an infrastructure strong enough to be configured using DG Broker, Observer and Fast Start Failover. Step by Step...

Wednesday, February 4, 2009 | HA | Read More

How to Synchronize the Standby When there is a Big gap, and all Required Archived Logs are Lost

This post describes a situation that should not happen, having a standby out of synch and not having the required logs to close the gap. If it is nice to have a procedure to deal with the situation, it is better to assure that archived logs are backed up on a schedule, and that the backup and cleanup script is automatically run every time a threshold of 20% free space on the archive log destination is reached before the next scheduled cleanup run. In any case this technique is...

Saturday, January 17, 2009 | Read More

Data Guard Fast Start Failover Implementation Step by Step

On this post I'm adding a step by step implementation procedure of a Dataguard site configured using fast start failover, it includes tips for configuring the observer process on a windows server and to configure automatic client failover in case of role change. All details on this document: Dataguard FSFO Step by Step

Saturday, January 17, 2009 | Read More

RAC and ASM Upgrade from to

11g is becoming more and more interesting for many Companies. Since last year I started to implement 11g Oracle Clusterware with 11g ASM and 10g RAC Databases; on the last months I did start to work on pure 11g environments soon to be production, so I thought that having experience on how to migrate 10g RAC environments to 11g was necessaire. On this post I'm presenting a step by step procedure for migrating Oracle Clusterware, ASM and RDBMS to 11g. Each step is...

Monday, December 22, 2008 | RAC | Read More

Data Guard Fast Start Failover Reference Implementation

This post contains a reference of parameter files and database parameters used on a healthy Data Guard Physical Standby implementation, configured with FSFO in Max Availability protection mode. This environment consistently succeeded to complete Switchover or FSFO, in case of crash, and reinstate the failed primary as soon it become available again. The details can be found in this document: DATA GUARD FSFO AND SWITCHOVER REFERENCE CONFIGURATION I was motivated to write this...

Saturday, December 6, 2008 | Data Guard | Read More

The Oracle-Validated rpm is available for all users

Preparing Linux for Oracle Clusterware and RAC installation is made fast and simple with the help of the Oracle Validated Rpm. The Oracle-Validated rpm automatize the install of the numerous components required to setup OCW and RAC on Linux, and execute most of the required OS configuration steps. That was a long and tedious task on the time before the rpm was made available. Until recently to be able to use the Oracle-Validated rpm was necessary to have an Unbreakable Linux...

Saturday, October 18, 2008 | Read More

Database Migration From Windows to Linux Using RMAN

This post describes the procedure required to migrate a database from Windows to Linux using the RMAN Convert Database command. Both Windows and Linux platforms have the same endian format, which makes possible to transfer the whole database, making the migration process very straightforward and simple. To migrate between platforms that have a different endian format, Cross Platform Transportable Tablespaces (XTTS) needs to be used instead.List of Steps Needed to Complete the...

Monday, October 6, 2008 | ASM | Read More

Real Application Testing on 10g/11g Step-by-Step

In this post I'm reviewing Real Application Testing Database Reply and SQL Performance Analyzer step by step. The initial understanding of the implementation process demand some work, but the results are excellent and worth the effort, we get valuable information to asses how our systems will behave after change, something much needed for production environments. The document I'm attaching to this post cover the functional tests I did of both Database Replay and SQL Performance...

Friday, October 3, 2008 | Read More

RAC and ASM on Linux Forum, July 16, Sheraton Hotel, Tel Aviv

On Wednesday, July 16 we will have our 3rd RAC and ASM on Linux Forum at the Sheraton Hotel in Tel Aviv, this time dedicated to Performance and Maximum Availability Architecture (MAA). We have several invited speakers from Oracle Israel: Oded Ilan, Gadi Chen and myself will speak about ASM performance; Eyal Druskin and Nickita Chernovsky will explain the concept of MAA, I'll present a short demo of Active Data Guard. Then we have two special invited speakers: Saar Maoz, from...

Monday, July 14, 2008 | Read More

Manually changing rebalance power on an ongoing ASM rebalance operation

A couple of times I've removed ASM disks from a diskgroup and found that the rebalance operation will take too long. This is not a problem when you don't want to disturb online users, the rebalance process only locks for writes 1mb of data at a time, so letting the rebalance to run slowly has no performance impact. My problem was that I needed to free the disks as fast as possible in order to finish some IO tests within a limited timeframe. The procedure to change the rebalance...

Wednesday, July 2, 2008 | ASM Administration | Read More

SQL Performance Analyzer and Database Replay are available for pre 11g Releases

Change Control is one of the most important aspects of managing a production database environment, the back port of this functionality is a major breakthrough, that will make easier to migrate 9i production environments to 10g and 11g. SQL Performance Analyzer permits to identify SQLperformance issues caused by system changes, the performance analyzer assess the impact of that changes and provide a report that highlight any  differences between the before and...

Saturday, June 7, 2008 | Read More

Oracle Recovery Manager ( RMAN) Hands On Practice

Oracle Recovery Manager, RMAN, is at the same time simple and complex, it's sintax can be as direct as 'backup database' to make a whole database backup, and as complex as it can be to make possible the most sophisticated and complex backup and recovery tasks. This post covers a series of tests I did prepare to show how Rman works, the tar of the scripts contained on the guide can be downloaded here. This RMAN Hands-On guide, once you have configured the environment, can be...

Thursday, June 5, 2008 | Read More

Bigfile Tablespaces

Bigfile Tablespaces may become a standard implementation on databases using high end storage and Oracle Automatic Storage Management. In order to avoid potential performance issues Bigfile Tablespaces must being built on top of a file system that can grow up dynamically and that support file striping, both characteristics are native to ASM.<meta http-equiv="CONTENT-TYPE" content="text/html; charset=utf-8"><title></title><meta name="GENERATOR" content="OpenOffice.org 2.3...

Wednesday, May 14, 2008 | Read More

Extended RAC on Pure Oracle Stack Summary

Extended RAC is a Tier 6 DRP technology that enable zero data loss in case of major disaster at a local site, and near 0 downtime if well configured and well managed. It is an expensive technology because of the costs associated with the required infrastructure, dark fiber over long distances and redundant components on all hardware layers.A strict change control procedure must be in place, with an identical test environment where all changes to the infrastructure and new...

Wednesday, March 19, 2008 | Read More

How To Resize the Online Redologs on a RAC Environment

Resizing the online redologs on a RAC environment is a simple task, that require a little more attention than that of a single instance. Usually we do resize when redolog switches happen too frequently. Take care to add all the new larger groups, then switch over until the current redolog for each instance is on the new larger groups before starting to remove the old groups. You can find a step by step description of the procedure on this document: How to Resize the Online...

Sunday, March 2, 2008 | Read More

Disaster Recovery Stories

In the last weeks I had the opportunity to work on two cases of disaster recovery. There are a couple of lessons learned from both cases: Check your backups Backup your archived logs Make backups of the database structure (full export without data) Make backups of your controlfiles (backup controlfile to trace) The first case was a development environment, where a month worth of work by a team of several developers was lost, despite that the database was working on archive log...

Tuesday, February 12, 2008 | Read More

How to Rename and Register with CRS a Clone of a RAC Database

To rename a database we set the new name recreating the controlfile, in the case of a RAC database the procedure is slightly more complex, as we need to recreate the controlfile in single instance mode using a modified pfile, and then recreate the spfile. Once the new controlfile is ready we need to register the database with CRS in order to enable its management as a RAC database. This document "How to Recreate the Controlfile on a RAC Environment" illustrate the steps...

Friday, January 11, 2008 | Read More

ASM Database Clones Naming Issues

There are several ways to clone an ASM based database, some of them result in a directory structure that do not match the current database name, i.e, the source database directory structure is built this way: +DATADG/proddb/datafile +DATADG/proddb/controlfile +DATADG/proddb/onlineredolog ... etc When cloning we will keep the same directory structure and change the database name to testdb or devdb or repdb, some dba's do not feel comfortable having a "testdb" database on...

Tuesday, January 8, 2008 | Read More

MAA Seminar: Questions and Answers

There were a couple of questions I did take home from the Seminar: 1. Which is the minimal retention time that can be used when using Flashback Database? 2. Is it possible to open a 10g Physical Standby read-write, perform application testing on it, and reinstantiate it as Physical Standby using flashback technologies? 3. Which is the correct High Availability configuration for tnsnames.ora when using Data Guard?4. How I can avoid redo shipping when the standby database is...

Sunday, January 6, 2008 | Read More

ASM Based Tablespaces Backup With Rman for Long Term Offline Storing

This post is the answer to a conversation with a Senior DBA. He asked me if it is possible to set a tablespace offline, use Rman to backup it, then remove the tablespace underlying datafiles, and, when needed restore the tablespace without performing recovery. The idea behind is to be able to free the space used by tablespaces that we do not need to have online, but we need to keep them, usually for legal reasons, for a period of several years.The standard solution would be...

Friday, January 4, 2008 | Read More

How to generate series of AWR reports

To checkup performance issues usually I do get awr and ash reports every 15 minutes along the period I want to check. This script helps to automatically generate a set of awr reports: Run_AWR_Reports

Wednesday, January 2, 2008 | Read More

MAA Seminar in Istanbul: using Cross Platform Transportable Tablespaces into ASM

One of the interesting questions I've got during the seminar was related to the possibility to migrate a terabyte database from a little endian machine to a big endian machine, using an ASM diskgroup to transfer the data.Unfortunately ASM diskgroups are not cross platform enabled. There is no problem to mount the same diskgroup on multiple servers at the same time, provided that the OS system for all the servers is the same. You can also unmount the ASM disks from one server...

Monday, December 31, 2007 | Read More

MAA Seminar in Istanbul: Using a Physical Standby For Read/Write Testing and Reporting

Another interesting subject we tested during the seminar was activating a physical standby database for test, reporting, etc. Usually we are using storage snapshots for this purpose; we mount a storage generated copy of the production database on a report server, perform recovery and use it during the day. The process of resynchronizing the storage copy is repeated each night.Using a physical standby for the same purpose is possible on 10g when flashback database is enabled...

Friday, December 28, 2007 | Read More

MAA Seminar in Istanbul: Database Migration to ASM

Last week I had the privilege to share experiences about Oracle technology with a group of colleagues in Istanbul. During the meeting we focused on Maximum Availability Architecture topics, one of the subjects I did present was Database Migration to ASM. I think this is an important subject, many critical production databases based on 8i and 9i are on the planning stage of migration to 10g with ASM. Rman provides a very efficient meanto perform the migration, with full...

Tuesday, December 25, 2007 | Read More

ASM Seminar, December 20, Hotel Daniel Hertzeliya

From December 16 to 20 will take place Oracle Week Israel. On the 20 we have a full day Seminar about ASM Features and Future, I did choose to be the main body of the Seminar a very good course offered until today to a limited number of people, it will be a unique opportunity to get a deep dive into ASM technology.So if you are around in Israel, I warmly invite you to come over. Hertzeliya, at the Mediterranean sea side  is a very nice place to spend the day discussing High...

Tuesday, November 27, 2007 | Read More

Mapping ASM disks to Physical Devices

Sometimes you may need to map ASM Disks to its physical devices.If they are based on ASMLib you will see their ASM name, ie: ORCL:VOL1 when querying v$asm_diskWhen running oracleasm querydisk VOL1 you will get in addition the major - minor numbers, that can be used to match the physical device, ie: [root@orcldb2 ~]# /etc/init.d/oracleasm querydisk VOL1 Disk "VOL1" is a valid ASM disk on device [8, 97] [root@orcldb2 ~]# ls -l /dev | grep 8, | grep 97brw-rw----   1 root disk    ...

Thursday, November 15, 2007 | Read More

10g database with 11g ASM

I just finished a test install on Oracle Enterprise Linux 5; I wanted to check how well ASM 11g  does work with a 10g database. I did install Linux 2.6.18-8.el5, Oracle, ASMLib 2.0.3-1.el5 and Oracle + patch set All the installs completed smoothly on my laptop virtual machine. Note that you can also download Oracle VM  from November 14. You can see here the Oracle Unveils Oracle® VM announcement.At last my configuration includes 11g ASM + 11g...

Tuesday, November 13, 2007 | Read More

CRS and ASM Fail to Start After Linux Kernel Upgrade

The case I'm describing apply to a specific configuration: RAC 10g R2 on Linux x86_64 with EMC storage using EMC Powerpath and ASM. It may be not relevant in other configurations, still it provides guidelines to diagnose and solve similar issues. In brief the problem was the following: CRS failed to start after the Linux Kernel was upgraded from version  2.6.9-22.ELsmp to 2.6.9-55.0.2.ELsmp; ASM also failed to start The solution steps were:Reinstall EMC Powerpath Reconfigure...

Tuesday, October 30, 2007 | Read More

How to Generate a Script to Kill the Sessions Holding an Object

The main issue in this post is a simple script, you can use to find sessions holding an object, and thus disabling the possibility to compile it. The script generate kill -9 commands to clear their shadow processes at the OS level. select 'kill -9 '||b.spid||' # username '||c.username||''from v$access a, v$process b, v$session c where a.object=upper('&name') and a.sid=c.sid and b.addr=c.paddr / If you want to have an example of using this script you can continue reading :-)This...

Tuesday, October 30, 2007 | Read More

10g Dataguard, Flashback and New Features Q&A

Last week I had the privilege to share some time with a selected group of DBA's. We did a review of some of the strong 10g features like ADDM, AWR, Dataguard, Flashback Technologies and ASM. There were many interesting questions been raised up, I did take note of some of them, and compiled them into this document;  10g Questions and Answers

Wednesday, October 24, 2007 | Read More

Oracle Flashback Technologies

Flashback Database, Table, Query, Transaction and Drop provides an extremely useful set of tools to the DBA.  Once understood they are easily configured, implemented and used, in case of need. They are not only useful for recovery purposes but sometimes for informational purposes as well, a question like: which was the value for some record the last 6 hours? can be easily answered using flashback records.Metalink Note 435998.1 : What Do All 10g Flashback Features Rely on and...

Monday, October 8, 2007 | Read More

Data Guard, Broker, Observer and Switchover, Step by Step

This post is the continuation of  the previous Data Guard, Physical Standby Implementation, Step by Step This time I did focus on configuring Data Guard Broker, the Observer and implement Switchover from the Primary to the Standby and back. The step by step instructions are contained on this document: Configuration of 10g Data Guard Broker and Observer for SwitchoverThis configuration do require a third server to run from it the Observer process on the background. It proved...

Tuesday, October 2, 2007 | Read More

Data Guard, Physical Standby Implementation, Step by Step

Although Enterprise Manager Grid Control provides the simplest way to implement Data Guard, in this post I'm presenting an exercise of manual implementation step by step. I think this is a valuable exercise that provides a close insight of the basic parameters and steps required to setup Data Guard, it provides also a good feeling of how this technology works, that the high level of automation we got from Grid Control cannot give.This exercise can be completed in a couple of...

Wednesday, September 19, 2007 | Read More

Data Guard or Storage Snapshots?

Customer sites often choose storage related technologies to ensure recoverability for large databases. I've been involved several times in recovering production databases from a BCV or SRDF backup. Mounting the storage copy back on the production server was done usually in about 30 minutes, after a period of deciding what to do that could take another 30 minutes to 1 hour. Then we needed to apply the relevant redo logs, that required a time that sometimes was of several...

Wednesday, September 19, 2007 | Read More

Oracle RAC on Linux Conference - Sep 4 - Tel Aviv

RAC, ASM and Linux: Configuration, Monitoring/Managing and Extended RAC configurations, and a Successfull implementation report are the topics of the conference of the RAC on Linux Forum to be held on September 4 at the Sheraton Hotel in Tel Aviv   If you are around I will be happy to see you there, the conference is free, you can register at the Oracle Israel, events page You can download my presentation here "Rac on Linux Configuration"  This presentation was compiled and...

Sunday, September 2, 2007 | Read More

IO Throughput Verification Before RAC Implementation: ORION

When we plan a RAC environment we need to define the throughput requirements early on the design phase, and then validate that our servers are able to perform at the required levels, even before we have create the RAC database. When performance problems related to poor performance at the storage level are detected in a late implementation stage the number of variables may make the task to detect the bottleneck difficult, and the time to find a solution scarce.There is a tool...

Thursday, August 9, 2007 | Read More

Log Miner Implementation Summary

This is a quick example of Log Miner implementation. Log Miner is a powerful tool and all DBA's should know and master it. Attached a summary of the first steps for mining archived logs or online redo logs Log Miner, first steps example This is the link to the complete reference on the Utilities Guide: Using Log Miner to Analyze Redo Log Files And this is an excellent article by Arup Nanda on OTN: Mining For Clues

Thursday, July 26, 2007 | Read More

ORA-1410 invalid row id

Problem solving strategies sometimes find clear causes and answers and sometimes not, in this case analyzing the facts did show the way to find a workaround. It happened on a multi TB RAC Database. Symptoms observed were that queries on a partitioned table started to fail with ORA-1410 on some partitions.The initial diagnostic was of a logical corruption affecting one specific datafile, in this case a shared raw device. Some of these partitions were dropped and recreated...

Sunday, July 22, 2007 | Read More

Database location on the storage frame

Is the location of database files irrelevant these days when the capabilities of high end storage seems to address all possible issues? The answer is still yes according to a good friend I was working with the last two weeks. It is extremely relevant and the performance gaining of  following this rule is high, as high is the penalty for not following it. Every production environment needs to be built according to this standard:      The formula for database location on the...

Sunday, July 22, 2007 | Read More

How to setup raw devices on Linux when /dev/raw does not exist

Sometimes you need to configure raw devices for OCR, Votingdisk and ASM Spfile, but the directory /dev/raw does not exist. As root create the /dev/raw directory:     mkdir /dev/raw Then create the required raw devices using the following sintax: mknod raw<raw_dev_number> c 162  <raw_dev_number> i.e.: mknod raw1 c 162 1 For setting up 12 raw devices use a loop:#!/bin/ksh x=1 cd /dev/raw while [ $x -lt 12 ]; do  mknod raw$x c 162 $x     x=`expr $x + 1 ` doneOnce configured create...

Thursday, July 5, 2007 | Read More

RAC on ASM replication using an NFS mounted File System

One of the RAC environments I'm working with is a very critical production database, there are several development, test and training clones of it, and from time to time a new clone needs to be created. The main backup strategy we are using is based on EMC snapshots, so we can easily create clones from snapshots. Still sometimes you get not standard requirements, or find circumstances that require creative solutions. The special request was: Replicate the 64 bit prod database...

Thursday, July 5, 2007 | Read More

Corrupted Archived Log While Performing Recovery of a Crashed Instance

When there is no more than the actual backup, and the database serves a critical online production facility, and the facility virtually stops because of the problem, then the last thing you want to meet is  a corrupted archived log! It started early in the morning, a very top industry had its ERP database down, a colleague was all the night working on the case, but the database was still down... Facts were: Database was performing a hot backup, While being backed up crashed Start...

Tuesday, July 3, 2007 | Read More

After Production Upgrade Thoughts, or Think Twice Before Dropping a Big Table

On a previous post "Production Upgrade from to"  I  did describe the preparation steps for a critical production database upgrade, I was responsible for the rdbms upgrade, a little part of the whole project that included several OS and Application layers. In few words, the bottom idea on this post is to summarize some lessons learned during the upgrade. They are: When executing a critical process, that needs to be finished within a restricted time window: Build...

Tuesday, July 3, 2007 | Read More

Moving a datafile from File System to ASM

Moving datafiles from file system to ASM can be done in different ways, one of them using the RMAN copy command. These are the steps:1. Check where to build a new file system based tablespace: [oracle@rac1 ~]$ cd /u01/oradata/racdb [oracle@rac1 ~]$ df -k . Filesystem           1K-blocks      Used Available Use% Mounted on /dev/vx/dsk/u01                       31457280  17540576  13819976  56% /u01 2. Connect to sqlplus and create a new tablespace[oracle@rac1 racdb]$ sqlplus /...

Tuesday, June 19, 2007 | Read More

RAC Proactive Monitoring Using OS Watcher

RAC and single instance as well ... OS Watcher is a monitoring application, written by Carl Davis from Oracle Center of Expertise. And I think is wonderful! install in minutes and work straight away without any problems. Is the kind of application you may hold on your disk-on-key and install on a client site so that after a couple of hours you will be able to check and see if there is an OS related issue, where the bottleneck is located.OSW gather statistics using OS commands...

Tuesday, June 19, 2007 | Read More

Production Upgrade from to

Is not uncommon that very critical databases remain on obsolete versions longer than it is convenient. If the database provide good service and reasonable performance, the risks related to any radical change make the people that need to make the decision to upgrade, to delay it as much as possible.Sooner or later the day come when the increasing amount of problems, usually related to a big increment on the load and complexity of the applications running on the database raise...

Wednesday, June 13, 2007 | Read More

Linux and RAC with ASM install on Virtual Nodes, Step 2 of 5

We did install the Oracle Distribution of Enterprise Linux on the previous post. This second post, Step By Step Configure Linux for RAC on a Virtual Node contains the configuration steps required to finish the setup of the first node: Install and configure VMware tools Configure the Network Oracle User Configuration Steps for RAC on a Virtual Node General Linux Configuration Steps for RAC on a Virtual Node Storage Configuration Steps for RAC on a Virtual Node Once these steps are...

Tuesday, June 5, 2007 | Read More

Linux and RAC with ASM install on Virtual Nodes, Step 1 of 5

Installing Linux and RAC with ASM on the constrained environment of a desktop PC or Laptop is a very interesting learning experience for people interested on RAC that lacks the possibility to install and try RAC on real servers. My first trial was made using the excellent step by step guide by Vincent Chan, that was published on OTN. Since then I tried this process a couple of times and developed a guide divided into 5 chapters: Linux Install  Configure Linux for RAC Create and...

Monday, June 4, 2007 | Read More

Clusterware Install hang when running root.sh on second node

Lately I've seen this behavior  twice, one on HPUX and another on Linux: Oracle clusterware install went on smoothly and then hang when running root.sh on the 2nd node, while displaying this message: Startup will be queued to init within 90 seconds. There were no errors logged on the second server, truss or strace showed the processes waiting...On the first server we saw errors logged on $ORA_CRS_HOME/log/<servername>/alert<servername>.log pointing to a problem...

Monday, May 21, 2007 | Read More

Enterprise Manager Database Control Reconfiguration

Enterprise Manager Database Control is of great help for RAC environments administration. For production environments it is advisable to setup a Grid Control dedicated server and have on it the EM repository. For test and training purposes Database Control is excellent. This link has a Guide to re-configure EM using Enterprise Manager Configuration Assistant (emca) when some misconfiguration makes Database Control incomplete or unavailable, removing it and reconfiguriong it...

Thursday, May 17, 2007 | Read More

RAC with ASM on Linux, Crash Scenario: All Oracle Homes Loss

On this 6th scenario we lose all Oracle Homes on one node. On this environment Oracle Homes are installed on this path: /oradisk/app01/oracle/product These are the installed Oracle Homes:ASM Home RDBMS Home CRS Home The scenario is triggered using the following script:cd /oradisk/app01 rm -rf *  Recovery from such a loss is greatly simplified by having a good backup of the Oracle Homes.In many sites you will not find such a backup, and if reinstalling the software is not an...

Wednesday, May 16, 2007 | Read More

RAC with ASM on Linux, Crash Scenario: ASM Spfile Loss

On this 5th scenario we lose the ASM Spfile. The ASM spfile  in this case is located on a Raw Device, so backup and recovery from and to a Raw Device is explained. Backup and restore using create pfile from spfile and create spfile from pfile can also be used. In this case it will be necessary to shutdown the database and ASM instances, then start the ASM instance with the pfile and recreate the spfile. To simulate the ASM spfile loss wewrite zeros to it's raw device; because...

Monday, May 7, 2007 | Read More

RAC with ASM on Linux, Crash Scenario: Voting Disk Loss

On this 4th scenario we lose the Voting Disk. The Voting Disk in this case is located on a Raw Device, so backup and recovery from and to a Raw Device is explained. To simulate the Voting Disk loss we write zeros to the raw device, this will cause CRS to fail, the database to crash and the server to get stuck. The Voting Disk and the OCR can and should be multiplexed. To multiplex a voting disk execute as root: crsctl add css votedisk <path> The recovery steps include: Reboot both...

Monday, May 7, 2007 | Read More