Alejandro Vargas' Blog

RAC, ASM and Linux Forum, December 15, 13:30 - 18:00 Beit HP Raanana

2009-12-13T17:04:43Z

It's time for our 2nd, 2009 RAC, ASM and Linux Forum in Tel Aviv!

I would like to warmly invite you to our next RAC, ASM and Linux forum to be held at Beit HP in Raanana, on December 15.

You can register on the Israel Oracle User Group site.

On this forum we will have a 11g R2 Technology Update and 2 very interesting Customer Presentations about minimum downtime VLDB Migration to RAC on Linux and Creating and Managing RPM's for Oracle Silent Install on Linux.

Panel on Grid Infrastructure, ASM, Clusterware and RAC 11g R2, Technology Update

Annie Flint, Principal Member of Technical Staff, Oracle Corporation
Ofir Manor, Senior Sales Consultant, Oracle Israel
Alejandro Vargas, Principal Support Consultant, Oracle Advanced Customer Services

In the few months since last forum on June many things happened, 11g Release 2 is already production and brought a revolution in terms of performance and deep changes to the world of ASM, Oracle Clusterware and RAC.

Exadata Release 2 was released opening the way for OLTP databases based on the new Oracle - Sun Database Machine.

In this seminar we will review the new face of ASM, Oracle Clusterware and RAC on 11g Release 2 and we will comment on some of the incredible performance enhancements of the new version.

Migration of a VLDB to RAC 11g with Minimal Downtime

Dotan Mor,
Senior DBA
Pelephone DBA Team

Dotan will tell us the whole story of migrating an 8TB Datawarehouse, with near 0 downtime, from Linux 3 with OCFS2 to Linux 5, with CRS, RAC and ASM 11g, and Infiniband; and how his customer still cannot believe the incredible improvements they got in performance.

He will tell us also all the several problems faced in the way to this big success and how all of them were overcome.

Packaging Application and Database Together On Linux for Super-Silent-Installs

Liron Amitzi,
VP Professional Services
Brillix

Liron will explain us how to build a Linux RPM that contains inside the whole set of files required to easily and fast deploy a complete application, from the database to last application executable.

See you there!

Best Regards,

Alejandro

Check IO Scripts

2009-12-10T09:32:46Z

These scripts are very useful to check throughput.

The original version can be found on My Oracle Support Note 422414.1 by Luca Canali.

set lines 250 pages 50000

alter session set nls_date_format='dd-mm-yyyy hh24:mi';

col Phys_Read_Total_Bps for 999999999999
col Phys_Write_Total_Bps for 999999999999
col Redo_Bytes_per_sec for 999999999999
col Phys_Read_IOPS for 999999999999
col Phys_write_IOPS for 999999999999
col Phys_redo_IOPS for 999999999999
col OS_LOad for 999999999999
col DB_CPU_Usage_per_sec for 999999999999
col Host_CPU_util for 999999999999
col Network_bytes_per_sec for 999999999999
col Phys_IO_Tot_MBps for 999999999999
col Phys_IOPS_Tot for 999999999999

spool io_max_checkup.log

select min(begin_time), max(end_time),
sum(case metric_name when 'Physical Read Total Bytes Per Sec' then maxval end) Phys_Read_Tot_Bps,
sum(case metric_name when 'Physical Write Total Bytes Per Sec' then maxval end) Phys_Write_Tot_Bps,
sum(case metric_name when 'Redo Generated Per Sec' then maxval end) Redo_Bytes_per_sec,
sum(case metric_name when 'Physical Read Total IO Requests Per Sec' then maxval end) Phys_Read_IOPS,
sum(case metric_name when 'Physical Write Total IO Requests Per Sec' then maxval end) Phys_write_IOPS,
sum(case metric_name when 'Redo Writes Per Sec' then maxval end) Phys_redo_IOPS,
sum(case metric_name when 'Current OS Load' then maxval end) OS_LOad,
sum(case metric_name when 'CPU Usage Per Sec' then maxval end) DB_CPU_Usage_per_sec,
sum(case metric_name when 'Host CPU Utilization (%)' then maxval end) Host_CPU_util, --NOTE 100% = 1 loaded RAC node
sum(case metric_name when 'Network Traffic Volume Per Sec' then maxval end) Network_bytes_per_sec,
snap_id
from dba_hist_sysmetric_summary
group by snap_id
order by snap_id;

spool off

spool io_maxtot_summary.log

select min(begin_time), max(end_time),
sum(case metric_name when 'Physical Read Total Bytes Per Sec' then maxval end)/1024/1024 +
sum(case metric_name when 'Physical Write Total Bytes Per Sec' then maxval end)/1024/1024 +
sum(case metric_name when 'Redo Generated Per Sec' then maxval end)/1024/1024 Phys_IO_Tot_MBps,
sum(case metric_name when 'Physical Read Total IO Requests Per Sec' then maxval end) +
sum(case metric_name when 'Physical Write Total IO Requests Per Sec' then maxval end) +
sum(case metric_name when 'Redo Writes Per Sec' then maxval end) Phys_IOPS_Tot,
sum(case metric_name when 'Current OS Load' then maxval end) OS_LOad,
sum(case metric_name when 'CPU Usage Per Sec' then maxval end) DB_CPU_Usage_per_sec,
sum(case metric_name when 'Host CPU Utilization (%)' then maxval end) Host_CPU_util, --NOTE 100% = 1 loaded RAC node
sum(case metric_name when 'Network Traffic Volume Per Sec' then maxval end) Network_bytes_per_sec,
snap_id
from dba_hist_sysmetric_summary
group by snap_id
order by snap_id;

spool off

RAC install root.sh fail on 2nd Node with error: "Timed out waiting for the CRS stack to start"

2009-12-01T10:51:05Z

Yesterday we spent the day trial to solve this problem.

The actual solution, once the problem was found, took a couple of minutes to solve.

Usually this issue will arise when there is a network misconfiguration. We did check that carefully and everything was well configured:

- Same device names at both ends
- No dropped packets or transmission errors
- Both sides configured full duplex

Then we decided to move the install software to the second node and used scp over the interconnect:

scp -r -priv:

scp progressed for a while and then become stalled.

Retrying using the public network worked without any problem.

Any other test using the interconnect got stalled after a while.

The network people checked what the problem can be and found that the switch was configured half duplex.

They corrected the problem and everything worked fine.

Oracle Support Note 745215.1 describe the problem and other areas to check as well.

ASM Hands-On Training, Server Enviroment Setup And Aliases

2009-09-12T18:37:41Z

The server where the labs of the ASM Hands-On Training works is configured with tcsh as the default oracle user shell.

A .cshrc file containing several aliases was setup to easy moving around and executing repetitive commands.

On this document there is a general explanation of how the environment can be used.

Details on this file: Server Enviroment Setup And Aliases

ASM Hands-On Training, Lab 21, ASMCMD Use And Options

2009-09-12T18:27:57Z

On this lab we will review some of the useful set of commands provided by the ASM command line utility.

Some of the asmcmd commands display information, these information is based on v$asm views, other commands actually make changes to the structure like mkdir or rm.

Details on file: ASMCMD Use And Options

ASM Hands-On Training, Lab 20, Storage Consolidation With ASM

2009-09-12T18:22:30Z

On this lab we will share our ASM disks with a second server and we will open the sati12 database on it.

To do that we copied the vm to another location, without including the ASM disks, that are located on a separate folder, this way the second vm is pointing to the same ASM disks as the first one.

Details on file: Storage Consolidation With ASM

ASM Hands-On Training, Lab18, ASM Metadata and Other Checkups

2009-09-12T18:01:47Z

On this lab we will review various utilities that provide further information for managing ASM.

Checksum Mismatch After Storage Crash, AMDU an 11g tool that can be used with 10g as well, Blockdumps, Asmiostats and asmdebug

Details on this file: ASM Metadata and Other Checkups

ASM Hands-On Training, Lab 17, Measuring Throughput

2009-09-12T16:55:15Z

On this lab we will review simple methods for getting input about the the database througput and response time.

We will produce the load using swingbench and we will gather AWR snapshots every 10 minutes. After some time we will be able to check througput statistics based on the AWR snapshots information.

Details on this Document: Measuring Throughput

Another useful script for checking IO:

set pages 50000 lines 250
alter session set nls_date_format='dd-mm-yy hh24:mi';
spool chkio1-Physical-rw.log
select min(begin_time), max(end_time),
sum(case metric_name when 'Physical Read Total Bytes Per Sec' then average end) Physical_Read_Total_Bps,
sum(case metric_name when 'Physical Write Total Bytes Per Sec' then average end) Physical_Write_Total_Bps,
snap_id
from dba_hist_sysmetric_summary
group by snap_id
order by snap_id;
spool off

select min(begin_time), max(end_time),
sum(case metric_name when 'Physical Read Total Bytes Per Sec' then maxval end) Physical_Read_Total_Bps,
sum(case metric_name when 'Physical Write Total Bytes Per Sec' then maxval end) Physical_Write_Total_Bps,
sum(case metric_name when 'Physical Read Total Bytes Per Sec' then maxval end) +
sum(case metric_name when 'Physical Write Total Bytes Per Sec' then maxval end) Total_IO,
snap_id
from dba_hist_sysmetric_summary
group by snap_id
order by snap_id
/

ASM Hands-On Training, Lab 16, Configuring And Running Swingbench And OSWatcher

2009-09-12T14:02:37Z

Whenever implementing new environments it is convenient to run stress tests and to gather OS statistics that will provide valuable diagnostics information.

Swingbench provide an easy configurable tool to run oltp or dss like stress tests; that is very useful when the customer does not have any possibility to benchmark the new configuration.

Oswatcher will collect statistics from the OS in a cyclic 2-day period by default, which can be extended to whatever period is convenient for you.

Details on this file:
Lab16-Configure-SwingBench-and-OsWatcher.pdf

ASM Hands-On Training, Lab 15, Cloning A Disk Group

2009-09-12T13:16:55Z

Sometimes it is required having several clones of the same database running on the same server. That was simple to do when the database was based on File System, and is still simple to do with the help of Rman.

But there are very large databases, which are usually cloned using storage tools like bcv, srdf or snapshots that cannot consider Rman as a viable possibility.

For these databases there is the possibility to implement the procedure we will test on this lab.

Note that the rename disk group script has not been made public.
The rename disk group functionality is available on 11g R2 that was made available to the general public on August 2009.

These are the steps required to complete the process:

1. Prepare a set of storage devices with enough space to hold a copy of the diskgroup to be copied
2. Shutdown the source database
3. Make a bit by bit copy of the source database on the target devices
4. Add kfed path to the path of user root
5. Execute the rename diskgroup script
6. Rename ASM disks of the cloned devices
7. Rescan ASM disks
8. Start the ASM instance and mount the new diskgroup
9. Recreate the controlfile and rename the database based on the cloned diskgroup
10.Open the clone with the resetlogs option

Details on this file:
Cloning Disk Group

ASM Hands-On Training, Lab 14, 11g Compatibility Parameters and Resilience Test

2009-09-12T10:48:37Z

On this lab we will make a review of the 11g Compatibility Parameters and 3 Resilience Tests

The compatibility parameters compatible.asm and compatible.rdbms define the minimum ASM and database versions that will be able to connect to a disk group. These parameters can be advanced only

On the resilience probe we will do the following tests

· Add ASM disks and bring down the ASM instance on the middle of the operation
· Remove ASM disks and bring down the ASM instance on the middle of the operation
· Destroy the ASM metadata and restore the disk

Details on this file:
11g Compatibility Parameters and Resilience Test

ASM Hands-On Training, Lab 13, ASM And Rman: Crash, Restore And Recovery Scenarios

2009-09-12T10:39:46Z

On this Lab we will review the following crash and recovery scenarios, and we will implement some of them:

· System tablespace loss
· Recover system tablespace
· User datafile loss
· Recover users tablespace
· Online redo loss
· Recover from redo loss
· Controlfile loss
· Recover from controlfile loss
· Database loss
· Recover from total loss

Details on this file
ASM and Rman Crash Restore and Recovery Scenarios

ASM Hands-On Training, Labs 11 and 12 : Restoring A Lost Database From Backup And Cleanup File Based ASM Diskgroups

2009-09-12T08:41:07Z

To work on the 11g part of the Labs we will start restoring an 11g database we have on a backup.
The backup consist of an incremental level 0 backupset, a controlfile and spfile backup and the archived logs required to recover the database.

The restore is done in 6 steps
· Re-create the ASM metada
· Restore the spfile
· Restore the controlfile
· Restore the datafiles
· Recover the database
· Open resetlogs

This is covered on this file:
Restoring a Lost Database From Backup

On the previous Labs we created ASM disks based on OS Files, before continuing with the next labs we need to clear them.

This is explained on this file:
Cleanup File based ASM Diskgroups

ASM Hands-On Training, Lab 10, Migrating From 10g ASM To 11g

2009-09-12T07:57:47Z

In this Lab we will migrate our 10g Disk Groups to 11g.

The first steps of the migration, Installing 11.1.0.6 on it's own Oracle Home and applying patch 11.1.0.7 are ready on this environment, so we can go ahead with the migration process.

In this lab we will migrate only ASM, the database migration is a little longer for the time frame we have so we will continue by moving to ASM an 11g database that already exist on this server.

Details on this document:
Migrating 10g ASM to 11g

ASM Hands-On Training, Lab 9, Migrating Raw And Block Devices To ASMLib

2009-09-12T07:50:34Z

ASMLib is best practice for managing ASM on Linux, it may happen that a database server that was configured a couple of years ago with ASM 10g R1 is still using raw devices, or that a new ASM 10g R2 or even 11g was configured using block devices instead of ASMLib.

ASMLib provides global open close call that will improve performance and transparently implement device labeling, an important aspect when managing storage on Linux.

This lab explain the procedure to migrate these devices to ASMLib.

Details on this document:
Migrating Raw and Block Devices to ASMLib