If you are interested to join the session, please see this link.

Rene Kundersma
Oracle Technology Services, The Netherlands

The presentation can be downloaded here

Rene Kundersma
Oracle Technology Services, The Netherlands

As an Oracle Consultant I feel very privileged to work with these great new products from the beginning !

Besides faster hardware and Flash Cache, Database Machine v2 is even more brainier then v1.
We now have Hybrid Columnar Compression, Storage Indexing and in-memory-parallel queries.

Please visit this url for more information or join my sessions at planboard.

If you want to find out the Sun Oracle Database Machine can help you in consolidating or killing performance problems, please contact me at Oracle Technology Services.

Rene Kundersma
Oracle Technology Services, The Netherlands

For me this is a great opportunity to tell about DBM v1 & v2.

Rene Kundersma
Oracle Technology Services, The Netherlands

I will however try to discuss some highlights or cool new things that changed since the previous (11gR1) release. 11gR2 Grid Infrastructure is one of those things.

11gR2 Grid Infrastructure combines Clusterware and ASM in one Oracle home and can be described as the next step in Grid Computing. If you are familiar with previous Clusterware and ASM releases, you will recognize the new functionality and way of working and realize this is indeed the next step in what we need for enabling Enterprise Grid. Deployment is simpler, faster and we are not talking about nodes anymore, but about services that live on resources.

One of the new features of 11gR2 is Grid Plug and Play, also called GPnP. Let me repeat what the documentation says about GPnP:

"Grid Plug and Play (GPnP) eliminates per-node configuration data and the need for explicit add and delete nodes steps. This allows a system administrator to take a template system image and run it on a new node with no further configuration. This removes many manual operations, reduces the opportunity for errors, and encourages configurations that can be changed easily. Removal of the per-node configuration makes the nodes easier to replace, because they do not need to contain individually-managed state.

Grid Plug and Play reduces the cost of installing, configuring, and managing database nodes by making their per-node state disposable. It allows nodes to be easily replaced with regenerated state"

Some of the key enablers for GPnP are GNS and DHCP. GNS, the Grid Naming Service is described here.

Since all of the requirements for a Grid Infrastructure installation are clearly documented in the "Grid Infrastructure installation guide", there is no need to discuss this.

This posting however is made to demo how to do an "Advanced Installation" of the Grid Infrastructure your self and show how to do an installation for education purposes, for example a situation at home where you want to test the setup of Oracle Grid Infrastructure with your own DNS and DHCP server. In real life, at customer sites, DNS and DHCP servers are all in place and Oracle Grid Infrastructure can leverage from these existing services.

Since most steps of the Oracle Grid Infrastructure installation are easy I will only only focus on the details I want to discuss regarding GNS and DHCP.

Oracle Grid Infrastructure can be downloaded here and when you made sure all prerequisites are checked you can start the installation by executing runInstaller.

It does makes sense to install the Oracle Grid Infrastructure with a different user id then the Oracle database. For this the Oracle documentation again has some sound examples. Because of this I had to make sure permissions for directories and for example ASM disks are setup with 'grid' permissions instead of 'oracle' (and both oinstall as group)

I used user "grid" to install the Oracle Grid Infrastructure and since I wanted to install and configure the Oracle Grid Infrastructure I chose the first option.

A typical installation does not have GNS and since the purpose of the posting is to explain about the setup with GNS, the "Advanced Installation" option was chosen.

Language, speaks for itself.

Okay, this basically is the most important step of the setup. At this step you have to define the name for your cluster. In my case "cluster01", that is an easy one as there are no relations for this.

The SCAN name however, is the "Simple Client Access Name" and will be setup by the Oracle Grid Infrastructure. This SCAN name will resolve to three ip addresses within the cluster. The good news is that you don't have to do much for it, just make up a name that your clients will use later to acces databases in your cluster. SCAN Port 1521 is the default port we always use for SQLNet. The SCAN name has to be in the GNS Sub Domain as explained below:

The option "Configure GNS" was checked. If this box was not checked, still SCAN could be used, but then, I had to setup the SCAN entries in DNS myself, with the SCAN name resolving to three different ip addresses.

However, since GNS is checked, the Grid Naming Service will be configured and GNS will setup my SCAN name. The only requirement is that a GNS Sub domain must be made and the DNS must be configured so that each request for this Sub Domain will be delegated to the GNS Sub Domain, so that GNS can handle the request.

The GNS VIP address is the ip address of the server that will host the GNS. You need to make sure this one is available for use.

You may ask yourself why this all is needed. Well, imagine yourself a cluster, where nodes are added and removed dynamically. In this situation, the complete administration with ip address management and name resolution management is done by the cluster itself. No need to do any manual work in updating connection strings, configuring ip numbers etc.

So how does it work:

First, my (named, linux) DNS is running on 10.161.102.40.
This DNS does the naming for cluster01.nl.oracle.com and pts.local.
For cluster01.nl.oracle.com a "delegation" is made, so that every request to a machine in the domain .cluster01.nl.oracle.com is delegated to the GNS. (with the GNS VIP).

In DNS:

cluster01.nl.oracle.com NS gns.cluster01.nl.oracle.com
gns.cluster01.nl.oracle.com. 10.161.102.55

So, once the cluster installation is done, the GNS in the cluster will be stared and a request to scan.cluster01.nl.oracle.com will be forwarded to the GNS. The GNS will then take care of the request and answer which three nodes in the cluster will serve as scan listeners:

[root@gridnode01pts05 ~]# nslookup scan.cluster01.nl.oracle.com
Server:         10.161.102.40
Address:        10.161.102.40#53
Non-authoritative answer:
Name:   scan.cluster01.nl.oracle.com
Address: 10.161.102.78
Name:   scan.cluster01.nl.oracle.com
Address: 10.161.102.79
Name:   scan.cluster01.nl.oracle.com
Address: 10.161.102.77

Also, with dig, you can see all information coming from GNS:

[root@dns-dhcp ~]# dig scan.cluster01.nl.oracle.com
; <<>> DiG 9.3.4-P1 <<>> scan.cluster01.nl.oracle.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46016
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 10, ADDITIONAL: 10
;; QUESTION SECTION:
;scan.cluster01.nl.oracle.com.  IN      A
;; ANSWER SECTION:
scan.cluster01.nl.oracle.com. 6 IN      A       10.161.102.78
scan.cluster01.nl.oracle.com. 6 IN      A       10.161.102.79
scan.cluster01.nl.oracle.com. 6 IN      A       10.161.102.77
;; AUTHORITY SECTION:
oracle.com.             10732   IN      NS      dns2.us.oracle.com.
oracle.com.             10732   IN      NS      dns3.us.oracle.com.
oracle.com.             10732   IN      NS      dns4.us.oracle.com.
oracle.com.             10732   IN      NS      dns1-us.us.oracle.com.
oracle.com.             10732   IN      NS      dnsmaster1.oracle.com.
oracle.com.             10732   IN      NS      dnsmaster2.oracle.com.
oracle.com.             10732   IN      NS      dnsmaster3.oracle.com.
oracle.com.             10732   IN      NS      dnsmaster4.oracle.com.
oracle.com.             10732   IN      NS      dnsmaster5.oracle.com.
oracle.com.             10732   IN      NS      dnsmaster6.oracle.com.
;; ADDITIONAL SECTION:
dns2.us.oracle.com.     3984    IN      A       130.35.249.52
dns3.us.oracle.com.     3984    IN      A       144.20.190.70
dns4.us.oracle.com.     3984    IN      A       138.2.202.15
dns1-us.us.oracle.com.  3984    IN      A       130.35.249.41
dnsmaster1.oracle.com.  1060    IN      A       192.135.82.4
dnsmaster2.oracle.com.  1060    IN      A       192.135.82.20
dnsmaster3.oracle.com.  1060    IN      A       192.135.82.36
dnsmaster4.oracle.com.  1060    IN      A       192.135.82.52
dnsmaster5.oracle.com.  1060    IN      A       192.135.82.70
dnsmaster6.oracle.com.  1060    IN      A       192.135.82.84
;; Query time: 0 msec
;; SERVER: 10.161.102.40#53(10.161.102.40)
;; WHEN: Sat Sep 19 17:15:47 2009
;; MSG SIZE  rcvd: 486

Later, when the database is installed, you can use the SCAN with SQLNet EZ connect to connect to the database. Can't wait, I just have to demo it now:

[oracle@gridnode01pts05 ~]$ sqlplus system/oracle@scan.cluster01.nl.oracle.com:1521/dbpts05.pts.local
SQL*Plus: Release 11.2.0.1.0 Production on Sat Sep 19 17:11:32 2009
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> 

After specification of the GNS details the details of the nodes within the cluster have to be entered. The node names you enter here also have to be able to resolve their name. The management of the virtual ip address will be done automatically as long as a working DHCP service is available to serve ip addresses within that network. You can see here that the nodes can be in another domain then the GNS Sub Domain.

Click here for my dhcp config.

New in 11gR2 is the ability to let the installer configure ssh for you ! Great !

Even if it says that it will take several minutes, most of the time the ssh setup is done within the minute.

As in 10g, and 11gR1, this step is used to specify the public and internal interface.
Same as in 10g and 11gR1, the private interface will be used for the interconnect.

11gR2 has the new option to place the OCR and Voting disks on ASM storage, so that is what I will do.

After choosing for ASM, the next step is specifying disks that will be used for the ASM diskgroup. This is also kind-of 10g/11gR1, however, at that time this step was in the DBCA.

Choosing 6 disks of 2GB with external redundancy.

You see the installer complaining about my not so complicated password that I chose. (since this not for production purposes)

Intelligent Planform Management is really cool, look at this. You choose to let the Grid Infrastructure work with it.

In the real world it does makes sense to separate the three groups !

Screen to specify the Oracle base and Software Location for the Grid Infrastructure. Remember, this location has to extist (read and write) on all nodes that you plan to install the software on.

Location of the inventory

This is really great. The Prerequisite checker now has the ability to generate a "fix" script.
So some (not all) requirements that are not setup okay can be corrected by running a fix-up script generated by the prerequisite checker.

In my situation, some kernel parameters needed to be changed. That could be done easily with this tool.

Run the script on both nodes.

My swap space seems to be 1KB too small. This is because I used an Oracle VM template (yes, although not officially certified yet I am running virtualized). I think I will manage with 1KB less then recommend.

Summary:

Progress:

Software being transferred to the other node:

And running the root scripts to setup permissions and configure the cluster.

Running oraInstRoot.sh on both nodes:

Running root.sh on node 1, this is where the cluster configuration is done.

Watch the Voting File on the first ASM disk in diskgroup DATA:

And root.sh on node 1 finished...

Don't forget node 2:

After running the root scripts, you have to click on okay. After this some small post configuration steps are performed by the installer and clufvy also runs. If this all runs fine, like it did here the screen will quickly continue to the last page telling you the installation was succesful

In my next postings i will show the installation and creation of a 11gR2 RAC database, and also adding nodes to the cluster and adding instances to the RAC database.

Rene Kundersma
Oracle Technology Services, The Netherlands

The mentioned pdf describes how to make sure a single instance database can failover with the use of Oracle Clusterware.

With this posting I want to show how to do this for a single instance RAC database where you have to manage instance1' and 'instance2' instead of just one instance.

Since the grid control agent may have some problems with an instance[number] travelling from node1 to node2 this choice was made on purpose for some customer. Other reason for this awkward solution is that the application for some reason cannot run with two instances concurrently.

Also, the fact that instance deployment with sequential instance numbers is standard in their grid environment the choice has been made to do this with different instance numbers instead of one.

Within this posting I will also show the required updates in tnsnames and spfile.

First, I made sure the OCR entries of the RAC database, the Services and the instances are removed from the OCR. Then a "resource group" will be created. This is the container for all the resources.

oracle@pts0138([crs]):/ora/product/11.1.0/crs> crs_profile -create \
oss.xdbprk.rg -t application -a \
/ora/product/11.1.0/crs/crs/public/act_resgroup.pl -o "ci=600" 
oracle@pts0138([crs]):/ora/product/11.1.0/crs> crs_register oss.xdbprk.rg

Now, let's verify the new entry:

oracle@pts0138([crs]):/ora/product/11.1.0/crs/crs/public> crsstat | grep rg
HA Resource                                   Target     State
oss.xdbprk.rg                                   OFFLINE    OFFLINE

The scripts mentioned in the pdf are placed in $CRS_HOME/crs/public on both nodes. I made sure the scripts are executable and tested them:

export CLUSTERWARE_HOME=/ora/product/11.1.0/crs/
export ORACLE_HOME=/ora/product/10.2.0/db_2
export _USR_ORA_LANG=$ORACLE_HOME 
export _USR_ORA_SRV=xdbprk2 
export _USR_ORA_FLAGS=1 
oracle@pts0138(xdbprk2):/tmp> $CLUSTERWARE_HOME/crs/public/act_db.pl start
SQL*Plus: Release 10.2.0.4.0 - Production on Fri Sep 18 10:12:21 2009
Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.
SQL> Connected to an idle instance.
SQL> ORACLE instance started.
Total System Global Area  536870912 bytes
Fixed Size                  2085360 bytes
Variable Size             150998544 bytes
Database Buffers          377487360 bytes
Redo Buffers                6299648 bytes
Database mounted.
Database opened.
SQL> Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,
Data Mining and Real Application Testing options

Also tested the stop function.

oracle@pts0138(xdbprk2):/tmp> $CLUSTERWARE_HOME/crs/public/act_db.pl stop
SQL*Plus: Release 10.2.0.4.0 - Production on Fri Sep 18 10:12:37 2009
Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.
SQL> Connected.
SQL> Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,
Data Mining and Real Application Testing options
oracle@pts0138(xdbprk2):/ora/product/11.1.0/crs/crs/public>

The action above was executed on both nodes. Just to verify if the instance could be started/stoped with the scripts. Only problem was: for each node, the ORACLE_SID had to be changed, as both nodes have another ORACLE_SID for that database. With only one SID you would not have the problem.

Then, the failover resource was created and registered.

oracle@pts0138([crs]):/tmp> crs_profile -create oss.xdbprk.db-cold-failover \
-t application -r oss.xdbprk.rg -a
 /ora/product/11.1.0/crs/crs/public/act_db.pl 
-o "ci=20,ra=5,osrv=xdbprk,ol=/ora/product/10.2.0/db_2,oflags=1,rt=600"
oracle@pts0138([crs]):/tmp> crs_register oss.xdbprk.db-cold-failover

The value osrv=xdbprk will never work as this is not the correct instance name for any of the nodes. Even if I made the value osrv=xdbprk1, then the script would only work on one of the nodes i.e. the node that had the appropriate init.ora etc.

So, leaving the value osrv=xdbprk to this, I actually hard-coded the ORACLE_SID on both sides of the cluster in the act_db.pl. Since the value is now hard coded, it should work. This clearly limits the option to re-use the script for other database, so I'd better change the name of the script to act_db_xdbprk.pl if I do this for real.

So, how will the resource profiles look now ?

oracle@pts0101([crs]):/var/opt/oracle> crs_profile -print oss.xdbprk.rg
NAME=oss.xdbprk.rg
TYPE=application
ACTION_SCRIPT=/ora/product/11.1.0/crs/crs/public/act_resgroup.pl
ACTIVE_PLACEMENT=0
AUTO_START=restore
CHECK_INTERVAL=600
DESCRIPTION=oss.xdbprk.rg
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=
OPTIONAL_RESOURCES=
PLACEMENT=balanced
REQUIRED_RESOURCES=
RESTART_ATTEMPTS=1
SCRIPT_TIMEOUT=60
START_TIMEOUT=0
STOP_TIMEOUT=0
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/ as sysdba
USR_ORA_DEBUG=0
USR_ORA_DISCONNECT=false
USR_ORA_FLAGS=
USR_ORA_IF=
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG=
USR_ORA_NETMASK=
USR_ORA_OPEN_MODE=
USR_ORA_OPI=false
USR_ORA_PFILE=
USR_ORA_PRECONNECT=none
USR_ORA_SRV=
USR_ORA_START_TIMEOUT=0
USR_ORA_STOP_MODE=immediate
USR_ORA_STOP_TIMEOUT=0
USR_ORA_VIP=

oracle@pts0101([crs]):/var/opt/oracle> crs_profile -print  oss.xdbprk.db-cold-failover
NAME=oss.xdbprk.db-cold-failover
TYPE=application
ACTION_SCRIPT=/ora/product/11.1.0/crs/crs/public/act_db.pl
ACTIVE_PLACEMENT=0
AUTO_START=restore
CHECK_INTERVAL=20
DESCRIPTION=oss.xdbprk.db-cold-failover
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=
OPTIONAL_RESOURCES=
PLACEMENT=balanced
REQUIRED_RESOURCES=oss.xdbprk.rg
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=60
START_TIMEOUT=600
STOP_TIMEOUT=0
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/ as sysdba
USR_ORA_DEBUG=0
USR_ORA_DISCONNECT=false
USR_ORA_FLAGS=1
USR_ORA_IF=
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG=/ora/product/10.2.0/db_2
USR_ORA_NETMASK=
USR_ORA_OPEN_MODE=
USR_ORA_OPI=false
USR_ORA_PFILE=
USR_ORA_PRECONNECT=none
USR_ORA_SRV=xdbprk
USR_ORA_START_TIMEOUT=0
USR_ORA_STOP_MODE=immediate
USR_ORA_STOP_TIMEOUT=0
USR_ORA_VIP=

Okay, and then the basic test of starting the resource, first all is down:

oracle@pts0101([crs]):/var/opt/oracle> crsstat
HA Resource                                   Target     State
-----------                                   ------     -----
oss.xdbprk.db-cold-failover                     OFFLINE    OFFLINE
oss.xdbprk.rg                                   OFFLINE    OFFLINE

Then, the start:

oracle@pts0101([crs]):/var/opt/oracle> crs_start  oss.xdbprk.db-cold-failover
Attempting to start `oss.xdbprk.db-cold-failover` on member `pts0138`
Start of `oss.xdbprk.db-cold-failover` on member `pts0138` succeeded.
oracle@pts0138([crs]):/var/opt/oracle> ps -ef | grep smon | grep dbprk
oracle   31568     1  0 11:51 ?        00:00:00 ora_smon_xdbprk2

And the relocate:

oracle@pts0101([crs]):/var/opt/oracle> crs_relocate -f oss.xdbprk.db-cold-failover
Attempting to stop `oss.xdbprk.db-cold-failover` on member `pts0138`
Stop of `oss.xdbprk.db-cold-failover` on member `pts0138` succeeded.
Attempting to stop `oss.xdbprk.rg` on member `pts0138`
Stop of `oss.xdbprk.rg` on member `pts0138` succeeded.
Attempting to start `oss.xdbprk.rg` on member `pts0101`
Start of `oss.xdbprk.rg` on member `pts0101` succeeded.
Attempting to start `oss.xdbprk.db-cold-failover` on member `pts0101`
Start of `oss.xdbprk.db-cold-failover` on member `pts0101` succeeded.

Let's verify if it runs on the other node:

oracle@pts0101([crs]):/var/opt/oracle> ps -ef | grep smon | grep dbprk
oracle   11568     1  0 11:55 ?        00:00:00 ora_smon_xdbprk1

And stopped on the original:

oracle@pts0138(xdbprk2):/ora/product/10.2.0/db_2/dbs> ps -ef | grep smon | grep dbprk

Okay, what is left to do from here:

Service names that used to be managed by CRS, now have to be coded hard in the spfile so that they register each time with the listener:

service_names in spfile added:

dbprk.xe.grid
dbprk.xe.supp
dbprk.xe.link

Also, for each node, another local listener will be used. I made sure this is in the spfile:

xdbprk1.local_listener='pts0101-LOCAL-LISTENER'
xdbprk2.local_listener='pts0138-LOCAL-LISTENER'

In order to be sure there can only be started one instance the cluster_database_instances_parameter is set to 1.

*.cluster_database_instances=1

For tnsnames a normail failover entry is created, if the first instance is down, the next will be found (and should be running):

oracle@pts0101(xdbprk1):/ora/dbprk/admin/xdbprk1/bdump> tnsping dbprk.xe.grid
TNS Ping Utility for Linux: Version 10.2.0.4.0 - Production on 18-SEP-2009 12:39:14
Copyright (c) 1997,  2007, Oracle.  All rights reserved.
Used parameter files:
/etc/oss/sqlnet.ora
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=pts0101-grid.nl.eu.abnamro.com)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=pts0138-grid.nl.eu.abnamro.com)(PORT=1521))(LOAD_BALANCE=ON))(CONNECT_DATA=(SERVICE_NAME=dbprk.xe.grid)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=20))))
OK (10 msec)

So, another basic test, what is the situation:

HA Resource                                   Target     State
-----------                                   ------     -----
oss.xdbprk.db-cold-failover                     ONLINE     ONLINE on pts0101
oss.xdbprk.rg                                   ONLINE     ONLINE on pts0101
oracle@pts0101([crs]):/ora/dbprk/admin/xdbprk1/bdump> crs_start  oss.xdbprk.db-cold-failover
Attempting to start `oss.xdbprk.db-cold-failover` on member `pts0101`
Start of `oss.xdbprk.db-cold-failover` on member `pts0101` succeeded.

In which instance will my session end ?

oracle@pts0101(xdbprk1):/ora/dbprk/admin/xdbprk1/bdump> sqlplus rk/rk@dbprk.xe.grid
SQL*Plus: Release 10.2.0.4.0 - Production on Fri Sep 18 12:39:55 2009
Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,
Data Mining and Real Application Testing options
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
xdbprk1

And after the relocate, the session should go to the other instance:

oracle@pts0101([crs]):/ora/dbprk/admin/xdbprk1/bdump> crs_relocate -f oss.xdbprk.db-cold-failover
Attempting to stop `oss.xdbprk.db-cold-failover` on member `pts0101`
Stop of `oss.xdbprk.db-cold-failover` on member `pts0101` succeeded.
Attempting to stop `oss.xdbprk.rg` on member `pts0101`
Stop of `oss.xdbprk.rg` on member `pts0101` succeeded.
Attempting to start `oss.xdbprk.rg` on member `pts0138`
Start of `oss.xdbprk.rg` on member `pts0138` succeeded.
Attempting to start `oss.xdbprk.db-cold-failover` on member `pts0138`
Start of `oss.xdbprk.db-cold-failover` on member `pts0138` succeeded.
oracle@pts0101([crs]):/ora/dbprk/admin/xdbprk1/bdump> crsstat
HA Resource                                   Target     State
-----------                                   ------     -----
oss.xdbprk.db-cold-failover                     ONLINE     ONLINE on pts0138
oss.xdbprk.rg                                   ONLINE     ONLINE on pts0138

oracle@pts0101(xdbprk1):/ora/dbprk/admin/xdbprk1/bdump> sqlplus rk/rk@dbprk.xe.grid
SQL*Plus: Release 10.2.0.4.0 - Production on Fri Sep 18 12:41:22 2009
Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,
Data Mining and Real Application Testing options
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
xdbprk2

As an extra test, to make sure the two instances cannot be started concurrently, started with a test to start the second instance after starting the first.
As you can see this is not possible.

oracle@pts0101(*):/var/opt/oracle> db xdbprk1
ORACLE_SID=xdbprk1
ORACLE_HOME=/ora/product/10.2.0/db_2
oracle@pts0101(xdbprk1):/var/opt/oracle> sqlplus / as sysdba
SQL*Plus: Release 10.2.0.4.0 - Production on Fri Sep 18 12:42:02 2009
Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.
Connected to an idle instance.
SQL> startup;
ORACLE instance started.
Total System Global Area  536870912 bytes
Fixed Size                  2085360 bytes
Variable Size             331353616 bytes
Database Buffers          197132288 bytes
Redo Buffers                6299648 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced

Rene Kundersma
Oracle Technology Services, The Netherlands

So how would one manage to mount that then ?

First, setup the loop, to see what's in it.

[root@pts05 ] losetup /dev/loop99  /OVS/running_pool/gridnode01/System.img
[root@pts05 ]# fdisk -lu /dev/loop99
Disk /dev/loop99: 6530 MB, 6530871808 bytes
255 heads, 63 sectors/track, 793 cylinders, total 12755609 sectors
Units = sectors of 1 * 512 = 512 bytes
       Device Boot      Start         End      Blocks   Id  System
/dev/loop99p1   *          63       64259       32098+  83  Linux
/dev/loop99p2           64260     8562644     4249192+  83  Linux
/dev/loop99p3         8562645    12739544     2088450   82  Linux swap / Solaris

So, three plain partitions, no logical volumes in it.

What to do next ? it seems NOT possible to inform the kernel about that three partitions:

[root@pts05 p]# partprobe -s /dev/loop99
Error: Error informing the kernel about modifications to partition /dev/loop99p1 -- Invalid argument.  This means Linux won't know about any changes you made to /dev/loop99p1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.
Error: Error informing the kernel about modifications to partition /dev/loop99p2 -- Invalid argument.  This means Linux won't know about any changes you made to /dev/loop99p2 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.
Error: Error informing the kernel about modifications to partition /dev/loop99p3 -- Invalid argument.  This means Linux won't know about any changes you made to /dev/loop99p3 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.
Warning: The kernel was unable to re-read the partition table on /dev/loop99 (Invalid argument).  This means Linux won't know anything about the modifications you made until you reboot.  You should reboot your computer before doing anything with /dev/loop99.

Okay, since I don't want to reboot, this is not an option.Maybe we need to setup the loop, but with an offset to the partition I need.

Since the sector size is 512 bytes and I start at 64260, my offset will be: 512 * 64260 = 32901120. Why do I start at 64260, since I guess partition one is the /boot partition.

[root@pts05 tools]# losetup /dev/loop98  /OVS/running_pool/gridnode01/System.img -o 32901120
[root@pts05 tools]# losetup -a | grep offset
/dev/loop98: [0811]:6127628 (/OVS/running_pool/gridnode01/System.img), offset 32901120

Yes, there it is ! Mounting is easy now !

[root@pts05 p]# mkdir /tmp/p; mount /dev/loop98 /tmp/p/
[root@pts05 p]# df -m | loop98
/dev/loop98               4013      2071      1901  53% /tmp/p
[root@pts05 p]# ls -l /tmp/p/
total 196
drwxr-xr-x  2 root root  4096 Sep  3 00:03 bin
drwxr-xr-x  2 root root  4096 Mar 25 05:44 boot
drwxr-xr-x  2 root root  4096 Sep  4 09:58 crs
drwxr-xr-x  2 root root  4096 Apr 10 07:14 dev
drwxr-xr-x 96 root root 12288 Sep 10 07:03 etc
drwxr-xr-x  4 root root  4096 Sep  8 21:30 home
drwxr-xr-x 13 root root  4096 Sep  3 00:03 lib
drwx------  2 root root 16384 Mar 25 05:44 lost+found
drwxr-xr-x  2 root root  4096 Jan  9  2009 media
drwxr-xr-x  2 root root  4096 Jan 21  2009 misc
drwxr-xr-x  3 root root  4096 Sep  4 09:40 mnt
dr-xr-xr-x  2 root root  4096 Sep  8 21:22 net
drwxr-xr-x  4 root root  4096 Sep  9 22:03 opt
drwxr-xr-x  2 root root  4096 Mar 25 05:44 proc
drwxr-x---  2 root root  4096 Sep  8 22:30 root
drwxr-xr-x  2 root root 12288 Sep  9 22:05 sbin
drwxr-xr-x  2 root root  4096 Mar 25 05:44 selinux
drwxr-xr-x  2 root root  4096 Jan  9  2009 srv
drwxr-xr-x  2 root root  4096 Mar 25 05:44 sys
drwxr-xr-x  3 root root  4096 Apr 10 07:03 tftpboot
drwxrwxrwt 17 root root  4096 Sep 10 14:57 tmp
drwxr-xr-x  2 root root  4096 Sep  2 22:13 u01
drwxr-xr-x 14 root root  4096 Apr 10 07:03 usr
drwxr-xr-x 21 root root  4096 Apr 10 07:05 var

Rene Kundersma
Oracle Technology Services, The Netherlands

To begin a short description: the picture below shows us Oracle VM can run both hardware and para-virtualized guests. Hardware virtualization uses device emulation. In general people get more performance using para-virtualization, as the quests VMs can use these as native drivers.

First one should verify if Oracle VM supports the platform. So, in this situation the Oracle VM online documentation says RedHat Enterprise Linux 3 is certified for hardware virtualization and can even run with para-virtualized drivers.
In order to use para-virtualized drivers the requirement is that at least RedHat Enterprise Linux 3 update 9 is used.

Having para-virtualized drivers in place makes the fully-virtualized guests VM 'para-virtualization-aware'. The benefit of this is the significant I/O performance improvement say 'speed' of the vm.

Please note that having para-virtualized drivers != running a para-virtualized VM. In this situation the only improvement you will see is on block (disk) and network performance.

In order to start using the paravirtualized drivers, you should download them first from the Oracle Unbreakable Linux Network (ULN). The file you need is kmod-xenpv-*el3.i686.rpm and you can find it in the appropriate channel. The file needs to be installed into each RHEL 3 HVM guest. (rpm -ivh kmod-xenpv-smp-0.1-9.el3.i686.rpm)

After this make sure the modules just installed with rpm are copied to /lib/modules/'uname -r'/extra/xenpv. After this the modules need to be loaded:

[root@gridnode01 /]# insmod xen-platform-pci.o
[root@gridnode01 /]# insmod xen-balloon.o
[root@gridnode01 /]# insmod xen-vbd.o
[root@gridnode01 /]# insmod xen-vnif.o

Also make sure the eth0 alias is setup in /etc/modules.conf

alias eth0 xen-vnif

So, for example, your vm.cfg for the HVM guest may look like this:

acpi = 1
apic = 1
boot = 'c'
bootloader = '/usr/bin/pygrub'
builder = 'hvm'
device_model = '/usr/lib/xen/bin/qemu-dm'
disk = ['file:/OVS/running_pool/112_pdrtest/system.img,hda,w',
'file:/OVS/running_pool/112_pdrtest/oh0.img,hdb,w',
]
disk_other_config = []
kernel = '/usr/lib/xen/boot/hvmloader'
keymap = 'en-us'
maxmem = 1024
memory = 1024
name = '12_pdrtest'
on_crash = 'restart'
on_reboot = 'restart'
pae = 1
serial = 'pty'
uuid = 'f2e29b25-391f-74e8-b0cf-cbc69bb7905'
vcpus = 1
vif = ['type=ioemu, mac=00:16:4E:1C:42:7E, bridge=xenbr1']
vnc = 1
vncconsole = 1
vnclisten = '0.0.0.0'
vncpasswd = oracle
vncunused = 1

What now needs to change is the entry "type=ioemu" from the vif configuration line.
The other item to change is for the IO. New physical devices need to be added to the vm.cfg file and one has to make sure they use the xen-vdb disk driver. Please note that once the new disk is created the data needs to be moved to that disk.

Now disk and network drivers are para-virtualized performance will be better.

The ability to keep running older operating systems in a virtualized environment is great.
Oracle Certifies most common operating systems on Oracle VM.
The Oracle software, for instance Real Application Clusters still needs to be certified with Oracle VM and the guest VM OS image. See also this link for best practices and certification.

Rene Kundersma
Oracle Technology Services, The Netherlands

On a daily basis database changes to production database systems happen everywhere. In order to provide rollback options for failed changes people most often use the regular rman restore (point in time recovery) strategy. However, restoring databases (or parts of databases) can take long, especially as databases are getting larger and larger. As longer downtime obviously is not something people look for, Flashback database maybe something to think of. I am very enthusiastic about this underrated feature

Background

Since Oracle version 10.1, "Oracle Flashback Database" was introduced. In the documentation Oracle stated the following:

"This feature introduces the FLASHBACK DATABASE statement in SQL. It let you quickly bring your database to a prior point in time by undoing all the changes that have taken place since that time. This operation is fast, because you do not need to restore the backups. This in turn results in much less downtime following data corruption or human error."

So, Flashback database is simpler and faster to use then point in time recovery because it does not require restoring datafiles from backup and it requires applying fewer changes from the archived redo logs.

Although most of us already know about availability of this option, it isn't used as often as rollback strategy for database changes.

To be complete in this background details, Flashback technology brings more options to the table:
· Oracle Flashback Table, which returns one or more tables to their contents at a previous time;
· Oracle Flashback Drop, which undoes the effects of the DROP TABLE operation;
· Oracle Flashback Query, which is used to query the contents of the database at a past time;
· Oracle Flashback Version Query, which lets you view past states of data;
· Oracle Flashback Transaction Query, which is used to review transactions affecting a table over time.

This blog entry is about database Flashback only because it is that functionality that comes in very handy for undoing database changes or testing cycles.

Normal and guaranteed restore points

For Flashback Database there are two type of restore points:
1. Normal restore points
2. Guaranteed restore points

The documentation says "Restore points provide capabilities related to Flashback Database as well as other recovery operations. Guaranteed restore points, in particular, provide a complementary capability to Flashback Database, allowing you to select an SCN and enforce the requirement that Flashback Database be usable to that SCN, though not necessarily to SCNs between the guaranteed restore point and the present SCN"

So, in order to make sure changes can be rolled back you can imagine the guarantee option is the one we need. Because, if space in the Flash recovery area is low, then Flashback logs may be deleted to free space for files required by the configured retention policy and that is not what we want.

During space pressure, the Flashback logs are deleted under two conditions:
1. When an archivelog is deleted, it would also delete the Flashback logs that are dependent
on the archived log.
2. When quota is shrunk, then Flashback logs are deleted to reclaim diskspace. This is decreasing the DB_RECOVERY_FILE_DEST_SIZE to such a value that the Flashback log itself will run into a space pressure and than the old Flashback logs will get deleted.

So, using guaranteed restore points is the only way to ensure that you can use Flashback Database to return to a specific point in time and guarantee the size of the Flashback window. I.e. NO DELETION of Flashback logs.

In the end, when a database is 'Flashed back' to its state is at some past target time using Flashback Database, each block changed since that time is restored from the copy of the block in the Flashback logs most immediately prior to the desired target time. The redo log is then used to re-apply changes since the time that block was copied to the Flashback logs. In one of the examples below you will see that redo is applied to roll forward within the Flashback procedure.

Be careful of the following remark: "if no files are eligible for deletion from the Flash recovery area because of the requirements imposed by your retention policy and the guaranteed restore point, then the database behaves as if it has encountered a disk full condition. In many circumstances, this causes your database to halt".

You may get messages about a full Flash Recovery Area (FRA) in your alert.log, also you may see this kind of error messages when your diskgroup is filled up with Flashback logs:

ORA-19624: operation failed, retry possible
ORA-19504: failed to create file "+RK"
ORA-17502: ksfdcre:4 Failed to create file +RK
ORA-15041: diskgroup space exhausted

How to setup and use Flashback

Using Flashback requires extra Flash Recovery Area diskspace as Flashback logs are not recycled when a Guaranteed Restore Point is in effect. Actually, you don't want any Flashback file from a restore point to be dropped until you are sure you don't need it again.

Remaining question is: How large should my FRA be sized extra in order to use Flashback ? Well, as a general rule of thumb the answer is: The total size of all the archive logs that would be created for the same change will suffice as required space for flashback logs. Please note that you need this size as additional space and this is just an estimate because how much space Flashback logs will take depends on your actions.

In one test I did, that only inserted data into a table 1.5GB of archives was created with only 85 megs of Flashback. Results of testing with updates and deletes will be added later.

In order to use Flashback, the two init.ora parameters DB_RECOVERY_FILE_DEST and DB_RECOVERY_FILE_DEST_SIZE should be set. The DB_RECOVERY_FILE_DEST_SIZE parameter specifies the maximum total bytes to be used for the FRA. DB_RECOVERY_FILE_DEST specifies the location of the FRA.

Please note, in case of RAC databases the value requires to be the same on all instances. Most of us already are using the DB_RECOVERY_FILE_DEST* parameters for their RMAN backups and archive logs. However, by default Flashback is disabled as you can see by querying v$database, so you still need to enable that.

SQL> SELECT flashback_on from v$database;
FLASHBACK_ON
------------
NO                                                                           

So, no Flashback is enabled and no Flashback logs exist:

SQL> SELECT * FROM v$flashback_database_log;
no rows selected

If you try to create your first guaranteed restore point while the database is running (open) you receive the following error:

SQL> create restore point rk1 guarantee flashback database;
create restore point rk1 guarantee flashback database
*
ERROR at line 1:
ORA-38784: Cannot create restore point 'RK1'.
ORA-38787: Creating the first guaranteed restore point requires mount mode when flashback database is off.

oracle@node01(xrku1):/var/opt/oracle> srvctl stop database -d xrku -o immediate
oracle@node01(xrku1):/var/opt/oracle> srvctl start instance -i xrku1 -d xrku -o mount

Now the restore point can be made and the database can be opened :

SQL> create restore point rk1 guarantee flashback database;
Restore point created.
SQL> alter database open;

You should now be able to see the restore point and identify the Flashback logfile:

SQL> SELECT flashback_on from v$database;
FLASHBACK_ON
------------
RESTORE POINT ONLY

SQL> SELECT NAME, SCN, TIME,
SQL> GUARANTEE_FLASHBACK_DATABASE
SQL> FROM V$RESTORE_POINT
SQL> WHERE GUARANTEE_FLASHBACK_DATABASE='YES';

NAME              SCN TIME                                                                        GUARANTEE_FL
---------- ---------- --------------------------------------------------------------------------- ------------
RK1           2066217 29-JUL-09 10.34.05.000000000 AM

SQL> select name from V$FLASHBACK_DATABASE_LOGFILE;
NAME
--------------------------------------------
+FLASH_RECOVERY_AREA/xrku/flashback/log_1.931.693484445

So, as an example, after creating or dropping some tables, in order to execute a Flashback database, the database has to be placed into mount mode again:

SQL> shutdown immediate
SQL> startup mount
SQL> flashback database to restore point rk1;
Flashback complete.

This is what you will see in the alert.log of your instance:

Wed Jul 29 12:03:38 2009
flashback database to restore point rk1
Wed Jul 29 12:03:38 2009
Flashback Restore Start
Flashback Restore Complete
Completed: flashback database to restore point rk1

After Flashing back your database, open it with an "alter database open resetlogs" command, you should now have the situation as of before the creation or dropping of the tables.

Benefits, specials and things to be careful of with Flashback database

As said, Flashback database can be used as a strategy for point in time recovery (PITR) and as an alternative to the rman restore and recover. As a matter of fact, the Flashback log captures old versions of changed blocks. You could look at this as a continuous backup. Then, the logs are 'replayed' to restore the database to a certain time. Flashback 'restores' just changed blocks. The process responsible for writing the Flashback log is the rvwr process:

oracle@node01(xrku1):/ora/rku/admin/xrku1/bdump> ps -ef | grep oracle | grep rvwr
oracle    2677     1  0 12:30 ?        00:00:00 ora_rvwr_xrku1

You will only see this process if Flashback is enabled.

Flashback database also has limitations that you should be aware of before using it:

1. Flashback Database can only undo changes to a datafile made by an Oracle database. For example:

Users created will be Flashed back as these are database changes. sysdba users however, will not be removed from password file (v$pwfile_users). The same counts for the init.ora file where Flashback doesn't Flashback init.ora settings.

2. Flashback Database cannot be used to repair media failures, or to recover from accidental deletion of datafiles.

3. You cannot use Flashback Database to undo a shrink datafile operation, actually, shrinking a datafile or dropping a tablespace can prevent Flashing back the Database. For example:

SQL> create tablespace x datafile size 10M;
Tablespace created

SQL> select file_name from dba_data_files where tablespace_name = 'X';
FILE_NAME
---------------------------------------------------------
+DATA_AREA01/xrku/datafile/x.329.693491457
1 row selected.

SQL> alter database datafile '+DATA_AREA01/xrku/datafile/x.329.693491457' resize 5M;
Database altered.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup mount;
ORACLE instance started.
Total System Global Area  536870912 bytes
Fixed Size                  2085360 bytes
Variable Size             331353616 bytes
Database Buffers          197132288 bytes
Redo Buffers                6299648 bytes
Database mounted.

SQL> flashback database to restore point rk2;
flashback database to restore point rk2
*
ERROR at line 1:
ORA-38766: cannot flashback data file 8; file resized smaller
ORA-01110: data file 8: '+DATA_AREA01/xrku/datafile/x.329.693491457'

As another example, tablespaces created after the restore point will be dropped and files will be deleted upon Flashback execution (as expected):

Wed Jul 29 12:18:20 2009
flashback database to restore point rk1
Wed Jul 29 12:18:20 2009
Flashback Restore Start
Deleted Oracle managed file +DATA_AREA01/xrku/datafile/y.305.693490587
Flashback: deleted datafile #9 in tablespace #9 from control file.
Flashback: dropped tablespace #9: 'Y' from the control file.
Deleted Oracle managed file +DATA_AREA01/xrku/datafile/x.329.693490579
Flashback: deleted datafile #8 in tablespace #8 from control file.
Flashback: dropped tablespace #8: 'X' from the control file.
Flashback Restore Complete
Completed: flashback database to restore point rk1
Wed Jul 29 12:18:25 2009

Tablespaces created after restore point X and before restore point X+1 will be Flashed back and 'recovered' automatically, Think about this and realize what a great idea it is from Oracle to combine rolling forwards and rolling backwards:

flashback database to restore point rk2
Wed Jul 29 12:30:57 2009
Flashback Restore Start
Flashback Restore Complete
Flashback Media Recovery Start
Media Recovery apply resetlogs offline range for datafile 1, incarnation : 0
Media Recovery apply resetlogs offline range for datafile 2, incarnation : 0
Media Recovery apply resetlogs offline range for datafile 3, incarnation : 0
Media Recovery apply resetlogs offline range for datafile 4, incarnation : 0
Media Recovery apply resetlogs offline range for datafile 5, incarnation : 0
Media Recovery apply resetlogs offline range for datafile 6, incarnation : 0
Media Recovery apply resetlogs offline range for datafile 7, incarnation : 0
 parallel recovery started with 3 processes
Wed Jul 29 12:30:57 2009
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: +DATA_AREA01/xrku/onlinelog/group_1.354.689779921
  Mem# 1: +DATA_AREA01/xrku/onlinelog/group_1.307.689779921
Successfully added datafile 8 to media recovery
Datafile #8: '+DATA_AREA01/xrku/datafile/x.329.693491457'
Wed Jul 29 12:30:57 2009
Incomplete Recovery applied until change 2066572
Flashback Media Recovery Complete
Completed: flashback database to restore point rk2

4. If the database control file is restored from backup or re-created, all accumulated Flashback log information is discarded. You cannot use FLASHBACK DATABASE to return to a point in time before the restore or re-creation of a control file.

5. When using Flashback Database with a target time at which a NOLOGGING operation was in progress, block corruption is likely in the database objects and datafiles affected by the NOLOGGING operation. For example, if you perform a direct-path INSERT operation in NOLOGGING mode, and that operation runs from 9:00 to 9:15 on April 3, 2005, and you later need to use Flashback Database to return to the target time 09:07 on that date, the objects and datafiles updated by the direct-path INSERT may be left with block corruption after the Flashback Database operation completes.

6. For combinations Flashback and features like streams and dataguard with Flashback you should be aware of some extra constraints:

Having the ability to enable your source (primary) database with Flashback Guaranteed Restore Points (and actually using Flashback) requires the physical standby database to be in full Flashback mode. Flashing back the 'primary' to 'SCN' requires to Flashback the target (to SCN-20) so that the log sequence can catch up.

Note that enabling full Flashback on the target database requires extra disk space.
As a general guideline Oracle advices 2x the total size of archives that will be created as a requirement for flashback data. You have to test this for your environment.

Another thing to watch out for: "If the switchover involved a physical standby database, the primary and standby database roles are preserved during the Flashback operation. That is, the role in which the database is running does not change when the database is Flashed back to the target SCN or time to which you Flashed back the database." i.e. you have to do this your self.

A last note on the combination flashback and dataguard: be sure to use the command 'flashback standby database ...' on your standy database

For streams this can of course only be done when there is a 1:1 relation ship between source and target. Note that the with Streams source and target database have their own log thread. It's obviously more difficult to find an SCN in source and in target that are on exact the same time.

Disk usage

The information about required disk space is collected after some basic tests and is only applicable to Flashback database with a guaranteed restore points. Each action (delete, update, insert, drop) on a database generates an different amount of flashback logs. Also, as with normal actions, the amount of flashback data also depends on whether complete tables are updated or only specific columns. Depending on the action and the amount of data you may need more or less space, though, in general this list of values can be used as a guide.

For my test a table with a size of 1000M was created, without an index.
For insertion, deletion, updating (and dropping) of the complete table information was collected on the amount of flashback logs. For each action the most possible redo was generated:

Summary

As discussed guaranteed restore points can come in very helpful to rollback database changes. It's a requirement to configure your Flash Recovery Area (FRA) if that's not already done. In order to use FRA DB_RECOVERY_FILE_DEST and DB_RECOVERY_FILE_DEST_SIZE need to be set. Flashback database could also come in very handy in testing cycles where you want to re-test performance for example over and over with the same set of data.

In order to Flashback, bring your database down to mount state and Flashback to that restore point. Also, be sure to understand the mentioned limitations of this wonderful option so that you know in what state your database is after Flashing it back.

Also know how to handle Flashback in combination with solutions like Dataguard and Streams. Carefully test this before heading for production. Make yourself aware with the technology.

Be sure to reserve at least 1 x the amount you would need for archives to hold your Flashback logs, since Flashback logs requires storage also. Active monitoring off free FRA space is highly advised because a full FRA will halt your database. For this use the view v$flash_recovery_area_usage;

Not using the FRA for you archives (i.e. using log_archive_dest_*) is not recommended, but brings us an extra option which maybe interesting for you as a temporary extra safety machanism: Since the location for archived logs is not FRA in this situation, a separate (temporary) diskgroup could be made for the FRA that will only be used for flashback logs.

The advantage of this solution is that filling up the diskgroup (with Flashback logs) will not risk database availability for other databases. In order to create this temporary diskgroup of course free disks need to be available. It's however recommend by Oracle to put archives as well as flashback logs all in FRA.

References

464542.1 Can I Open A Physical Standby Db For R/W And Use Flashback to Put It Back?
728374.1 How To Flashback Primary Database In Standby Configuration
330535.1 Restore Points in Oracle10g Release2
565535.1 Flashback Database Best Practices & Performance
305648.1 What is a Flash Recovery Area and how to configure it
369759.1 FLASH RECOVERY AREA and FLASHBACK database
369755.1 Flashback Logs-Space management
Oracle® Database Backup and Recovery Basics 10g Release 2 (10.2) - 7.5 Reversing Database Changes with Flashback Database

Rene Kundersma
Oracle Expert Services, The Netherlands

Until that time, I like to share some thoughts.

First: why do you possibly want "Jumbo Frames" ?

Answer: Jumbo Frames (in my Oracle world) are Ethernet frames that do not have a conventional payload of 1500 bytes, but 9000 bytes.

As you can imagine the standard 1500 bytes is too small for our 'regular' Oracle database block of 8k. When you make sure the frames are 9000 byte, a block should fit in and this way eliminate the overhead.

Eliminating the overhead would potentially improve performance and decrease CPU usage. This sound like a good theory and very applicable to use for the RAC interconnect where we do sent 8k (UDP) blocks from instance A to instance B.

Investigation tells us Jumbo Frames are not (yet) an IEEE standard. This means, you could (I am not saying you will) have problems between the devices that supposed to be configured to handle frames of this size.

The OS, network card as well as the switch all need to 'talk' the same size of "Jumbo Frames". Note that some vendors may call 4000 bytes Jumbo, some call 9000 bytes Jumbo.

Theory tells us properly configured Jumbo Frames can eliminate 10% of overhead on UDP traffic.

So how to test ?

I guess an 'end to end' test would be best way. So my first test is a 30 minute Swingbench run against a two node RAC, not too much stress in the begin.

The MTU configuration of the network bond (and the slave nics will be 1500 initially).

After the test, collect the results on the total transactions, the average transactions per second, the maximum transaction rate (results.xml), interconnect traffic (awr) and cpu usage. Then, do exactly the same, but now with an MTU of 9000 bytes. For this we need to make sure the switch settings are also modified to use an MTU of 9000.

B.t.w.: yes, it's possible to measure network only, but real-life end-to-end testing with a real Oracle application talking to RAC feels like the best approach to see what the impact is on for example the avg. transactions per second.

In order to make the test as reliable as possible some remarks:
- use guaranteed snapshots to flashback the database to its original state.
- stop/start the database (clean the cache)

B.t.w: before starting the test with an MTU of 9000 bytes the correct setting had to be proofed.

One way to do this is using ping with a packet size (-s) of 8972 and prohibiting fragmentation (-M do).
One could send Jumbo Frames and see if they can be sent without fragmentation.

[root@node01 rk]# ping -s 8972 -M do node02-ic -c 5
PING node02-ic. (192.168.23.32) 8972(9000) bytes of data.
8980 bytes from node02-ic. (192.168.23.32): icmp_seq=0 ttl=64 time=0.914 ms

As you can see this is not a problem. While for packages larger then 9000 bytes, this is a problem:

[root@node01 rk]# ping -s 8973 -M do node02-ic -c 5
--- node02-ic. ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4003ms
rtt min/avg/max/mdev = 0.859/0.955/1.167/0.109 ms, pipe 2
PING node02-ic. (192.168.23.32) 8973(9001) bytes of data.
From node02-ic. (192.168.23.52) icmp_seq=0 Frag needed and DF set (mtu = 9000)

Bringing back the MTU size to 1500 should also prohibit sending of fragmented 9000 packages:

[root@node01 rk]# ping -s 8972 -M do node02-ic -c 5
PING node02-ic. (192.168.23.32) 8972(9000) bytes of data.
--- node02-ic. ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms

Bringing back the MTU size to 1500 and sending 'normal' packages should work again:

[root@node01 rk]# ping node02-ic -M do -c 5
PING node02-ic. (192.168.23.32) 56(84) bytes of data.
64 bytes from node02-ic. (192.168.23.32): icmp_seq=0 ttl=64 time=0.174 ms

--- node02-ic. ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.174/0.186/0.198/0.008 ms, pipe 2

An other way to verify the correct usage of the MTU size is the command 'netstat -a -i -n' (the column MTU size should be 9000 when you are performing tests on Jumbo Frames):

Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
bond0 1500 0 10371535 0 0 0 15338093 0 0 0 BMmRU
bond0:1 1500 0 - no statistics available - BMmRU
bond1 9000 0 83383378 0 0 0 89645149 0 0 0 BMmRU
eth0 9000 0 36 0 0 0 88805888 0 0 0 BMsRU
eth1 1500 0 8036210 0 0 0 14235498 0 0 0 BMsRU
eth2 9000 0 83383342 0 0 0 839261 0 0 0 BMsRU
eth3 1500 0 2335325 0 0 0 1102595 0 0 0 BMsRU
eth4 1500 0 252075239 0 0 0 252020454 0 0 0 BMRU
eth5 1500 0 0 0 0 0 0 0 0 0 BM

As you can see my interconnect in on bond1 (build on eth0 and eth2). All 9000 bytes.

Not finished yet, no conclusions yet, but here is my first result.
You will notice the results are not that significantly.

MTU 1500:
TotalFailedTransactions : 0
AverageTransactionsPerSecond : 1364
MaximumTransactionRate : 107767
TotalCompletedTransactions : 4910834

MTU 9000:
TotalFailedTransactions : 1
AverageTransactionsPerSecond : 1336
MaximumTransactionRate : 109775
TotalCompletedTransactions : 4812122

In a chart this will look like this:

As you can see, the number of transactions between the two tests isn't really that significant, but the UDP traffic is less ! Still, I expected more from this test, so I have to put more stress to the test.

I noticed the failed transaction, and found "ORA-12155 TNS-received bad datatype in NSWMARKER packet". I did verify this and I am sure this is not related to the MTU size. This is because I only changed the MTU size for the interconnect and there is no TNS traffic on that network.

As said, I will now continue with tests that have much more stress on the systems:
- number of users changed from 80 to 150 per database
- number of databases changed from 1 to 2
- more network traffic:
- rebuild the Swingbench indexes without the 'REVERSE' option
- altered the sequences and lowered increment by value to 1 and cache size to 3. (in stead of 800)
- full table scans all the time on each instance
- run longer (4 hours in stead of half an hour)

Now, what you see is already improving. For the 4 hour test, the amount of extra UDP packets sent with an MTU size of 1500 compared to an MTU size of 9000 is about 2.5 to 3 million, see this chart:

Imagine yourself what an impact this has. Each package you not send save you the network-overhead of the package itself and a lot of CPU cycles that you don't need to spend.

The load average of the Linux box also decreases from an avg of 16 to 14.

In terms of completed transactions on different MTU sizes within the same timeframe, the chart looks like this:

To conclude this test two very high load runs are performed. Again, one with an MTU of 1500 and one with an MTU of 9000.

In the charts below you will see less CPU consumption when using 9000 bytes for MTU.

Also less packets are sent, although I think that number is not that significant compared to the total number of packets sent.

My final thoughts on this test:

1. you will hardly notice the benefits of using Jumbo on a system with no stress
2. you will notice the benefits of Jumbo using Frames on a stressed system and such a system will then use less CPU and will have less network overhead.

This means Jumbo Frames help you scaling out better then regular frames.

Depending on the interconnect usage of your applications the results may vary of course. With interconnect traffic intensive applications you will see the benefits earlier then with application that have relatively less interconnect activity.

I would use Jumbo Frames to scale better, since it saves CPU and reduces network traffic and this way leaves space for growth.

Rene Kundersma
Oracle Expert Services, The Netherlands

Then, probably, your next step is plan how to actually move your existing database to Exadata / HP Oracle DB-Machine (from now on called Exadata in this blog).

For this scenario, there are some small things to take care of.

My personal experience is that, preparing such a migration is all about knowing Oracle's MAA strategy. With that knowledge you know what options you have and don't have.

Off-line or On-line
First of al one has to decide whether the migration is 'off-line' or on-line.
'On-line' in this case is a matter of a couple of minutes downtime, you always have to switch from source database 'A' to target database 'B'.

Consider this time, for example, as the time it takes to switch over to a standby database.

'Off-line' is the most comfy solution, but only of course when the off-line window is large enough to migrate your data. In case of migrations to Exadata the data bulk you have to migrate will probably be 'large', so consider yourself lucky if you have a 'off-line' option and a window large enough to do the action.

For Exadata it is recommend to have an extent size that is a multiple of 4M. The ASM AU size should be 4M also. This is to make sure that at database level (for the extents) and at ASM level (for the AU's) at least 4M of contiguous chunks will be read. This is required to make Exadata perform best.

So, if you want to stick to this recommendation, you have to check the current size of your AU's and extents.
When you find out the size is not the recommend size then choice number two pops-up: will you do a 'logical' migration and stick to the recommendation or not and do a 'physical' migration. Of course, when your source system 'A' is not on ASM there isn't much AU's to check.

For the extents sizes the query should look like this:

select segment_name,
from dba_exents
where bytes < (4*1024*1024)
and owner = 'the name of the schema you are migrating';

If the extents already have the recommended size, you of course still can choose for a logical migration, but my personal feeling would be to do a physical migration in that case.

Physical migration
Physical migration to me means 'transferring the datafiles from platform A to platform B'. This situation leads us to new choices, like the choice of how to transfer the datafiles from A to B. This depends because platform A en B can both have a different architecture.

As we know Exadata/DB-Machine is based on Linux on Intel X86-64 and the source can be RISC (Sun Sparc, IBM P-series) for example (big endian). (where Intel and Itanium is little endian)

Fortunately Oracle has some good answers for this kind of questions:
- transportable tablespaces
- DBMS_STREAMS_TABLESPACE_ADM.PULL_TABLESPACES

To a certain level physical migrations can be done 'on-line' (or at least with as less downtime as possible). Think of dataguard for example.

As a last step; which solution you choose also depends on the infrastructure you are working in. Question you may ask yourself are:
- Do I have staging space on the source system
- Do I have network attached storage (NAS) available that I can use.
- Can this NAS be connected to source and target ?
- Are platform A en B actually on the same network anyway and
- If on the same network: can I safely use that network to tranfer my terabytes of data (not hurting the performance of other systems)

These are all realistic questions you have to deal with when you are talking about migrations.

Logical migration
You have to investigate into available solutions for logical migrations as well.
Before I continue I should first explain what my definition of logical migration is.

To me logical migration is 'exporting' the data from the source and 'importing' that into the target. Export can be any kind of tool such as: datapump, exp or even SQL*Loader or CTAS over a db-link. A 'logical' export can even be done 'on-line' if you would consider solutions like for example logical Standby database (same endian) or Oracle Streams (endian independent). 'Logical' migrations often need 'staging' (a location where you temporary dump you data). Note that datapump exports/imports can be done over the network without dumping anything to file.

As said, if you set yourself the target to change the extent sizes to the recommended values, you automatically end up in a logical migration if you not already have extents of this size.

There may be even more things to consider. For example.
- you need to build more then one replica of the source database on Exadata.
- you want to put as less load on the source system as possible because it's a highly critical production system.

When dealing with these kind of questions always know that Exadata can do the job quicker, because it so powerful.

Say for instance: you may realize exporting a 4TB database with datapump will cause to much load on the prodution system. But you still want to do a logical migration. For these kind of questions my answer would be to just transfer the datafiles to Exadata (and endian convert them if needed) and perform the export (expdp) and import (impdp) there. You probably don't even need to transfer the files, perhaps you can retrieve them from backup.

Exporting data with datapump is something you prefer to do in a read consistent way. For that think of the expdp arguments:
- flashback_time
- undo_rentention
And undo guarentee on tablespace level.

Note 1. Note that exporting lobs in a read consistent way can not be accomplished only by tuning the undo retention of the database. For lobs undo is a property of the column. In order to make sure the retention of these objects is set well you may have to alter your table first. For more information on this see "The application developers guide - Large objects"

Note 2. transferring or exporting indexes to Exadata seems rather useless to me. First, you have to consider yourself if the index is still needed in the first place. And if so, why not just recreate it on the target, since Exadata has probably got the muscles for it to recreate it in a small fraction of the time it took on your source database !

Summary
Migrations to Exadata are not more complicated then normal migrations. When dealing with large amount of data one should always think of a strategy before htting the keybord. As production environments have their limitations it proofed to be helpful to workout alternative scenarios as well. With expdp, transportable tablespaces, transportable databases, Streams, physical, logical standby databases, Oracle has all thinkable options available to migrate to this wonderful piece of database !

Helpful documents:
- Technical White Paper: Best Practices for Migrating to HP Oracle Exadata Storage Server
- Oracle Database High Availability Documentation - Features and Best Practices

Rene Kundersma
Oracle Expert Services, The Netherlands

The whitepaper details on VCPU strategies and storage solutions.
The document also describes some best practices you should know using RAC in an Oracle VM environment.

Check it out:

http://www.oracle.com/technology/products/database/clusterware/pdf/oracle_rac_in_oracle_vm_environments.pdf

Rene Kundersma
Oracle Expert Services, The Netherlands

#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.sample  
#
ddns-update-style none;
allow booting; 
allow bootp;   

subnet 192.168.200.0 netmask 255.255.255.0 {
    option routers             192.168.200.1;
    option subnet-mask         255.255.255.0;
    option nis-domain          "nl.oracle.com";
    option domain-name         "nl.oracle.com";
    option domain-name-servers 192.135.82.60;

    default-lease-time 60;
    max-lease-time 60;
 
    next-server 192.168.200.173;
    filename "/pxelinux.0";

    host RK{
    hardware ethernet 00:16:3e:62:39:d3;
    fixed-address 192.168.200.177;
    }
}

service dhcpd restart 

cp /usr/lib/syslinux/pxelinux.0 /tftpboot/
cp /usr/lib/syslinux/mboot.c32 /tftpboot/

cp $MOUNT_OEL_DISTR/images/xen/* /tftpboot/

[root@gridnode03 pxelinux.cfg]# gethostip -x 192.168.200.177
C0A8C8B1

[root@gridnode03 ~]# cat /tftpboot/pxelinux.cfg/C0A8C8B1 
default linux
prompt 1
timeout 120
label linux
  kernel vmlinuz
  append initrd=initrd.img lang=en_US keymap=us \
  ks=nfs:192.168.200.200:/vol/vol1/distrib/linux32/workshop-ovs/oel/OEL5U2/ks.cfg \  
  ksdevice=eth0 ip=dhcp

[root@nlhpblade07 pxe]# cat rk.cfg 
name = "RK"
memory = "1024"
disk = [ 'file:/OVS/running_pool/pxe/system.img,xvda,w',]
vif = [ 'mac=00:16:3e:62:39:d3,bridge=xenbr0', '', ]
vfb = ["type=vnc,vncunused=1,vnclisten=0.0.0.0"]
#bootloader="/usr/bin/pygrub"
bootloader="/usr/bin/pypxeboot"
bootargs=vif[0]
vcpus=1
on_reboot   = 'restart'
on_crash    = 'restart'

[root@nlhpblade07 pxe]# dd if=/dev/zero of=system.img bs=1M count=8000
8000+0 records in
8000+0 records out
8388608000 bytes (8.4 GB) copied, 165.725 seconds, 50.6 MB/s
[root@nlhpblade07 pxe]# 

[root@nlhpblade07 pxe]# cat rk.cfg
name = "RK"
memory = "1024"
disk = [ 'file:/OVS/running_pool/pxe/system.img,xvda,w',]
vif = [ 'mac=00:16:3e:62:39:d3,bridge=xenbr0', '', ]
vfb = ["type=vnc,vncunused=1,vnclisten=0.0.0.0"]
bootloader="/usr/bin/pygrub"
vcpus=1
on_reboot   = 'restart'
on_crash    = 'restart'

The existing Oracle VM Manager, is rather quick and light weight, so you will probably think why do we need EMGC for that.

Well, think about the following advantages:
- All in one management console
- EMGC runs also on Windows, so now you can manage Oracle VM from windows, which was not possible before
- You can set thresholds
- Use the Single Sign On that EMGC has
- Use the security features EMGC has
- Setting notifications and alerts so that admins can manage on exception basis.
- Context based flows (even if manual currently).
For example, change the vCPU on para-virtualized linux guests if database diagnostics (ADDM) suggests additional CPU
- Specific EM features like Configuration compare, search, policies for host and hypervisor. The policies can span multiple tiers and can be user defined. Example, do not run mixed workloads (database and middleware) on the same Hypervisor.
- Guest patching with integration with Unbreakable Linux Network (ULN)

Some quick requirements had to be filled in before I could begin:
- install a one-off patch to the OMS (patch 8244731)
- install VnCViewer into $ORACLE_HOME/j2ee/OC4J_EM/applications/em/em
- also had to be sure Oracle VM Agent was updated to version 2.2-70 or higher.

This all went smoothly. Please note that it makes sense to install the patch and the VnCViewer at the same time when the OMS is down.

Please read note 781879.1 for full details (as information may change and there is only one singe source of truth)

First I had to create a server pool. After creating a server pool, it was just a matter of making the Oracle VM Server aware of the Vm images I already had on disk.

And exactly for this was an option, which you can see in the capture below.

This is the place where you can see all the VM Images discovered in your virtual server pool.
After this, it is just a matter of selecting the images and "importing" them into your vm "admin" page.

From here you can do the usual; start, stop, pause, suspend, clone and get to the console.

To me this seems a new step taken for EMGC to be the complete management framework.

Rene Kundersma
Oracle Expert Services, The Netherlands