Tuesday Sep 16, 2014

Oracle Database Documentation - From Past to Present

Yesterday one of my colleagues, well known Data Guard Guru Larry "Murphy" Carpenter, sent out an internal email with links to older and recent Oracle Database Documentation. Reason why you should use the below links: Some sites in the www have stored copies of the doc - but some may represent not the most recent state or may not be complete. So stay with the official Oracle docs only:

-Mike

PS: For those who haven't upgraded for the past 17 years or for those with nostalgic feelings this is the link to the Oracle 7.3.4 documentation :-)

Thursday Sep 04, 2014

OOW 2014 - Upgrade and Data Pump Talks

Oracle Open World (OOW) 2014 in San Francisco is coming ... just a few weeks to go ... everybody is in the prep phase for demos, presentations, labs etc. 

If you'd like to get in touch with us to discuss your upgrade and migration strategies please feel free to contact either Roy Swonger or myself directly. We'll be happy to assist you. And of course you are welcome to stop by at our combined Upgrade/DataPump booth at the demo grounds and visit one of our talks.

Our group is happy to deliver the following talks and labs:

How to Upgrade, Migrate, and Consolidate to Oracle Database 12c [CON7647]
Monday, Sep 29, 5:15 PM - 6:00 PM - Moscone South - 102

The most widely anticipated feature of Oracle Database 12c is now available, and you may be wondering just how you can move your current databases to pluggable databases in a multitenant architecture. Whether you are just starting to explore the world of pluggable databases or are planning a production upgrade in the near future to Oracle Database 12c, this presentation by Oracle Database upgrade and migration experts gives you all the details: what methods are available; how they work; and which is the best for your particular upgrade, migration, or consolidation scenario.

How an Oracle Database 12c Upgrade Works in a Multitenant Environment [CON7648]
Tuesday, Sep 30, 12:00 PM - 12:45 PM - Moscone South - 306

With the first patch set of Oracle Database 12c, you will be able to choose between various methods of upgrading a multitenant container database and its pluggable databases. In this session, you will hear from Oracle upgrade experts about all the details of how a database upgrade works in a multitenant environment. You will learn what your options are, how parallelism works for database upgrades, and what is new for database upgrades in the first patch set of Oracle Database 12c.

How and Why to Migrate from Schema Consolidation to Pluggable Databases [CON7649]
Wednesday, Oct 1, 11:30 AM - 12:15 PM - Moscone South - 306

One important use case for pluggable databases is to enable you to move from schema consolidation with multiple applications in the same database to a more secure environment with Oracle Multitenant and pluggable databases. In this technical session, you will hear from Oracle development experts about the methods available for migrating from schema consolidation to a multitenant database environment with Oracle Data Pump, transportable tablespaces, or new features in Oracle Multitenant.

Oracle Database 12c Upgrade: Tools and Best Practices from Oracle Support [CON8236]
This talk is not done by us but by our Global Tech Lead for Upgrades in Support, Agrim Pandit
Tuesday, Sep 30, 5:00 PM - 5:45 PM - Moscone South - 310

You’ve heard about Oracle Database 12c and its new capabilities. Now come hear from Oracle experts about all the great tools and resources Oracle offers to help you upgrade to Oracle Database 12c efficiently and effectively. This session’s presenters, from Oracle Support, bring years of database experience and recent lessons learned from Oracle Database 12c upgrades at companies of all sizes all around the world. You are sure to leave with valuable information that will help you plan and execute your upgrade. What's more, most, if not all, of the tools and resources they discuss are available to current customers at no additional cost through their standard product support coverage.

.

I will publish the schedules for the Hands-On-Lab (4x) and the location of the demo ground's booth as soon as I'll get it.

-Mike

Tuesday Sep 02, 2014

Unified Auditing - is it ON or OFF in Oracle 12c?

Don't trust our slides - only believe what you've verified by yourself ;-)

Actually one of our slides gives a parameter recommendation to set AUDIT_TRAIL since Oracle 11g explicitly to the value you want as otherwise it may switch to "DB" and you may not be aware of it. In conjunction with this setting we explain the new Oracle Database 12c feature Unified Auditing - which is not linked into the kernel and therefore should be off.

Should be ... well ... thanks to Marco Patzwahl who asked me why he still has over 100 audit records in V$UNIFIED_AUDIT_TRAIL? Good question - and I've had no answer. But Carol, my manager, knew the right person to ask. And Naveen replied within minutes (thanks!!!).

Here are the facts: 

  • Unified Auditing is not linked into the Oracle 12c kernel by default to offer people the choice to use it and to avoid conflicts in case somebody has auditing ON already - so neither during an upgrade nor with a fresh 12c database you'll see it included into the kernel. It will have to be linked in manually (see our slides)
    • Check if Unified Auditing is present in your environment:
      SQL> select VALUE from V$OPTION where PARAMETER='Unified Auditing';
    • In case you'll link it into the kernel
      cd $ORACLE_HOME/rdbms/lib
      make -f ins_rdbms.mk
      uniaud_on ioracle ORACLE_HOME=$ORACLE_HOME

      make sure you set AUDIT_TRAIL=NONE afterwards as otherwise both auditing mechanisms will run concurrently
  • But even though it is not linked into the kernel a bit of Unified Auditing is ON by default in MIXED MODE when you create a fresh Oracle 12c database.
    • MIXED MODE auditing?
      • See the documentation for further information
      • Just two policies are enabled by default: ORA_SECURECONFIG and ORA_LOGON_FAILURES
  • Turn Unfiied Auditing OFF?
    • If is has been linked in into the kernel, unlink it:
      cd $ORACLE_HOME/rdbms/lib
      make -f ins_rdbms.mk 
      uniaud_off ioracle ORACLE_HOME=$ORACLE_HOME
    • Disable the two default policies - this will turn off any Unified Auditing features:
      SQL> noaudit policy ORA_SECURECONFIG;
      Noaudit succeeded.
      SQL> noaudit policy ORA_LOGON_FAILURES;
      Noaudit succeeded.


-Mike

Friday Aug 22, 2014

Automatic Maintenance Jobs in every PDB?
New SPM Evolve Advisor Task in Oracle 12.1.0.2

A customer checking out our slides from the OTN Tour in August 2014 asked me a finicky question the other day:

"According to the documentation the Automatic SQL Tuning Advisor maintenance task gets executed only within the CDB$ROOT, but not within each PDB - but the slides are not clear here. So what is the truth?"

Ok, that's good question. In my understanding all tasks will get executed within each PDB - that's why we recommend (based on experience) to break up the default maintenance windows when using Oracle Multitenant. Otherwise all PDBs will have the same maintenance windows, and guess what will happen when 25 PDBs start gathering object statistics at the same time ...

The documentation indeed says:

Automatic SQL Tuning Advisor data is stored in the root. It might have results about SQL statements executed in a PDB that were analyzed by the advisor, but these results are not included if the PDB is unplugged. A common user whose current container is the root can run SQL Tuning Advisor manually for SQL statements from any PDB. When a statement is tuned, it is tuned in any container that runs the statement.

This sounds reasonable. But when we have a look into our PDBs or into the CDB_AUTOTASK_CLIENT view the result is different from what the doc says. In my environment I did create just two fresh empty PDBs (CON_ID 3 and 4):

SQL> select client_name, status, con_id from cdb_autotask_client;

CLIENT_NAME                           STATUS         CON_ID
------------------------------------- ---------- ----------
auto optimizer stats collection       ENABLED             1
sql tuning advisor                    ENABLED             1
auto space advisor                    ENABLED             1
auto optimizer stats collection       ENABLED             4
sql tuning advisor                    ENABLED             4
auto space advisor                    ENABLED             4
auto optimizer stats collection       ENABLED             3
sql tuning advisor                    ENABLED             3
auto space advisor                    ENABLED             3

9 rows selected.

I haven't verified the reason why this is different from the docs but it may have been related to one change in Oracle Database 12.1.0.2: The new SPM Evolve Advisor Task ( SYS_AUTO_SPM_EVOLVE_TASK) for automatic plan evolution for SQL Plan Management. This new task doesn't appear as a stand-alone job (client) in the maintenance window but runs as a sub-entity of the Automatic SQL Tuning Advisor task. And (I'm just guessing) this may be one of the reasons why every PDB will have to have its own Automatic SQL Tuning Advisor task 

Here you'll find more information about how to enable, disable and configure the new Oracle 12.1.0.2 SPM Evolve Advisor Task:

-Mike

Grid Infrastructure Management Repository (GIMR)
database now mandatory in Oracle GI 12.1.0.2

During the installation of Oracle Grid Infrastructure 12.1.0.1 you've had the following option to choose YES/NO to install the Grid Infrastructure Management Repository (GIMR) database MGMTDB:

With Oracle Grid Infrastructure 12.1.0.2 this choice has become obsolete and the above screen does not appear anymore. The GIMR database has become mandatory

What gets stored in the GIMR?

See the changes in Oracle Clusterware 12.1.0.2 here:

  • Automatic Installation of Grid Infrastructure Management Repository

    The Grid Infrastructure Management Repository is automatically installed with Oracle Grid Infrastructure 12crelease 1 (12.1.0.2). The Grid Infrastructure Management Repository enables such features as Cluster Health Monitor, Oracle Database QoS Management, and Rapid Home Provisioning, and provides a historical metric repository that simplifies viewing of past performance and diagnosis of issues. This capability is fully integrated into Oracle Enterprise Manager Cloud Control for seamless management.

Furthermore what the doc doesn't say explicitly:

  • The -MGMTDB has now become a single-tenant deployment having a CDB with one PDB
    • This will allow the use of a Utility Cluster that can hold the CDB for a collection of GIMR PDBs
  • When you've had already an Oracle 12.1.0.1 GIMR this database will be destroyed and recreated
    • Preserving the CHM/OS data can be acchieved with OCULMON to dump it out into node view
  • The data files associated with it will be created within the same disk group as OCR or VOTING
    •  The OUI will get the disk groups for OCR and Voting and chooses the first one - which usually is the first OCR. This may lead to serious space issues. It is tracked internally as Bug:19661882  In a future release there may be an option offered to put in into a separate disk group.
      Workaround would be to move the affected OCR to another disk group (use ocrconfig command for it) - see MOS Note:1589394.1
  • Some important MOS Notes:
    • MOS Note 1568402.1
      FAQ: 12c Grid Infrastructure Management Repository, states there's no supported procedure to enable Management Database once the GI stack is configured
    • MOS Note 1589394.1
      How to Move GI Management Repository to Different Shared Storage
      (shows how to delete and recreate the MGMTDB)
    • MOS Note 1631336.1
      Cannot delete Management Database (MGMTDB) in 12.1
  • Average growth size per day per node is roughly 650-750 MB. E.g. a 4 node cluster would lead at the default retention of 3 days to an approximate size of  5.9-6.8 GB
  • Change the retention:
    $CRS_HOME/bin/oclumon manage -repos checkretentiontime 260000

-Mike

PS: Kudos to Sebastian Solbach who updated me on the things to add (retention, average growth, OUI choosing the first disk group displayed for the MGMTDB) - cheers!

Thursday Aug 14, 2014

RMAN Catalog requires Enterprise Edition (EE)
since Oracle Database 12.1.0.2

Credits go to Cameron Hodge, Malcom and Martin Mäs who all highlighted issues to me following my previous entry about RMAN - and sorry for any disappointment but I wasn't aware of all these nice little things. 

Ok, you'd upgrade your RMAN Catalog to be ready to backup/recover Oracle Database 12.1.0.2 databases and you see this error:

RMAN> upgrade catalog;

error creating create_deleted_object_seq
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================

RMAN-06004: ORACLE error from recovery catalog database: ORA-00439: feature not enabled: Partitioning

Now you start to wonder as your Catalog Database had been an Oracle Standard Edition (SE) database for quite a while - and of course an SE database does not have Partitioning on. And on the side the Oracle Partitioning Option is licensable option.

First of all this new behavior gets introduced with Oracle Database 12.1.0.2. And as far as I know it is not documented in relation to the "upgrade catalog". 

The valid workaround leads to a new feature called Infrastructure Repository Database - which is always an EE database without the need for extra licensening as all feature will be used only by Oracle internal mechanisms. 

Licensing doc:
http://docs.oracle.com/database/121/DBLIC/editions.htm#DBLIC119

Infrastructure Repository Databases

A separate single instance Oracle Database can be installed and used as an infrastructure repository for RMAN, Oracle Enterprise Manager Cloud Control, Automatic Workload Repository (AWR) Warehouse, Global Data Services Catalog, and Grid Infrastructure Management Repository without additional license requirements, provided that all the targets are correctly licensed. It may not be used or deployed for other uses.

The infrastructure repositories for RMAN, Oracle Enterprise Manager Cloud Control, AWR Warehouse, and Global Data Services Catalog can be shared in one database, or deployed in separate databases. Enterprise Edition must be used for the infrastructure repository database(s).

Furthermore the Recovery Manager documentation mentions this:

Creating and Managing Virtual Private Catalogs with Oracle Database 12c Release 1 (12.1.0.2)

Note:

Starting with Oracle Database 12c Release 1 (12.1.0.2), virtual private catalogs can be used only with the Enterprise Edition of the Oracle Database. This functionality is not supported for the Standard Edition.

Now this is a bit misleading as most of you won't use the Virtual Private Catalogs - but even though you may not use it still some of the functionality is in your catalog already. And therefore every RMAN catalog beginning with Oracle Database 12.1.0.2 must be hosted in an Enterprise Edition database which does not require an EE license.

The next question is:
How do I migrate my catalog into an EE database?

There are two options whereas I'd prefer the one mentioned in the RMAN documentation:

And there are more things to mention:

  • DBMS_LOB package must be present
  • UTL_HTTP must be present

.

To summarize to successfully upgrade your RMAN catalog:

  • You need to execute this script before upgrading the catalog:
    SQL> @$ORACLE_HOME/rdbms/admin/dbmsrmansys.sql
  • Your database hosting the RMAN Catalog beginning with Oracle 12.1.0.2:
    • Must be an Enterprise Edition database - no EE license required
    • Oracle Partitioning Option must be installed - no Partitioning lincese required
    • DBMS_LOB and UTL_HTTP packages must be present

-Mike                  

PS:  Credits for this addition go to Malcom!!! In addition to RMAN Catalog Database now requiring to be in an EE database this database will require the Partitioning Option linked in as well. See MOS Note:1927265.1

Monday Aug 11, 2014

RMAN Catalog Upgrade to Oracle 12.1.0.2

It sounds so simple - but in this specific case a bit of doc reading is required as upgrading the RMAN catalog to handle Oracle 12.1.0.2 databases is not as trivial as in the past. 

Thanks to a German customer and a friendly colleague from Sales Consulting in Stuttgart I have learned my RMAN lesson for Oracle Database 12.1.0.2 this week. A simple "upgrade catalog" is not the correct step once you'd like to handle backups of Oracle 12.1.0.2 databases in your current catalog schema. 

Even though you may not have ever heard before about Virtual Private Catalogs you need to follow this guideline:

http://docs.oracle.com/database/121/BRADV/rcmcatdb.htm#BRADV848

The doc tells you to run this step first:

SQL> @$ORACLE_HOME/rdbms/admin/dbmsrmansys.sql

  • Please ignore the "-all" option printed in the doc - this is a known docu bug
  • If you miss this step the upgrade of the catalog will fail with a warning that your user lacks privileges 

$ rman CATALOG my_catalog_owner@catdb
recovery catalog database Password: 
RMAN> UPGRADE CATALOG; 
RMAN> EXIT; 

It should work now :-)

-Mike 

Wednesday Aug 06, 2014

Upgrade PDBs - Everything At Once (Full CDB Upgrade)

As referred to it before there are two techniques to upgrade an Oracle Multitenant environment:

In this post I will explain the method of "Everything At Once" and describe all the steps. The benefit of this approach is simplicity and ease of maintenance. In an upgrade workshop in Melbourne earlier this year a DBA from Germany came by in one of the breaks explaining that he takes care on over 100 developer databases - and it would ease his life a lot if he could "hit" all of the databases at the same time with one patch set or PSU, or even upgrade them to a higher release. This is the use case where it will make a lot of sense to leverage from this approach. But be aware and don't over-consolidate as the pain point is common downtime. If you plan to use this approach you need to check before if your application owners can agree on common downtime windows - otherwise you may end in trouble. Big trouble!

The technique is easy to describe:

  • CDB$ROOT will always get upgraded first - I call this CYCLE 1
    • The "-n" parameter of catctl.pl will define how many parallel workers run the upgrade - 8 is the current maximum
    • The "-M" option will decide whether the CDB$ROOT stays in UPGRADE mode throughout the entire process of upgrade or becomes available for access after being upgraded leading to the fact that PDBs become available for access as well once they got upgraded - set "-M" and the CDB$ROOT will stay in UPGRADE mode throughout the entire process
  • Afterwards we can upgrade multiple PDBs (including the PDB$SEED) in parallel at the same time - I call this CYCLE 2,3, ...
    • The "-n" parameter (divided by 2) of catctl.pl will determine how many PDBs will be upgraded in parallel
    • Your limiting factor is CPU power and cores
    • The "-N" parameter can alter the number of parallel workers per PDB - default is 2
      .

A few simple examples will demonstrate how the parameters work hand-in-hand: 

  • CDB has (always) one PDB$SEED and 25 PDBs numbered PDB1 .. PDB25
  • That means we'll have CYCLE 1 (for the CDB$ROOT) and between one additional CYCLE 2 (for all remaining 26 PDBs in parallel) up to 27 CYCLES once you decide to have no PDBs upgraded at the same time together with another PDB
    .
  • catctl.pl -M -n 16 would lead to:
    • CYCLE 1: CDB$ROOT
    • CYCLE 2: PDB$SEED, PDB1-PDB7 (-n 16 divided by 2 = 8 PDBs to upgrade in parallel)
    • CYCLE 3: PDB8-PDB15
    • CYCLE 4: PDB16-PDB23
    • CYCLE 5: PDB24 and PDB25
    • Each PDB will be upgraded with 2 workers in parallel as -N default is 2
    • The CDB$ROOT will remain in UPGRADE mode until the last PDB is upgraded due to -M setting

  • catctl.pl -M -n 26 would lead to:
    • CYCLE 1: CDB$ROOT
    • CYCLE 2: PDB$SEED, PDB1-PDB12 (-n 26 divided by 2 = 13 PDBs to upgrade in parallel)
    • CYCLE 3: PDB13-PDB25
    • Each PDB will be upgraded with 2 workers in parallel as -N default is 2
    • The CDB$ROOT will remain in UPGRADE mode until the last PDB is upgraded due to -M setting

  • catctl.pl -n 52 -N 1 would lead to:
    • CYCLE 1: CDB$ROOT
    • CYCLE 2: PDB$SEED, PDB1-PDB25 (-n 52 divided by 2 = 26 PDBs to upgrade in parallel)
    • Each PDB will be upgraded with 1 workerl as -N is 1
    • The CDB$ROOT will be available once upgraded - same applies for PDBs once the upgrade is finished

Step-by-step instructions:

Preupgrade Steps

  • Create a guaranteed restore point in order to flashback in case of failure
    CREATE RESTORE POINT UPGRADE GUARANTEE FLASHBACK DATABASE;
  • Copy preupgrd.sql and utluppkg.sql from the Oracle 12.1.0.2 home's ?/rdbms/admin into the source Oracle 12.1.0.1 ?/rdbms/admin directoryTHIS STEP IS EXTREMLY IMPORTANT as otherwise the preupgrd.sql and the utluppkg.sql from 12.1.0.1 will be loaded into the 12.1.0.1 database - but the concept requires always to use the preupgrd.sql/utluppkg.sql from the higher version. If you fail you see it (a) hanging and (b) getting plenty of errors.
  • Execute preuprd.sql within the source database - database needs to be up and running:
    $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl -n 1 -d $ORACLE_HOME/rdbms/admin -l /home/oracle/mike -b preupgrd preupgrd.sql
    • It will create 3 files which combine all information for the root, the seed and all pdbs together into one preupgrade.log, one preupgrade_fixups.sql and one postupgrade_fixups.sql. Default location for those files is $ORACLE_HOME/cfgtoollogs/<SID>/preupgrade
    • Verify the preupgrad.log and follow all advices
  • Execute the preupgrade_fixups.sql while all PDBs are open:
    ALTER PLUGGABLE DATABASE ALL OPEN;
    $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl -n 1 -d $ORACLE_HOME/cfgtoollogs/cdbupgr/preupgrade -l /home/oracle/mike -b preupgrade_fixups preupgrade_fixups.sql
  • Copy the init<sid>.ora into the new $ORACLE_HOME/dbs
  • Specific stepts for RAC environments:
    • Set cluster_database=false
      ALTER SYSTEM SET CLUSTER_DATABASE=FALSE SCOPE=SPFILE;
    • Stop all instances
      srvctl stop database -d <db_name>

Upgrade

  • Switch to the new $ORACLE_HOME including all necessary env variables
  • Connect with sqlplus:
    sqlplus / as sysdba
  • Bring the CDB$ROOT instance into upgrade mode:
    STARTUP UPGRADE
  • Bring all PDBs into upgrade mode:
    ALTER PLUGGABLE DATABASE ALL OPEN UPGRADE;
  • Control it with:
    SHOW PDBS
    Status should be MIGRATE for all PDBs
  • Exit from SQL*Plus and cd to $ORACLE_HOME/rdbms/admin :
    EXIT
    cd $ORACLE_HOME/rdbms/admin
  • Perform the upgrade in parallel:
    $ORACLE_HOME/perl/bin/perl catctl.pl -d $ORACLE_HOME/rdbms/admin -n 16 -M -l /home/oracle/mike catupgrd.sql
  • The important file with timings per PDB and to for a quick check is called upg_summary.log and can be found in:
    $ORACLE_HOME/cfgtoollogs/<SID>/upgrade/upg_summary.log

Postupgrade Steps


  • Only in case -M hasn't been used then the CDB remains open during the upgrade of the PDBs and will need to be shutdown manually post upgrade:
    SHUTDOWN IMMEDIATE
  • Followed by a startup all PDBs must be opened now for recompilation
    STARTUP
    ALTER PLUGGABLE DATABASE ALL OPEN;
  • Execute the postupgrade_fixups.sql:
    $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl -n 1 -d $ORACLE_HOME/cfgtoollogs/cdbupgr/preupgrade -l /home/oracle/mike -b postupgrade_fixups postupgrade_fixups.sql
  • Exit from SQL*Plus and cd into $ORACLE_HOME/rdbms/admin :
    EXIT
    cd $ORACLE_HOME/rdbms/admin
  • The recompilation is done via catcon.pl using the utlrp.sql script from within ?/rdbms/admin:
    $ORACLE_HOME/perl/bin/perl catcon.pl -n 1 -e -b utlrp -d '''.''' utlrp.sql
    1. utl.rp does not parallelize between PDBs
    2. Default parallel degree is: cpu_cores x 2
    3. It takes roughly 45 sec per PDB PLUS another 30 seconds to initialize XDB - as this happens serially it takes approx a while to complete recompilation past upgrade 
  • Drop the guaranteed restore point
    DROP RESTORE POINT UPGRADE;
  • In RAC environments only:
    • Set cluster_database=true again
      ALTER SYSTEM SET CLUSTER_DATABASE=TRUE SCOPE=SPFILE;
    • Start all instances
      srvctl start database -d <db_name>

Seems to be a lot of work, too? But in fact most of the steps are applicable to any upgrade such as the preupgrd.sql etc. Only remarkable change is the need to start scripts with catcon.pl - and not directly within SQL*Plus. Please remember that this approach will upgrade as many PDBs as you have depending on your CPU power in parallel.

In my test environment (a very outdated Exadata V1 without Flash, 6 year old disks, 2 physical CPU sockets) I upgrade 25 PDBs each roughly 25 GB in size with user data in it and all options present in less than 3 hours including recompilation. Please repeat this exercise with 25 independent databases consolidated on the same node within the same time ;-) Try it :-)

-Mike 

Tuesday Aug 05, 2014

Upgrade PDBs - One at a Time (unplug/plug)

Basically there are two techniques to upgrade an Oracle Multitenant environment:

In this post I will refer to the "One at a Time" approach and describe the steps. During some presentations, discussions etc people were left with the impression that it will be a very simple approach to unplug one or many PDBs from a CDB in lets say Oracle 12.1.0.1 and plug it into an Oracle 12.1.0.2 Container Database. Bingo, upgraded!

Well, unfortunately this is not true. In fact it is completely wrong.


If you want to upgrade via unplug/plug the following steps will have to be followed:

  • In CDB1 environment - e.g. Oracle 12.1.0.1 with an PDB1
    • In SQL*Plus: 
      • alter session set container=PDB1;
      • @$ORACLE_HOME_12102/rdbms/admin/preupgrd.sql
        (The output of the preupgrade.log will show you the location of the fixups)
      • @/u01/app/oracle/cfgtoollogs/CDB1/preupgrade/preupgrade_fixups.sql
        (If ORACLE_BASE is not set the files will be created under $ORACLE_HOME/cfgtoollogs instead of $ORACLE_BASE/cfgtoollogs)
      • exec dbms_stats.gather_dictionary_stats;
        (plus include all additional treatments recommended by the preupgrade.log)
      • alter sesstion set container=CDB$ROOT; 
      • alter pluggable database PDB1 close;
      • alter pluggable database PDB1 unplug into '/stage/pdb1.xml';
      • exit
        .
  • In CDB2 environment - e.g. Oracle 12.1.0.2
    • In SQL*Plus:
      • alter session set container=CDB$ROOT;
      • At this point we "could" do a Plug In Check but as the COMPATIBLE of the new CDB2 created as per recommendation with DBCA defaults to "12.1.0.2" the Plug In Check will result in "NO" - but obviously the plugin operation will work. Just for the records here's the procedure to check plugin compatibility
        • SET SERVEROUTPUT ON
          DECLARE
            compatible CONSTANT VARCHAR2(3) := CASE DBMS_PDB.CHECK_PLUG_COMPATIBILITY(
            pdb_descr_file => '/stage/pdb1.xml',
            pdb_name => 'PDB1')
            WHEN TRUE THEN 'YES' ELSE 'NO'
          END;
          BEGIN
          DBMS_OUTPUT.PUT_LINE(compatible);
          END;
          /

          .
          select message, status from pdb_plug_in_violations where type like '%ERR%';
          .
      • create pluggable database pdb1 using '/stage/pdb1.xml' file_name_convert=('/oradata/CDB1/pdb1', '/oradata/CDB2/pdb1');
      • alter pluggable database PDB1 open upgrade;
      • exit
    • On the command prompt:
      • cd $ORACLE_HOME/rdbms/admin 
      • $ORACLE_HOME/perl/bin/perl catctl.pl -c "PDB1" -l /home/oracle/upgrade catupgrd.sql
    • Back into SQL*Plus:
      • alter session set container=pdb1;
      • startup
      • @?/rdbms/admin/utlrp.sql
      • @/u01/app/oracle/cfgtoollogs/CDB1/preupgrade/postupgrade_fixups.sql
        (If ORACLE_BASE is not set the files will be created under $ORACLE_HOME/cfgtoollogs instead of $ORACLE_BASE/cfgtoollogs)
Of course this technique will work also with more than one PDB at a given time. You'll have to repeat the steps, and your upgrade call on the command line will look like this:

      • $ORACLE_HOME/perl/bin/perl catctl.pl -c "PDB1, PDB2" -l /home/oracle/upgrade catupgrd.sql

Well, not really unplug+plug=upgraded ;-)

-Mike 

PS: I did add a few pieces of information based on the excellent feedback given to me by Frank Kobylanski from the MAA Team - cheers, Frank!!! 

Monday Jul 07, 2014

Remote Cloning of Pluggable Databases
in Oracle Database 12.1.0.1

Cloning of Pluggable Database in Oracle Multitenant is a great feature and very useful. It had just a tiny flaw in Oracle Database 12.1.0.1: Remote cloning from one to another CDB fails

This will work flawless beginning with the first patch set Oracle 12.1.0.2 - but if you want to use that feature still in Oracle 12.1.0.1 you need to apply at least PSU3 or a single patch 18898403.

See the documentation

-Mike 

Friday Jun 06, 2014

Save Upgrade downtime: Upgrade APEX upfront

With almost every patch or release upgrade of the Oracle Database a new version of Oracle Application Express (APEX) will be installed. And as APEX is part of the database installation it will be upgraded as part of the component upgrades after the ORACLE SERVER component has been successfully upgraded to the new releases.

APEXBut the APEX upgrade can take a bit (several minutes or even more in some cases). Therefore it is a common advice to upgrade APEX upfront before upgrading the database as this can be done online while the database is in production (unless your databases serves just as an APEX application backend - in this case upgrading APEX upfront won't save you anything).

To upgrade Oracle APEX upfront you'll have to follow
MOS Note:1088970.1.

It explains that you'll have to:

  1. Determine the installation type by running this query:
    select count(*) from <SCHEMA>.WWV_FLOWS where id = 4000;
    whereas <SCHEMA> can be one of the following:
    FLOWS_010500 1.5.X 
    FLOWS_010600 1.6.X 
    FLOWS_020000 2.0.X 
    FLOWS_020100 2.1.X 
    FLOWS_020200 2.2.X 
    FLOWS_030000 3.0.X 
    FLOWS_030100 3.1.X  
    APEX_030200 3.2.X 
    APEX_040000 4.0.X
    APEX_040100 4.1.X
    APEX_040200 4.2.X

    If the query returns 0 then you'll need to run apxrtins.sql
    If the query returns 1 then you'll need to execute apexins.sql

  2. Download the newest APEX package and install it.

-Mike

Monday May 26, 2014

New interesting White Paper:
Oracle Multitenant Efficiency Study

There's a new White Paper available interesting for those who'd like to learn a bit more about the efficiency and the overhead of Oracle Multitenant in a fairly large environment:

http://www.oracle.com/technetwork/database/multitenant/learn-more/oraclemultitenantt5-8-final-2185108.pdf

-Mike

Tuesday May 13, 2014

More than one PDB in the same directory?

Can you create more than one pluggable database (PDB) within the same directory?
And how does the file naming work? Considering the fact each PDB's SYSTEM tablespace will be named system01.dbf by default the question is not trivial. 

This question got asked by a customer during one of the workshops in Switzerland last week. And the solution is straight forward. Thanks to Roy for trying it out yesterday at 170 km/h on our way back from Stuttgart :-)

Thanks :-)

-Mike 

Additional information:

Within ASM with OMF the file structure looks like this:

 1  select con_id, substr(file_name,1,90),tablespace_name from cdb_data_files
  2* order by 1

    CON_ID SUBSTR(FILE_NAME,1,90)                                                           TABLESPACE_NAME
---------- -------------------------------------------------------------------------------- ---------------
         1 +DA1/CDBUPGR/DATAFILE/system.394.845632641                                       SYSTEM
         1 +DA1/CDBUPGR/DATAFILE/users.475.845632685                                        USERS
         1 +DA1/CDBUPGR/DATAFILE/undotbs4.448.845632683                                     UNDOTBS4
         1 +DA1/CDBUPGR/DATAFILE/sysaux.392.845632651                                       SYSAUX
         1 +DA1/CDBUPGR/DATAFILE/undotbs2.393.845632679                                     UNDOTBS2
         1 +DA1/CDBUPGR/DATAFILE/undotbs1.471.845632657                                     UNDOTBS1
         1 +DA1/CDBUPGR/DATAFILE/undotbs3.478.845632681                                     UNDOTBS3
         2 +DA1/CDBUPGR/F7B70DCBF2D4ECEAE0437A28890AE4D8/DATAFILE/sysaux.472.845632655      SYSAUX
         2 +DA1/CDBUPGR/F7B70DCBF2D4ECEAE0437A28890AE4D8/DATAFILE/system.398.845632647      SYSTEM
         3 +DA1/CDBUPGR/F6A142792168D540E0437A28890A4707/DATAFILE/system.493.845643325      SYSTEM
         3 +DA1/CDBUPGR/F6A142792168D540E0437A28890A4707/DATAFILE/sysaux.468.845643325      SYSAUX
         3 +DA1/CDBUPGR/F6A142792168D540E0437A28890A4707/DATAFILE/soets.452.845643325       SOETS
         4 +DA1/CDBUPGR/F7B9BDC2AEC4411EE0437A28890A2B81/DATAFILE/system.491.845643937      SYSTEM
         4 +DA1/CDBUPGR/F7B9BDC2AEC4411EE0437A28890A2B81/DATAFILE/sysaux.488.845643937      SYSAUX
         4 +DA1/CDBUPGR/F7B9BDC2AEC4411EE0437A28890A2B81/DATAFILE/soets.484.845643937       SOETS
         5 +DA1/CDBUPGR/F7B9CA6B92804A56E0437A28890A2721/DATAFILE/system.485.845644149      SYSTEM
         5 +DA1/CDBUPGR/F7B9CA6B92804A56E0437A28890A2721/DATAFILE/sysaux.490.845644149      SYSAUX
         5 +DA1/CDBUPGR/F7B9CA6B92804A56E0437A28890A2721/DATAFILE/soets.487.845644149       SOETS
         6 +DA1/CDBUPGR/F7B9D727715B5B4AE0437A28890AB3D9/DATAFILE/system.486.845644363      SYSTEM
         6 +DA1/CDBUPGR/F7B9D727715B5B4AE0437A28890AB3D9/DATAFILE/sysaux.483.845644363      SYSAUX
         6 +DA1/CDBUPGR/F7B9D727715B5B4AE0437A28890AB3D9/DATAFILE/soets.481.845644363       SOETS
         7 +DA1/CDBUPGR/F7B9E3D23CFC67F1E0437A28890A5A68/DATAFILE/system.453.845644575      SYSTEM
         7 +DA1/CDBUPGR/F7B9E3D23CFC67F1E0437A28890A5A68/DATAFILE/sysaux.482.845644575      SYSAUX
         7 +DA1/CDBUPGR/F7B9E3D23CFC67F1E0437A28890A5A68/DATAFILE/soets.467.845644575       SOETS
         8 +DA1/CDBUPGR/F7B9F051E81B7892E0437A28890AD3A3/DATAFILE/system.465.845644785      SYSTEM
         8 +DA1/CDBUPGR/F7B9F051E81B7892E0437A28890AD3A3/DATAFILE/sysaux.455.845644785      SYSAUX
         8 +DA1/CDBUPGR/F7B9F051E81B7892E0437A28890AD3A3/DATAFILE/soets.479.845644785       SOETS
         9 +DA1/CDBUPGR/F7BA2D0F2F17A755E0437A28890A72C6/DATAFILE/system.464.845645805      SYSTEM
         9 +DA1/CDBUPGR/F7BA2D0F2F17A755E0437A28890A72C6/DATAFILE/sysaux.500.845645805      SYSAUX
         9 +DA1/CDBUPGR/F7BA2D0F2F17A755E0437A28890A72C6/DATAFILE/soets.498.845645805       SOETS
        10 +DA1/CDBUPGR/F7BA3A179DAFB12FE0437A28890ABBF3/DATAFILE/system.499.845646023      SYSTEM
        10 +DA1/CDBUPGR/F7BA3A179DAFB12FE0437A28890ABBF3/DATAFILE/sysaux.504.845646023      SYSAUX
        10 +DA1/CDBUPGR/F7BA3A179DAFB12FE0437A28890ABBF3/DATAFILE/soets.502.845646023       SOETS
        11 +DA1/CDBUPGR/F7BA46A1A6B7B9C2E0437A28890AE021/DATAFILE/system.503.845646233      SYSTEM
        11 +DA1/CDBUPGR/F7BA46A1A6B7B9C2E0437A28890AE021/DATAFILE/sysaux.508.845646233      SYSAUX
        11 +DA1/CDBUPGR/F7BA46A1A6B7B9C2E0437A28890AE021/DATAFILE/soets.506.845646233       SOETS
...


Wednesday Apr 23, 2014

"Mr. Upgrade" on DOAG.TV

Mike may be too humble to post this himself, so I will point out that the Oracle user group in Germany (DOAG) has published a video of an interview with Mike about upgrade to Oracle Database 12c. The interview is in German, so if you speak the language, please enjoy:

DOAG.TV screenshot

Those who  are inclined to view the video may also want to pay attention to the announcement about upgrade workshops in Germany. See Mike's blog entry about our upcoming events for more information!

Friday Apr 11, 2014

Why cows can fly! Believe it or not ...

A colleague of mine, Murilo from Oracle Brasil talked about our roles at Oracle (but this applies to any other software company in the market I'd guess).

Do you believe in flying cows?

  • Sales
    In Sales you'll sell a customer that cows can fly
  • Presales
    The presales consultant will have to do a presentation showing the ROI, TCO and KPIs of flying cows. She or he may have also a screenshot demo in Powerpoint showing flying cows. 
  • Consulting
    The consultant is the person to install the flying-cow piece of software at the customer site and embedd it into the customer's environment. Unfortunately the poor consultant finds out that in this environment cows actually don't fly well. In fact, cows don't fly at all. So the consultant may raise an SR before going home.
  • Support
    The SR gets into the queue of a Support engineer somewhere in the world. The Support engineer wonders about this SR as she or he has never heard of flying cows before. So she/he takes a look into the doc. And there's nothing mentioned about flying cows. But the doc doesn't say anything about "cows can't fly" so a testcase will be required from the customer or the consultant to show the non-flying cows. Poor Support engineer: she/he can't even file a bug for it as the doc doesn't say anything about the flying-cow-feature. 

Guess how that story will end?
Does it sound familiar to you? To me it does ... a bit ;-)
And I'm glad that I'm working in Development :-)

So you may watch this tiny little video in your spare time ...

-Mike

PS: Thanks to Murilo - and I phrased it in my own words as I still burst out laughing and crying at the same time when thinking about our conversation :-) But credits go to him - obgrigado!!!

PPS: No offense to anybody in any role in a software company ... 

Thursday Mar 13, 2014

Do you know?
Premier Support for Oracle 11.2 will end soon!!!

------------------------------------------------

*** Please read this newer blog post as well ***
https://blogs.oracle.com/UPGRADE/entry/premier_support_for_oracle_11
*** Please read this newer blog post as well ***

------------------------------------------------

Do you know that Oracle Database 11.2 will run out of PREMIER SUPPORT in less than 11 months?

Well. you may mention now the waived first year of free EXTENDED SUPPORT but this will apply to Oracle Database 11.2.0.4 (and to Oracle Database 11.2.0.3 until Aug 27, 2015) only!!! Now if you think of upgrading to Oracle 11.2.0.4 within the next months start thinking more into the direction of Oracle Database 12c instead. There won't be much difference in efforts and the amount of time you'll spend, regardless if you'd like to Oracle Database 11.2.0.4 or Oracle Database 12c.

Don't waste your time, resources and money - start evaluation Oracle Database 12c today!!! 

Now a bit of clarification as somebody has raised this question:
MOS Note:209768.1 Error Correction Policy describes clearly what you can expect after the Premier Support for Oracle Database 11.2 has ended.

  • Error correction is only available for the: 
  • Current patch set, and 
  • Previous patch set, for the duration of the Grace Period
  •  What does the Grace Period mean?

    Grace Period: up to 1 year for first patch set (minimum 3 months), and up to 2 years for second and subsequent patch sets. 

    • First Patch Set: Customers on all platforms have up to one year from the release of the first patch set on the first platform of a major release to install it.

    • Second and Subsequent Patch Sets: For the second and any subsequent patch set release, customers on all platforms have up to two years from the release of a new patch set to install it. 

    As an example, for release X.Y (base release version X.Y.0.1): 

    • First patch set for (X.Y.0.2) comes out on Linux x86-64 in September 2010. Customers on all platforms have until September 2011 to install it. 

    • Second patch set (X.Y.0.3) comes out in September 2011. Customers would have two years – until September 2013 – to install X.Y.0.3. 

    • Third patch set (X.Y.0.4) comes out in July 2013, customers have until July 2015 to install it.

    During the grace period we will create new bug fixes for the previous patch set and the current patch set.

    Patching end dates for current and most recent patch sets can be found on My Oracle Support in Release Schedule of Current Database Releases (Doc ID 742060.1)

    --------
    CORRECTION:

    I appologize but I got used to Support policies for 18 years now and therefore got to learn about something which got removed with Oracle Database 10.2.0.5. The direct combination of "Terminal Patch Set" and "Extended Support" has been removed a while back (and I didn't know it). That means (and the above posting has been corrected already):

    In order to get Extended Support for Oracle Database 11.2 you will have to be on Oracle Database 11.2.0.4 (the terminal patch set) or on Oracle Database 11.2.0.3. In the latter case patching will end by August 27, 2015 - which is described clearly in MOS Note: 742060.1


    Release Patching Ends

    Exceptions*

    11.2.0.4 31-Jan-2018
    HP-UX Itanium: Patching ends Dec 2020. Beginning Feb 1, 2018, Sev 1 fixes only (no PSU or CPU will be produced).
    11.2.0.3 27-Aug-2015

    11.2.0.2 31-Oct-2013
    End date extended beyond normal.

    Tuesday Mar 04, 2014

    Upgrade to Grid Infrastructure - but OCR and VOTING on RAW?

    Just received this question from a colleague these days:

    "The current customer environment is 10.2.0.5 on Linux with a 2 node RAC cluster having OCR and Voting Disks on RAW devices. Customer is concerned about the possibility of upgrading to 11gR2 Grid infrastructure first before they could upgrade to 12c Grid infrastructure."

    Now the answer is written down in MOS Note 1572925.1:
    How to Upgrade to 12c Grid Infrastructure if OCR or Voting File is on Raw/Block Devices

    Basically the MOS Note says:
    You will have to relocate your OCR/Voting to a supported device BEFORE you can upgrade to Oracle Grid Infrastructure 12c. No way out. A bit more clarification (thanks to Markus Michalewicz):

    The assumption of the note (which you might want to state) is that the customer has pre-12c GI with OCR / Voting Disk on RAW.

    In this case, Option A is always an option.

    For Option B, however, you need to distinguish. So the note should say: 

    • Option B: Customer is on 11g Rel. 2 still using RAW Devices 
      • Then move the OCR and Voting Disks to ASM
    • Option C: Customer is on pre-11g Rel. 2 and does not want to introduce a CFS (bad idea anyways) or use NFS
      • Then upgrade to 11g Rel. 2 and move the Clusterware files into ASM as mentioned under Option B.

    Unfortunately another example to add to my collection of "What happens if you stay too long on older releases?". In this case it simply increases complexity and drives costs as well. And please no complaints: Oracle Database 10.2 went out of PREMIER SUPPORT in July 2010 - 3.5 years ago!!!

    -Mike

    Thursday Feb 27, 2014

    CSX Corporation Upgrades Databases 2x Faster With Oracle Real Application Testing

    Wow - one of our reference projects went into the Yahoo Finance webpages. Carol Tagliaferri (my direct manager) worked with CSX, one of the largest US based railway companies, over a longer period together with the RAT team, and helped CSX to successfully upgrade and migrate their database landscape to Oracle Database 11.2.
    Plus: We did present about CSX at Oracle Open World 2012 as this project is a perfect showcase for:

    (a) an Upgrade project with many databases
    (b) over a longer period of time
    (c) using Real Application Testing decreasing the testing effort by factors!

    Read the published article on Yahoo Finance!

    And that is what Maritza Gonzalez, Technical Director at CSX said:

    “The Real Application Testing tool provided a comprehensive and flexible solution for assessing the impact of the Oracle 11g database upgrade  into CSX systems.  At CSX we were able to capture real production workloads,  replay it in the 11g environment,  identify poor performing  queries and, fine tune these queries in a test environment  before the production implementation.“

     More details on the actual project are included in our OOW 2012 talk (slides 35-42).

    -Mike
    Graphics are courtesy linked directly from the CSX public webpage

    Tuesday Feb 25, 2014

    Simplify your Migration from AIX to Solaris

    Oracle Solaris 11There is a brand new white paper available that will be of interest to anybody contemplating a database and/or application migration from AIX to Solaris. Simplify the Migration of Oracle Database and Oracle Applications from AIX to Oracle Solaris gives a good description of the steps involved in planning and executing a migration project, along with the benefits you can expect to achieve and a solid example of migration using Oracle Data Pump, complete with scripted steps.

    Of course, if your migration will include moving up to Oracle Database 12c, don't forget to sign up for Mike's webcast on February 26!

    Thursday Feb 13, 2014

    Collaborate '14 Early Bird Registration ends Today!

    Mike and I are in Sydney, Australia delivering an upgrade workshop today, but I am also looking forward to the Collaborate '14 conference in April. I will have a talk about upgrading to Oracle Database 12c, and of course there will be plenty of information about Oracle Multitenant, applications, and other aspects of your Oracle environment. 

    Collaborate 14 banner

    Early bird registration ends today, so if you want to save some money and attend a conference that is always chock full of good technical presentations, register today!

    Tuesday Feb 11, 2014

    New 11.2.0.4 Parameter: ENABLE_GOLDENGATE_REPLICATION

    Just learned something new I couldn't find actually in the doc at the first glance:

    There's a new init.ora parameter introduced in Oracle Database 11.2.0.4 named:
    ENABLE_GOLDENGATE_REPLICATION

    By default it is set to FALSE and the parameter got introduced because not only the external use of Oracle GoldenGate requires a valid license but also the use of the internal APIs. For example, XStream provides high performance APIs that enable client applications to receive and send real-time data changes from an Oracle database. Other APIs were added for encryption support, trigger suppression, etc. None of these APIs are licensed with the Oracle database itself - the license is included with an Oracle GoldenGate license, and most of the APIs are not public.

    So please make sure if you turn ENABLE_GOLDENGATE_REPLICATION=TRUE that you'll need to have a valid Oracle GoldenGate license in place.

    That will apply to 3rd party products such as Dell Shareplex as well in case they'll switch it to true.

    -Mike

    ---
    Addition: As a customer talked to me about that during the Sydney workshop last week I'd like to point out that this paramater does not effect the use of Oracle GoldenGate as a stand-alone tool but only the internal interfaces inside the database.

    Monday Dec 16, 2013

    Why you shouldn't set OPTIMIZER_FEATURES_ENABLE

    Roy today answered an interesting question on one of our internal mailing lists. And this reminds me to pick up that piece of information as we see this quite often on customer sites, especially after upgrades or migrations. People set OPTIMIZER_FEATURES_ENABLE (OFE) to revert the optimizer's behaviour to another pre-current release. That's what a lot of people think this parameter does.

    But in fact this is not true. Even though our documentation states it:

    OPTIMIZER_FEATURES_ENABLE acts as an umbrella parameter for enabling a series of optimizer features based on an Oracle release number.

    For example, if you upgrade your database from release 10.1 to release 11.1, but you want to keep the release 10.1 optimizer behavior, you can do so by setting this parameter to 10.1.0. At a later time, you can try the enhancements introduced in releases up to and including release 11.1 by setting the parameter to 11.1.0.6.

    In my experience the following is true:
    Setting this parameter reverts the optimizer settings in terms of parameters - but we still use the new optimizer code from that release you are on now. And as far as I know nobody ever tests if switching back OFE will turn back to exactly the behaviour as it was known in the release OFE is now switched to. So it's simply guessing and assuming. But as code got changed there's very little chance to get back to the old behaviour. 

    What people sometimes  experience:
    Turning back OFE brings back good performance. This can happen. But if we act really really really precise than the performance should always be better with the optimizer running with the new settings - and not when the wheel is turned back. So in some cases this should be treated as a bug unless new behaviour leads to predictable worse results (such as more buffer gets etc). And I get so angry when Oracle Support people recommend to switch OFE to this or that setting to cure one or the otther misbehaviour. That's like throwing a big rock to kill a fly. 

    But the real danger is described in the following note - and I'm pretty sure a lot of people are not aware of this:
    Reverting the optimizer parametrization with OFE can turn back to misbehaviour which got fixed already in the current release. MOS Note:1581465.1 describes this pretty well.

    And in addition regarding upgrades you may want to read this note here as well:
    MOS Note: 1362332.1
    Use Caution if Changing the OPTIMIZER_FEATURES_ENABLE Parameter After an Upgrade
    (Thanks Roy!)

    Recommendation:
    Stay away from tweaking anything with OFE. Use Real Application Testing's SQL Performance Analyzer (SPA) to find out which plans get changed and use SQL Plan Management (SPM) to nail down misbehaving plans in 11g or 12c.

    -Mike 

    Friday Dec 06, 2013

    INTERACTIVE Database 12c QUICK REFERENCE

    The Oracle Database 12c Interactive Quick Reference is a nice little tool which may help you quickly to find a view you are searching for or to get more information about background processes in Oracle Database 12c.

    Click through it:
    Oracle Database 12c: INTERACTIVE QUICK REFERENCE

    -Mike

    Tuesday Dec 03, 2013

    Starting up 252 PDBs automatically?

    In my recent posting I have explained the startup of many PDBs at the same time.

    But once you startup the container database CDB$ROOT the PDBs will stay in MOUNT status. So how do you start them during CDB$ROOT startup (or immediately afterwards) in an automatic fashion?

    A startup trigger will do this job.

    CREATE OR REPLACE TRIGGER startup_all_pdbs
    AFTER STARTUP ON DATABASE

    BEGIN

    EXECUTE IMMEDIATE 'ALTER PLUGGABLE DATABASE ALL OPEN';

    END;

    /

    And of course you can use the EXCEPT command option to exclude one or more PDBs from the automatic startup.

    CREATE OR REPLACE TRIGGER startup_all_pdbs_except_a_few
    AFTER STARTUP ON DATABASE

    BEGIN
    EXECUTE IMMEDIATE 'ALTER PLUGGABLE DATABASE ALL OPEN EXCEPT PDB100, PDB101';
    END;
    /

    How does this work in an Oracle Real Application Clusters environment?
    In an RAC environment you won't need the startup trigger as clusterware takes over this role of ensuring the automatic startup of a PDB on designated nodes within the CDB$ROOT's instances.

    srvctl add service -db rac -service pdbrac_srv -pdb pdbrac -preferred "rac1,rac2"

    A snipet from the crsctl status output will look like this:

       crsctl status resource -t
        :
       ora.rac.db
             1    ONLINE  ONLINE   rac-server01       Open,STABLE
             2    ONLINE  ONLINE   rac-server02       Open,STABLE
       ora.rac.pdbrac_srv.svc
             1    ONLINE  ONLINE   rac-server01       STABLE
             2    ONLINE  ONLINE   rac-server02       STABLE
        :

    -Mike

    Friday Nov 29, 2013

    Starting up 252 PDBs in Oracle Multitenant

    What happens when you start up 252 PDBs (Pluggable Databases) with the Oracle Multitenant Option for the first time?

    Interesting question, isn't it? The expectation would be that this will happen within seconds as the SGA and therefore the shared memory segments are already allocated from within the CDB$ROOT (Container Database). But ...

    The following happens:
    It takes minutes ... hours .... In my tiny lab environment with just as little as 20 PDBs due to space constraints it takes over 30 minutes to startup 21 PDBs. Takashi Ikeda from Fujitsu Hokoriku Systems who did a great demo with the new Fujitsu M10 servers at OOW this year told me that it took over two hours to start up 252 PDBs for the first time.
    Why is that?

    Let's have a closer look into the alert.log during startup. After issueing the command:

    ALTER PLUGGABLE DATABASE ALL OPEN;

    I'd expect all PDBs to get started. With an EXCEPT PDB1, PDB2, PDB3 clause I could exclude some PDBs from this action. Now a look into the alert.log shows a very promising message:

    I'm just wondering about the opening sequence of PDBs. I'd expect 1 ... 2 ... 3 ... 4 ... ... ... 21. But the "order" is 3 ... 10 ... 16 ... 15 ... 20 ... 21 etc. telling me that the Resource Manager is not active (which is a must if you take Multitenant serious).
    OK, for that strange order there's an explanation:
    The open action gets distrubuted to slaves so PDBs may opened in a random order.
    Fuuny thing apart from that: I can access the PDB but the system seems to be really under heavy pressure. CPUs are all at 100%. What the heck is going on here in the background?

    Well, XDB needs to be installed (at least that is what the message says). Strange, isn't it, as the PDB$SEED has XDB in it and all my PDBs got provisioned from it. The awkward thing here is that the XDB messages appear over 20 minutes AFTER the PDBs signaled the Opening message into the alert.log (see the time stamps above).

    Now after exchanging a few emails with some very helpful people in development there's an explanation for the XDB messages as well. Actually it doesn't get really installed but the SGA needs to be initialized for XDB. And I'm guessing that this action takes a lot of resources plus may cause contention when many PDBs get opened at the same time. And there's optimization work going on right now meaning that a problem with port initialization within the PDB will get fixed in a future patch set. So this issue with the very long startups of PDBs because of XDB should disappear in 12.1.0.2 most likely :-)

    Finally it took another while to get the PDBs really into OPEN mode. Even though they were showing OPEN before already in V$PDBS. But as the CPUs all went to 100% as XDB got installed/initiallized at more or less the same time in all PDBs you really can't do anything.

    Finally ...

    ... all PDBs got opened and the command ALTER PLUGGABLE DATABASE ALL OPEN returned completed.

    The good news:
    It takes just this long during the initial startup of a newly provisioned PDB. And you may see this issue only when you try to open many PDBs at the same time. But have a close look into your alert.log if you'll spot the message after creating a fresh PDB.

    And btw, just for the records: I was using Oracle Database 12.1.0.1 with Oct 2013 PSU in it.

    -Mike

    About

    Mike Dietrich - Oracle Mike Dietrich
    Senior Principal Technologist - Database Upgrade Development Group - Oracle Corporation

    Based near Munich/Germany and spending plenty of time in airplanes to run either upgrade workshops or work onsite with reference customers. Acting as interlink between customers and the Upgrade Development.

    Contact me either via XING or LinkedIn

    Search

    Archives
    « March 2015
    SunMonTueWedThuFriSat
    1
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
        
           
    Today
    Oracle related Tech Blogs
    Slides Download Center
    Visitors since 17-OCT-2011
    White Paper and Docs
    Workshops
    Viewlets and Videos
    This week on my Rega/iPod/CD
    Workshop Map
    Upgrade Reference Papers