Friday Aug 22, 2014

Automatic Maintenance Jobs in every PDB?
New SPM Evolve Advisor Task in Oracle 12.1.0.2

A customer checking out our slides from the OTN Tour in August 2014 asked me a finicky question the other day:

"According to the documentation the Automatic SQL Tuning Advisor maintenance task gets executed only within the CDB$ROOT, but not within each PDB - but the slides are not clear here. So what is the truth?"

Ok, that's good question. In my understanding all tasks will get executed within each PDB - that's why we recommend (based on experience) to break up the default maintenance windows when using Oracle Multitenant. Otherwise all PDBs will have the same maintenance windows, and guess what will happen when 25 PDBs start gathering object statistics at the same time ...

The documentation indeed says:

Automatic SQL Tuning Advisor data is stored in the root. It might have results about SQL statements executed in a PDB that were analyzed by the advisor, but these results are not included if the PDB is unplugged. A common user whose current container is the root can run SQL Tuning Advisor manually for SQL statements from any PDB. When a statement is tuned, it is tuned in any container that runs the statement.

This sounds reasonable. But when we have a look into our PDBs or into the CDB_AUTOTASK_CLIENT view the result is different from what the doc says. In my environment I did create just two fresh empty PDBs (CON_ID 3 and 4):

SQL> select client_name, status, con_id from cdb_autotask_client;

CLIENT_NAME                           STATUS         CON_ID
------------------------------------- ---------- ----------
auto optimizer stats collection       ENABLED             1
sql tuning advisor                    ENABLED             1
auto space advisor                    ENABLED             1
auto optimizer stats collection       ENABLED             4
sql tuning advisor                    ENABLED             4
auto space advisor                    ENABLED             4
auto optimizer stats collection       ENABLED             3
sql tuning advisor                    ENABLED             3
auto space advisor                    ENABLED             3

9 rows selected.

I haven't verified the reason why this is different from the docs but it may have been related to one change in Oracle Database 12.1.0.2: The new SPM Evolve Advisor Task ( SYS_AUTO_SPM_EVOLVE_TASK) for automatic plan evolution for SQL Plan Management. This new task doesn't appear as a stand-alone job (client) in the maintenance window but runs as a sub-entity of the Automatic SQL Tuning Advisor task. And (I'm just guessing) this may be one of the reasons why every PDB will have to have its own Automatic SQL Tuning Advisor task 

Here you'll find more information about how to enable, disable and configure the new Oracle 12.1.0.2 SPM Evolve Advisor Task:

-Mike

Tuesday Aug 05, 2014

Upgrade PDBs - One at a Time (unplug/plug)

Basically there are two techniques to upgrade an Oracle Multitenant environment:

In this post I will refer to the "One at a Time" approach and describe the steps. During some presentations, discussions etc people were left with the impression that it will be a very simple approach to unplug one or many PDBs from a CDB in lets say Oracle 12.1.0.1 and plug it into an Oracle 12.1.0.2 Container Database. Bingo, upgraded!

Well, unfortunately this is not true. In fact it is completely wrong.


If you want to upgrade via unplug/plug the following steps will have to be followed:

  • In CDB1 environment - e.g. Oracle 12.1.0.1 with an PDB1
    • In SQL*Plus: 
      • alter session set container=PDB1;
      • @$ORACLE_HOME_12102/rdbms/admin/preupgrd.sql
        (The output of the preupgrade.log will show you the location of the fixups)
      • @/u01/app/oracle/cfgtoollogs/CDB1/preupgrade/preupgrade_fixups.sql
        (If ORACLE_BASE is not set the files will be created under $ORACLE_HOME/cfgtoollogs instead of $ORACLE_BASE/cfgtoollogs)
      • exec dbms_stats.gather_dictionary_stats;
        (plus include all additional treatments recommended by the preupgrade.log)
      • alter sesstion set container=CDB$ROOT; 
      • alter pluggable database PDB1 close;
      • alter pluggable database PDB1 unplug into '/stage/pdb1.xml';
      • exit
        .
  • In CDB2 environment - e.g. Oracle 12.1.0.2
    • In SQL*Plus:
      • alter session set container=CDB$ROOT;
      • At this point we "could" do a Plug In Check but as the COMPATIBLE of the new CDB2 created as per recommendation with DBCA defaults to "12.1.0.2" the Plug In Check will result in "NO" - but obviously the plugin operation will work. Just for the records here's the procedure to check plugin compatibility
        • SET SERVEROUTPUT ON
          DECLARE
            compatible CONSTANT VARCHAR2(3) := CASE DBMS_PDB.CHECK_PLUG_COMPATIBILITY(
            pdb_descr_file => '/stage/pdb1.xml',
            pdb_name => 'PDB1')
            WHEN TRUE THEN 'YES' ELSE 'NO'
          END;
          BEGIN
          DBMS_OUTPUT.PUT_LINE(compatible);
          END;
          /

          .
          select message, status from pdb_plug_in_violations where type like '%ERR%';
          .
      • create pluggable database pdb1 using '/stage/pdb1.xml' file_name_convert=('/oradata/CDB1/pdb1', '/oradata/CDB2/pdb1');
      • alter pluggable database PDB1 open upgrade;
      • exit
    • On the command prompt:
      • cd $ORACLE_HOME/rdbms/admin 
      • $ORACLE_HOME/perl/bin/perl catctl.pl -c "PDB1" -l /home/oracle/upgrade catupgrd.sql
    • Back into SQL*Plus:
      • alter session set container=pdb1;
      • startup
      • @?/rdbms/admin/utlrp.sql
      • @/u01/app/oracle/cfgtoollogs/CDB1/preupgrade/postupgrade_fixups.sql
        (If ORACLE_BASE is not set the files will be created under $ORACLE_HOME/cfgtoollogs instead of $ORACLE_BASE/cfgtoollogs)
Of course this technique will work also with more than one PDB at a given time. You'll have to repeat the steps, and your upgrade call on the command line will look like this:

      • $ORACLE_HOME/perl/bin/perl catctl.pl -c "PDB1, PDB2" -l /home/oracle/upgrade catupgrd.sql

Well, not really unplug+plug=upgraded ;-)

-Mike 

PS: I did add a few pieces of information based on the excellent feedback given to me by Frank Kobylanski from the MAA Team - cheers, Frank!!! 

Tuesday Dec 03, 2013

Starting up 252 PDBs automatically?

In my recent posting I have explained the startup of many PDBs at the same time.

But once you startup the container database CDB$ROOT the PDBs will stay in MOUNT status. So how do you start them during CDB$ROOT startup (or immediately afterwards) in an automatic fashion?

A startup trigger will do this job.

CREATE OR REPLACE TRIGGER startup_all_pdbs
AFTER STARTUP ON DATABASE

BEGIN

EXECUTE IMMEDIATE 'ALTER PLUGGABLE DATABASE ALL OPEN';

END;

/

And of course you can use the EXCEPT command option to exclude one or more PDBs from the automatic startup.

CREATE OR REPLACE TRIGGER startup_all_pdbs_except_a_few
AFTER STARTUP ON DATABASE

BEGIN
EXECUTE IMMEDIATE 'ALTER PLUGGABLE DATABASE ALL OPEN EXCEPT PDB100, PDB101';
END;
/

How does this work in an Oracle Real Application Clusters environment?
In an RAC environment you won't need the startup trigger as clusterware takes over this role of ensuring the automatic startup of a PDB on designated nodes within the CDB$ROOT's instances.

srvctl add service -db rac -service pdbrac_srv -pdb pdbrac -preferred "rac1,rac2"

A snipet from the crsctl status output will look like this:

   crsctl status resource -t
    :
   ora.rac.db
         1    ONLINE  ONLINE   rac-server01       Open,STABLE
         2    ONLINE  ONLINE   rac-server02       Open,STABLE
   ora.rac.pdbrac_srv.svc
         1    ONLINE  ONLINE   rac-server01       STABLE
         2    ONLINE  ONLINE   rac-server02       STABLE
    :

-Mike

Friday Nov 29, 2013

Starting up 252 PDBs in Oracle Multitenant

What happens when you start up 252 PDBs (Pluggable Databases) with the Oracle Multitenant Option for the first time?

Interesting question, isn't it? The expectation would be that this will happen within seconds as the SGA and therefore the shared memory segments are already allocated from within the CDB$ROOT (Container Database). But ...

The following happens:
It takes minutes ... hours .... In my tiny lab environment with just as little as 20 PDBs due to space constraints it takes over 30 minutes to startup 21 PDBs. Takashi Ikeda from Fujitsu Hokoriku Systems who did a great demo with the new Fujitsu M10 servers at OOW this year told me that it took over two hours to start up 252 PDBs for the first time.
Why is that?

Let's have a closer look into the alert.log during startup. After issueing the command:

ALTER PLUGGABLE DATABASE ALL OPEN;

I'd expect all PDBs to get started. With an EXCEPT PDB1, PDB2, PDB3 clause I could exclude some PDBs from this action. Now a look into the alert.log shows a very promising message:

I'm just wondering about the opening sequence of PDBs. I'd expect 1 ... 2 ... 3 ... 4 ... ... ... 21. But the "order" is 3 ... 10 ... 16 ... 15 ... 20 ... 21 etc. telling me that the Resource Manager is not active (which is a must if you take Multitenant serious).
OK, for that strange order there's an explanation:
The open action gets distrubuted to slaves so PDBs may opened in a random order.
Fuuny thing apart from that: I can access the PDB but the system seems to be really under heavy pressure. CPUs are all at 100%. What the heck is going on here in the background?

Well, XDB needs to be installed (at least that is what the message says). Strange, isn't it, as the PDB$SEED has XDB in it and all my PDBs got provisioned from it. The awkward thing here is that the XDB messages appear over 20 minutes AFTER the PDBs signaled the Opening message into the alert.log (see the time stamps above).

Now after exchanging a few emails with some very helpful people in development there's an explanation for the XDB messages as well. Actually it doesn't get really installed but the SGA needs to be initialized for XDB. And I'm guessing that this action takes a lot of resources plus may cause contention when many PDBs get opened at the same time. And there's optimization work going on right now meaning that a problem with port initialization within the PDB will get fixed in a future patch set. So this issue with the very long startups of PDBs because of XDB should disappear in 12.1.0.2 most likely :-)

Finally it took another while to get the PDBs really into OPEN mode. Even though they were showing OPEN before already in V$PDBS. But as the CPUs all went to 100% as XDB got installed/initiallized at more or less the same time in all PDBs you really can't do anything.

Finally ...

... all PDBs got opened and the command ALTER PLUGGABLE DATABASE ALL OPEN returned completed.

The good news:
It takes just this long during the initial startup of a newly provisioned PDB. And you may see this issue only when you try to open many PDBs at the same time. But have a close look into your alert.log if you'll spot the message after creating a fresh PDB.

And btw, just for the records: I was using Oracle Database 12.1.0.1 with Oct 2013 PSU in it.

-Mike

About

Mike Dietrich - Oracle Mike Dietrich
Master Product Manager - Database Upgrade & Migrations - Oracle Corp

Based near Munich/Germany and spending plenty of time in airplanes to run either upgrade workshops or work onsite or remotely with reference customers. Acting as interlink between customers/partners and the Upgrade Development.

Follow me on TWITTER

Contact me via LinkedIn or XING

Search

Archives
« April 2015
SunMonTueWedThuFriSat
   
2
3
4
5
6
7
8
9
11
12
13
15
17
18
19
20
21
22
23
24
25
26
27
28
29
30
  
       
Today
Oracle related Tech Blogs
Slides Download Center
Visitors since 17-OCT-2011
White Paper and Docs
Workshops
Viewlets and Videos
This week on my Rega/iPod/CD
Workshop Map
Upgrade Reference Papers