Wednesday Jul 01, 2015

Log Writer Slave Issues in 12.1.0.2 - mainly on IBM AIX

Currently we see a lot of issues with the Multiple Logwriter feature in Oracle Database 12.1.0.2, especially on the IBM AIX platform.

What are Multiple LGWRs?

You will see multiple LGnn (Log Writer Worker) processes on multiprocessor systems, The LGWR (Log Writer) creates worker processes (slaves) to improve the performance of writing to the redo log. LGnn (Log Writer Worker) processes are not used when there is a SYNC standby destination. Possible processes include LG00-LG99.

Here is a list of new background processes in Oracle Database 12.1:
MOS Note 1625912.1 - New Background Processes In 12c 

For deep dive information about how to trace the LGWR (and DBWR) and the differences 11.2/12.1 and wait events and much more, please see this Deep Dive PDF from an UKOUG talk from Frits Hoogland.

Known Symptoms:


Solution:

Turn off multiple logwriters on IBM AIX at the moment - in some cases this may solve issues on other platforms as well but please don't take this as a generic recommendation for all platforms. We are seeing the issues mainly on IBM AIX.

Set in your spfile: 

_use_single_log_writer=true

.

--Mike 

Tuesday Jun 23, 2015

Package Differences between Oracle 11.2.0.4 and 12.1.0.2?

The question sounds pretty trivial:

Which packages did exist in Oracle 11.2.0.4 but don't exist in Oracle 12.1.0.2 anymore?

And going a bit deeper into the topic,

Which procedures and functions calls of SYS packages have been changed since then?

.

A colleague of mine (thanks to Thomas Kempkens from ACS Support for writing up a fancy procedure - all credits go to him) had to dig this out for a customer,

See the result.
These packages don't exist in Oracle 12.1.0.2 anymore but did exist in Oracle 11.2.0.4 (without PSUs):

  • CWM2_OLAP_INSTALLER
  • DBMS_AMD
  • DBMS_APPCTX
  • DBMS_DBLINK
  • DBMS_DM_IMP_INTERNAL
  • DBMS_DM_UTIL_INTERNAL
  • DBMS_DUMA_INTERNAL
  • DBMS_JDM_INTERNAL
  • DBMS_RULE_COMPATIBLE_90
  • DBMS_SQL2
  • DBMS_XDSUTL
  • DBMS_XS_SECCLASS_INT
  • DBMS_XS_SECCLASS_INT_FFI
  • EXF$DBMS_EXPFIL_SYSPACK
  • HTMLDB_SYSTEM
  • ODM_ABN_MODEL
  • SQLJUTL2

17 packages don't exist anymore.

And this is the list of
Procedures and Functions Calls inside of Oracle SYS packages which have changed in Oracle 12.1.0.2:

  • PACKAGE_NAME               SUBTYPE    SUBTYPE_NAME
  • -------------------------- ---------- ------------------------------------------------
  • CWM2_OLAP_INSTALLER        PROCEDURE  VALIDATE_CWM2_INSTALL
  • DBMS_AMD                   PROCEDURE  MOVE_OLAP_CATALOG
  • DBMS_APPCTX                PROCEDURE  CLEAR_CONTEXT
  • DBMS_APPCTX                PROCEDURE  SET_CONTEXT
  • DBMS_AQADM_SYS             PROCEDURE  GET_CORR_MSG
  • DBMS_AQADM_SYS             PROCEDURE  RESET_CORR_MSG
  • DBMS_AQADM_SYSCALLS        PROCEDURE  KWQA_3GL_UPDPERSTATS
  • DBMS_DATA_MINING           FUNCTION   GET_MODEL_DETAILS_ABN RETURNS DM_ABN_DETAILS
  • DBMS_DATA_MINING_INTERNAL  PROCEDURE  CREATE_TEMP_META_TABLE
  • DBMS_DBLINK                PROCEDURE  UPGRADE
  • DBMS_DDL_INTERNAL          FUNCTION   GEN_NEW_OBJECT_ID RETURNS BINARY_INTEGER
  • DBMS_DM_EXP_INTERNAL       FUNCTION   GET_TRACE RETURNS BOOLEAN
  • DBMS_DM_EXP_INTERNAL       PROCEDURE  DELETE_FROM_TEMP
  • DBMS_DM_EXP_INTERNAL       PROCEDURE  DO_TEMP_TABLE
  • DBMS_DM_EXP_INTERNAL       PROCEDURE  DUMP_STATUS
  • DBMS_DM_EXP_INTERNAL       PROCEDURE  GET_LIST_MODELS
  • DBMS_DM_EXP_INTERNAL       PROCEDURE  INSERT_LIST_MODELS
  • DBMS_DM_EXP_INTERNAL       PROCEDURE  SET_TRACE
  • DBMS_DM_IMP_INTERNAL       PROCEDURE  REGISTER_TEMP_TABLE
  • DBMS_DM_IMP_INTERNAL       PROCEDURE  UNREGIST_TEMP_TABLE
  • DBMS_DM_MODEL_EXP          FUNCTION   INSTANCE_INFO_EXP RETURNS VARCHAR2
  • DBMS_DM_MODEL_EXP          FUNCTION   SCHEMA_INFO_EXP RETURNS VARCHAR2
  • DBMS_DM_MODEL_EXP          PROCEDURE  SCHEMA_CALLOUT
  • DBMS_DM_UTIL               PROCEDURE  DISPLAY_MSG
  • DBMS_DM_UTIL               PROCEDURE  DUMP_MSG
  • DBMS_DM_UTIL_INTERNAL      PROCEDURE  DUMP_DM_TRACE
  • DBMS_DUMA_INTERNAL         FUNCTION   GET_PROPERTY RETURNS NUMBER
  • DBMS_DUMA_INTERNAL         FUNCTION   READ_LONG_TO_BLOB RETURNS BLOB
  • DBMS_DUMA_INTERNAL         PROCEDURE  CONVERT_LONG
  • DBMS_DUMA_INTERNAL         PROCEDURE  SCAN_TABLE
  • DBMS_DUMA_INTERNAL         PROCEDURE  WRITE_BLOB_TO_LONG
  • DBMS_INTERNAL_LOGSTDBY     PROCEDURE  EDS_EVOLVE_TABLE_1_FINISH
  • DBMS_INTERNAL_LOGSTDBY     PROCEDURE  END_INSTANTIATION
  • DBMS_INTERNAL_LOGSTDBY     PROCEDURE  INSTANTIATE_FEATURE
  • DBMS_INTERNAL_LOGSTDBY     PROCEDURE  LOCK_TABLES
  • DBMS_INTERNAL_LOGSTDBY     PROCEDURE  PREPARE_INSTANTIATION
  • DBMS_INTERNAL_LOGSTDBY     PROCEDURE  REGISTER_SCHEMA
  • DBMS_INTERNAL_LOGSTDBY     PROCEDURE  UNSUPPORTED_DML
  • DBMS_JAVA                  FUNCTION   DEPLOY_INVOKE RETURNS VARCHAR2
  • DBMS_JAVA                  FUNCTION   DROP_JAR RETURNS NUMBER
  • DBMS_JAVA                  FUNCTION   FINISH_LOADING_JAR RETURNS NUMBER
  • DBMS_JAVA                  FUNCTION   GETSOURCECHUNK RETURNS VARCHAR2
  • DBMS_JAVA                  FUNCTION   HANDLEMD5 RETURNS RAW
  • DBMS_JAVA                  FUNCTION   INITGETSOURCECHUNKS RETURNS NUMBER
  • DBMS_JAVA                  FUNCTION   JAR_STATUS RETURNS NUMBER
  • DBMS_JAVA                  FUNCTION   RJBC_INIT RETURNS VARCHAR2
  • DBMS_JAVA                  FUNCTION   RJBC_NORMALIZE RETURNS NUMBER
  • DBMS_JAVA                  FUNCTION   RJBC_REQUEST RETURNS NUMBER
  • DBMS_JAVA                  FUNCTION   RJBC_RESPOND RETURNS NUMBER
  • DBMS_JAVA                  FUNCTION   START_LOADING_JAR RETURNS NUMBER
  • DBMS_JAVA                  PROCEDURE  DEPLOY_COPY
  • DBMS_JAVA                  PROCEDURE  DEPLOY_OPEN
  • DBMS_JAVA                  PROCEDURE  RJBC_DONE
  • DBMS_JAVA                  PROCEDURE  RJBC_OUTPUT
  • DBMS_JDM_INTERNAL          FUNCTION   GET_JDM_ATTR_NAMES RETURNS JDM_ATTR_NAMES
  • DBMS_JDM_INTERNAL          FUNCTION   GET_JDM_NUM_VALS RETURNS JDM_NUM_VALS
  • DBMS_JDM_INTERNAL          FUNCTION   GET_JDM_STR_VALS RETURNS JDM_STR_VALS
  • DBMS_JDM_INTERNAL          FUNCTION   TO_CHAR_VARRAY RETURNS VARCHAR2
  • DBMS_JDM_INTERNAL          FUNCTION   UNIQUE_OBJECT_NAME RETURNS VARCHAR2
  • DBMS_JDM_INTERNAL          PROCEDURE  APPLY_TASK
  • DBMS_JDM_INTERNAL          PROCEDURE  BUILD_TASK
  • DBMS_JDM_INTERNAL          PROCEDURE  EXPLAIN_TASK
  • DBMS_JDM_INTERNAL          PROCEDURE  EXPORT_TASK
  • DBMS_JDM_INTERNAL          PROCEDURE  IMPORT_TASK
  • DBMS_JDM_INTERNAL          PROCEDURE  PREDICT_TASK
  • DBMS_JDM_INTERNAL          PROCEDURE  PROFILE_TASK
  • DBMS_JDM_INTERNAL          PROCEDURE  SQL_APPLY_TASK
  • DBMS_JDM_INTERNAL          PROCEDURE  TEST_TASK
  • DBMS_JDM_INTERNAL          PROCEDURE  XFORM_SEQ_TASK
  • DBMS_JDM_INTERNAL          PROCEDURE  XFORM_TASK
  • DBMS_LOGMNR_INTERNAL       PROCEDURE  ADDXTENDEDPKLOGGROUP
  • DBMS_LOGMNR_INTERNAL       PROCEDURE  DROPXTENDEDPKLOGGROUP
  • DBMS_LOGMNR_INTERNAL       PROCEDURE  KRVGDRM2
  • DBMS_LOGMNR_INTERNAL       PROCEDURE  LOGMNR_KRVILD
  • DBMS_LOGMNR_INTERNAL       PROCEDURE  LOGMNR_KRVIULD
  • DBMS_LOGMNR_INTERNAL       PROCEDURE  SET_PARAMETER
  • DBMS_LOGMNR_INTERNAL       PROCEDURE  UNSET_PARAMETER
  • DBMS_LOGSTDBY              FUNCTION   VALIDATE_AUTH RETURNS BOOLEAN
  • DBMS_LOGSTDBY              PROCEDURE  EDS_EVOLVE_TABLE
  • DBMS_METADATA_INT          PROCEDURE  CLEAR_CACHE
  • DBMS_METADATA_INT          PROCEDURE  CLEAR_XSL_CACHE
  • DBMS_OBJECTS_UTILS         FUNCTION   SPLIT_SOURCE RETURNS NUMBER
  • DBMS_OBJECTS_UTILS         PROCEDURE  RECOMPILE_TYPES
  • DBMS_OBJECTS_UTILS         PROCEDURE  UPDATE_TYPES
  • DBMS_REPORT                FUNCTION   STORE_FILE RETURNS NUMBER
  • DBMS_REPORT                PROCEDURE  CREATE_SHARED_DIRECTORY
  • DBMS_REPORT                PROCEDURE  DROP_SHARED_DIRECTORY
  • DBMS_RMIN                  PROCEDURE  INSTALL
  • DBMS_RMIN                  PROCEDURE  RM$_CALIBRATE_IO
  • DBMS_RMIN                  PROCEDURE  RM$_CLRPAREA
  • DBMS_RMIN                  PROCEDURE  RM$_CONSUMER_GROUP_MAPPING_PRI
  • DBMS_RMIN                  PROCEDURE  RM$_CRTPAREA
  • DBMS_RMIN                  PROCEDURE  RM$_CUP_CATEGORY
  • DBMS_RMIN                  PROCEDURE  RM$_CUP_GROUP
  • DBMS_RMIN                  PROCEDURE  RM$_CUP_PLAN
  • DBMS_RMIN                  PROCEDURE  RM$_CUP_PLAN_DIRECTIVE
  • DBMS_RMIN                  PROCEDURE  RM$_CUP_STORAGE_POOL_MAPPING
  • DBMS_RMIN                  PROCEDURE  RM$_DRP_CATEGORY
  • DBMS_RMIN                  PROCEDURE  RM$_DRP_GROUP
  • DBMS_RMIN                  PROCEDURE  RM$_DRP_PLAN
  • DBMS_RMIN                  PROCEDURE  RM$_DRP_PLAN_CSD
  • DBMS_RMIN                  PROCEDURE  RM$_DRP_PLAN_DIRECTIVE
  • DBMS_RMIN                  PROCEDURE  RM$_SET_CAPABILITY
  • DBMS_RMIN                  PROCEDURE  RM$_SET_CONSUMER_GROUP_MAPPING
  • DBMS_RMIN                  PROCEDURE  RM$_SET_INSTANCE_CAPABILITY
  • DBMS_RMIN                  PROCEDURE  RM$_SUBPAREA
  • DBMS_RMIN                  PROCEDURE  RM$_SWTCH_SESSION_GROUP
  • DBMS_RMIN                  PROCEDURE  RM$_SWTCH_USR_GROUP
  • DBMS_RMIN                  PROCEDURE  RM$_VLDPAREA
  • DBMS_RMIN                  PROCEDURE  UNINSTALL
  • DBMS_RULE_COMPATIBLE_90    FUNCTION   INCOMPATIBLE_RULES_ENGINE_OBJ RETURNS NUMBER
  • DBMS_RULE_COMPATIBLE_90    PROCEDURE  DOWNGRADE_RULE_OBJECTS
  • DBMS_RULE_COMPATIBLE_90    PROCEDURE  INCOMPATIBLE_EVCTXS
  • DBMS_RULE_COMPATIBLE_90    PROCEDURE  INCOMPATIBLE_RULES
  • DBMS_RULE_COMPATIBLE_90    PROCEDURE  INCOMPATIBLE_RULESETS
  • DBMS_RULE_COMPATIBLE_90    PROCEDURE  UPGRADE_RULE_OBJECTS
  • DBMS_SPM_INTERNAL          FUNCTION   FETCH_PLAN_INFO RETURNS REF CURSOR
  • DBMS_SPM_INTERNAL          PROCEDURE  GET_SPM_HINTSET
  • DBMS_SQL2                  PROCEDURE  EXPAND_SQL_TEXT
  • DBMS_SQL2                  PROCEDURE  PARSE
  • DBMS_STATS                 FUNCTION   TO_CASCADE_TYPE RETURNS BOOLEAN
  • DBMS_STATS                 FUNCTION   TO_DEGREE_TYPE RETURNS NUMBER
  • DBMS_STATS                 FUNCTION   TO_ESTIMATE_PERCENT_TYPE RETURNS NUMBER
  • DBMS_STATS                 FUNCTION   TO_PUBLISH_TYPE RETURNS BOOLEAN
  • DBMS_STATS_INTERNAL        FUNCTION   COMPOSE_HASHVAL_CLOB RETURNS CLOBTAB
  • DBMS_STATS_INTERNAL        FUNCTION   GET_CHTAB RETURNS CHTAB
  • DBMS_STATS_INTERNAL        FUNCTION   GET_SYNOPSIS_BLKCNT RETURNS NUMBER(38)
  • DBMS_STATS_INTERNAL        PROCEDURE  DELETE_TABLE_SYNOPSIS
  • DBMS_STATS_INTERNAL        PROCEDURE  EXPORT_SYNOPSIS_HEAD
  • DBMS_SWRF_REPORT_INTERNAL  FUNCTION   TO_1000S RETURNS VARCHAR2
  • DBMS_SWRF_REPORT_INTERNAL  FUNCTION   TO_1024S RETURNS VARCHAR2
  • DBMS_WORKLOAD_REPLAY       FUNCTION   SYNCPOINT_WAIT_TO_POST RETURNS NUMBER
  • DBMS_WRR_INTERNAL          FUNCTION   DATE_TO_EPOCH_SECS RETURNS BINARY_INTEGER
  • DBMS_WRR_INTERNAL          FUNCTION   IS_CONSOLIDATED_DIR RETURNS BOOLEAN
  • DBMS_XDS                   PROCEDURE  REFRESH_DSD
  • DBMS_XDSUTL                PROCEDURE  INVALIDATE_DSD_CACHE
  • DBMS_XDSUTL                PROCEDURE  INVALIDATE_DSD_CACHE_BY_ACLID
  • DBMS_XS_SECCLASS_INT       PROCEDURE  DELETE_SEC_CLASS
  • DBMS_XS_SECCLASS_INT       PROCEDURE  GET_LEAF_PRIVILEGES
  • DBMS_XS_SECCLASS_INT_FFI   PROCEDURE  DELETE_SEC_CLASS
  • DBMS_XS_SECCLASS_INT_FFI   PROCEDURE  GET_LEAF_PRIVILEGES
  • DBMS_XS_SESSIONS           FUNCTION   CHECK_ROLE RETURNS BOOLEAN
  • DBMS_XS_SESSIONS           FUNCTION   CHECK_SESSION RETURNS BOOLEAN
  • DBMS_XS_SESSIONS           FUNCTION   GET_SESSIONID_FROM_COOKIE RETURNS RAW
  • DBMS_XS_SESSIONS           PROCEDURE  ABORTATTACH_SESSION
  • DBMS_XS_SESSIONS           PROCEDURE  GET_SESSION_COOKIE
  • DBMS_XS_SESSIONS           PROCEDURE  SET_NAMESPACE_ACL
  • DBMS_XS_SESSIONS           PROCEDURE  SET_NAMESPACE_HANDLER
  • DBMS_XS_SESSIONS_FFI       PROCEDURE  ABORTATTACH_SESSION
  • DBMS_XS_SESSIONS_FFI       PROCEDURE  CHECK_ROLE
  • DBMS_XS_SESSIONS_FFI       PROCEDURE  CHECK_SESSION
  • DBMS_XS_SESSIONS_FFI       PROCEDURE  GET_SESSION_COOKIE
  • DBMS_XS_SESSIONS_FFI       PROCEDURE  SET_NAMESPACE_ACL
  • DBMS_XS_SESSIONS_FFI       PROCEDURE  SET_NAMESPACE_HANDLER
  • DMP_SEC                    FUNCTION   GET_MODEL_DETAILS_ABN RETURNS VARCHAR2(32767)
  • DMP_SYS                    PROCEDURE  DUMP_MODEL_REC
  • DMP_SYS                    PROCEDURE  DUMP_TRACE
  • DMP_SYS                    PROCEDURE  DUMP_VARCHAR2A
  • DMP_SYS                    PROCEDURE  DUMP_VARCHAR2S
  • DM_QGEN                    PROCEDURE  LS_DUMP
  • EXF$DBMS_EXPFIL_SYSPACK    FUNCTION   GET_CMTSCN_FOR_XID RETURNS NUMBER
  • EXF$DBMS_EXPFIL_SYSPACK    FUNCTION   GET_EXPRID_4M_TEXTID RETURNS VARCHAR2
  • EXF$DBMS_EXPFIL_SYSPACK    FUNCTION   GET_SPATIAL_MDATAREP RETURNS VARCHAR2
  • EXF$DBMS_EXPFIL_SYSPACK    FUNCTION   GET_TAB_NAME RETURNS VARCHAR2
  • EXF$DBMS_EXPFIL_SYSPACK    FUNCTION   GET_TAB_OWNER RETURNS VARCHAR2
  • EXF$DBMS_EXPFIL_SYSPACK    FUNCTION   GET_TYPELV_STATUS RETURNS VARCHAR2
  • EXF$DBMS_EXPFIL_SYSPACK    FUNCTION   GET_TYPE_OID RETURNS RAW
  • EXF$DBMS_EXPFIL_SYSPACK    FUNCTION   PROC_IS_DEFINERS RETURNS NUMBER
  • EXF$DBMS_EXPFIL_SYSPACK    FUNCTION   TAB_COLUMN_EXISTS RETURNS NUMBER
  • EXF$DBMS_EXPFIL_SYSPACK    PROCEDURE  CLR_EXP_DEPACTION
  • EXF$DBMS_EXPFIL_SYSPACK    PROCEDURE  CLR_OBJECT_NOEXP
  • EXF$DBMS_EXPFIL_SYSPACK    PROCEDURE  CLR_USER_NOEXP
  • EXF$DBMS_EXPFIL_SYSPACK    PROCEDURE  DOWNGRADE_COMPILED_SPARSE
  • EXF$DBMS_EXPFIL_SYSPACK    PROCEDURE  PARSE_PROC_AS_OWNER
  • EXF$DBMS_EXPFIL_SYSPACK    PROCEDURE  PARSE_PROC_AS_OWNER2
  • EXF$DBMS_EXPFIL_SYSPACK    PROCEDURE  SET_EXP_DEPACTION
  • EXF$DBMS_EXPFIL_SYSPACK    PROCEDURE  SET_OBJECT_NOEXP
  • HTMLDB_SYSTEM              FUNCTION   VERIFY_USER RETURNS BINARY_INTEGER
  • INITJVMAUX                 FUNCTION   DO_JIS_DROP RETURNS BOOLEAN
  • KUPD$DATA                  FUNCTION   CONVENTIONAL_LOAD_CALLOUT RETURNS BINARY_INTEGER
  • KUPF$FILE_INT              FUNCTION   GET_ENCPWD RETURNS RAW
  • KUPF$FILE_INT              FUNCTION   GET_OBFPWD RETURNS RAW
  • KUPF$FILE_INT              PROCEDURE  ENCRYPT_PWD
  • KUPP$PROC                  PROCEDURE  INTALGCONV
  • LOGMNR_DICT_CACHE          PROCEDURE  PURGE_SCN
  • LOGSTDBY_INTERNAL          PROCEDURE  EDS_EVOLVE_TABLE_1_I
  • ODM_ABN_MODEL              PROCEDURE  BUILD
  • ODM_ABN_MODEL              PROCEDURE  ESTIMATE_FEATUREBLD_TIME
  • ODM_ABN_MODEL              PROCEDURE  GET_MODEL_STATE
  • ODM_CLUSTERING_UTIL        FUNCTION   IS_DEBUG_ENABLED RETURNS BOOLEAN
  • ODM_MODEL_UTIL             FUNCTION   UNIQUE_SEQUENCE_NAME RETURNS VARCHAR2(30)
  • ODM_MODEL_UTIL             PROCEDURE  CREATE_SEQUENCE
  • ODM_MODEL_UTIL             PROCEDURE  DEBUG_DUMP
  • ODM_MODEL_UTIL             PROCEDURE  DROP_SEQUENCE
  • ODM_MODEL_UTIL             PROCEDURE  GET_TIME_MILLISECONDS
  • ODM_UTIL                   FUNCTION   COMPUTE_LOG_COMBINATIONS RETURNS NUMBER
  • PRVT_RTADDM                FUNCTION   EXTRACTOSSTAT RETURNS NUMBER
  • PRVT_RTADDM                FUNCTION   GETDATA RETURNS CLOB_T
  • PRVT_RTADDM                FUNCTION   GETVERSION RETURNS VARCHAR2
  • SQLJUTL2                   FUNCTION   EVALUATE RETURNS LONG RAW
  • SQLJUTL2                   FUNCTION   INVOKE RETURNS LONG RAW
  • SQLJUTL2                   FUNCTION   REFLECT RETURNS LONG
  • SQLJUTL2                   FUNCTION   REFLECT2 RETURNS CLOB
  • STANDARD                   FUNCTION   ASCII RETURNS BINARY_INTEGER

204 procedure and function calls have been changed.

-- Mike

PS: The above listening is without PSUs applied - I learned that the list my vary slightly based on the PSUs or BPs you have applied. Credits to my colleague Katsumi-san, who pointed my to the DBMS_OPTIM_BUNDLE package which seems to be introduced with the October 2014 PSU. Philippe Fierens has blogged about the package: http://pfierens.blogspot.co.uk/2014_10_01_archive.html

Tuesday May 26, 2015

Oracle 12.1.0.2 - Security Behavior Change with non-SYSDBA Triggers

Oracle Database SecuritySometimes things get revealed at unexpected occasions. This one happened during a recent customer upgrade to Oracle Database 12c with a 3rd party geospatioanl application installed (ESRI).

At the very end of the upgrade the customer saw many ORA-1031 (insufficient privileges) errors and it seemed to be that nothing was working correctly anymore. 

This happened during the run of catupend.sql. The following code path in  catupend.sql causes the error.

cursor ddl_triggers is                                       
   select o.object_id from dba_triggers t, dba_objects o     
    where t.owner = o.owner and t.trigger_name = o.object_name
      and o.object_type = 'TRIGGER'                          
      and (t.triggering_event like '%ALTER%' or              
    t.triggering_event like '%DDL%');     


ERROR at line 1:
ORA-04045: errors during recompilation/revalidation of
SDE.DB_EV_ALTER_ST_METADATA
ORA-01031: insufficient privileges
ORA-06512: at "SYS.DBMS_UTILITY", line 1279
ORA-06512: at line 20

Apparently there's no access to an application trigger anymore - which got deployed as a system trigger (for more information about ESRI's system trigger please click this link). Even though this is strange it doesn't seem like a big issue. But in fact it is as this procedure failed and caused other stuff not getting validated correctly. So subsequent actions (for instance the run of utlu121s.sql, the post upgrade script) failed with ORA-1031 as well pointing to DBMS_UTILITY.

The customer [Danke Andy!!!] itself found the workaround by pattern matching similar issues in MOS and trying some grants - the 3rd one did the trick:

  • GRANT ADMINISTER DATABASE TRIGGER to SDE;


So it was obvious that something in the security architecture in Oracle Database 12.1.0.2 had been changed - and somebody forgot to document it. Later on I've learned that this change got introduced with the July 2013 PSU/CPU as well. I don't blame the customer for not applying PSUs since almost two years - I knew that upfront and we are implementing a 2-PSUs-per-year strategy now with the upgrade to Oracle Database 12c. 

The system trigger ESRI had created couldn't be validated anymore under the context of the SDE (ESRI's application) user. Therefore it failed but caused other actions to fail as well.  

This behavior change is related to "SYSDBA privilege should not be available in non-SYS owned DR procedure / trigger execution" which is first fixed into 12.1.0.2, and then backported as part of CPU July-2013.
When SYS executes a non-SYS owned DR procedure or a Trigger, the SYS privileges would not available during the procedure/trigger execution. The procedure/trigger owner privileges prevail.

--Mike

Thursday Apr 23, 2015

CDBs with less options now supported in Oracle 12.1.0.2

When Oracle Multitenant was launched Roy and I amongst many other people always mentioned that the requirement of having all options in a Container Database (CDB$ROOT), and therefore also for the PDB$SEED with the immediate result that all PDBs provisioned from the PDB$SEED will have all options as well, will hinder customer adoption significantly. 

Almost all customers I have talked to in the past 3-4 years about Oracle Multitenant mentioned immediately that it will be a huge problem for them to install all options as (1) their policy is to install only things they are licensed for to (2) prevent developers, users and DBAs to use things accidentally without even knowing that this or that will require a license.

As it is not allowed to manipulate and change the PDB$SEED the workaround - as PDBs were allowed to have less options - has been to create a stand-alone Oracle 12c database with exactly the options you'd like to have configured as your gold standard - and then plug it in under a remarkable name, for instance PDB$MASTER. Switch it to read only and make sure from now on you'll provision a new PDB always as a clone from PDB$MASTER, and not from PDB$SEED.

All Options in a CDB

That would have even worked in the Single Tenant case, which does not require licensing the Oracle Multitenant option and where you have only one active PDB. For this purpose you would have unplugged your PDB$MASTER after making it a pluggable database and provision new PDBs with just your desired options set as plugging in PDB$MASTER under a new name (e.g. PDB26) using the COPY option of the command.

Now this will become all obsolete as from now you it is allowed to have a CDB installation with less options. This applies to linked kernel modules (e.g. RAT) as well as to configured database components (e.g. JAVA, OWM, SPATIAL etc).

Please see the following new/rephrased MOS Notes:

MOS Note:2001512.1 basically explains the following steps:

  • Do all the click work in DBCA (Database Creation Assistant) to create a container database - but let DBCA only create the scripts
  • Edit the <SID>.sql script and remove the unwanted options according to the dependency table in the MOS Note
  • Edit the CreateDBCatalog.sql in case you want to remove OWM (Oracle Workspace Manager) creation as well 
  • Add the Oracle PERL $ORACLE_HOME/perl/bin in front of your $PATH variable
  • Start the <SID>.sh script on the shell prompt

Here's an example of a CreateDBCatalog.sql and a XXXX.sql creating a CDB with no options except XDB (which is mandatory in Oracle Database 12c):

cat CreateDBCatalog.sql

SET VERIFY OFF
connect "SYS"/"&&sysPassword" as SYSDBA
set echo on
spool /u01/app/oracle/admin/XXXX/scripts/CreateDBCatalog.log append
alter session set "_oracle_script"=true;
alter pluggable database pdb$seed close;
alter pluggable database pdb$seed open;
host perl /u01/app/oracle/product/12.1.0.2/rdbms/admin/catcon.pl -n 1 -l /u01/app/oracle/admin/XXXX/scripts -b catalog /u01/app/oracle/product/12.1.0.2/rdbms/admin/catalog.sql;
host perl /u01/app/oracle/product/12.1.0.2/rdbms/admin/catcon.pl -n 1 -l /u01/app/oracle/admin/XXXX/scripts -b catproc /u01/app/oracle/product/12.1.0.2/rdbms/admin/catproc.sql;
host perl /u01/app/oracle/product/12.1.0.2/rdbms/admin/catcon.pl -n 1 -l /u01/app/oracle/admin/XXXX/scripts -b catoctk /u01/app/oracle/product/12.1.0.2/rdbms/admin/catoctk.sql;
-- host perl /u01/app/oracle/product/12.1.0.2/rdbms/admin/catcon.pl -n 1 -l /u01/app/oracle/admin/XXXX/scripts -b owminst /u01/app/oracle/product/12.1.0.2/rdbms/admin/owminst.plb;
host perl /u01/app/oracle/product/12.1.0.2/rdbms/admin/catcon.pl -n 1 -l /u01/app/oracle/admin/XXXX/scripts -b pupbld -u SYSTEM/&&systemPassword /u01/app/oracle/product/12.1.0.2/sqlplus/admin/pupbld.sql;
connect "SYSTEM"/"&&systemPassword"
set echo on
spool /u01/app/oracle/admin/XXXX/scripts/sqlPlusHelp.log append
host perl /u01/app/oracle/product/12.1.0.2/rdbms/admin/catcon.pl -n 1 -l /u01/app/oracle/admin/XXXX/scripts -b hlpbld -u SYSTEM/&&systemPassword -a 1  /u01/app/oracle/product/12.1.0.2/sqlplus/admin/help/hlpbld.sql 1helpus.sql;
spool off
spool off

.

cat XXXX.sql

set verify off
ACCEPT sysPassword CHAR PROMPT 'Enter new password for SYS: ' HIDE
ACCEPT systemPassword CHAR PROMPT 'Enter new password for SYSTEM: ' HIDE
host /u01/app/oracle/product/12.1.0.2/bin/orapwd file=/u01/app/oracle/product/12.1.0.2/dbs/orapwXXXX force=y format=12
@/u01/app/oracle/admin/XXXX/scripts/CreateDB.sql
@/u01/app/oracle/admin/XXXX/scripts/CreateDBFiles.sql
@/u01/app/oracle/admin/XXXX/scripts/CreateDBCatalog.sql
-- @/u01/app/oracle/admin/XXXX/scripts/JServer.sql
-- @/u01/app/oracle/admin/XXXX/scripts/context.sql
-- @/u01/app/oracle/admin/XXXX/scripts/ordinst.sql
-- @/u01/app/oracle/admin/XXXX/scripts/interMedia.sql
-- @/u01/app/oracle/admin/XXXX/scripts/cwmlite.sql
-- @/u01/app/oracle/admin/XXXX/scripts/spatial.sql
-- @/u01/app/oracle/admin/XXXX/scripts/labelSecurity.sql
-- @/u01/app/oracle/admin/XXXX/scripts/apex.sql
-- @/u01/app/oracle/admin/XXXX/scripts/datavault.sql
-- @/u01/app/oracle/admin/XXXX/scripts/CreateClustDBViews.sql

@/u01/app/oracle/admin/XXXX/scripts/lockAccount.sql
@/u01/app/oracle/admin/XXXX/scripts/postDBCreation.sql
@/u01/app/oracle/admin/XXXX/scripts/PDBCreation.sql
@/u01/app/oracle/admin/XXXX/scripts/plug_PDB.sql
@/u01/app/oracle/admin/XXXX/scripts/postPDBCreation_PDB.sql

 .

This results in a database having only these components - the minimal component set in Oracle 12.1.0.2: 

COMP ID  NAME
-------- --------------------------------------

CATALOG  Oracle Database Catalog View

CATPROC  Oracle Database Packages and

XDB      Oracle XML Database

   .

-- Mike 

Friday Mar 13, 2015

EM Cloud Control 12.1.0.4 (or newer) OMS Repository certified with Oracle Database 12.1.0.2 plus patches

The certification for Oracle Database 12.1.0.2 as Repository Database underneath of Oracle Enterprise Manager Cloud Control 12.1.0.4 (OMS) had been revoked for a while. One reason may have been the potential removal of the SYSMAN user during a DBUA upgrade ;-) But this is just a rumor. I don't know the exact reasons.

After checking with the certification matrix earlier today I realized:

Cloud Control Repository Database Certification Oracle 12.1.0.2

Oracle Database 12.1.0.2 is certified as a repository database for Cloud Control resp. OMS.

But ... with a few important restrictions (see MOS Note:1987905.1 - 12c Database has been Certified as a 12cR4 Repository with Certain Patchset Restrictions):

  • Database PSU April 2015 (12.1.0.2.3)

    OR:

  • Database PSU October 2014 (12.1.0.2.1) or newer
  • Database Patch 20243268 on top of the PSU
    • Bug20243268:
      EM QUERY WITH SQL_ID 4RQ83FNXTF39U PERFORMS POORLY ON ORACLE 12C RELATIVE TO 11G

Please be aware that the above mentioned MOS Note has a typo - it refers to PSU 12.1.0.2.3 (correct) as the Oct PSU (incorrect - it's the April 2015 one). 

--Mike 

PS: Thanks for the anonymous comment ... I wasn't aware of the MOS Note and updated the patch requirements

Monday Mar 09, 2015

Applying a PSU or BP to a Single-/Multitenant Environment

I have already explained in broad details a while ago how to:

But one may miss the steps for applying a PSU (Patch Set Update) or BP (Bundled Patch) to a Single-/Multitenant Environment. At first everything will work just the same if you choose the Everything-at-Once strategy as datapatch will adjust all the required things regardless of being executed in a stand-alone or a singe/Multitenant environment.

But what happens if you apply a PSU or a BP to one of your Multitenant environments and want to move PDBs one after another (or a few at the same time) to the new environment?
Or revert a PSU by plugging out from a CDB with the PSU inside - and plug it back into a CDB with a lower version or no PSU at all? 

First step - Check Plug In Compatibility 

Before you can even start your unplug/plug operation you should always perform the plugin check. This is divided in two simple steps:

  1. Create the XML description file for your PDB in CDB_SOURCE
    exec DBMS_PDB.DESCRIBE ('/home/oracle/PDB1_unplug.xml', 'PDB1');
  2. Run the plug in check in CDB_DEST
    begin
      if DBMS_PDB.CHECK_PLUG_COMPATIBILITY('/home/oracle/PDB1_unplug.xml','PDB1') then  
      DBMS_OUTPUT.PUT_LINE('No violations found - you can relax');
    else
      DBMS_OUTPUT.PUT_LINE('Violations found - check PDB_PLUG_IN_VIOLATIONS');
    end if;
    end;
    /

No Plugin Violations?

Then please follow the procedure described in:
http://blogs.oracle.com/UPGRADE/entry/upgrade_pdbs_one_at_a
without the upgrade part as you don't need to upgrade anything in this case of course. 


Higher patch in CDB_DEST than in CDB_SOURCE?

Then run this query:

select TYPE,MESSAGE,ACTION from PDB_PLUG_IN_VIOLATIONS where NAME='PDB1';

It will tell you to execute datapatch:

TYPE    MESSAGE
------  ----------------------------------------------------------------------------
ERROR   PSU bundle patch 1 (PSU Patch 4711): Installed in the CDB but not in the PDB

ACTION
------------------------------------------------
Call datapatch to install in the PDB or the CDB

Lower patch in CDB_DEST than in CDB_SOURCE?

Now this becomes a bit more tricky. See the output of PDB_PLUG_IN_VIOLATIONS:

TYPE  MESSAGE
----- ----------------------------------------------------------------------------
ERROR PSU bundle patch 1 (PSU Patch 4711): Installed in the PDB but not in the CDB

ACTION
------------------------------------------------
Call datapatch to install in the PDB or the CDB

Huh? Install???
What does this mean? Should I install now the current CDB/PDB's PSU into my target environment before being able to step down? 

Actually I think this message is misleading. And when you look into the MyOracle Support Note describing this under scenario 3 (MOS Note:1935365.1 - Multitenant Unplug/Plug Best Practices) you'll see that the author silently assumed as well that is is more likely that you'll remove the patch from the PDB. 

But how do you remove changes which came in with datapatch from within a PDB only?

You will need to run datapatch -rollback on the affected PDBs only:

$> datapatch -rollback <patch id> –force [–bundle_series] -pdbs <pdb1,pdb2,...,pdbn>

For further information see:

--Mike 

Tuesday Mar 03, 2015

ORAchk 12.1.0.2.3 with new checks + Restart support

ORAchk

Never used or heard about ORAchk? Then it's time to give it a try! 

ORAchk 12.1.0.2.3 has now be released. ORAchk 12.1.0.2.3 has so many great features, especially more than 50 new health checks, OL7 Support - and is now aware of Oracle Restart environments.

New features include:

  • Linux on System Z (RHEL 6, RHEL 7, SuSE 12)
  • Oracle Enterprise Linux 7
  • Single Instance ASM Checks
  • Upgrade Validation checks for 12.1.0.2
  • Enhanced Golden Gate Checks 
  • Enterprise Manager Agent Checks
  • Enhanced Reporting to highlight checks that ORAchk cannot gain full visibility for
    (checks that require manual validation)
  • Improved health Score Calculation (you can now hit 100%)
  • Over 50 new health checks
  • Bug Fixes

Download the most recent version via MOS Note:1268927.1.

Learn more about the Upgrade Readiness Assessment with ORAckh in MOS Note: 1457357.1

--Mike 

PS: Updated the post on Mar 7, 2015, as ORAchk 12.1.0.2.3 became production!

Friday Feb 20, 2015

Downgrade Oracle Restart 12c - Grid Infrastructure only?

Oracle RestartCan you downgrade your Oracle Restart installation from Oracle 12c back to Oracle 11g?

Actually there's no real direct downgrade supported for Oracle Restart. But of course there's a way to do it.

Basically it is: 

  • Deconfigure Oracle Restart in 12c
  • Configure Oracle Restart in 11g 

But wait a minute. It is very important to know if you have upgraded your database already. If that is the case then first you MUST downgrade your database(s) as you can't manage a higher version Oracle Database with a lower version Clusterware.

So first of all, please downgrade your Oracle database(s) first: 

At the next stage you'll need to "downgrade" the Oracle Clusterware resp Grid Infrastructure for Oracle Restart: 

Before you attempt this you'll need to deconfigure the Restart resources - and please be aware that here's a small difference in commands between Oracle 12.1.0.2 and Oracle 12.1.0.1.

This is from the documentation for Oracle 12.1.0.2

  • Deconfigure Oracle Restart:
    • Log in as root
    • cd /u01/app/12.1.0.2/grid/crs/install
    • roothas.sh -deconfig -force
  • Once this is complete you can now deinstall the Grid Infrastructure with the deinstall tool
  • Then you will need either to reinstall the previous - for instance Oracle 11.2.0.4 - Grid Infrastructure or - if it's still present on the machine - reconfigure it by running root.sh from the previous Clusterware's home
  • And finally reconfigure the database(s) again with Clusterware
    • $ srvctl downgrade database -d db-unique-name -o oraclehome -t to_version

.
In case you'd plan to do this exercise back from Oracle 12.1.0.1 instead then you'll have different steps to follow to deconfigure Oracle Restart 12.1.0.1:

  • Deconfigure Oracle Restart 12.1.0.1
    • # cd /u01/app/12.1.0.1/grid/crs/install
    • # /u01/app/12.1.0.1/grid/perl/bin/perl /u01/app/12.1.0.1/grid/crs/install/roothas.pl -deconfig
      .

--Mike

PS: See bug18160024 or the GI install guide, section A11.4 for the Standalone GI downgrade instructions from 12.1.0.2.0. 

Thursday Jan 22, 2015

Non-CDB architecture of Oracle databases is DEPRECATED since Oracle Database 12.1.0.2

Beginning with Oracle Database 12.1.0.2 a non-CDB architecture is deprecated.

non-CDB deprecated in Oracle 12.1.0.2

--

What does this mean?

Deprecation first of all does not mean "desupported". It means in this case that you of course can have still stand-alone Oracle databases as you know the architecture from previous releases. But you can also have a single-tenant deployment (a CDB with one PDB - no Multitenant license required) or multitenant databases (a CDB with up to 252 PDBs - Multitenant Option license required).

But in a future Oracle release single-tenant and multitenant databases only may be allowed. Right now there's no information available when this might happen. --

Which features are not supported at the moment?

  • Database Change Notification
  • Continuous Query Notification (CQN)
  • Client Side Cache
  • Heat Map
  • Automatic Data Optimization
  • Oracle Streams
  • Oracle Fail Safe
  • Flashback Pluggable Database (Flashback Database works but will flashback CDB$ROOT including all PDBs)
  • DBVERIFY
  • Data Recovery Advisor (DRA)
  • Flashback Transaction Backout 

-Mike 

---

From the Upgrade Guide:
By deprecate, we mean that the feature is no longer being enhanced but is still supported for the full life of the 12.1 release. By desupported, we mean that Oracle will no longer fix bugs related to that feature and may remove the code altogether. Where indicated, a deprecated feature may be desupported in a future major release

Friday Jan 09, 2015

Oracle 12.1.0.2 is now certified with Oracle EBS 12.2

Just copying this information from Steven Chang's blog: 

Please note that Oracle Database 12.1.0.2 is already certified with EBS 12.1. 

-Mike 

Tuesday Sep 02, 2014

Unified Auditing - is it ON or OFF in Oracle 12c?

Don't trust our slides - only believe what you've verified by yourself ;-)

Actually one of our slides gives a parameter recommendation to set AUDIT_TRAIL since Oracle 11g explicitly to the value you want as otherwise it may switch to "DB" and you may not be aware of it. In conjunction with this setting we explain the new Oracle Database 12c feature Unified Auditing - which is not linked into the kernel and therefore should be off.

Should be ... well ... thanks to Marco Patzwahl who asked me why he still has over 100 audit records in V$UNIFIED_AUDIT_TRAIL? Good question - and I've had no answer. But Carol, my manager, knew the right person to ask. And Naveen replied within minutes (thanks!!!).

Here are the facts: 

  • Unified Auditing is not linked into the Oracle 12c kernel by default to offer people the choice to use it and to avoid conflicts in case somebody has auditing ON already - so neither during an upgrade nor with a fresh 12c database you'll see it included into the kernel. It will have to be linked in manually (see our slides)
    • Check if Unified Auditing is present in your environment:
      SQL> select VALUE from V$OPTION where PARAMETER='Unified Auditing';
    • In case you'll link it into the kernel
      cd $ORACLE_HOME/rdbms/lib
      make -f ins_rdbms.mk
      uniaud_on ioracle ORACLE_HOME=$ORACLE_HOME

      make sure you set AUDIT_TRAIL=NONE afterwards as otherwise both auditing mechanisms will run concurrently
  • But even though it is not linked into the kernel a bit of Unified Auditing is ON by default in MIXED MODE when you create a fresh Oracle 12c database.
    • MIXED MODE auditing?
      • See the documentation for further information
      • Just two policies are enabled by default: ORA_SECURECONFIG and ORA_LOGON_FAILURES
  • Turn Unfiied Auditing OFF?
    • If is has been linked in into the kernel, unlink it:
      cd $ORACLE_HOME/rdbms/lib
      make -f ins_rdbms.mk 
      uniaud_off ioracle ORACLE_HOME=$ORACLE_HOME
    • Disable the two default policies - this will turn off any Unified Auditing features:
      SQL> noaudit policy ORA_SECURECONFIG;
      Noaudit succeeded.
      SQL> noaudit policy ORA_LOGON_FAILURES;
      Noaudit succeeded.


-Mike

Friday Aug 22, 2014

Automatic Maintenance Jobs in every PDB?
New SPM Evolve Advisor Task in Oracle 12.1.0.2

A customer checking out our slides from the OTN Tour in August 2014 asked me a finicky question the other day:

"According to the documentation the Automatic SQL Tuning Advisor maintenance task gets executed only within the CDB$ROOT, but not within each PDB - but the slides are not clear here. So what is the truth?"

Ok, that's good question. In my understanding all tasks will get executed within each PDB - that's why we recommend (based on experience) to break up the default maintenance windows when using Oracle Multitenant. Otherwise all PDBs will have the same maintenance windows, and guess what will happen when 25 PDBs start gathering object statistics at the same time ...

The documentation indeed says:

Automatic SQL Tuning Advisor data is stored in the root. It might have results about SQL statements executed in a PDB that were analyzed by the advisor, but these results are not included if the PDB is unplugged. A common user whose current container is the root can run SQL Tuning Advisor manually for SQL statements from any PDB. When a statement is tuned, it is tuned in any container that runs the statement.

This sounds reasonable. But when we have a look into our PDBs or into the CDB_AUTOTASK_CLIENT view the result is different from what the doc says. In my environment I did create just two fresh empty PDBs (CON_ID 3 and 4):

SQL> select client_name, status, con_id from cdb_autotask_client;

CLIENT_NAME                           STATUS         CON_ID
------------------------------------- ---------- ----------
auto optimizer stats collection       ENABLED             1
sql tuning advisor                    ENABLED             1
auto space advisor                    ENABLED             1
auto optimizer stats collection       ENABLED             4
sql tuning advisor                    ENABLED             4
auto space advisor                    ENABLED             4
auto optimizer stats collection       ENABLED             3
sql tuning advisor                    ENABLED             3
auto space advisor                    ENABLED             3

9 rows selected.

I haven't verified the reason why this is different from the docs but it may have been related to one change in Oracle Database 12.1.0.2: The new SPM Evolve Advisor Task ( SYS_AUTO_SPM_EVOLVE_TASK) for automatic plan evolution for SQL Plan Management. This new task doesn't appear as a stand-alone job (client) in the maintenance window but runs as a sub-entity of the Automatic SQL Tuning Advisor task. And (I'm just guessing) this may be one of the reasons why every PDB will have to have its own Automatic SQL Tuning Advisor task 

Here you'll find more information about how to enable, disable and configure the new Oracle 12.1.0.2 SPM Evolve Advisor Task:

-Mike

Grid Infrastructure Management Repository (GIMR)
database now mandatory in Oracle GI 12.1.0.2

During the installation of Oracle Grid Infrastructure 12.1.0.1 you've had the following option to choose YES/NO to install the Grid Infrastructure Management Repository (GIMR) database MGMTDB:

With Oracle Grid Infrastructure 12.1.0.2 this choice has become obsolete and the above screen does not appear anymore. The GIMR database has become mandatory

What gets stored in the GIMR?

See the changes in Oracle Clusterware 12.1.0.2 here:

  • Automatic Installation of Grid Infrastructure Management Repository

    The Grid Infrastructure Management Repository is automatically installed with Oracle Grid Infrastructure 12crelease 1 (12.1.0.2). The Grid Infrastructure Management Repository enables such features as Cluster Health Monitor, Oracle Database QoS Management, and Rapid Home Provisioning, and provides a historical metric repository that simplifies viewing of past performance and diagnosis of issues. This capability is fully integrated into Oracle Enterprise Manager Cloud Control for seamless management.

Furthermore what the doc doesn't say explicitly:

  • The -MGMTDB has now become a single-tenant deployment having a CDB with one PDB
    • This will allow the use of a Utility Cluster that can hold the CDB for a collection of GIMR PDBs
  • When you've had already an Oracle 12.1.0.1 GIMR this database will be destroyed and recreated
    • Preserving the CHM/OS data can be acchieved with OCULMON to dump it out into node view
  • The data files associated with it will be created within the same disk group as OCR or VOTING
    •  The OUI will get the disk groups for OCR and Voting and chooses the first one - which usually is the first OCR. This may lead to serious space issues. It is tracked internally as Bug:19661882  In a future release there may be an option offered to put in into a separate disk group.
      Workaround would be to move the affected OCR to another disk group (use ocrconfig command for it) - see MOS Note:1589394.1
  • Some important MOS Notes:
    • MOS Note 1568402.1
      FAQ: 12c Grid Infrastructure Management Repository, states there's no supported procedure to enable Management Database once the GI stack is configured
    • MOS Note: 1921105.1
      Managing the Cluster Health Monitor Repository (incl how to resize)  
    • MOS Note 1589394.1
      How to Move GI Management Repository to Different Shared Storage
      (shows how to delete and recreate the MGMTDB)
    • MOS Note 1631336.1
      Cannot delete Management Database (MGMTDB) in 12.1
    • MOS Note 1945558.1
      _mgmtdb Service Registered with All Local Listeners in a Grid Infrastructure Environment
  • Average growth size per day per node is roughly 650-750 MB. E.g. a 4 node cluster would lead at the default retention of 3 days to an approximate size of  5.9-6.8 GB
  • Change the retention 12.1.0.1:
    $CRS_HOME/bin/oclumon manage -repos changeretentiontime 260000
  • Change the retention 12.1.0.2:
    $CRS_HOME/bin/oclumon manage -repos checkretentiontime 260000

-Mike

PS: Kudos to Sebastian Solbach who updated me on the things to add (retention, average growth, OUI choosing the first disk group displayed for the MGMTDB) - cheers!

Thursday Aug 14, 2014

RMAN Catalog requires Enterprise Edition (EE)
since Oracle Database 12.1.0.2

Credits go to Cameron Hodge, Malcom and Martin Mäs who all highlighted issues to me following my previous entry about RMAN - and sorry for any disappointment but I wasn't aware of all these nice little things. 

Ok, you'd upgrade your RMAN Catalog to be ready to backup/recover Oracle Database 12.1.0.2 databases and you see this error:

RMAN> upgrade catalog;

error creating create_deleted_object_seq
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================

RMAN-06004: ORACLE error from recovery catalog database: ORA-00439: feature not enabled: Partitioning

Now you start to wonder as your Catalog Database had been an Oracle Standard Edition (SE) database for quite a while - and of course an SE database does not have Partitioning on. And on the side the Oracle Partitioning Option is licensable option.

First of all this new behavior gets introduced with Oracle Database 12.1.0.2. And as far as I know it is not documented in relation to the "upgrade catalog". 

The valid workaround leads to a new feature called Infrastructure Repository Database - which is always an EE database without the need for extra licensening as all feature will be used only by Oracle internal mechanisms. 

Licensing doc:
http://docs.oracle.com/database/121/DBLIC/editions.htm#DBLIC119

Infrastructure Repository Databases

A separate single instance Oracle Database can be installed and used as an infrastructure repository for RMAN, Oracle Enterprise Manager Cloud Control, Automatic Workload Repository (AWR) Warehouse, Global Data Services Catalog, and Grid Infrastructure Management Repository without additional license requirements, provided that all the targets are correctly licensed. It may not be used or deployed for other uses.

The infrastructure repositories for RMAN, Oracle Enterprise Manager Cloud Control, AWR Warehouse, and Global Data Services Catalog can be shared in one database, or deployed in separate databases. Enterprise Edition must be used for the infrastructure repository database(s).

Furthermore the Recovery Manager documentation mentions this:

Creating and Managing Virtual Private Catalogs with Oracle Database 12c Release 1 (12.1.0.2)

Note:

Starting with Oracle Database 12c Release 1 (12.1.0.2), virtual private catalogs can be used only with the Enterprise Edition of the Oracle Database. This functionality is not supported for the Standard Edition.

Now this is a bit misleading as most of you won't use the Virtual Private Catalogs - but even though you may not use it still some of the functionality is in your catalog already. And therefore every RMAN catalog beginning with Oracle Database 12.1.0.2 must be hosted in an Enterprise Edition database which does not require an EE license.

The next question is:
How do I migrate my catalog into an EE database?

There are two options whereas I'd prefer the one mentioned in the RMAN documentation:

And there are more things to mention:

  • DBMS_LOB package must be present
  • UTL_HTTP must be present

.

To summarize to successfully upgrade your RMAN catalog:

  • You need to execute this script before upgrading the catalog:
    SQL> @$ORACLE_HOME/rdbms/admin/dbmsrmansys.sql
  • Your database hosting the RMAN Catalog beginning with Oracle 12.1.0.2:
    • Must be an Enterprise Edition database - no EE license required
    • Oracle Partitioning Option must be installed - no Partitioning lincese required
    • DBMS_LOB and UTL_HTTP packages must be present

-Mike                  

PS:  Credits for this addition go to Malcom!!! In addition to RMAN Catalog Database now requiring to be in an EE database this database will require the Partitioning Option linked in as well. See MOS Note:1927265.1

Monday Aug 11, 2014

RMAN Catalog Upgrade to Oracle 12.1.0.2

It sounds so simple - but in this specific case a bit of doc reading is required as upgrading the RMAN catalog to handle Oracle 12.1.0.2 databases is not as trivial as in the past. 

Thanks to a German customer and a friendly colleague from Sales Consulting in Stuttgart I have learned my RMAN lesson for Oracle Database 12.1.0.2 this week. A simple "upgrade catalog" is not the correct step once you'd like to handle backups of Oracle 12.1.0.2 databases in your current catalog schema. 

Even though you may not have ever heard before about Virtual Private Catalogs you need to follow this guideline:

http://docs.oracle.com/database/121/BRADV/rcmcatdb.htm#BRADV848

The doc tells you to run this step first:

SQL> @$ORACLE_HOME/rdbms/admin/dbmsrmansys.sql

  • Please ignore the "-all" option printed in the doc - this is a known docu bug
  • If you miss this step the upgrade of the catalog will fail with a warning that your user lacks privileges 

$ rman CATALOG my_catalog_owner@catdb
recovery catalog database Password: 
RMAN> UPGRADE CATALOG; 
RMAN> EXIT; 

It should work now :-)

-Mike 

Tuesday Aug 05, 2014

Upgrade PDBs - One at a Time (unplug/plug)

*** I have added an important change on May 26, 2015 ***
***      Please see below marked in YELLOW           *** 
********************************************************

Basically there are two techniques to upgrade an Oracle Multitenant environment:

In this post I will refer to the "One at a Time" approach and describe the steps. During some presentations, discussions etc people were left with the impression that it will be a very simple approach to unplug one or many PDBs from a CDB in lets say Oracle 12.1.0.1 and plug it into an Oracle 12.1.0.2 Container Database. Bingo, upgraded!

Well, unfortunately this is not true. In fact it is completely wrong.


If you want to upgrade via unplug/plug the following steps will have to be followed:

  • In CDB1 environment - e.g. Oracle 12.1.0.1 with an PDB1
    • In SQL*Plus: 
      • alter session set container=PDB1;
      • @$ORACLE_HOME_12102/rdbms/admin/preupgrd.sql
        (The output of the preupgrade.log will show you the location of the fixups)
      • @/u01/app/oracle/cfgtoollogs/CDB1/preupgrade/preupgrade_fixups.sql
        (If ORACLE_BASE is not set the files will be created under $ORACLE_HOME/cfgtoollogs instead of $ORACLE_BASE/cfgtoollogs)
      • exec dbms_stats.gather_dictionary_stats;
        (plus include all additional treatments recommended by the preupgrade.log)
      • alter session set container=CDB$ROOT; 
      • alter pluggable database PDB1 close;
      • alter pluggable database PDB1 unplug into '/stage/pdb1.xml';
      • drop pluggable database PDB1 keep datafiles;
        The reason why you will need to DROP the PDB afterwards is simply to cleanup leftovers in the CDB views. It is under observation if this is a bug or not. The information does not get removed to allow quick plugin again but the leftovers may cause plenty of trouble once you'll try to upgrade this CDB1 later on.
      • exit
        .
  • In CDB2 environment - e.g. Oracle 12.1.0.2
    • In SQL*Plus:
      • alter session set container=CDB$ROOT;
      • At this point we "could" do a Plug In Check but as the COMPATIBLE of the new CDB2 created as per recommendation with DBCA defaults to "12.1.0.2" the Plug In Check will result in "NO" - but obviously the plugin operation will work. Just for the records here's the procedure to check plugin compatibility
        • SET SERVEROUTPUT ON
          DECLARE
            compatible CONSTANT VARCHAR2(3) := CASE DBMS_PDB.CHECK_PLUG_COMPATIBILITY(
            pdb_descr_file => '/stage/pdb1.xml',
            pdb_name => 'PDB1')
            WHEN TRUE THEN 'YES' ELSE 'NO'
          END;
          BEGIN
          DBMS_OUTPUT.PUT_LINE(compatible);
          END;
          /

          .
          select message, status from pdb_plug_in_violations where type like '%ERR%';
          .
      • create pluggable database pdb1 using '/stage/pdb1.xml' file_name_convert=('/oradata/CDB1/pdb1', '/oradata/CDB2/pdb1');
      • alter pluggable database PDB1 open upgrade;
      • exit
    • On the command prompt:
      • cd $ORACLE_HOME/rdbms/admin 
      • $ORACLE_HOME/perl/bin/perl catctl.pl -c "PDB1" -l /home/oracle/upgrade catupgrd.sql
    • Back into SQL*Plus:
      • alter session set container=pdb1;
      • startup
      • @?/rdbms/admin/utlrp.sql
      • @/u01/app/oracle/cfgtoollogs/CDB1/preupgrade/postupgrade_fixups.sql
        (If ORACLE_BASE is not set the files will be created under $ORACLE_HOME/cfgtoollogs instead of $ORACLE_BASE/cfgtoollogs)
Of course this technique will work also with more than one PDB at a given time. You'll have to repeat the steps, and your upgrade call on the command line will look like this:

      • $ORACLE_HOME/perl/bin/perl catctl.pl -c "PDB1, PDB2" -l /home/oracle/upgrade catupgrd.sql

Well, not really unplug+plug=upgraded ;-)

-Mike 

PS: I did add a few pieces of information based on the excellent feedback given to me by Frank Kobylanski from the MAA Team - cheers, Frank!!! 

Friday Aug 01, 2014

New (some undocumented) Parameters in Oracle 12.1.0.2

Every release offers some surprises - even to myself ;-)

Right now Roy and I are in the final steps to refresh our big slide deck to the new layout, but more important, to have Oracle 12.1.0.2 information included as well (were necessary). So I did my usual "compare parameters" query between releases - getting unusual surprises this time.

This is the list of new parameters introduced with the patch set Oracle Database 12.1.0.2. Where applicable I have added the link to the doc.

But as you may recognize not all of them are explained in the doc ;-)

  • DBFIPS_140
    • Default: FALSE
    • DBFIPS_140 enables Transparent Data Encryption (TDE) and DBMS_CRYPTO PL/SQL package program units to run in a mode compliant to the Federal Information Processing Standard (subsequently known as "FIPS mode";)
  • COMMON_USER_PREFIX
    • Default: c##
    • Specifies a prefix that the names of common users, roles, and profiles in a multitenant container database (CDB) must start with. If COMMON_USER_PREFIX is set to an empty string, Oracle will not enforce any restrictions on the names of common or local users, roles, and profiles.
  • DB_PERFORMANCE_PROFILE <<updated Dec 16>>
    • Undocumented
    • See bug17861171, bug18406144 and bug19817284 - IORM feature on Exadata only
  • ENABLE_GOLDENGATE_REPLICATION
    • Default: FALSE
    • Controls services provided by the RDBMS for Oracle GoldenGate (both capture and apply services). Set this to true to enable RDBMS services used by Oracle GoldenGate
    • Introduced with Oracle 11.2.0.4 and Oracle 12.1.0.2
  • EXCLUDE_SEED_CDB_VIEW
    • Undocumented
    • Per feedback by the Multitenant team:
      • Default: TRUE
      • Setting this parameter to FALSE would return results for the seed database when querying against the CDB views
  • INMEMORY_CLAUSE_DEFAULT
    • Default: an empty string
    • Enables you to specify a default In-Memory Column Store (IM column store) clause for new tables and materialized views. If the INMEMORY_CLAUSE_DEFAULT parameter is unset or set to an empty string (the default), only tables and materialized views explicitly specified asINMEMORY will be populated into the IM column store. Setting the value of the INMEMORY_CLAUSE_DEFAULT parameter to NO INMEMORY has the same effect as setting it to the default value.
  • INMEMORY_FORCE
    • Default: DEFAULT
    • Allows you to specify whether tables and materialized view that are specified as INMEMORY are populated into the In-Memory Column Store (IM column store) or not. The default value is DEFAULT. When this value is in effect, the IM column store is populated only with tables and materialized views specified as INMEMORY. If OFF is specified, then even if the IM column store is configured on this instance, no tables or materialized are populated in memory.
  • INMEMORY_MAX_POPULATE_SERVERS
    • DefaultHalf the effective CPU thread count or the PGA_AGGREGATE_TARGET value divided by 512M, whichever is less.
    • Specifies the maximum number of background populate servers to use for In-Memory Column Store (IM column store) population, so that these servers do not overload the rest of the system
  • INMEMORY_QUERY
    • Default: ENABLE
    • Used to enable or disable in-memory queries for the entire database at the session or system level. This parameter is helpful when you want to test workloads with and without the use of the In-Memory Column Store (IM column store)
  • INMEMORY_SIZE
    • Default: 0
    • Sets the size of the In-Memory Column Store (IM column store) on a database instance. If a database does not have automatic memory management enabled, this parameter must be set to a nonzero value that reserves the amount of memory to use for the database's IM column store. The default value is 0, which means that the IM column store is not used. The database must be restarted after setting this parameter to enable the IM column store. The minimum size to which this parameter can be set is 100 MB.
  • INMEMORY_TRICKLE_REPOPULATE_SERVERS_PERCENT
    • Default: 1
    • Limits the maximum number of background populate servers used for In-Memory Column Store (IM column store) repopulation, as trickle repopulation is designed to use only a small percentage of the populate servers. The value for this parameter is a percentage of the INMEMORY_MAX_POPULATE_SERVERS initialization parameter value
  • OPTIMIZER_INMEMORY_AWARE
    • Default: TRUE
    • Enables or disables all of the optimizer cost model enhancements for in-memory. Setting the parameter to false causes the optimizer to ignore the in-memory property of tables during the optimization of SQL statements. This behavior can also be achieved by setting theOPTIMIZER_FEATURES_ENABLE initialization parameter to values lower than 12.1.0.2
  • PDB_LOCKDOWN
    • Undocumented
    • Per feedback by the Multitenant team:
      • Not functional in Oracle 12.1.0.2
  • PDB_OS_CREDENTIAL
    • Undocumented
    • Per feedback by the Multitenant team:
      • Not functional in Oracle 12.1.0.2
      • May be functional with a future PSU allwoing then OS user verfication/validation for PDBs

-Mike

PS: Forgot to mention this one as a parameter which had been disappeared in Oracle 12.1.0.2:

  • PARALLEL_FAULT_TOLERANCE_ENABLED
    • Undocumented in Oracle 12.1.0.2
    • Disappeared in Oracle 12.1.0.2 but did exist in Oracle 12.1.0.1 

Thursday Jul 24, 2014

Why "We'll wait for the 2nd release!" is a misconception ...

Oh, how often have I heard this phrase:

"We'll wait for the second release!"

And sometimes it makes me really anxious and angry at the same time when I hear that.

Anxious because it means that somebody has no strategy for the database upgrades/migrations and is just postponing necessary tasks to sometime in the future. Easy deal but not very clever

Angry because this is a way of thinking from the 90s/00s when Oracle had this "10.1", then later "10.2" strategy in database releases. But we are in 2014 now. And things have changed. Changed a lot in fact.

I'm probably not the only person who would love to see if we'd remove this "first" and "second" release tags. This has become obsolete with Oracle 11.1. We were telling customers officially that this is the brand new fantastic Oracle Database 11g release. But in fact it was - from the coding perspective - more or less a very stable 10g. In my workshops I did call it Oracle 10.3 with a marketing sticker on it. And as far as I can see the customer's I've had helped with going live on Oracle Database 11.1.0.7 were quite happy. Some really large shops still work with this release today with hundreds of databases in production.

But honestly most of the changes got introduced with Oracle Database 11.2. And not for a small number of customers this meant waiting for the first patch set (which since then has become a full release). Plenty of people went live with Oracle Database 11.2.0.2. But a lot of the remarkable changes got introduced not in Oracle 11.1 but in Oracle 11.2. Just remember things such as the move from Clusterware to Grid Infrastructure. But also minor things such as DEFERRED_SEGMENT_CREATION and plenty of optimizer news.

Now with Oracle Database 12c and the first patch set (full release) 12.1.0.2 I have heard this again - and I see it on the mailing list as well once a week:

"When will Oracle 12.2 be available?"

Well, that's the misconception. It's true, Oracle Database 12.1.0.2 has new features and extensions. But it has also many fixes over the already very stable Oracle 12.1.0.1. Why should anybody wait for Oracle 12.2 now? Because it's supposed to be THE SECOND release? Forget this - this is thinking from the old days.
We are in year 2014 now.

And then spend a minute to look closer to the Support Policy.

  • Oracle Database 11.2 will go out of Premier Support in 6 months. Yes!!! 6 months
  • And correct, we'll give everybody on Oracle 11.2.0.4 one full year of Extended Support for free
  • For Oracle 11.2.0.3 Extended Support will end 28-AUG-2015

That means if you plan to stay on Oracle 11.2.0.3/4 for a longer period you'll either have to calculate 20% extra of your support fee for the 2nd year of Extended Support. Or you prefer to "hope".

I can't tell you when Oracle 12.2 will be available - and I don't care. Usually people wait for the first patch set anyways which gets releases based on experience from the past 3 databases releases roughly a year and a bit after the initial release. Just do the math and you'll see where you end up with this strategy.

My recommendations are:

  • Stop thinking about THE SECOND release
  • Evaluate Oracle Database 12.1.0.2 now - not tomorrow
  • Look at the Support Policy - you need to start your upgrades as soon as possible
-Mike 

 

 

About

Mike Dietrich - Oracle Mike Dietrich
Master Product Manager - Database Upgrade & Migrations - Oracle Corp

Based near Munich/Germany and spending plenty of time in airplanes to run either upgrade workshops or work onsite or remotely with reference customers. Acting as interlink between customers/partners and the Upgrade Development.

Follow me on TWITTER

Contact me via LinkedIn or XING

Search

Archives
« July 2015
SunMonTueWedThuFriSat
   
4
5
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
 
       
Today
Oracle related Tech Blogs
Slides Download Center
Visitors since 17-OCT-2011
White Paper and Docs
Workshops
Viewlets and Videos
This week on my Rega/iPod/CD
Workshop Map
Upgrade Reference Papers