Wednesday May 04, 2016

Are BPs. PSUs and Proactive BPs cumulative?

Are Bundle Patches (BPs) and Patch Set Updates (PSUs) cumulative?

That is a question sounding trivial to many people but actually it does get asked quite often. And sometimes I forget to mention this during the workshops - and luckily usually somebody asks the question reminding me to explain it.

Yes, Bundle Patches and Patch Set Updates (and of course Proactive Bundle Patches and Critical/Security Patch Updates (CPUs/SPUs) are all cumulative.

You'll find this mentioned in the first paragraph of MOS Note: 854428.1 - Patch Set Updates for Oracle Products:


...

Interesting note on the side:
I would have expected this important piece of information in MOS Note:1962125.1 - Oracle Database - Overview of Database Patch Delivery Methods but I couldn't find it. So it's no wonder why people ask such a trivial question ... [irony!]  
.

Two simple examples:

  • You have the October 2015 PSU applied
  • You'd like to apply the April 2016 PSU on top
    • Then you don't need the January 2016 PSU as it is included in the April 2016 PSU already
      .
  • You never applied a Procative Bundle Patch
  • You'd like to apply the April 2016 Proactive Bundle because a guy recommended it on an Oracle blog - and actually MOS notes mention it as well as highly recommended
    • You don't need to apply anything beforehand.
      The April 2016 Proactive BP has all the fixes from all previous BPs included on top of Oracle Database 12.1.0.2.0
      .

Further Information?

You'll find recent related postings on this blog here: 

 --Mike
.

Tuesday May 03, 2016

Can I apply a BP on top of a PSU? Or vice versa?

This question was in my inbox this morning raised by a customer via a colleague. 

Our feeling said:
Yes, but you'll have to deinstall the PSU first before applying the BP.

Having a feeling is one thing, knowing the truth is better. And as I have so much fun by applying PSUs and BPs back and forth since two weeks I thought "let's give it a try". So here we go ...
.

Apply a BP on top of a PSU

This is my starting point - Oracle Database 12.1.0.2 with PSU October 2015, the last state in my VBox environment since I experiment with parameters and packages at the moment.

And this is the result when trying to apply the BP from April 2016 on top: 

[CDB2] oracle@localhost:/media/sf_CTEMP/22899531/22899531/22806133
$ opatch apply
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/12.1.0.2
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.1.0.2/oraInst.loc
OPatch version    : 12.1.0.1.10
OUI version       : 12.1.0.2.0
Log file location : /u01/app/oracle/product/12.1.0.2/cfgtoollogs/opatch/opatch2016-05-03_10-26-37AM_1.log

Verifying environment and performing prerequisite checks...

Conflicts/Supersets for each patch are:

Sub-Patch : 20243804

        Bug Conflict with Sub-Patch 21359755
        Conflicting bugs are:
        18966843, 19468991, 19032777, 19587324

[..]


Following patches have conflicts: [   21359755   22806133 ]
Refer to My Oracle Support Note 1299688.1 for instructions on resolving patch conflicts.

UtilSession failed: Patch 21359755 is a composite patch which could not be rollback.

Log file location: /u01/app/oracle/product/12.1.0.2/cfgtoollogs/opatch/opatch2016-05-03_10-26-37AM_1.log

OPatch failed with error code 73


Ok, our suspicion was correct.

There are a lot of conflicts - I have to remove the PSU first. And of course the recommended conflict check would have given me the same result. Furthermore I think I have read this a while back in one of the central notes about PSUs and BPs in MOS as well.
.

Removing the PSU from October 2015 first

Removing a PSU or BP is very simple and straight forward (and well described in the ReadMe.html coming coming with the patch).

[CDB2] oracle@localhost:/media/sf_CTEMP/21359755/21359755
$ opatch rollback -id 21359755
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/12.1.0.2
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.1.0.2/oraInst.loc
OPatch version    : 12.1.0.1.10
OUI version       : 12.1.0.2.0

Log file location : /u01/app/oracle/product/12.1.0.2/cfgtoollogs/opatch/21359755_May_03_2016_10_41_54/rollback2016-05-03_10-41-54AM_1.log

Patches will be rolled back in the following order:
   21359755   20831110   20299023   19769480
The following patch(es) will be rolled back: 21359755  20831110  20299023  19769480
Sub-patches of a composite series are being rolled back. The system will be returned to a state where all subpatches are rolled back.

[..]

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/12.1.0.2')

Is the local system ready for patching? [y|n]
y

User Responded with: Y
Rolling back patch 21359755...

[..]

RollbackSession removing interim patch '19769480' from inventory Log file location: /u01/app/oracle/product/12.1.0.2/cfgtoollogs/opatch/21359755_May_03_2016_10_41_54/rollback2016-05-03_10-41-54AM_1.log OPatch succeeded.

Apply the BP from April 2016

I don't want to repeat myself as I wrote already about this positive experience a few days ago:

Oracle Database BP April 2016 applied successfully (Apr 21, 2016)
https://blogs.oracle.com/UPGRADE/entry/oracle_database_bp_april16_applied

Further Information?

You'll find recent related postings on this blog here:

--Mike

Tuesday Apr 26, 2016

MOS Note 1454618.1: Quick Reference to Database PSUs, CPUs, BPs and Patchsets

Sometimes my mouse arm gets tired by clicking myself through MOS notes just to download a specific PSU or BP - and as I experiment a lot with PSUs and BPs right now I clicked a lot in the past days and nights. 

Usually I'd start with either MOS Note:161818.1 - then click on the release link (e.g. 12.1.0.x) in the left-most column, then into the Availability and Known Issues not (e.g. MOS Note:1683799.1 for Oracle Database 12.1.0.2.0) and then select the most recent patch from the list of Current Recommended Patches.

Even though we all agree that you should be ideally always on the most recent BP (or at least PSU) there may be situations where you need to access an older PSU or BP or just the CPU.
So what if you need for instance the October 2015 PSU?
This is obviously not linked from the above note.

My usual click routine guides me via the Critical Patch Updates and Security Alerts page, from there via the particular release date (e.g. October 2015) to the Database Server Version (e.g. linked from here is MOS Note:2037108.1) and then to the patch number I'm searching for.

Just learned a few days ago by accident that there's a MOS Note which I have on my favorites since then:

containing all this information - even for 8.1.7.4 patches if you still need them ;-)

After listing the base release first followed by the available patch sets it offers you the links to the PSUs, CPUs and BPs - and if you are looking for the patches containing the OJVM you'll find them by scrolling down towards the end of the note as well in a separate section. 

*** Don't get puzzled by the note's title - it lists the Bundle Patches for Windows only inside, not the general link to all BPs. Myself and a colleague left already feedback for the note owner to add the BP links (or Proactive Bundle Patch links) as well *** 

MOS Note Patches PSUs CPUs SPUs BPs

In fact a very useful MOS Note.
.

--Mike

Thursday Apr 21, 2016

Oracle Database BP April16 applied successfully

Usually I don't post twice a day but as my post scriptum for the previous blog post got longer and longer I decided to write an entry about it - maybe simply because I feel soooo happy that my patch application succeeded flawless.

For many of you the following steps may look very boring as you have done this many times. But I use the blog also to brain-dump information for myself ;-) And for those who'd like to play with it, I summarized the steps fitting exactly into our Hands-On-Lab environment.

Download

I downloaded the following patches: 

Unzip

Then unzipped the patch into an empty directory and OPatch directly into the $ORACLE_HOME/OPatch directory in my 12.1.0.2 home.  As I run everything in our HOL environment (download it here) I stored the files on my local host and mapped the directories via a VBox shared folder. 

  • BP into: /media/sf_CTEMP/22899531/22899531/
  • OPatch into: unzip <OPATCH-ZIP> -d <ORACLE_HOME>
    .

Prechecks

Not sure if the 2nd conflict check was really necessary as it is a CRS patch. 

  • System Space Check 
    • Created a file /tmp/patch_list_dbhome.txt with the following content:
      /media/sf_CTEMP/22899531/22899531/22806133
      /media/sf_CTEMP/22899531/22899531/23006522
    • Run opatch to check if enough free space is available in the Database Home:
      $ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_dbhome.txt

My first attempt failed - I had to clean up a bit.

After I removed a bit of stuff (I actually deleted the 12.1.0.1 home with $ORACLE_HOME/deinstall/deinstall after switching to the 12.1.0.1 environment with . cdb1 in the HOL environment) I could install the BP. 
.

Patch Installation

Here it struck me a bit that the logical layout of the patches readme is still very misleading. I was scrolling up and down to find the section for "Database Home only" install but realized that is is still buried under Patching Oracle RAC Database Homes and GI Together which does not make much sense. Anyhow, I stopped arguing with the owners. 

The section I'm interested in here is under Case 3: Non-Oracle RAC Database homes.

  • Shutdown my database(s) and my listener - in the lab env this means:
    • . cdb2
    • sqlplus / as sysdba
    • SQL> shutdown immediate
    • SQL> exit
    •  lsnrctl stop
  • Add OPatch to the PATH 
    • export PATH=$PATH:/u01/app/oracle/product/12.1.0.2/OPatch
  • Switch to the patches directory and apply the BP
    • cd /media/sf_CTEMP/22899531/22899531/22806133
    • opatch apply

 And after a while ... 


.

SQL Changes 

Almost done. But don't forget to run ./datapatch -verbose afterwards - see the readme's section:

2.6.2 Loading Modified SQL Files into the Database

As it is a Multitenant database in my case I'll have to start the database and the listener first, then start all pluggable databases: 

  • . cdb2
  • sqlplus / as sysdba
  • SQL> startup
  • SQL> alter pluggable database all open;
  • SQL> exit
  • lsnrctl start
  • cd $ORACLE_HOME/OPatch
  • ./datapatch -verbose

And finally ...

... after another little while ...

Done!
.

--Mike

Oracle April 2016 PSU and Proactive BPs are there

Hurray, it's Patching Day!

Sounds a bit like D-Day ;-) But April 19, 2016 the most recent April PSUs (Patch Set Updates) and BPs (Bundle Patches) got released.

Find all the necessary information with the below links: 

The important change in the April PSU/BP release:
The database patch for "Engineered Systems and Database In-Memory 12.1.0.2" luckily got renamed into "Proactive Bundle Patch 12.1.0.2". That is not only a rebranding but it should express that we would like to encourage you to apply the Bundle Patches
instead of the PSUs. Simple reason is that the BPs will contain optimizer fixes. 

In the MOS Note: 2102148.1 (Patch Set Update and Critical Patch Update April 2016 Availability Document) you'll find a section 3.1.4 linking to the database patches

This is the recommended one for Oracle Database 12.1.0.2:

  • Database Proactive Bundle Patch 12.1.0.2.160419 (Apr2016) Patch 22899531,

But right now it is available for Linux-x86-64, zLinux and Intel Solaris only. Not sure when the others will get released. Please find links to the regular PSUs and other ports and releases such as 11.2.0.4 and Windows etc in the above MOS Note: 2102148.1.

This is the list of fixes included in this Bundle Patch:

And don't worry about the name - I found out yesterday that not all MOS Notes have adopted the new naming convention to rename "Bundle Patches for Engineeered Systems and DB In-Memory" which was very misleading anyway into the new "Proactive Bundle Patches" naming. This may take a few additional days I'd guess ...

I will download it right now and patch my HOL environment.

And as usual don't forget the most recent version of opatch (Patch 6880880).

opatch download MOS

.

--Mike 
.


Tuesday Feb 02, 2016

How to find out if a PSU has been applied? DBMS_QOPATCH

pflaster.jpgSince we change the PSU and BP patch numbering from Oracle Database 12.1.0.2.PSU6 to 12,1,0,2,160119 it is almost impossible to distinguish from the patch name only if you have applied a PSU or a BP.

But:
In Oracle Database 12c there's a package available which is very useful to query plenty of information about patches from within the database: DBMS_QOPATCH.

Here are a few helpful examples which I created by checking in our DBaaS Cloud database.

Which patches have been applied (or rolled back)?

SQL> set serverout on

SQL> exec dbms_qopatch.get_sqlpatch_status;

Patch Id : 20415564
        Action : APPLY
        Action Time : 24-JUN-2015 06:19:23
        Description : Database PSU 12.1.0.2.3, Oracle JavaVM Component (Apr2015)
        Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/20415564/18617752/
                  20415564_apply_ORCL_CDBRO
OT_2015Jun24_06_18_09.log
        Status : SUCCESS

Patch Id : 20299023
        Action : APPLY
        Action Time : 24-JUN-2015 06:19:23
        Description : Database Patch Set Update : 12.1.0.2.3 (20299023)
        Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/20299023/18703022/
                  20299023_apply_ORCL_CDBRO
OT_2015Jun24_06_18_11.log
        Status : SUCCESS

Patch Id : 20848415
        Action : APPLY
        Action Time : 24-JUN-2015 06:19:23
        Description :
        Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/20848415/18918227/
                  20848415_apply_ORCL_CDBRO
OT_2015Jun24_06_18_15.log
        Status : SUCCESS

Patch Id : 20848415
        Action : ROLLBACK
        Action Time : 24-JUN-2015 06:52:31
        Description :
        Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/20848415/18918227/
                  20848415_rollback_ORCL_CD
BROOT_2015Jun24_06_52_29.log
        Status : SUCCESS

Patch Id : 20618595
        Action : APPLY
        Action Time : 24-JUN-2015 13:52:13
        Description :
        Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/20618595/18956621/
                  20618595_apply_ORCL_CDBRO
OT_2015Jun24_13_52_12.log
        Status : SUCCESS

Patch Id : 20618595
        Action : ROLLBACK
        Action Time : 24-JUN-2015 14:37:11
        Description :
        Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/20618595/18956621/
                  20618595_rollback_ORCL_CD
BROOT_2015Jun24_14_37_10.log
        Status : SUCCESS

Patch Id : 20415564
        Action : ROLLBACK
        Action Time : 27-JAN-2016 17:43:18
        Description : Database PSU 12.1.0.2.3, Oracle JavaVM Component (Apr2015)
        Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/20415564/18617752/
                  20415564_rollback_MIKEDB_
CDBROOT_2016Jan27_17_42_16.log
        Status : SUCCESS

Patch Id : 21555660
        Action : APPLY
        Action Time : 27-JAN-2016 17:43:18
        Description : Database PSU 12.1.0.2.5, Oracle JavaVM Component (Oct2015)
        Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/21555660/19361790/
                  21555660_apply_MIKEDB_CDB
ROOT_2016Jan27_17_42_17.log
        Status : SUCCESS

Patch Id : 21359755
        Action : APPLY
        Action Time : 27-JAN-2016 17:43:18
        Description : Database Patch Set Update : 12.1.0.2.5 (21359755)
        Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/21359755/19194568/
                  21359755_apply_MIKEDB_CDB
ROOT_2016Jan27_17_42_18.log
        Status : SUCCESS

Patch Id : 21962590
        Action : APPLY
        Action Time : 27-JAN-2016 17:43:18
        Description :
        Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/21962590/19426224/
                  21962590_apply_MIKEDB_CDB
ROOT_2016Jan27_17_42_21.log
        Status : SUCCESS

PL/SQL procedure successfully completed.
.

Where's my home and inventory?

SQL> set pagesize 0

SQL> set long 1000000 

SQL> select xmltransform(dbms_qopatch.get_opatch_install_info, dbms_qopatch.get_opatch_xslt) "Home and Inventory" from dual;

Home and Inventory
-------------------------------------------------------------

Oracle Home     : /u01/app/oracle/product/12.1.0/dbhome_1
Inventory    
    : 
/u01/app/oraInventory


Has a specific patch been applied?

Lets check for the latest PSU. 

SQL> select xmltransform(dbms_qopatch.is_patch_installed('21359755'), dbms_qopatch.get_opatch_xslt) "Patch installed?" from dual;

Patch installed?
-------------------------------------------------------

Patch Information:
         21359755:   applied on 2015-10-22T21:48:17Z

.

What's tracked in my inventory?

The equivalent of opatch lsinventory -detail ...

SQL> select xmltransform(dbms_qopatch.get_opatch_lsinventory, dbms_qopatch.get_opatch_xslt) from dual; 

Oracle Querayable Patch Interface 1.0
----------------------------------------------------------------
Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
Inventory         : /u01/app/oraInventory
----------------------------------------------------------------

Installed Top-level Products (1):
                                    12.1.0.2.0
Installed Products ( 135)
                               ...

.

Additional Information and Patches

If you need more helpful examples you may check this excellent blog post by Simon Pane (Pythian):

And credits to Martin Berger for sending me this important information:

Just in case there are multiple DBs running from the same O_H, and someone      
queries dbms_qopatch.get_opatch_lsinventory automated from all DBs (as in       
automated monitoring/reporting scripts) I'd recommend Patch 20599273 -          
otherwise there might be strange XM errors due to race conditions. 

.

--Mike 

Wednesday Jan 20, 2016

Oracle January 2016 CPU PSU BP available now - BE AWARE OF CHANGES IN PATCH NUMBERING

Last night the PSUs and BPs for January 2016 have been made available for download on support.oracle.com.

Oracle Critical Patch Update Advisory - January 2016

http://www.oracle.com/technetwork/topics/security/cpujan2016-2367955.html 

It contains 248 security fixes across all products and platforms. And of course important non-security fixes - and that's why we recommend to apply the PSUs (or the BPs in case you are on Exadata or an Oracle In-Memory user) as soon as possible. 

Change in Patch Numbering

Please be aware that as of November 2015 there's been a change in patch numbering introduced which most of you may not be aware of. A database PSU was named 12.1.0.2.5 before (or I used to call it 12.1.0.2.PSU5 before to make clear that a PSU and not a BP has been applied). But the new notation will change the 5th digit to a 6-digit-number to include the date. See MOS Note:2061926.1 for details.

Example:

  • Before: Oracle Database 12c PSU October 2015 ... 12.1.0.2.5
  • Now: Oracle Database 12c PSU January 2016 ... 12.1.0.2.160119 

More Information? 

--Mike

Thursday Oct 22, 2015

PSU (and CPU/SPU) October 2015 got released

October 21, 2015 - Oracle released the October 2015 SPU/CPU, PSU and BPs. 

See:

For Oracle Database 12.1.0.2 and Oracle Grid Infrastructure access the PSUs from here - if you have only a single instance database you can safely take the Combo patch as you'll get downtime anyways - but for customers running RAC you need to evaluate the OJVM component as this will incur downtime which the database-only patch does not require as it can be applied rolling.

Patch Set Updates

Document Description Rolling RAC Patch Download
Note:21555660.8 Oracle JavaVM Component 12.1.0.2.5 Database PSU (Oct 2015) (OJVM PSU) No Patch:21555660
Note:21520444.8 Combo of 12.1.0.2.5 OJVM PSU and 12.1.0.2.5 DB PSU (Oct 2015) Part Patch:21520444

No patch found at the moment [Mike]
Note:21359755.8 12.1.0.2.5 (Oct 2015) Database Patch Set Update (DB PSU) Yes Patch:21359755

Grid Infrastructure

Document Description Rolling RAC Patch Download
Note:21523260.8 Combo of 12.1.0.2.5 OJVM PSU and 12.1.0.2.5 GI PSU (Oct 2015) Part Patch:21523260
Note:21523234.8 12.1.0.2.5 (Oct 2015) Grid Infrastructure Patch Set Update (GI PSU) Yes Patch:21523234


And be aware to patch your DBaaS Cloud databases as well - this is not done automatically ;-)

--Mike

.

Wednesday Sep 02, 2015

No patches anymore for Oracle Database 11.2.0.3

Just in case you've missed the date please be aware:

No bug fixes, no patches, no CPU/SPUs, no PSUs, no BPs will be produced for Oracle Database 11.2.0.3 since Augugst 28, 2015

I know from the many emails I'm receiving that people are a bit disturbed since the Oracle Lifetime Support Policy brochure (Lifetime Support Policy and Brochure for Technology Products ) doesn't talk explicitly about patch sets but offers support for Oracle 11.2 until January 2018.

Oracle 11.2 Lifetime Support Policy

But this - as always - applies to the terminal patch set only - i.e. Oracle Database 11.2.0.4.

For a detailed description please see MOS Note:742060.1 and my previous blog posts from:

I have visited customers with hundreds of Oracle 11.2.0.3 databases in the past months. If you are now sitting there with one or many or many many Oracle 11.2.0.3 instances don't even think of moving to Oracle 11.2.0.4. The amount of work you'll spend in evaluating Oracle 11.2.0.4 is exactly the same as testing Oracle 12.1.0.2. Move to Oracle Database 12.1.0.2 - NOW!

And keep in mind: there's no such thing as a 2nd release anymore. Every patch set release is a full release - no matter if we talk about 11.2.0.4 or 12.1.0.2. Both have new features, new behavior etc etc.

--Mike

Monday Apr 20, 2015

Oracle PSU and BP April 2015 is available

As of April 14, 2015:

The April 2015 PSU/BP is available!

Here's the most important information:

Please find below the links to the Recommended Patches and Patch Numbers for each of your database releases:

12.1.0.2
 Availability and Known issues for 12.1.0.2      Note:1683799.1
12.1.0.1
 Availability and Known issues for 12.1.0.1      Note:1565082.1 
11.2.0.4
 Availability and Known issues for 11.2.0.4      Note:1562139.1
 
11.2.0.3
 Availability and Known issues for 11.2.0.3      Note:1348336.1

11.1.0.7
 List of fixes included in 11.1.0.7              Note:601739.1

-Mike

Wednesday Feb 18, 2015

Grid Infrastructure PSU Jan 2015 - Am I too intolerant?

Germans are not only known for being precise and timely - but sometimes also for being too direct. Well, Roy could tell you stories ... and I always honor his politeness :-)

Enough about stereotypes. I work with a customer at the moment on their 12c upgrades. And I did recommend the most recent PSU (Patch Set Updates) for their Grid Infrastructure environments running Oracle Restart. Same of course for the database homes but this blog post will just name some findings I've had the other night when trying to apply the January 2015 GI PSU to my Oracle Restart environment.

First of all, start here:

  • Drill down from MOS Note:161818.1 (Click on the Release Link for 12.1.0!) into
  • Oracle 12.1 Support Status and Alerts MOS Note:1565065.1 and further into 
  • 12.1.0.2 Patch Set - Availability and Known Issues MOS Note:1683799.1 to get
    • Recommendations for patches on top of Oracle Database 12.1.0.2 - and of course for Grid Infrastructure (GI), Engineered Systems and Database In-Memory as well.

For GI you'll get the following recommendations:

Grid Infrastructure

Document Description Rolling RAC Patch Download
Note:20132450.8 Combo of 12.1.0.2.2 OJVM PSU and 12.1.0.2.2 GI PSU (Jan 2015) Part Patch:20132450
Note:19954978.8 12.1.0.2.2 (Jan 2015) Grid Infrastructure Patch Set Update (GI PSU) Yes Patch:19954978

Well, first question:
Do you need the Combo Patch or the non-Combo? 

My personal recommendation: Take the non-Combo Patch as the combo patch includes not only the GI PSU, but also the Database PSU, the OJVM Patch and some other things you will not need for patching your Grid Infrastructure only. Of course I can see the benefit of downloading just one big piece and applying everything all together to my environment. But first of all parts of the patch (speaking of OJVM) are not rolling applicable - and second I'd like to control and script things separately. But please feel free to see and do this in a different way.

So I did download Patch:19954978 to my environment. Unzipped it. 
.

You all know it already - you will need a new OPatch!

Of course my OPatch version is too old. You will need at least OPatch 12.1.0.1.5 - the most recent one is OPatch 12.1.0.1.6 - and you simply download it via MOS patch 6880880 and install it after removing the old directory into your $ORACLE_HOME. 
.

First learning experience? OPatch doesn't do anything without a response file.

The reason for this (at least for me as I don't patch daily) new requirement seems to be the new opatchauto call which scripts the entire apply process in a silent way. Well, at least the patch readme tells me what to do. Please read MOS Note:966023.1 to learn about how to create this response file.

If you do not have the OCM response file (ocm.rsp), see the following My Oracle Support Document 966023.1 How To Create An OCM Response File For Opatch Silent Installation.

I did create it with OCM OFF as I don't see a benefit in my environment for using OCM. And just on the side: I was a bit worried that this note does not contain the new opatchauto syntax but instead lists an example (which is always good and nice and helpful) from the old days:

$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /u02/unconfig.rsp

-

Second learning experience? Analyze analyzes ... not everything!

In the readme under 2.2 you'll find also the requirement to run an anylze for conflict detection and resolution first. Maybe I'm too naive as I'm so happy with all the stuff ORAchk (credits for this gem of a tool go to Sandesh Rao's team - I will write something about it later) can deliver - and expected too much. In my case the anylyze signaled: All good, sky is bright and nice and clear - ready to fly!

#GRID_HOME/OPatch/opatchauto apply <UNZIPPED_PATCH_LOCATION>/19954978 -analyze -ocmrf <ocm response file>

You'll read below why my assumption was incorrect.
.

Third learning experience? An error is an error is an error!

Can you expect a patch to get applied correctly in the first run?

opatchauto apply /oradata/grid/12.1.0/19954978 -oh /oradata/grid/12.1.0 -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp 

I did. Well ... mine failed. Must be Murphy's Law See the log below - I have marked the failing part in red.

2015-02-16_19-45-09 :
Failed to run this command :
/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp
oracle.opatchauto.gi.RunExecutionSteps.runGenericShellCommands(RunExecutionSteps.java:913)
oracle.opatchauto.gi.RunExecutionSteps.processAllSteps(RunExecutionSteps.java:215)
oracle.opatchauto.gi.GIPatching.processPatchingSteps(GIPatching.java:544)
oracle.opatchauto.gi.OPatchautoExecution.main(OPatchautoExecution.java:141)
Command "/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp" execution failed:
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.

Log file Location for the failed command: /oradata/grid/12.1.0/cfgtoollogs/opatch/opatch2015-02-16_19-45-03PM_1.log

2015-02-16_19-45-09 :

--------------After fixing the cause of failure you have two options shown below:
Run 'opatchauto resume'
  or
Manually run the commands listed below
---------------------------------------------------------------------------------

/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp (Run as oracle) - (TRIED BUT FAILED)

.
Now the question is:
How much space does it really require to apply this PSU? 

At this point I was wondering as the analyze passed successfully - and I couldn't find anything in the readme about specific space requirements. My disk had roughly 8GB of free space - and as my GI Restart installation's footprint was around 6GB I don't had any bad thoughts. 
.

Forth learning experience? Always see the logfile first ...

Just rerunning the mentioned command gave me the correct information (just wondering why OPatch couldn't tell me this during the failed run?). So I did execute as oracle user:

/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp

Receiving finally this result:

Verifying environment and performing prerequisite checks...
Prerequisite check "CheckSystemSpace" failed.
The details are:
Required amount of space(12773.858MB) is not available.
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.
Log file location: /oradata/grid/12.1.0/cfgtoollogs/opatch/opatch2015-02-16_19-49-04PM_1.log

Ouch - 12.7GB of free space??? Really? Seriously?? Now I was scared. 

The patch for Linux x86-64 has a tip size of 873 MB - but unzipped it takes 3.3 GB. So why does it require 12.7 GB of free space? Actually I don't know the answer yet but I will follow up here as soon I know the details. Colleagues in Development ensure that especially on AIX you will need even more free space, such as in the area of 22GB!!!

One of the reasons for such a huge space requirement may be this:

Starting with 10.2, Opatch does not backup only the affected modules, it also takes a backup of the complete affected libraries to $ORACLE_HOME/.patch_storage/<patchid>/backup/<lib_directory_name>/<library_name>.

.

Fifth learning experience? There are workarounds ...

Blogs are sometimes VERY helpful. I found two helpful entries from external bloggers (but didn't bookmark them so I can't credit them here - sorry).

/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp OPatch.SKIP_VERIFY_SPACE=true

Of course you can't give this variable at the end to opatchauto as it wouldn't understand.
.

Final learning lesson? Clean up is a great idea!

This is not as simple as I did expect it. Of course, it's not. But please see this MOS Note:550522.1 - How To Avoid Disk Full Issues Because OPatch Backups Take Big Amount Of Disk Space
.

Famous Last Words?

Patches are great. In fact YOU NEED TO PATCH. Take the PSUs. There's no way out as it will help you to avoid plenty of known issues. But I hope that my above learning experience may help you to sail around one or the other pitfall :-)

--Mike

Friday Nov 07, 2014

Sleeping Beauties - Upgrade to 11.2.0.4 can be slow

A customer from the US did contact me past week via LinkedIn and raised a question:

"Is it expected that my patch set upgrade from Oracle 11.2.0.3 to Oracle 11.2.0.4 takes over 3 hours?"

Of course, no - this is not expected

This is the upgrade stats gathered post upgrade with utlu112s.sql:

SQL> @?/rdbms/admin/utlu112s.sql ; .
Oracle Database 11.2 Post-Upgrade Status Tool 10-31-2014 10:05:29
Component Current Version Elapsed Time
Name Status Number HH:MM:SS
Oracle Server  
. VALID 11.2.0.4.0 02:46:19
JServer JAVA Virtual Machine
. VALID 11.2.0.4.0 00:08:34
[..]
Final Actions
. 00:00:00
Total Upgrade Time: 03:06:47

No, this is not really expected. So we tried to nail down the root cause finding out these statements in the upgrade script c1102000.sql are causing the trouble:

194 -- wri$_optstat_histhead_history2.
195 execute immediate
196 q'#create unique index i_wri$_optstat_hh_obj_icol_st on
197 wri$_optstat_histhead_history (obj#, intcol#, savtime, colname)
198 tablespace sysaux #';
199
200 execute immediate
201 q'#create index i_wri$_optstat_hh_st on
202 wri$_optstat_histhead_history (savtime)
203 tablespace sysaux #';
204 end;
205 /

It's index rebuilds on histogram tables. And the customer has a large amount of stats data in his database as the default stats retention is 31 days.

Obviously the index rebuild is not done very efficiently (not done in parallel, no nologging clause). Those things can happen and sometimes this may not cause any issues. But in this case it lead to over 2 hours for just those index rebuilds.  

Luckily my colleague Cindy is an excellent resource for such things - after asking our team I've got the reply that this is tagged with a bug number and code fix already got checked in (under review right now):

bug19855835:
Upgrade from 11.2.0.2 to 11.2.0.4 is slow 

-Mike

PS: Credits go to Tan for bringing this to my attention - and sorry for the inconvenience! 

Wednesday Mar 05, 2014

PSU1 and PSU2: Datapatch Issues coverd in MOS Note

You may have read a posting disrecommending PSU1 and PSU2 for Oracle Multitenant especially in RAC/GI environments earlier this week. Actually following a lot of internal discsussions I will post some advice and clarification later this week.

Now I have an useful update:
Datapatch Issues are covered within a separate MOS Note making it easier to keep track and find workarounds for known issues.
Please see MOS Note:1609718.1 Datapatch Known Issues

-Mike

Tuesday Jan 28, 2014

MS Windows Bundle Patches - a VERY different story

I like Upgrade Workshops for a very different reason:
Hopefully people feed back with questions and information I wasn't aware off before. So for myself these workshops are also a learning experience. And thanks to a customer in Singapore (Thanks a lot, Hanh!!) I learned something about Oracle's Bundle Patches for MS Windows.

Customer asked about a specific issue when using RMAN Duplicate command in 12c failing with a nice ORA-600 [KSRPCSEXEC_1]. This got logged a while ago as bug16883554 - and it will be fixed in database patch set 12.1.0.2. But this customer asked for an earlier delivery.

So after a quick check of the SR and the bug it was clear that this patch will be also included for the MS Windows platform into the Bundle Patch 12.1.0.1.4 on Win.
Well, my first thought: Ouch! That will be the July 2014 PSU.

But in fact it isn't meant here. So I learned that on Windows patching runs on a different schedule and notation. 

MOS Note:161549.1 will tell you all the details about:
Oracle Database, Networking and Grid Agent Patches for Microsoft Platforms

64-Bit x64 Patches :

Patch

Patch Location Bug Fix List Notes
12.1.0.1.4 Target date End January 2014
12.1.0.1.3 Bug:17977915 Patch:17977915 Requires 12.1.0.1.0

-Mike 

Wednesday Oct 16, 2013

October 2013 PSUs and CPUs - News for 12c

Last night CET the most recent Patch Set Updates (PSU) and Critical Patch Updates (CPU aka SPU) got published on MOS. And there's a significant and remarkable change for Oracle Database 12c onwards. MOS Note: 1571391.1 - Patch Set Update and Critical Patch Update October 2013 Availability Document says:

2.1 Database Security Patching from 12.1.0.1 Onwards

Starting with Oracle Database version 12.1.0.1, Oracle only provides Patch Set Updates (PSU) to meet the Critical Patch Update (CPU) program requirements for security patching. Security Patch Updates (SPU) will no longer be available. Oracle has moved to this simplified model due to popular demand. The PSU is Oracle's preferred proactive patching mechanism since 2009.

For more information, see My Oracle Support Note 1581950.1, Database Security Patching from 12.1.0.1 onwards.

That's a real change. It's not harmful at all as our recommendation for years now is to apply PSUs as they contain not only the security patches but also the important fixes for critical issues. And apply them asap - the day we publish the Security Patch Advisory with some details is the day when external security experts go public as well with their findings.

MOS Note: 756671.1 - Oracle Recommended Patches -- Oracle Database will guide you to the databases patches for your platform. If you miss the PSU for Oracle Database 10.2.0.5 (the Pre-Release Announcement stated that there will be a PSU for 10.2.0.5) my personal understanding is: as Oracle 10.2.0.5 went out of Extended Support in July 2013 there won't be any October PSU released anymore.

And I'll apply the new 12.1.0.1 Oct13 PSU now to my Oracle Database 12c Multitenant environment as well :-)

Don't forget:
MOS Note: 224346.1 - OPatch: Where Can I Find the Latest Version of OPatch?
Find it via Patch Placeholder 6880880

-Mike

Wednesday Feb 15, 2012

Why is every patchset now a full release?

This question got posted by Naveen in the previous entry today - and I found it worth it to create a new blog entry as this question gets raised in nearly every 2nd workshop as well - so others might be interested as well.

Mike,

A question has been bothering me for a while and I thought I'll throw it out there. I didn't raise a support ticket, I knew what the response is going to be.

So here is the question. I have an 11g ORACLE HOME that was built as 11.2.0.1. Now if I want to upgrade to 11.2.0.2 or 11.2.0.3, I guess there is no easy way to upgrade the existing ORACLE HOME. Why didn't Oracle give the option to just apply a patch (opatch) and upgrade to 11.2.0.2 (or .3). For a customer to create a new ORACLE HOME is just a lot of work and breaks a lot of things. We have to copy over so many configs from the old home to the new home.

I'm sure there is a good reason, I'm just trying to understand what they are.

Thanks
Naveen

Naveen, thanks for your question. And the answer has many aspects.

Install into your existing ORACLE Home

First of all you can still install the new patch set (which is now a full release since Oracle 11.2.0.2) into your existing $ORACLE_HOME. But you'll have to detach your current home from the OUI's inventory first. Please see slide 41 of the Upgrade and Migration workshop deck. Please backup the contents of $ORACLE_HOME/dbs and $ORACLE_HOME/network/admin first as you'll have to copy them back later.

$ ./runInstaller -detachHome ORACLE_HOME=/u01/orahomes/11.2.0

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 10047 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/orabase

'DetachHome' was successful.

Once you've done that you'll install into your existing (old) directory.  And finally you'll copy back your dbs and network/admin files.

Why customers might use this procedure?

There are basically two reason why a customer chooses this strategy:
(a) Home naming conventions not allowing an Oracle 11.2.0.2 home and another Oracle 11.2.0.3 home
(b) Space issues
Both are valid reasons and therefore you can stay with the old strategy.

Why did we change from patchsets to full release patchsets?

Simple reason: Customers did ask us for a looooong time why we are delivering 3GB large patchsets instead of full releases. So we'd follow this wish. And second it will decrease the overall downtime when you'd install into your new Oracle Home. If you apply patchset software to your existing Oracle Home every Oracle process serviced by that home must be shut down as well - obviously including the database(s). And furthermore if something fails you'll have to restore your OUI's inventory plus all home contents - or reinstall your previous Oracle software.

Besides that we recommend to patch this home first with important one-offs (see links in MOS Note 161818.1) and the latest PSU or CPU. If you'd do that to your already emptied old home it will simply increase again your overall downtime.

And this is the reason why we recommend:
Always install into a new Oracle home beginning with Oracle Database 11.2 - don't erase your old home to reuse it unless there's no significant need for that procedure.

Tuesday Apr 26, 2011

Patches, Patch Sets, Patch Bundles, Bundled Patches ...

[Read More]

Thursday Jan 20, 2011

11.2.0.2 Bundled Patch 3 for Linux x86-64bit released


NOTE ADDED ON 25-MAY-2011


Please be aware that this entry has been published a long while ago. As of today (end of May 2011) Exadata BP10 (and not BP3) is the most recent Bundle Patch. And me and some colleagues have learned in the past months that applying Exadata BPs to a non-Exadata environment might cause some trouble once you request an one-off/interim patch on top of an Exadata BP.


Officially this Bundle Patch for Oracle Database 11.2.0.2 is titled "Exadata Database recommended patch" and got released yesterday. But I would recommend this one to all customers using 11.2.0.2 Grid Infrastructure, RAC and ASM.

pflaster.jpg

Reason?
Please see the readme.

 

 


  • Total of 42 bug fixes are included in this bundle

  • 40 Database fixes( 4 are diskmon, 12 are ASM)

  • 2 Database fixes from PSU/CPU Jan2011( 17 of the content has already been included in the previous release,BP2)

  • The patch contains Database, Diskmon and CRS patches packaged as a single patch.

  • This bundle is both DataGuard Rolling and RAC Rolling Installable.

  • This patch will be installed using "Opatch Auto" feature of Opatch.

 

Information on the bugs included, install requirements and known issues are documented in the README. Tracking bug is 10387939.
To download this Bundle Patch login to support.oracle.com and proceed to Patches and Updates. Fill in the bug number 10387939 and search to access the download screen.

Note 1:
Make sure you'll use the latest version of opatch (Patch 6880880) if you'd like to use the new auto functionality. Otherwise you'll see this error:
Error :
Undefined subroutine &main::unlockCRSHomeforpatch called at /u01/app/11.2.0/grid/OPatch/crs/patch112.pl line 1529


Note 2:
Per discussion with Oracle Support I would like to emphasize that applying this BP3 to RAC systems is my personal recommendation. Oracle Support will still recommend applying the PSU+GI Bundle Patch instead of the Exadata Bundle Patch. But nevertheless applying the Exadata Bundle Patch is supported as well.

Thursday Dec 16, 2010

Patch Set 11.2.0.2 for Win32 and Win64 now available

[Read More]

Monday Jun 14, 2010

Grid Infrastructure PSU 11.2.0.1.1 has been exchanged

[Read More]

Friday Aug 28, 2009

Have you checked our OS patch recommendations?

[Read More]

Monday Aug 24, 2009

Are you using RAT?

[Read More]
About

Mike Dietrich - Oracle Mike Dietrich
Master Product Manager - Database Upgrade & Migrations - Oracle

Based in Germany. Interlink between customers/partners and the Upgrade Development. Running workshops between Arctic and Antartica. Assisting customers in their reference projects onsite and remotely. Connect via:

- -

Search

Archives
« May 2016
SunMonTueWedThuFriSat
1
2
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    
       
Today
Slides Download Center
Visitors since 17-OCT-2011
White Paper and Docs
Workshops
Viewlets and Videos
Workshop Map
x Oracle related Tech Blogs
This week on my Rega & Pono
Upgrade Reference Papers