Monday Apr 20, 2015

Oracle PSU and BP April 2015 is available

As of April 14, 2015:

The April 2015 PSU/BP is available!

Here's the most important information:

Please find below the links to the Recommended Patches and Patch Numbers for each of your database releases:

12.1.0.2
 Availability and Known issues for 12.1.0.2      Note:1683799.1
12.1.0.1
 Availability and Known issues for 12.1.0.1      Note:1565082.1 
11.2.0.4
 Availability and Known issues for 11.2.0.4      Note:1562139.1
 
11.2.0.3
 Availability and Known issues for 11.2.0.3      Note:1348336.1

11.1.0.7
 List of fixes included in 11.1.0.7              Note:601739.1

-Mike

Wednesday Feb 18, 2015

Grid Infrastructure PSU Jan 2015 - Am I too intolerant?

Germans are not only known for being precise and timely - but sometimes also for being too direct. Well, Roy could tell you stories ... and I always honor his politeness :-)

Enough about stereotypes. I work with a customer at the moment on their 12c upgrades. And I did recommend the most recent PSU (Patch Set Updates) for their Grid Infrastructure environments running Oracle Restart. Same of course for the database homes but this blog post will just name some findings I've had the other night when trying to apply the January 2015 GI PSU to my Oracle Restart environment.

First of all, start here:

  • Drill down from MOS Note:161818.1 (Click on the Release Link for 12.1.0!) into
  • Oracle 12.1 Support Status and Alerts MOS Note:1565065.1 and further into 
  • 12.1.0.2 Patch Set - Availability and Known Issues MOS Note:1683799.1 to get
    • Recommendations for patches on top of Oracle Database 12.1.0.2 - and of course for Grid Infrastructure (GI), Engineered Systems and Database In-Memory as well.

For GI you'll get the following recommendations:

Grid Infrastructure

Document Description Rolling RAC Patch Download
Note:20132450.8 Combo of 12.1.0.2.2 OJVM PSU and 12.1.0.2.2 GI PSU (Jan 2015) Part Patch:20132450
Note:19954978.8 12.1.0.2.2 (Jan 2015) Grid Infrastructure Patch Set Update (GI PSU) Yes Patch:19954978

Well, first question:
Do you need the Combo Patch or the non-Combo? 

My personal recommendation: Take the non-Combo Patch as the combo patch includes not only the GI PSU, but also the Database PSU, the OJVM Patch and some other things you will not need for patching your Grid Infrastructure only. Of course I can see the benefit of downloading just one big piece and applying everything all together to my environment. But first of all parts of the patch (speaking of OJVM) are not rolling applicable - and second I'd like to control and script things separately. But please feel free to see and do this in a different way.

So I did download Patch:19954978 to my environment. Unzipped it. 
.

You all know it already - you will need a new OPatch!

Of course my OPatch version is too old. You will need at least OPatch 12.1.0.1.5 - the most recent one is OPatch 12.1.0.1.6 - and you simply download it via MOS patch 6880880 and install it after removing the old directory into your $ORACLE_HOME. 
.

First learning experience? OPatch doesn't do anything without a response file.

The reason for this (at least for me as I don't patch daily) new requirement seems to be the new opatchauto call which scripts the entire apply process in a silent way. Well, at least the patch readme tells me what to do. Please read MOS Note:966023.1 to learn about how to create this response file.

If you do not have the OCM response file (ocm.rsp), see the following My Oracle Support Document 966023.1 How To Create An OCM Response File For Opatch Silent Installation.

I did create it with OCM OFF as I don't see a benefit in my environment for using OCM. And just on the side: I was a bit worried that this note does not contain the new opatchauto syntax but instead lists an example (which is always good and nice and helpful) from the old days:

$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /u02/unconfig.rsp

-

Second learning experience? Analyze analyzes ... not everything!

In the readme under 2.2 you'll find also the requirement to run an anylze for conflict detection and resolution first. Maybe I'm too naive as I'm so happy with all the stuff ORAchk (credits for this gem of a tool go to Sandesh Rao's team - I will write something about it later) can deliver - and expected too much. In my case the anylyze signaled: All good, sky is bright and nice and clear - ready to fly!

#GRID_HOME/OPatch/opatchauto apply <UNZIPPED_PATCH_LOCATION>/19954978 -analyze -ocmrf <ocm response file>

You'll read below why my assumption was incorrect.
.

Third learning experience? An error is an error is an error!

Can you expect a patch to get applied correctly in the first run?

opatchauto apply /oradata/grid/12.1.0/19954978 -oh /oradata/grid/12.1.0 -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp 

I did. Well ... mine failed. Must be Murphy's Law See the log below - I have marked the failing part in red.

2015-02-16_19-45-09 :
Failed to run this command :
/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp
oracle.opatchauto.gi.RunExecutionSteps.runGenericShellCommands(RunExecutionSteps.java:913)
oracle.opatchauto.gi.RunExecutionSteps.processAllSteps(RunExecutionSteps.java:215)
oracle.opatchauto.gi.GIPatching.processPatchingSteps(GIPatching.java:544)
oracle.opatchauto.gi.OPatchautoExecution.main(OPatchautoExecution.java:141)
Command "/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp" execution failed:
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.

Log file Location for the failed command: /oradata/grid/12.1.0/cfgtoollogs/opatch/opatch2015-02-16_19-45-03PM_1.log

2015-02-16_19-45-09 :

--------------After fixing the cause of failure you have two options shown below:
Run 'opatchauto resume'
  or
Manually run the commands listed below
---------------------------------------------------------------------------------

/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp (Run as oracle) - (TRIED BUT FAILED)

.
Now the question is:
How much space does it really require to apply this PSU? 

At this point I was wondering as the analyze passed successfully - and I couldn't find anything in the readme about specific space requirements. My disk had roughly 8GB of free space - and as my GI Restart installation's footprint was around 6GB I don't had any bad thoughts. 
.

Forth learning experience? Always see the logfile first ...

Just rerunning the mentioned command gave me the correct information (just wondering why OPatch couldn't tell me this during the failed run?). So I did execute as oracle user:

/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp

Receiving finally this result:

Verifying environment and performing prerequisite checks...
Prerequisite check "CheckSystemSpace" failed.
The details are:
Required amount of space(12773.858MB) is not available.
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.
Log file location: /oradata/grid/12.1.0/cfgtoollogs/opatch/opatch2015-02-16_19-49-04PM_1.log

Ouch - 12.7GB of free space??? Really? Seriously?? Now I was scared. 

The patch for Linux x86-64 has a tip size of 873 MB - but unzipped it takes 3.3 GB. So why does it require 12.7 GB of free space? Actually I don't know the answer yet but I will follow up here as soon I know the details. Colleagues in Development ensure that especially on AIX you will need even more free space, such as in the area of 22GB!!!

One of the reasons for such a huge space requirement may be this:

Starting with 10.2, Opatch does not backup only the affected modules, it also takes a backup of the complete affected libraries to $ORACLE_HOME/.patch_storage/<patchid>/backup/<lib_directory_name>/<library_name>.

.

Fifth learning experience? There are workarounds ...

Blogs are sometimes VERY helpful. I found two helpful entries from external bloggers (but didn't bookmark them so I can't credit them here - sorry).

/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp OPatch.SKIP_VERIFY_SPACE=true

Of course you can't give this variable at the end to opatchauto as it wouldn't understand.
.

Final learning lesson? Clean up is a great idea!

This is not as simple as I did expect it. Of course, it's not. But please see this MOS Note:550522.1 - How To Avoid Disk Full Issues Because OPatch Backups Take Big Amount Of Disk Space
.

Famous Last Words?

Patches are great. In fact YOU NEED TO PATCH. Take the PSUs. There's no way out as it will help you to avoid plenty of known issues. But I hope that my above learning experience may help you to sail around one or the other pitfall :-)

--Mike

Friday Nov 07, 2014

Sleeping Beauties - Upgrade to 11.2.0.4 can be slow

A customer from the US did contact me past week via LinkedIn and raised a question:

"Is it expected that my patch set upgrade from Oracle 11.2.0.3 to Oracle 11.2.0.4 takes over 3 hours?"

Of course, no - this is not expected

This is the upgrade stats gathered post upgrade with utlu112s.sql:

SQL> @?/rdbms/admin/utlu112s.sql ; .
Oracle Database 11.2 Post-Upgrade Status Tool 10-31-2014 10:05:29
Component Current Version Elapsed Time
Name Status Number HH:MM:SS
Oracle Server  
. VALID 11.2.0.4.0 02:46:19
JServer JAVA Virtual Machine
. VALID 11.2.0.4.0 00:08:34
[..]
Final Actions
. 00:00:00
Total Upgrade Time: 03:06:47

No, this is not really expected. So we tried to nail down the root cause finding out these statements in the upgrade script c1102000.sql are causing the trouble:

194 -- wri$_optstat_histhead_history2.
195 execute immediate
196 q'#create unique index i_wri$_optstat_hh_obj_icol_st on
197 wri$_optstat_histhead_history (obj#, intcol#, savtime, colname)
198 tablespace sysaux #';
199
200 execute immediate
201 q'#create index i_wri$_optstat_hh_st on
202 wri$_optstat_histhead_history (savtime)
203 tablespace sysaux #';
204 end;
205 /

It's index rebuilds on histogram tables. And the customer has a large amount of stats data in his database as the default stats retention is 31 days.

Obviously the index rebuild is not done very efficiently (not done in parallel, no nologging clause). Those things can happen and sometimes this may not cause any issues. But in this case it lead to over 2 hours for just those index rebuilds.  

Luckily my colleague Cindy is an excellent resource for such things - after asking our team I've got the reply that this is tagged with a bug number and code fix already got checked in (under review right now):

bug19855835:
Upgrade from 11.2.0.2 to 11.2.0.4 is slow 

-Mike

PS: Credits go to Tan for bringing this to my attention - and sorry for the inconvenience! 

Wednesday Mar 05, 2014

PSU1 and PSU2: Datapatch Issues coverd in MOS Note

You may have read a posting disrecommending PSU1 and PSU2 for Oracle Multitenant especially in RAC/GI environments earlier this week. Actually following a lot of internal discsussions I will post some advice and clarification later this week.

Now I have an useful update:
Datapatch Issues are covered within a separate MOS Note making it easier to keep track and find workarounds for known issues.
Please see MOS Note:1609718.1 Datapatch Known Issues

-Mike

Tuesday Jan 28, 2014

MS Windows Bundle Patches - a VERY different story

I like Upgrade Workshops for a very different reason:
Hopefully people feed back with questions and information I wasn't aware off before. So for myself these workshops are also a learning experience. And thanks to a customer in Singapore (Thanks a lot, Hanh!!) I learned something about Oracle's Bundle Patches for MS Windows.

Customer asked about a specific issue when using RMAN Duplicate command in 12c failing with a nice ORA-600 [KSRPCSEXEC_1]. This got logged a while ago as bug16883554 - and it will be fixed in database patch set 12.1.0.2. But this customer asked for an earlier delivery.

So after a quick check of the SR and the bug it was clear that this patch will be also included for the MS Windows platform into the Bundle Patch 12.1.0.1.4 on Win.
Well, my first thought: Ouch! That will be the July 2014 PSU.

But in fact it isn't meant here. So I learned that on Windows patching runs on a different schedule and notation. 

MOS Note:161549.1 will tell you all the details about:
Oracle Database, Networking and Grid Agent Patches for Microsoft Platforms

64-Bit x64 Patches :

Patch

Patch Location Bug Fix List Notes
12.1.0.1.4 Target date End January 2014
12.1.0.1.3 Bug:17977915 Patch:17977915 Requires 12.1.0.1.0

-Mike 

Wednesday Oct 16, 2013

October 2013 PSUs and CPUs - News for 12c

Last night CET the most recent Patch Set Updates (PSU) and Critical Patch Updates (CPU aka SPU) got published on MOS. And there's a significant and remarkable change for Oracle Database 12c onwards. MOS Note: 1571391.1 - Patch Set Update and Critical Patch Update October 2013 Availability Document says:

2.1 Database Security Patching from 12.1.0.1 Onwards

Starting with Oracle Database version 12.1.0.1, Oracle only provides Patch Set Updates (PSU) to meet the Critical Patch Update (CPU) program requirements for security patching. Security Patch Updates (SPU) will no longer be available. Oracle has moved to this simplified model due to popular demand. The PSU is Oracle's preferred proactive patching mechanism since 2009.

For more information, see My Oracle Support Note 1581950.1, Database Security Patching from 12.1.0.1 onwards.

That's a real change. It's not harmful at all as our recommendation for years now is to apply PSUs as they contain not only the security patches but also the important fixes for critical issues. And apply them asap - the day we publish the Security Patch Advisory with some details is the day when external security experts go public as well with their findings.

MOS Note: 756671.1 - Oracle Recommended Patches -- Oracle Database will guide you to the databases patches for your platform. If you miss the PSU for Oracle Database 10.2.0.5 (the Pre-Release Announcement stated that there will be a PSU for 10.2.0.5) my personal understanding is: as Oracle 10.2.0.5 went out of Extended Support in July 2013 there won't be any October PSU released anymore.

And I'll apply the new 12.1.0.1 Oct13 PSU now to my Oracle Database 12c Multitenant environment as well :-)

Don't forget:
MOS Note: 224346.1 - OPatch: Where Can I Find the Latest Version of OPatch?
Find it via Patch Placeholder 6880880

-Mike

Wednesday Feb 15, 2012

Why is every patchset now a full release?

This question got posted by Naveen in the previous entry today - and I found it worth it to create a new blog entry as this question gets raised in nearly every 2nd workshop as well - so others might be interested as well.

Mike,

A question has been bothering me for a while and I thought I'll throw it out there. I didn't raise a support ticket, I knew what the response is going to be.

So here is the question. I have an 11g ORACLE HOME that was built as 11.2.0.1. Now if I want to upgrade to 11.2.0.2 or 11.2.0.3, I guess there is no easy way to upgrade the existing ORACLE HOME. Why didn't Oracle give the option to just apply a patch (opatch) and upgrade to 11.2.0.2 (or .3). For a customer to create a new ORACLE HOME is just a lot of work and breaks a lot of things. We have to copy over so many configs from the old home to the new home.

I'm sure there is a good reason, I'm just trying to understand what they are.

Thanks
Naveen

Naveen, thanks for your question. And the answer has many aspects.

Install into your existing ORACLE Home

First of all you can still install the new patch set (which is now a full release since Oracle 11.2.0.2) into your existing $ORACLE_HOME. But you'll have to detach your current home from the OUI's inventory first. Please see slide 41 of the Upgrade and Migration workshop deck. Please backup the contents of $ORACLE_HOME/dbs and $ORACLE_HOME/network/admin first as you'll have to copy them back later.

$ ./runInstaller -detachHome ORACLE_HOME=/u01/orahomes/11.2.0

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 10047 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/orabase

'DetachHome' was successful.

Once you've done that you'll install into your existing (old) directory.  And finally you'll copy back your dbs and network/admin files.

Why customers might use this procedure?

There are basically two reason why a customer chooses this strategy:
(a) Home naming conventions not allowing an Oracle 11.2.0.2 home and another Oracle 11.2.0.3 home
(b) Space issues
Both are valid reasons and therefore you can stay with the old strategy.

Why did we change from patchsets to full release patchsets?

Simple reason: Customers did ask us for a looooong time why we are delivering 3GB large patchsets instead of full releases. So we'd follow this wish. And second it will decrease the overall downtime when you'd install into your new Oracle Home. If you apply patchset software to your existing Oracle Home every Oracle process serviced by that home must be shut down as well - obviously including the database(s). And furthermore if something fails you'll have to restore your OUI's inventory plus all home contents - or reinstall your previous Oracle software.

Besides that we recommend to patch this home first with important one-offs (see links in MOS Note 161818.1) and the latest PSU or CPU. If you'd do that to your already emptied old home it will simply increase again your overall downtime.

And this is the reason why we recommend:
Always install into a new Oracle home beginning with Oracle Database 11.2 - don't erase your old home to reuse it unless there's no significant need for that procedure.

Tuesday Apr 26, 2011

Patches, Patch Sets, Patch Bundles, Bundled Patches ...

[Read More]

Thursday Jan 20, 2011

11.2.0.2 Bundled Patch 3 for Linux x86-64bit released


NOTE ADDED ON 25-MAY-2011


Please be aware that this entry has been published a long while ago. As of today (end of May 2011) Exadata BP10 (and not BP3) is the most recent Bundle Patch. And me and some colleagues have learned in the past months that applying Exadata BPs to a non-Exadata environment might cause some trouble once you request an one-off/interim patch on top of an Exadata BP.


Officially this Bundle Patch for Oracle Database 11.2.0.2 is titled "Exadata Database recommended patch" and got released yesterday. But I would recommend this one to all customers using 11.2.0.2 Grid Infrastructure, RAC and ASM.

pflaster.jpg

Reason?
Please see the readme.

 

 


  • Total of 42 bug fixes are included in this bundle

  • 40 Database fixes( 4 are diskmon, 12 are ASM)

  • 2 Database fixes from PSU/CPU Jan2011( 17 of the content has already been included in the previous release,BP2)

  • The patch contains Database, Diskmon and CRS patches packaged as a single patch.

  • This bundle is both DataGuard Rolling and RAC Rolling Installable.

  • This patch will be installed using "Opatch Auto" feature of Opatch.

 

Information on the bugs included, install requirements and known issues are documented in the README. Tracking bug is 10387939.
To download this Bundle Patch login to support.oracle.com and proceed to Patches and Updates. Fill in the bug number 10387939 and search to access the download screen.

Note 1:
Make sure you'll use the latest version of opatch (Patch 6880880) if you'd like to use the new auto functionality. Otherwise you'll see this error:
Error :
Undefined subroutine &main::unlockCRSHomeforpatch called at /u01/app/11.2.0/grid/OPatch/crs/patch112.pl line 1529


Note 2:
Per discussion with Oracle Support I would like to emphasize that applying this BP3 to RAC systems is my personal recommendation. Oracle Support will still recommend applying the PSU+GI Bundle Patch instead of the Exadata Bundle Patch. But nevertheless applying the Exadata Bundle Patch is supported as well.

Thursday Dec 16, 2010

Patch Set 11.2.0.2 for Win32 and Win64 now available

[Read More]

Monday Jun 14, 2010

Grid Infrastructure PSU 11.2.0.1.1 has been exchanged

[Read More]

Friday Aug 28, 2009

Have you checked our OS patch recommendations?

[Read More]

Monday Aug 24, 2009

Are you using RAT?

[Read More]
About

Mike Dietrich - Oracle Mike Dietrich
Master Product Manager - Database Upgrade & Migrations - Oracle Corp

Based near Munich/Germany and spending plenty of time in airplanes to run either upgrade workshops or work onsite or remotely with reference customers. Acting as interlink between customers/partners and the Upgrade Development.

Follow me on TWITTER

Contact me via LinkedIn or XING

Search

Archives
« August 2015
SunMonTueWedThuFriSat
      
2
3
6
7
8
9
10
11
12
13
15
16
22
23
25
26
27
28
29
30
31
     
Today
Oracle related Tech Blogs
Slides Download Center
Visitors since 17-OCT-2011
White Paper and Docs
Workshops
Viewlets and Videos
Workshop Map
This week on my Rega & Pono
Upgrade Reference Papers