Friday Jul 03, 2015

ORAchk 12.1.0.4 released - a MUST USE Tool

ORAchk I have blogged a while back in March 2015 about ORAchk 12.1.0.3  as it had added the support for Oracle Restart amongst other great features.

ORAchk (short for: ORA check) is such a great tool to check your environment before any upgrade, but also on a regular basis for almost everything. I'd call it a MUST USE Tool not only for Oracle Grid Infrastructure or Oracle Restart upgrades or patches or changes. But also for EM Cloud Control 12c, EBS, Siebel, and of course your hardware.

The newest version of ORAchk 12.1.0.4 has been released.

The new ORAchk release 12.1.0.2.4 is now available to download.

General ORAchk features

  • Proactively scans for the most impactful problems across the various layers of your stack
  • Simplifies and streamlines how to investigate and analyze which known issues present a risk to you
  • Lightweight tool that runs within your environment without requiring config data to be sent to Oracle
  • High level reports show your system health risks with the ability to drill down into specific problems and understand their resolutions
  • Can be configured to send email notifications when it detects problems
  • Collection Manager, a companion Application Express web app, provides a single dashboard view of collections across your entire enterprise 
  • It does not cost anything!!!

 

Details of new features in ORAchk 12.1.0.2.4

Auto update ORAchk when newer version is available

New in this release, if ORAchk is older than 120 days and a newer version is not available locally it will check to see if a newer version is available on My Oracle Support and automatically download and upgrade.

Download of latest version directly from My Oracle Support can also be specifically triggered with “./orachk –download”.

If ORAchk is running in automated mode the daemon will automatically upgrade from local location defined by RAT_UPGRADE_LOC just before the next scheduled run. Email notification will be sent about the upgrade then ORAchk will continue with the scheduled run using the upgraded version, all without requiring you to restart the ORAchk daemon.

Expanded Oracle Product Support

ORAchk 12.1.0.2.4 now brings wider and deeper support throughout the Oracle product stack, with newly added support for the following product areas:

  • Enterprise Manager OMS
  • E-Business Suite Oracle Fixed Assets
  • E-Business Suite Oracle Human Resources
  • E-Business Suite Oracle Receivables
  • Siebel CRM Application

See Document 1268927.2 for further details of the new product support.

Over 60 New Health Checks

This release of ORAchk adds new checks for some of the most impactful problems seen to Oracle Customer Support specifically in the areas of:

  • Systems hardware settings to optimize encryption performance for the Database and E-Business Suite.
  • Solaris & Siebel CRM Object Manager to ensure page sizes are set appropriately for Siebel CRM to handle large numbers of users.
  • Database optimization of memory and resource related configurations and Application Continuity checks.
  • Enterprise Manager OMS High impact problems that cause functional failure or difficulty with patching or upgrade.
  • E-Business Suite Receivables detection of non-validated Receivables Accounting Definitions, which might prevent the Create Accounting process from functioning.
  • E-Business Suite Fixed Assets checks for any books with an errored or incomplete depreciation run, to allow for resolution prior to month end close.
  • E-Business Suite Human Resources verification of Setup Business Group configuration.
  • Siebel Applications verification of the database configuration for stability, best practices and performance optimization.

--Mike

Friday Feb 20, 2015

Downgrade Oracle Restart 12c - Grid Infrastructure only?

Oracle RestartCan you downgrade your Oracle Restart installation from Oracle 12c back to Oracle 11g?

Actually there's no real direct downgrade supported for Oracle Restart. But of course there's a way to do it.

Basically it is: 

  • Deconfigure Oracle Restart in 12c
  • Configure Oracle Restart in 11g 

But wait a minute. It is very important to know if you have upgraded your database already. If that is the case then first you MUST downgrade your database(s) as you can't manage a higher version Oracle Database with a lower version Clusterware.

So first of all, please downgrade your Oracle database(s) first: 

At the next stage you'll need to "downgrade" the Oracle Clusterware resp Grid Infrastructure for Oracle Restart: 

Before you attempt this you'll need to deconfigure the Restart resources - and please be aware that here's a small difference in commands between Oracle 12.1.0.2 and Oracle 12.1.0.1.

This is from the documentation for Oracle 12.1.0.2

  • Deconfigure Oracle Restart:
    • Log in as root
    • cd /u01/app/12.1.0.2/grid/crs/install
    • roothas.sh -deconfig -force
  • Once this is complete you can now deinstall the Grid Infrastructure with the deinstall tool
  • Then you will need either to reinstall the previous - for instance Oracle 11.2.0.4 - Grid Infrastructure or - if it's still present on the machine - reconfigure it by running root.sh from the previous Clusterware's home
  • And finally reconfigure the database(s) again with Clusterware
    • $ srvctl downgrade database -d db-unique-name -o oraclehome -t to_version

.
In case you'd plan to do this exercise back from Oracle 12.1.0.1 instead then you'll have different steps to follow to deconfigure Oracle Restart 12.1.0.1:

  • Deconfigure Oracle Restart 12.1.0.1
    • # cd /u01/app/12.1.0.1/grid/crs/install
    • # /u01/app/12.1.0.1/grid/perl/bin/perl /u01/app/12.1.0.1/grid/crs/install/roothas.pl -deconfig
      .

Regarding ASM backwards compatibility please be aware that changing the COMPATIBLE attributes within ASM will get you in trouble - see the Oracle Database Upgrade Guide for further information:
http://docs.oracle.com/database/121/UPGRD/afterup.htm#CEGJDCDD

--Mike

PS: See bug18160024 or the GI install guide, section A11.4 for the Standalone GI downgrade instructions from 12.1.0.2.0. 

Wednesday Feb 18, 2015

Grid Infrastructure PSU Jan 2015 - Am I too intolerant?

Germans are not only known for being precise and timely - but sometimes also for being too direct. Well, Roy could tell you stories ... and I always honor his politeness :-)

Enough about stereotypes. I work with a customer at the moment on their 12c upgrades. And I did recommend the most recent PSU (Patch Set Updates) for their Grid Infrastructure environments running Oracle Restart. Same of course for the database homes but this blog post will just name some findings I've had the other night when trying to apply the January 2015 GI PSU to my Oracle Restart environment.

First of all, start here:

  • Drill down from MOS Note:161818.1 (Click on the Release Link for 12.1.0!) into
  • Oracle 12.1 Support Status and Alerts MOS Note:1565065.1 and further into 
  • 12.1.0.2 Patch Set - Availability and Known Issues MOS Note:1683799.1 to get
    • Recommendations for patches on top of Oracle Database 12.1.0.2 - and of course for Grid Infrastructure (GI), Engineered Systems and Database In-Memory as well.

For GI you'll get the following recommendations:

Grid Infrastructure

Document Description Rolling RAC Patch Download
Note:20132450.8 Combo of 12.1.0.2.2 OJVM PSU and 12.1.0.2.2 GI PSU (Jan 2015) Part Patch:20132450
Note:19954978.8 12.1.0.2.2 (Jan 2015) Grid Infrastructure Patch Set Update (GI PSU) Yes Patch:19954978

Well, first question:
Do you need the Combo Patch or the non-Combo? 

My personal recommendation: Take the non-Combo Patch as the combo patch includes not only the GI PSU, but also the Database PSU, the OJVM Patch and some other things you will not need for patching your Grid Infrastructure only. Of course I can see the benefit of downloading just one big piece and applying everything all together to my environment. But first of all parts of the patch (speaking of OJVM) are not rolling applicable - and second I'd like to control and script things separately. But please feel free to see and do this in a different way.

So I did download Patch:19954978 to my environment. Unzipped it. 
.

You all know it already - you will need a new OPatch!

Of course my OPatch version is too old. You will need at least OPatch 12.1.0.1.5 - the most recent one is OPatch 12.1.0.1.6 - and you simply download it via MOS patch 6880880 and install it after removing the old directory into your $ORACLE_HOME. 
.

First learning experience? OPatch doesn't do anything without a response file.

The reason for this (at least for me as I don't patch daily) new requirement seems to be the new opatchauto call which scripts the entire apply process in a silent way. Well, at least the patch readme tells me what to do. Please read MOS Note:966023.1 to learn about how to create this response file.

If you do not have the OCM response file (ocm.rsp), see the following My Oracle Support Document 966023.1 How To Create An OCM Response File For Opatch Silent Installation.

I did create it with OCM OFF as I don't see a benefit in my environment for using OCM. And just on the side: I was a bit worried that this note does not contain the new opatchauto syntax but instead lists an example (which is always good and nice and helpful) from the old days:

$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /u02/unconfig.rsp

-

Second learning experience? Analyze analyzes ... not everything!

In the readme under 2.2 you'll find also the requirement to run an anylze for conflict detection and resolution first. Maybe I'm too naive as I'm so happy with all the stuff ORAchk (credits for this gem of a tool go to Sandesh Rao's team - I will write something about it later) can deliver - and expected too much. In my case the anylyze signaled: All good, sky is bright and nice and clear - ready to fly!

#GRID_HOME/OPatch/opatchauto apply <UNZIPPED_PATCH_LOCATION>/19954978 -analyze -ocmrf <ocm response file>

You'll read below why my assumption was incorrect.
.

Third learning experience? An error is an error is an error!

Can you expect a patch to get applied correctly in the first run?

opatchauto apply /oradata/grid/12.1.0/19954978 -oh /oradata/grid/12.1.0 -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp 

I did. Well ... mine failed. Must be Murphy's Law See the log below - I have marked the failing part in red.

2015-02-16_19-45-09 :
Failed to run this command :
/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp
oracle.opatchauto.gi.RunExecutionSteps.runGenericShellCommands(RunExecutionSteps.java:913)
oracle.opatchauto.gi.RunExecutionSteps.processAllSteps(RunExecutionSteps.java:215)
oracle.opatchauto.gi.GIPatching.processPatchingSteps(GIPatching.java:544)
oracle.opatchauto.gi.OPatchautoExecution.main(OPatchautoExecution.java:141)
Command "/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp" execution failed:
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.

Log file Location for the failed command: /oradata/grid/12.1.0/cfgtoollogs/opatch/opatch2015-02-16_19-45-03PM_1.log

2015-02-16_19-45-09 :

--------------After fixing the cause of failure you have two options shown below:
Run 'opatchauto resume'
  or
Manually run the commands listed below
---------------------------------------------------------------------------------

/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp (Run as oracle) - (TRIED BUT FAILED)

.
Now the question is:
How much space does it really require to apply this PSU? 

At this point I was wondering as the analyze passed successfully - and I couldn't find anything in the readme about specific space requirements. My disk had roughly 8GB of free space - and as my GI Restart installation's footprint was around 6GB I don't had any bad thoughts. 
.

Forth learning experience? Always see the logfile first ...

Just rerunning the mentioned command gave me the correct information (just wondering why OPatch couldn't tell me this during the failed run?). So I did execute as oracle user:

/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp

Receiving finally this result:

Verifying environment and performing prerequisite checks...
Prerequisite check "CheckSystemSpace" failed.
The details are:
Required amount of space(12773.858MB) is not available.
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.
Log file location: /oradata/grid/12.1.0/cfgtoollogs/opatch/opatch2015-02-16_19-49-04PM_1.log

Ouch - 12.7GB of free space??? Really? Seriously?? Now I was scared. 

The patch for Linux x86-64 has a tip size of 873 MB - but unzipped it takes 3.3 GB. So why does it require 12.7 GB of free space? Actually I don't know the answer yet but I will follow up here as soon I know the details. Colleagues in Development ensure that especially on AIX you will need even more free space, such as in the area of 22GB!!!

One of the reasons for such a huge space requirement may be this:

Starting with 10.2, Opatch does not backup only the affected modules, it also takes a backup of the complete affected libraries to $ORACLE_HOME/.patch_storage/<patchid>/backup/<lib_directory_name>/<library_name>.

.

Fifth learning experience? There are workarounds ...

Blogs are sometimes VERY helpful. I found two helpful entries from external bloggers (but didn't bookmark them so I can't credit them here - sorry).

/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp OPatch.SKIP_VERIFY_SPACE=true

Of course you can't give this variable at the end to opatchauto as it wouldn't understand.
.

Final learning lesson? Clean up is a great idea!

This is not as simple as I did expect it. Of course, it's not. But please see this MOS Note:550522.1 - How To Avoid Disk Full Issues Because OPatch Backups Take Big Amount Of Disk Space
.

Famous Last Words?

Patches are great. In fact YOU NEED TO PATCH. Take the PSUs. There's no way out as it will help you to avoid plenty of known issues. But I hope that my above learning experience may help you to sail around one or the other pitfall :-)

--Mike

Tuesday Dec 23, 2014

Upgrading ORACLE RESTART from 11.2 to 12.1

First of all, Oracle Restart is deprecated in Oracle Database 12c. Deprecation means the product is still supported, you can open SRs, you'll get bug fixes - but we won't do further development of the product or feature.

But a customer (Thanks to Jaco de Graaf for bringing this to my attention!) had a question via the blog about the best approach to upgrade his Oracle Restart configuration from Oracle 11.2.0.3 to Oracle 12.1.0.2 - and apply the October PSU as well.

What is Oracle Restart?
Please find an overview here in the doc. 
.

How to upgrade your Oracle Restart configuration?
The standard way is simple: Choose "Upgrade" in the starting page of the Grid Infrastructure's OUI screen.
Grid Infrastructure OUI Upgrade Oracle Restart
You won't see anything saying "Upgrade Stand Alone Installation" or "Upgrade Oracle Restart" - but the general "Upgrade ..." option will cover this topic as well.
.

Upgrade Oracle Restart and include a PSU (Patch Set Update)?
This is a bit more tricky and as far as I see not documented entirely. But Jaco did open an SR and Support could help.

  1. Install 12.1 GI ‘software only’
  2. Patch that software set to the latest PSU (using: opatch apply ….-local)
  3. Remove the Oracle Restart configuration (using the ‘old’ 11.2 GRID_HOME)
    Please find a description on the excellent blog of my support colleague, Helmut Hutzler:
    http://www.hhutzler.de/blog/reconfigure-12-1-0-2-has-stack-running-11-2-0-4-database/
    or in MOS Note:986740.1
    It is important to mention that you need to use the 'old' GI Home to remove the Restart configuration - the MOS Note doesn't say this explicitlely.
  4. Reconfigure the Oracle Restart configuration (using the ‘new’ 12.1 GRID_HOME)
  5. Add all required items back to the Oracle Restart stack (ASM, Listener, databases).

Steps 3, 4, and 5 will require a bit of downtime. 
.

-Mike 

---

Added MOS Note 1584742.1Support Impact of the Deprecation Announcement of Oracle Restart with Oracle Database 12c on Feb 22, 2015

Friday Dec 21, 2012

Creating ASM for test purposes in the file system

First of all, I'm back after pausing for a while - sorry for not updating the blog in the past weeks ... and you won't see many updates in the following weeks as it'll be holiday season (and we Germans have sooooo many public holidays) :-)

Anyway, back to tech topics. Today I want to test Oracle Restart upgrades. Oracle Restart internally is called SIHA (Single Instance High Availability) which explains the topic a bit more. Basically it means having your database reside in ASM and let Oracle Clusterware take care on it, even though you don't have a cluster. Not a bad idea as this can be very helpful in real world environments. But I did realize that the entire process is not documented in all details. So I'd thought I should give this a try.

The first challenge I do face: I have just one disk in my machine - so I'll have to tweak ASM a bit to make it work with files on the file system.

Creating two empty strawman files in file system with dd is not a big deal:
$ dd if=/dev/zero of=/oradata/ASM/dg_DATA bs=8192 count=1000000 oflag=direct
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB) copied, 336.371 seconds, 24.4 MB/s
[V112] oracle@localhost:/oradata
$ dd if=/dev/zero of=/oradata/ASM/dg_BCK bs=8192 count=500000 oflag=direct
500000+0 records in
500000+0 records out
4096000000 bytes (4.1 GB) copied, 246.021 seconds, 16.6 MB/s

But the next step is to start the cssd (Cluster Synchronization Services Demon) in my Oracle Database 10.2.0.5 installation from within the $ORACLE_HOME/bin directory:
[root@localhost bin]# . localconfig add
/etc/oracle does not exist. Creating it now.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configuration for local CSS has been initialized
Adding to inittab
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
        localhost
CSS is active on all nodes.
Oracle CSS service is installed and running under init(1M)

Otherwise no chance for ASM to start up.

Now my attempts to use simply DBCA (Database Configuration Assistant) to creare the ASM instance on these two strawman files did not work as the DBCA didn't want to recognize the "disks". So back to good old command line. By the way, there's a MOS Note out there which may be helpful as well (but didn't work in my case).
How To Create ASM Diskgroups using NFS/NAS Files? (Doc ID 731775.1)

  1. Create a password file for ASM instance in $ORACLE_HOME/dbs
  2. Create a fresh init.ora for ASM within the same directory having the following parameters set:
    _asm_allow_only_raw_disks='FALSE'
    asm_diskstring='/oradata/ASM/dg*'
    asm_power_limit=4
    instance_type=asm
  3. With these parameter set I could bring the instance into MOUNT state ready to create the two disk groups after setting the ORACLE_SID=+ASM in the environment:
    SYS:+ASM> create diskgroup DATA external redundancy disk '/oradata/ASM/dg_DATA';
    Diskgroup created.
    SYS:+ASM>  create diskgroup BCK  external redundancy disk '/oradata/ASM/dg_BCK';
    Diskgroup created.

Starting up ASM did work now well after shutting it down first - and a check for SELECT path from V$ASM_DISK did show me my disks.

Next step - simply - is to create a database with DBCA inside of ASM. So the first part of my test did complete.

... to be continued soon ...

About

Mike Dietrich - Oracle Mike Dietrich
Master Product Manager - Database Upgrade & Migrations - Oracle Corp

Based near Munich/Germany and spending plenty of time in airplanes to run either upgrade workshops or work onsite or remotely with reference customers. Acting as interlink between customers/partners and the Upgrade Development.

Follow me on TWITTER

Contact me via LinkedIn or XING

Search

Archives
« September 2015
SunMonTueWedThuFriSat
  2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Oracle related Tech Blogs
Slides Download Center
Visitors since 17-OCT-2011
White Paper and Docs
Workshops
Viewlets and Videos
Workshop Map
This week on my Rega & Pono
Upgrade Reference Papers