Tuesday Feb 04, 2014

Master Note for Database Startup/Shutdown (Doc ID 1270450.1)

Concepts

During Startup, the instance is started first (nomount stage) which then mounts the database by accessing the control files specified in the CONTROL_FILES parameter. The next step will be to open the database which includes opening the datafiles and redo log files. Applying the redo (rollforward) and the rollback of uncommitted data (rollback) also happens if the previous shutdown was not a clean one (shutdown normal or immediate).

Similarly the database and instance undergoes multiple phases during shutdown. First the database is closed ie, the datafiles and the redolog files are closed. Then the database will be dismounted  to disassociate it from the instance. After a database is unmounted, Oracle Database closes the control files of the database. The next step will be to shutdown the instance by removing the SGA from the memory and terminating the background processes.

There are various shutdown modes such are normal, immediate,transactional and abort. When a database is closed as part of a SHUTDOWN with any option other than ABORT, Oracle Database writes data in the SGA to the data files and online redo log files. If a SHUTDOWN ABORT or abnormal termination occurs, then the instance of an open database closes and shuts down the database instantaneously. Oracle Database does not write data in the buffers of the SGA to the data files and redo log files. The subsequent reopening of the database requires instance recovery, which Oracle Database performs automatically.

Database startup and shutdown are powerful administrative options that are restricted to users with administrator privileges.

For details you can refer Note 1505155.1 - Master Note: Overview of Database Startup and Shutdown

Common Issues During Database Startup

This Section list out the common issues encountered during startup. The error can be reported in any of the above mentioned phases like nomount,mount or open. Since transaction recovery continues even after opening the database, you may observe issues like hang, high CPU usage etc, even after the database is open. The documents mentioned in the below section can be specific to platform or database versions.

Startup upgrade/migrate or setting _system_trig_enabled = FALSE will disable startup triggers to rule out startup trigger problems.

12c Startup/Shutdown

Startup and shutdown of the container database is the same as it has always been for regular instances. 

For PDBs

Also Refer to the following documents for more information on 12c Startup and Shutdown: 
  • Note 1582453.1 When to use startup / shutdown database and alter database command in 12c

ORA-27102 Errors

ORA-27102 errors normally occurs due to memory issues.The Common causes could be due to Semaphore Kernel misconfigurations,Memory related Ulimit settings,RAM or swap configurations.


ORA-27300 Errors

These errors are generally reported when the Operating System called for error or when there was a connection killed or a network interconnection failures or an OS configuration issue.The error ORA-27300 will also be accompanied by ora-27301 and ora-27302

Note 579365.1 Troubleshooting ORA-27300 ORA-27301 ORA-27302 errors

ORA-64 Errors

This error could occur when the database init.ora parameter calling for more resources than the Operating System is configured to provide.The parameters could be PROCESSES,DB_BLOCK_SIZE,SGA and more.

Note 1457812.1 ORA-00064 Error Reported After Increasing Processes Parameter value


ORA-27123 Errors

Note 115753.1 UNIX: Resolving the ORA-27123 error

ORA-1081 Errors

This error could occur when we try to startup an instance that is already running or if the shared segments/semaphores already exist.

Note 1010214.6 ORA-1081: Starting Instance
Note 18657.1 OERR: ORA 1081 cannot start already-running Oracle - shut it down first

NFS Related Issues

Note 8418190.8 Bug 8418190 - Direct NFS warnings during database startup
Note 236794.1 NFS Locking Problems Encountered During Database Startup
Note 971406.1 DATABASE STARTUP HANGS AT MOUNTING CONTROLFILE WHEN DNFS IS ENABLED
Note 1430654.1 Database Startup Failed with "Direct NFS: please check that oradism is setuid"
Note 430920.1 NetApp: Using 'nolock' NFS Mount Option with non-RAC Systems Results in Database Corruption

Transaction Recovery Slowness

There could be slowness in the database during the open phase when the database is busy performing transaction recovery.

Note 1494886.1 Database Transaction Recovery
Note 414242.1 Database Hangs Because SMON Is Taking 100% CPU Doing Transaction Recovery
Note 12934890.8 Bug 12934890 - Startup hangs waiting for row cache lock due to open transaction against UNDO$

ORA-704 Errors

This is a general error reported at startup when there is some problem during processing of bootstrap information.There should be an accompanying error/s. Plug the ORA=704 and the other errors it reports into the Oracle Support search engine for a solution.

ORA-9968 Errors

There are some client shadow processes hanging. Although the lk< SID> file is deleted the hanging processes still have a lock on the open file handle. This prevents the database to startup although a new lk<SID> file can be created successfully. An oracle process (background or shadow process) that exists while the instance is not started (crashed or not cleanly stopped) can have a lock on a file while this file is actually removed from the system. This is because on UNIX there is still a lock on the open file handle.

Note 160395.1 Database Startup Fails with ORA-1102 and ORA-9968
Note 1488147.1 Instance Startup Raises Error ORA-09968: unable to lock file (Doc ID 1488147.1)

ORA-12547 Errors

The error ORA-12547 indicates that the communication channel has been broken. It's most often thrown because the other end of the process went away unexpectedly.

Note 1307075.1 Oracle Database Fails to Start with Error ORA-12547
Note 381566.1 connect / as sysdba Fails with Ora-12547 And Tns-12514
Note 744512.1 Ora-12547: Tns:Lost Contact Creating Database After Clean Installation

ORA-1157 Errors 

The background process was not able to find one of the datafiles.The database will prohibit access to this file but other files will be unaffected.However, the first instance to open the database will need to access all online datafiles.Accompanying messages from the operating system will describe why the file was not found.

Note 184327.1 ORA-1157 Troubleshooting

Common Errors / Issues During Database Shutdown

The most common issue observed while bringing down the database is shutdown immediate hang. The main reasons for Shutdown immediate hang is:
- processes still continue to be connected to the database and do not terminate.
- SMON is cleaning temp segments or performing delayed block cleanouts.
- Uncommitted transactions are being rolled back.

The below section provides the consolidated list of known issues during shutdown. The documents mentioned in the below section can be specific to platform or database versions.

Transaction Recovery

Note 375935.1 What To Do and Not To Do When 'shutdown immediate' Hangs
Note 1076161.6 Shutdown Normal or Shutdown Immediate Hangs. SMON disabling TX Recovery
Note 414242.1 Database Hangs Because SMON Is Taking 100% CPU Doing Transaction Recovery
Note 100054.1 Transaction Rollback after a failed operation or during Database Shutdown

ORA-24324 Errors

Note 794293.1 ORA-24324 During Shutdown

DISM

Note 1001248.1 On Solaris 9 Systems, Oracle Shutdown May Hang If Utilizing Dynamic Initmate Shared Memory (DISM) 

Issues specific to automatic shutdown and startup

This section is specific to the automatic shutdown and startup that can be configured with the dbora / dbshut / dbstart scripts.

Automatic Startup Failure

The key to diagnosing automatic startup failures is to determine where startup fails. This can be done via the following steps:

  1. Determine if instance starts manually as Oracle software owner.
  2. Determine if instance starts via dbstart command run as Oracle software owner.
  3. Determine if instance starts when root runs following dbstart command:
    su - $ORA_OWNER -c $ORA_HOME/bin/dbstart

    where $ORA_OWNER is set to Oracle software owner.
  4. Determine if instance starts when running as root the OS script which calls dbstart, ie "/etc/init.d/dbora start". NOTE: Running via sh -x command will show each command as it is run from script to better see what is going on.
  1. #> sh -x /etc/init.d/dbora start

Automatic Shutdown Failure

As with automatic startup, the key to diagnosing automatic shutdown failures is to determine where shutdown fails. This can be done via following steps:

  1. Determine if instance stops manually as Oracle software owner.
  2. Determine if instance stops via dbshut command run as Oracle software owner.
  3. Determine if instance stops when root runs command
    su - $ORA_OWNER -c $ORA_HOME/bin/dbshut
    where $ORA_OWNER is set to Oracle software owner.
  4. Determine if instance stops when running as root the OS script which calls dbshut, ie "/etc/init.d/dbora stop". NOTE: Running via sh -x command will show each command as it is run from script to better see what is going on.
    #> sh -x /etc/init.d/dbora stop

Further Diagnostics

If you were not able to resolve the issue with the details provided in this document, please raise a Service Request for further assistance from Oracle Support. The diagnostic information required to troubleshoot the issue is listed in the below document:

Troubleshooting Database Startup/Shutdown Problems (Doc ID 851057.1).

References

NOTE:1505155.1 - Master Note: Overview of Database Startup and Shutdown
About

News and Troubleshooting tips for Oracle Database and Enterprise Manager

Search

Categories
Archives
« April 2015
SunMonTueWedThuFriSat
   
1
2
3
4
5
6
7
8
9
10
11
12
13
14
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
  
       
Today