Oracle Support Master Note for 10g Grid Control OMS Process Control (Start, Stop and Status) & Configuration (Doc ID 1081865.1)

 

 

For most current information refer Master Note for 10g Grid Control OMS Process Control (Start, Stop and Status) & Configuration (Doc ID 1081865.1)

 

 

In this Document
  Purpose
  
Scope and Application
  
Master Note for 10g Grid Control OMS Process Control (Start, Stop and Status) & Configuration
     
OMS Concepts
     
OMS Process Control (How to start, stop and check OMS status?)
     
Modifying OMS configuration files (for changing sysman pwd, port etc)
     
Diagnostic Tools Available for Troubleshooting OMS Process Control and Configuration issues
     
Troubleshooting OMS Process Control
     
Troubleshooting OMS Configuration
     
Best Practices (Maintenance Activities, OCM, Healthcheck, CPU & PSU and Certification)


Applies to:

Enterprise Manager Grid Control - Version: 10.1.0.2 to 10.2.0.5 - Release: 10.1 to 10.2
Enterprise Manager Grid Control - Version: 10.1.0.2 to 10.2.0.5   [Release: 10.1 to 10.2]
Information in this document applies to any platform.

 

Purpose

This Master Note helps understand OMS Process Control and Configuration and provides assistance in using diagnostics effectively to debug/troubleshoot and resolve issues encountered.

 

Scope and Application

This document is intended to assist Enterprise Manager Grid Control Administrators effectively troubleshoot OMS Process Control and Configuration issues. This document covers the following topics:

1. OMS Concepts
2. OMS Process Control (How to start, stop and check OMS status)
3. Modifying OMS configuration files for changing SYSMAN password, Ports, Repository connection details etc.
4. Diagnostic Tools
5. Troubleshooting and using Diagnostics effectively
6. Best Practices (Maintenance Activities, OCM, Healthcheck, CPU & PSU and Certification)

 

Master Note for 10g Grid Control OMS Process Control (Start, Stop and Status) & Configuration

 

OMS Concepts



The Oracle Management Service (OMS) renders the user interface for Oracle Enterprise Manager Grid Control, interacts with Management Agents on monitored hosts and stores persistent data in the Oracle Management Repository. The Grid Control installation comprises of these components:

  • OC4J instances (OC4J_EM, OC4J_EMPROV)
    • OC4J_EM instance contains the EM application.
    • OC4J_EMPROV instance contains the Agent Deploy application (new from 10.2.0.1 Grid Control onwards).
  • HTTP Server: serves client HTTP(s) requests

Other optional components include WebCache, OCMRepeater (for OCM configuration in OMS home from 10.2.0.4 Version onwards).

The Enterprise Manager user-interface is called the Grid Console and can be accessed in any certified Web browser via Oracle HTTP Server (default port - 7778) or WebCache (default port - 7777). The WebCache is not a mandatory component for the OMS functionality, it is present as part of the Application server installation but does not provide significant performance improvement for the Grid Console pages.

The Management Agents on monitored hosts communicate with the OMS through HTTP_Server, bypassing WebCache. The HTTP_Server listens for Oracle Management Agent upload traffic on default ports - 4889 (http) and 1159 (https).

For additional details regarding the OMS, refer to:

  • Configuration Files:

Note 1080406.1: Description of Important Configuration files in the 10g Oracle Management Service (OMS) Home of the Enterprise Manager Grid Control

====================================================================

OMS Process Control (How to start, stop and check OMS status?)

The OMS can be started using two utilities:

<<OMS_HOME>>/opmn/bin/opmnctl
<<OMS_HOME>>/bin/emctl

The usage of these utilities is described in detail in the following document:

Note 298991.1: How to Process Control (start, stop, check status) the Oracle Management Service (OMS) Component in Enterprise Manager Grid Control

====================================================================

Modifying OMS configuration files (for changing sysman pwd, port etc)

After the OMS installation, there may be a need to modify some of the OMS configuration details.
The following documents describe the OMS configuration changes that can be done, grouped by the type of activity:

  • Changing the SYSMAN password in the Repository
    • Note 270516.1: How to Change the Password of SYSMAN User in Grid Control?
  • OMS to Repository Database Connection and Communication
    • Note 736061.1: How To Re-configure the OMS After Hostname/IP Address change on the repository database server
    • Note 369997.1: How to Re-configure the OMS After Port Change for the Listener Servicing the Grid Control Repository Database
    • Note 738075.1: How to Re-configure OMS After Change in the Grid Control Repository Database Name
    • Note.457854.1: How to configure OMS using existing DB when RAC database is used as Repository
  • Re-creating OMS configuration files
    • Note 733401.1: How to Recreate the emoms.properties File from Scratch
    • Note 431410.1: How to Recreate the 'opmnctl' file from Scratch on Unix Machines
  • Changing hostname of the OMS Machine
    • Note 371865.1: Is it possible to Modify the EM 10g Grid Control OMS Hostname After the Installation?
  • Modification of Ports used by OMS
    • Note 235298.1: Overview of Default Ports Used by EM 10g Grid Control, DB Control and AS Control
  • Controlling the Log and Trace Files for OMS
    • Note 229627.1: How to Locate 10g Grid Control OMS Log / Trace Files and Control their Size and Other Details

====================================================================

Diagnostic Tools Available for Troubleshooting OMS Process Control and Configuration issues

  • RDA

The Remote Diagnostic Agent (RDA) can be executed specifically with the Grid Control / OMS profile name: GridControl in order to reduce the number of questions that need to be answered and also to collect all details of the OMS home correctly.

The steps to execute the RDA with GridControl profile is explained in:

Note 1057051.1: How to Run the RDA against a Grid Control Installation

It is highly recommended that the latest EMDiagkit is installed and executed in the OMS home, before running the RDA. This will ensure that the RDA picks up the latest data collected by the EMDiagkit.

  • EMDiagkit

The EMDiagkit is a diagnostic tool developed to assist in diagnosis and correction of Enterprise Manager 10g Framework issues. At present, the tool allows us to extract necessary troubleshooting data from the EM Repository Schema using the repvfy utility.

The details for installation, usage of EMDiagkit are available in

Note 421053.1: EMDiagkit Download and Master Index

====================================================================

Troubleshooting OMS Process Control

OMS is a J2EE application deployed in a Application Server OC4J instance and needs to connects to an Oracle Database hosting the Repository, for its operations.

  • Failures during Startup

The OMS startup failure could occur at any of the following component level:

1. OPMN and OC4J_EM components in the Application Server.
2. HTTP_Server component in the Application Server.
3. OMS Application and Repository Database Connectivity.
4. Specific Objects / Configuration in the Grid Control Repository.
5. WebCache component in the Application Server.

My Oracle Support contains several notes with steps to resolve the specific issue being faced during OMS startup, at each of the above component levels. The best way to find these notes is to login to My Oracle Support portal and query the 'Knowledge' with the following keywords:

OMS Startup Fails <symptom that you have found in the log/trace files>

Some examples:

OMS Startup Fails with "Communication error with the OPMN server local port."
OMS Startup Fails: 'opmnctl status' Shows OC4J_EM Component in 'Stop' Status
OMS Startup Fails at HTTP_Server Component if the HTTP_Server Logs are > 2  GB in Size
OMS Startup Fails at HTTP_Server Component Startup with "Address already in use:make_sock: could not bind to port"

For steps to identify the symptoms and navigate through the log/trace files, refer to the troubleshooting steps suggested in:

Note 730308.1: How to Troubleshoot Process Control (start, stop, check status) the 10g Oracle Management Service(OMS) Component in 10 Enterprise Manager Grid Control
Section: OMS Startup

  • Failures during Shutdown

All the processes that are started as part of the OMS startup should always be shutdown cleanly (no stuck / defunct processes) so that the state information held by the opmn is cleaned out and not re-used during the next startup. Also, all the sessions started by the OMS application in the repository should be closed successfully.

Sometimes, the OC4J_EM component may fail to shutdown gracefully within the default timeout period, due to a heavily loaded OMS server or latency on the network between the remote OMS machines and the repository. In such cases, the opmn timeout value can be increased using the steps in:

Note 358849.1: OMS Stops Fails with 'time out while waiting for a managed process to stop' When Using 'opmnctl stopall', How to increase the Timeout?

To find documents related to problems during OMS stop, login to My Oracle Support portal and query the 'Knowledge' with the following keywords:

OMS Stop Fails <symptom that you have found in the log/trace files>

For more troubleshooting steps, refer to:

Note 730308.1: How to Troubleshoot Process Control (start, stop, check status) the 10g Oracle Management Service(OMS) Component in 10g Enterprise Manager Grid Control
Section: OMS Stop / shutdown

  • Failures during Status check
    • opmnctl status

The 'opmnctl status' command shows the status of the necessary opmn components needed for OMS functionality - HTTP_Server and OC4J_EM component etc. The status of the components could be in these status : 

Alive: indicates that the component is up and is responding to the opmn status check.
Down: indicates that the component is not running.
Init: indicates that the component is still in the process of initialization. This is a transitional status only.
Stopped: indicates that the component has been stopped by the opmn as it is unable to start completely or the opmn is trying to clean up the state information. This is a transitional status only.

If the OC4J_EM component is in Alive status, it only means that the opmn-managed OC4J_EM component is running - the 'em' / OMS application may or may not be actually running.

    • emctl status oms

This command helps in verifying:

-  whether the 'em' / OMS application is actually running or not.
-  whether the HTTP server is running or not.
-  whether the OMS is able to successfully connect to the sysman schema in the Grid Control Repository Database.
-  whether the emkey is configured properly.

Note: If the OC4J_EM status is Alive, but the 'emctl status oms' returns an error that the Management Service is not Up, then the Grid Console cannot be accessed from the browser.


To find documents related to problems during OMS status check, login to My Oracle Support portal and query the 'Knowledge' with the following keywords:

'emctl status oms' Fails <symptom that you have found in the log/trace files>

For steps to identify the symptoms and navigate through the log/trace files, refer to the troubleshooting steps suggested in:

Note 730308.1: How to Troubleshoot Process Control (start, stop, check status) the 10g Oracle Management Service(OMS) Component in 10g Enterprise Manager Grid Control
Section: OMS Status Check

  • Searching My Oracle Support Documents for OMS Process Control

As the search is specific to Enterprise Manager Grid Control issues, we recommend that the search be performed only under the Grid Control section, using the following navigation:

Login to My Oracle Support then Click Knowledge -> Enterprise Management -> Enterprise Manager Consoles - Packs - and Plugins -> Enterprise Manager Grid Control ->All of Enterprise Manager Grid Control.

  • Using RDA and EMDiagkit for troubleshooting OMS Process Control

The RDA output generated with the GridControl profile is very useful in obtaining all the configuration files and log/trace files together. Specific logs which will be helpful for each component are described in Note 730308.1.

The EMDiagkit output is very useful in diagnosing problems with Grid Control Repository objects, which can cause the OMS startup failures.

Note: It is highly recommended that the latest EMDiagkit is installed and executed in the OMS home, before running the RDA. This will ensure that the RDA picks up the latest data collected by the EMDiagkit.

====================================================================

Troubleshooting OMS Configuration

Modifying the OMS configuration in the emoms.properties, httpd_em.conf or any other configuration files may be done incorrectly, resulting in OMS startup failure or problems during its operations:

  • Note 418159.1: Problem: OMS Stops Working Due to emoms.properties Becoming Empty
  • Note 794084.1: Sysman Password Change Fails when using case sensitive password with 11g repository
  • Note 420275.1: Problem: emctl: "YOU HAVEN'T DISABLED SET-ID SCRIPTS IN THE KERNEL YET!" ERROR WHEN EMCTL IS RUN
  • Note 740387.1: The emoms.trc File Shows Many "getFormattedDateWithTZ.441 - Invalid region name format" Errors for Console
    Operations

====================================================================

Best Practices (Maintenance Activities, OCM, Healthcheck, CPU & PSU and Certification)

This section lists some of the best practices which will help prevent problems with OMS Process Control and Configuration:

EM Certification Checker

It is strongly recommended that you always use a certified combination of OMS, Agent and Repository Database for managing Targets which are certified with this combination.
The Enterprise Manager certification details are available in:

Note 412431.1: Oracle Enterprise Manager 10g Grid Control Certification Checker

Maintenance Activities

  • Enable Log Rotation for the access_log and error_log files created by the httpd_em.conf file:
          
    Note 339819.1: How to Enable Rotation for the HTTP_Server and OC4J_EM Logfiles in
           the 10g Grid Control OMS Home?
  • Execute EMDiagKit and regular intervals (once per week or more frequently, depending on your setup) and check for any new problems that are reported.
  • Take valid backups of the OMS and Repository Database Homes at regular intervals, to restore back any configuration files that are deleted by accident.
  • Remove the emkey details from the repository and store the <<OMS_HOME>>/sysman/config/emkey.ora file in a safe place. If the emkey is lost, there is no way to recover it except for re-installing all the OMS and Repository.

OCM

Oracle Configuration Manager (OCM) works with My Oracle Support to enable proactive support capability that helps you organize, collect and manage your Oracle configurations by providing Proactive configuration-specific notification of Security and General Alerts, HealthCheck recommendations based on Support Best practices when using configuration auto-collection, Simplified Service Request logging, tracking and reporting and Project cataloging of key milestones and contacts associated with your configurations.

  • Among these the following topics are related to the Enterprise Manager Components:
    • 2.52 Oracle Enterprise Manager 10g Grid Control Management Agent:
    • 2.54 Oracle Enterprise Manager 10g Grid Control Management Service
    • 2.53 Oracle Enterprise Manager 10g Grid Control Management Repository
    • 2.72 Oracle Grid Control Repository (for oracle_emrep target)
    • 2.38 Oracle Agent Deployment Configuration (oracle_emd target)
    • 2.73 Oracle Home
    • 2.23 Host

 

Note: The above list is expected to be expanded as and when new collections are introduced in future.

 

  • It is also advisable to review the collections available for the Database instance, so that the Database hosting the repository can be monitored as well:
    • 2.10 Database Instance
    • 2.78 Oracle Listener

Healthcheck

Healthchecks are executed dynamically against the Oracle Configuration Manager uploaded configurations in My Oracle Support. These checks, based on Oracle Best practices, will proactively notify you of potential problems in your environment, and provide recommendations that help you improve system performance and avoid problems in your Oracle environment.

  • If you are receiving any Healthcheck alerts in My Oracle support, then refer to the following document for the alert details and its corresponding document for resolving the same:

Note 868955.1: My Oracle Support Health Checks Catalog

  • For Healthchecks specific to the Enterprise Manager and Repository Database, refer to the sections titled:
    • Enterprise Manager (for the OMS)
    • Oracle Database (for the Database hosting the Repository)
  • Some examples of Enterprise Manager specific healthchecks which will alert potential OMS process control and configuration issues are:
    • job_queue_processes May Be Set Too Low
      To resolve, refer
      Note 957560.1: Health Check Alert: JOB_QUEUE_PROCESSES is set to a value lower than 10
    • UNDO_TABLESPACE is NULL
      To resolve, refer
      Note 957569.1: Health Check Alert: UNDO_TABLESPACE is NULL

CPU and PSU

  • CPU

Critical Patch Updates (CPU) is the primary means of releasing security fixes for Oracle products. They are released on the Tuesday closest to the 15th day of January, April, July and October. This page lists all the currently available Critical Patch Updates (CPUs) in chronological order and is updated whenever new Critical Patch is released. You can also subscribe to the CPU Email Alerts using the steps listed here.

To obtain the latest CPU patch details for the Enterprise Manager Grid Control and its dependent products - Oracle Application Server and Oracle Database:

- In the page, click on the link shown for the latest CPU in the table under the 'Critical Patch Updates'.
- The next page, lists all the products which have security fixes in the chosen CPU release. Scroll down to 'Patch Availability Table ..' topic and find the table with details for the Product Group and Patch Availability and Installation Information.
- In the table, find the row related to Product Group: 'Oracle Enterprise Manager' and pick up the document number given in the Patch Availability and Installation Information column. In the document, navigate to:

"Critical Patch Update Availability for Oracle Products" and then to
"Oracle Enterprise Manager Grid Control"

  • PSU

Patch Set Updates (PSU) are proactive cumulative patches containing recommended bug fixes that are released on a regular and predictable schedule. PSUs are on the same quarterly schedule as the Critical Patch Updates (CPU), specifically the Tuesday closest to the 15th of January, April, July, and October. The PSUs serve as a new baseline version for reporting issues to Oracle, hence it is always recommended to be on the latest PSU release.

·         For more details on PSU, refer Note 854428.1: Patch Set Updates for Oracle Products

·         For Enterprise Manager specific PSU, refer Note 822485.1: Oracle Recommended Patches -- Oracle Enterprise Manager

  • Choosing between CPU / PSU patches

The PSU and CPU released each quarter contain the same security content. However, the patches employ different patching mechanisms, so customers need to choose wisely which patch satisfies their needs better:

·         A PSU can be applied on the CPU released at the same time or on an any earlier CPU for the base release version. A PSU can be applied on any earlier PSU or the base release version. CPUs are only created on the base release version.

·         Once a PSU has been installed, the recommended way to get future security content is to apply subsequent PSUs. Reverting from PSU back to CPU, while possible, would require significant effort, and so is not advised.

  • Getting CPU / PSU patch recommendations via OCM

    OCM also collects and recommends the latest CPU and PSU patch that can be applied to a particular Oracle Home. These details can be seen in the My Oracle Support ->Patches and Updates -> Patch Recommendations section

- 'Security' patch recommendations include the CPU patches.
- 'Other Recommendations' include the PSU patches.

 

Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

News and Troubleshooting tips for Oracle Database and Enterprise Manager

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today