Oracle Support Master Note for 10g Grid Control Enterprise Manager Communication and Upload issues (Doc ID 1086343.1)

 

 

For most current information refer Master Note for 10g Grid Control Enterprise Manager Communication and Upload issues (Doc ID 1086343.1)

 

 

 

In this Document
  Purpose
  
Scope and Application
  
Master Note for 10g Grid Control Enterprise Manager Communication and Upload issues
     
Inter-Communication between Enterprise Manager Components
     
Data Flow in Enterprise Manager
     
Configuring Proxy / Firewall for Communication Between EM Components
     
Modifying EM Configuration for Data Upload Changes
     
Diagnostic Tools Available for Troubleshooting EM Communication and Data Upload Issues
     
Troubleshooting Communication Between Grid Control Components
     
Troubleshooting Data Upload Issues in Grid Control
     
Best Practices (Certification, Maintenance Activities, OCM, Healthcheck, CPU & PSU)
  
References


Applies to:

Enterprise Manager Grid Control - Version: 10.1.0.2 to 10.2.0.5 - Release: 10.1 to 10.2
Enterprise Manager Grid Control - Version: 10.1.0.2 to 10.2.0.5   [Release: 10.1 to 10.2]
Information in this document applies to any platform.

Purpose

This Master Note helps to understand Inter-Communication and Data Upload in Enterprise Manager Grid Control and provides assistance in using diagnostics effectively to debug/troubleshoot and resolve issues encountered.

Scope and Application

This document is intended to assist Enterprise Manager Grid Control Administrators effectively troubleshoot Enterprise Manager Inter-Communication and Data Upload issues. This document covers the following topics:

1. Inter-Communication between EM Components and Data Upload Concepts.
2. Configuring Proxy / Firewall for Communication between EM Components.
3. Modifying EM Configuration for Data Upload changes.
4. Diagnostic Tools.
5. Troubleshooting and using Diagnostics effectively
6. Best Practices (Certification, Maintenance Activities, OCM, Healthcheck, CPU & PSU).

Master Note for 10g Grid Control Enterprise Manager Communication and Upload issues

Inter-Communication between Enterprise Manager Components

  • The 10g Grid Console uses the default OracleAS Web Cache port (for example, port 7777 on UNIX systems and port 80 on Windows systems) to connect to the Oracle HTTP Server component of the OMS. The connection can be via http / https.
  • The OMS communicates with the Repository Schema present in an Oracle Database via thin JDBC.
  • The Agent and OMS communicate with each other via http / https.
  • Administrators use the Grid Console to monitor and administer the managed targets that are discovered by the Management Agents on each host. The Grid Console communicates with a Managed Database Target via TCP/IP (sqlnet connection).
  • An Agent communicates with another Agent during certain operations such as Database Cloning / Standby Database creation via http / https.

Note: The scope of this article does not include the communication between an Agent and its managed target, communication between two Agents for cloning (or similar) operations and the communication between OMS and a target Database.

For additional details regarding the Grid Control Communication, refer to: 

  • Note 1084777.1: Description of Important Communication Components in a 10g Enterprise Manager Grid Control Agent
  • Note 358953.1: What ports are used in communication between the Grid Control OMS and a Management Agent?
  • Note 763844.1: How to Verify the Hostname/IP Address Resolution Between the 10g Enterprise Manager Grid Control Components? 
  • Note 1088393.1: How to Verify the Communication Between the 10g Enterprise Manager Grid Control Components via Firewall/Proxy?
  • Note 762747.1: Sample Java Program to Test Thin JDBC Connection from the 10g OMS to the Repository Database


********************************************************************************

Data Flow in Enterprise Manager

 

 

  • When all the Grid Control components are installed on a single host, the management data travels along the following paths:

1. The OMS uses thin JDBC connections to retrieve data from the Management Repository for being displayed in the Grid Console, as requested by the Administrator.

2. The Agent loads its data (which includes data about all the managed targets on the host) to the OMS via the Oracle HTTP Server upload URL, bypassing the OracleAS Web Cache. The default http port for the OMS upload URL is 4889 and the https port is 1159. The port used by the Agent is defined by the REPOSITORY_URL parameter in the <AGENT_HOME>/sysman/config/emd.properties configuration file.

For more details, refer Note 235290.1: Understanding the Enterprise Manager Management Agent 10g 'emd.properties' File

3. The OMS uses thin JDBC connections to load data into the Management Repository database via a component called Loader which is responsible for loading the data sent by the Agents into the Repository tables. The Management Repository connection information is defined in the <OMS_HOME>/sysman/config/emoms.properties configuration file.

For more details, refer to

·         Note 1080085.1: Understanding the 10g Oracle Management Service (OMS) 'emoms.properties' File

·         Note 550615.1: Understanding The EM 10g Grid Control Loader System 

4. The OMS sends data to the Agent via http / https using the Agent URL. The Agent software includes a built-in HTTP listener that listens on the Agent URL for messages from the OMS. As a result, the OMS can bypass the Oracle HTTP Server and communicate directly with the Agent. 

The OMS uses the Agent URL to monitor the availability of the Agent, submit Enterprise Manager jobs, and other management functions. The Agent URL can be identified by the EMD_URL property in the <AGENT_HOME>/sysman/config/emd.properties configuration file.

In addition, the name of the Agent as it appears in the Grid Console consists of the Agent host name and the port used by the Agent.

For more details, refer to

·         Note 1084777.1: Description of Important Communication Components in a 10g Enterprise Manager Grid Control Agent

·         Note 287276.1: Understanding the Files in the Agent's <EMHOME>/sysman/emd/upload Directory

  • Additional Reference:
    Note 427666.1: Is it Possible to Manually Load the Files from <OMS_HOME>/sysman/recv/errors to the Repository?


********************************************************************************

Configuring Proxy / Firewall for Communication Between EM Components

  • Configuring Firewall for Enterprise Manager Communication
  • Configuring Proxy for Enterprise Manager Communication
    • Note 313181.1: Overview of the Parameters For Proxy Configuration of EM Grid Control Components 
    • Note 471842.1: Understanding Proxy Settings in Enterprise Manager Grid Control 


********************************************************************************

Modifying EM Configuration for Data Upload Changes

After the OMS installation, there may be a need to modify some of the upload parameters at the Agent / OMS side.

  • Configuring Shared File Areas for Management Services

    The Grid Control 10g Release 2 has a high availability feature called the Shared Filesystem Loader, to be used when there are multiple OMS for load-balancing in a large setup. In this feature, the data files sent by all the Agents are stored temporarily on a common shared location called the shared receive directory. All the OMS are configured to use the same storage location for the  receive directory and they coordinate internally to distribute amongst themselves the workload of uploading files into the Management Repository. The value is identified by the ReceiveDir parameter in the <OMS_HOME>/sysman/config/emoms.properties file. If a particular OMS goes down, its workload is taken up by surviving OMS(s).
    Refer the Enterprise Manager documentation topic: 
    Configuring the Management Services

  • Modifying the parameters at the Agent / OMS side related to upload performance

    Refer to Note 550615.1: Understanding The EM 10g Grid Control Loader System 
    Section: Loader Related Parameters in Agent and OMS Configuration Files


********************************************************************************

Diagnostic Tools Available for Troubleshooting EM Communication and Data Upload Issues

  • RDA

The Remote Diagnostic Agent (RDA) can be executed specifically with the Grid Control / OMS profile name: GridControl and the Agent profile name: AGT in order to reduce the number of questions that need to be answered and also to collect all details of the OMS / Agent Home correctly. 

The steps to execute the RDA with GridControl and AGT profiles are explained in:

Note 1057051.1: How to Run the RDA against a Grid Control Installation

It is highly recommended that the latest EMDiagkit is installed and executed in the OMS home, before running the RDA. This will ensure that the RDA picks up the latest data collected by the EMDiagkit.

  • EMDiagkit

The EMDiagkit is a diagnostic tool developed to assist in diagnosis and correction of Enterprise Manager 10g Framework issues. At present, the tool allows us to extract necessary troubleshooting data from the EM Repository Schema using the repvfy utility.

The details for installation, usage of EMDiagkit are available in 

Note 421053.1: EMDiagkit Download and Master Index

  • Wget

    GNU Wget (or just Wget) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get, connotative of its primary function. It currently supports downloading via HTTP, HTTPS, and FTP protocols, the most popular TCP/IP-based protocols used for web browsing.

    For more details, refer:


Note: wget is a third party tool and problems faced while using this tool cannot be supported by Oracle Support. Also, the above mentioned download links are not maintained by Oracle and hence are subject to change.


********************************************************************************

Troubleshooting Communication Between Grid Control Components

The Communication failure could occur at any of the below channels:

  • Grid Console (UI) to the Oracle Management Service (OMS)

    Incorrect communication setup between the Grid Console (User Interface) and the OMS can result in the following problems:
    • Trying to access the Grid Console URL from a certified web-browser results in the error:

"page cannot be displayed"

OR

502 Bad Gateway
The following error occurred:

[code=DNS_HOST_NOT_FOUND] The host name was not found during DNS lookup. Contact your system administrator if the problem is not found by retrying the URL.

 

    • Trying to access the Grid console using unsecured URL (using http) fails with:

Forbidden
You don't have permission to access /em/console/home on this server.
-----------------------------------------------
Oracle-Application-Server-10g/10.1.2.2.0 Oracle-HTTP-Server Server at omsmachine.domain: 7778


But access to the secured URL (using https) works fine.

    • Accessing the Grid Console URL returns no errors but shows a blank page without the Login screen.

For troubleshooting steps, refer to Note 1089443.1: How to Troubleshoot Communication Between the Grid Console (UI) and Oracle Management Service (OMS) Components in 10g Enterprise Manager Grid Control 

To find documents related to communication problems between the Grid Console (UI) and OMS, login to My Oracle Support portal and query the 'Knowledge' with the following keywords:

Accessing the 10g Grid Console URL Results <actual error message seen>

Some examples:

Accessing the 10g Grid Console URL Results in 502 Bad Gateway
Accessing the 10g Grid Console URL Results in 'page cannot be displayed'


  • Oracle Management Service (OMS) to the Grid Control Repository Database

    Incorrect communication setup between the OMS and the Repository Database can result in the following problems:
    • The OMS is unable to initialize completely. The OC4J_EM component is able to startup but the 'emctl status oms' shows:

Oracle Enterprise Manager 10g Release 3 Grid Control
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Oracle Management Server is not functioning because of the following reason:
Connection to the repository failed. Verify that the repository connection information provided is correct.

 

    • The OMS can crash if it is unable to connect to the Repository Database during its operations.
    • Trying to access the Grid Console URL from the browser, may return an error such as:

503 Service Unavailable
Servlet error: Service is not initialized correctly.
Verify that the repository connection information provided is correct.

OR

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, you@your.address and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.
--------------------------------------------------------------------------------
Oracle-Application-Server-10g/10.1.2.0.2 Oracle-HTTP-Server Server at omsmachine.domain Port 7777


For troubleshooting steps, refer to Note 1089693.1: How to Troubleshoot Communication Between the Oracle Management Service (OMS) and Grid Control Repository Database Components in 10g Enterprise Manager Grid Control 

To find documents related to communication problems between the OMS and the Repository Database, login to My Oracle Support portal and query the 'Knowledge' with the following keywords:

conn.ConnectionService <error seen in the emoms.trc for connection attempt>


  • Oracle Management Service (OMS) to the Grid Agent

    Incorrect communication setup between the OMS and the remote Agents can result in the following problems:
    • In the Grid Console, Setup -> Agents -> clicking on the 'Agent name' displays a red warning/error message such as:

Communication between the Oracle Management Service host to the Agent host is unavailable. Any functions or displayed information requiring this communication link will be unavailable. For example: deleting/configuring/adding targets, uploading metric data, or displaying Agent home page information such as Agent to Management Service Response Time (ms).

 

    • All the targets monitored by the Agent, including the Agent itself are not shown in the correct status in the Grid Console.
    • Many pages related to this Agent's monitored targets, in the Grid Console return errors such as:

Error getting data for target <Host target name>. Exception: No route to host

OR

oracle.sysman.emSDK.emd.comm.CommException: UnknownHostException sending request:: agentmachine.domain:agentmachine.domain

 

    • Trying to configure a database monitored by the remote Agent from Grid Console results in the error:

oracle.sysman.emSDK.emd.comm.CommException: Cannot establish proxy connection: 302 Moved Temporarily

OR

oracle.sysman.emSDK.emd.comm.CommException: javax.net.ssl.SSLException: SSL handshake failed: SSLIOClosedOverrideGoodbyeKiss

OR

Communication from the Oracle Management Service host to the Agent host failed. Refer to help for details. IOException in sending Request :: Connection refused: connect

 

    • Trying to access any of the database target pages or testing the preferred credentials could return:

Io exception: The Network Adapter could not establish the connection

 

    • Any EM-level job submitted against the targets monitored by the agent machine hangs in 'Running' status. The <OMS_HOME>/sysman/log/emoms.trc shows errors such as:

2007-06-27 15:08:08,841 [JobWorker 4046:Thread-30] INFO em.jobs executeCommand.526 - remoteOp:
oracle.sysman.emSDK.emd.comm.CommException: Agent HTTP version HTTP/1.0 not HTTP/1.1, can't send 0
byte streaming request.
at oracle.sysman.emSDK.emd.comm.EMDClient.getStreamResponse_
(EMDClient.java:2986)
at oracle.sysman.emSDK.emd.comm.EMDClient.getStreamResponse_
(EMDClient.java:2870)
....


This affects operations such as cloning, creating a standby database etc.

For troubleshooting steps, refer to Note 1088414.1: How to Troubleshoot Communication Between the Oracle Management Service (OMS) and Grid Agent Components in 10g Enterprise Manager Grid Control

To find documents related to communication problems between the OMS and the Grid Agent, login to My Oracle Support portal and query the 'Knowledge' with the following keywords:

OMS to Agent Communication Fails <actual error message seen>

Some Examples:

OMS to Agent Communication Fails 'IOException in sending Request :: Broken pipe'
OMS to Agent Communication Fails 'Cannot Establish Proxy Connection'


  • Grid Agent to the Oracle Management Service (OMS)

    Incorrect communication setup between the Grid Agent and the OMS Components can result in the following problems:
    • The command
      <AGENT_HOME>/bin/emctl upload
      does not return message "EMD upload completed successfully".
    • The <AGENT_HOME>/sysman/log/emagent.trc will have errors such as:

Thread-3071241136 WARN http: snmehl_connect: connect failed to (omsmachine.domain:1159): Connection refused (error = 111)

Thread-3071241136 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://omsmachine.domain:1159/em/upload: retStatus=-32>.

 

    • Trying to secure the agent fails with below errors in the <AGENT_HOME>/sysman/log/secure.log:

[main] ERROR sec.GetWallet - caught Exception java.lang.Exception: szOmsSeed
exception java.net.ConnectException: Connection refused
java.lang.Exception: szOmsSeed exception java.net.ConnectException: Connection refused at oracle.sysman.eml.sec.emd.GetWallet.secureAgentWithOMS(GetWallet.
java:282)
at oracle.sysman.eml.sec.emd.GetWallet.<init>(GetWallet.java:95)
at oracle.sysman.eml.sec.emd.GetWallet.main(GetWallet.java:385)
[25-02-2009 13:33:53] USERINFO :: Failed.
[25-02-2009 13:33:53] USERINFO ::Failed to contact the OMS at the HTTP URL set
in /u01/app/oracle/product/10.2.0/agent10g/sysman/config/emd.properties

 

For troubleshooting steps, refer to Note 951076.1: How to Troubleshoot Communication Between the Grid Agent and Oracle Management Service (OMS) Components in 10g Enterprise Manager Grid Control

To find documents related to communication problems between the Agent and the OMS, login to My Oracle Support portal and query the 'Knowledge' with the following keywords:

OMS to Agent Communication Fails <actual error message seen in the emagent.trc file>

Some Examples:

Agent to OMS Communication Fails with "Common Name = "omshost.domain" Does not Match Hostname = "omshost" " in the emagent.trc
Agent to OMS Communication Fails if the Agent's REPOSITORY_URL Parameter has Incorrect Value

  • Searching My Oracle Support Documents for EM Communication 

    As the search is specific to Enterprise Manager Grid Control issues, we recommend that the search be performed only under the Grid Control section, using the following navigation:

    Login to My Oracle Support then Click Knowledge -> Enterprise Management -> Enterprise Manager Consoles - Packs - and Plugins -> Enterprise Manager Grid Control ->All of Enterprise Manager Grid Control.
  • Using RDA and EMDiagkit for troubleshooting EM Communication
    • RDA output generated with the GridControl profile - is useful for the following:
      • Obtaining the Hostnames and IP addresses of the machines where the EM components are running.
      • Performing a network ping test to one or more remote machines including the Grid console machine, Repository Database machine to test their accessibility:

Do you want RDA to perform the network ping tests (Y/N)?
Hit 'Return' to accept the default (N)
> Y
Enter the name of the node the script is running on (used for ping tests)
Hit 'Return' to accept the default (myomshost)
> [Return]
Enter a remote node connecting to this server (used for ping tests)
> myremotenote.fr.oracle.com
Enter the node hosting the database instance (used for ping tests)
Hit 'Return' to accept the default (myomhost)
> [Return]
Enter the node where the Web Server/Forms server is on (used for ping tests)
> [Return]
Enter a typical client node connecting to the database (used for ping tests)
> mylaptop.fr.oracle.com


The results can be seen in the RDA output under Network -> Network Performance

      • Performing a network ping test to one or more Agent machines to test their accessibility:

Do you want to perform agent ping test (Y/N)?
Hit 'Return' to accept the default (N)
> Y
You can enter multiple host names one by one and finish the list by entering a . at last. Enter an agent host name or . to end.
> myclienthost.fr.oracle.com
You can enter multiple host names one by one and finish the list by entering a . at last. Enter an agent host name or . to end.
> .
Enter the Oracle Enterprise Manager Grid Control Agent home directory location
Hit 'Return' to accept the default (C:\app\oracle\agent10g)


The results can be seen in the RDA output under Enterprise Manager Server -> Agent Ping Test 

      • Collection of the OMS log/trace files which will show any connectivity related errors to the Repository Database and/or Agent
    • RDA output generated with the AGT profile - is useful for the following:
      • Obtaining the Hostnames and IP addresses of the machines where the EM components are running.
      • Performing a network ping test to the OMS machine to test its accessibility. 
        The result can be seen in the RDA output under EM Agent (agentmachine.domain:1830) -> OMS Ping Test
      • Collection of the OMS log/trace files which will show any connectivity related errors to the OMS.
    • EMDiagkit output - is useful in obtaining a list of agents which are unable to upload. The Agent Upload failure could actually be due to a communication failure.


********************************************************************************

Troubleshooting Data Upload Issues in Grid Control

For the Data Upload to work fine from the Agent to OMS and OMS to Repository, the Communication between these components should be working fine. Immediately after starting up, the Agent will send an initial heartbeat, which the OMS must respond to by giving the version and protocol definitions of how to transfer the files. The Agent also attempts to pingthe OMS Upload URL, before the Upload attempt.

  • Agent to OMS Upload Problems

    If the Agent is unable to upload its data to the OMS, the following symptoms will be seen:
    • The command  <AGENT_HOME>/bin/emctl upload

      does not return message "EMD upload completed successfully".
      but returns errors such as 

EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet..

OR

upload failed with ret=-10


The 'emctl status agent' output shows a large number for:

Number of XML files pending upload 
Size of XML files pending upload(MB)

and is increasing gradually.

    • The <AGENT_HOME>/sysman/log/emagent.trc will have errors such as:

Thread-3071241136 WARN http: snmehl_connect: connect failed to (omsmachine.domain:1159): Connection refused (error = 111)

Thread-3071241136 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://omsmachine.domain:1159/em/upload: retStatus=-32>.

 

    • Though the Grid Agent has discovered all the targets in its machine, none of these targets are seen in the Grid Console.
    • If the targets are all discovered in the console, the status of all the targets in the Console are incorrect or shown as 'Agent Unreachable'.
    • Even if there is a problem in one of the managed targets at the Agent-side, for example the Database has crashed, no Metric Alert is raised in the Grid Console.

For troubleshooting steps, refer to Note 550617.1: How To Effectively Investigate & Diagnose 10g Grid Agent Upload Problems to the Oracle Management Service (OMS).

To find documents related to Upload problems from the Agent to the OMS, login to My Oracle Support portal and query the 'Knowledge' with the following keywords:

Agent Upload to the OMS <symptom seen or error message>

Some examples:

Agent Upload to OMS Works Only Once 24 Hours or when Forced Manually
Agent Upload to OMS Fails if the OMS is Locked in Secure Mode and Agent is not Secured


  • OMS to Repository Database Upload Problems

    If the Agent is able to upload files to the OMS but the files are being rejected due to a problem at the OMS / Repository, the <AGENT_HOME>/sysman/log/emagent.trc file will show:

2007-04-09 16:58:28 Thread-96 WARN upload: FxferSend: received http error in header from repository: http://omsmachine.domain:4889/em/upload/
ERROR-400|ORA-04091: table SYSMAN.MGMT_CREDENTIALS2 is mutating, trigger/function may not see it


The high-lighted part indicates that the error in the Agent trace file is actually the feedback received from the OMS / Repository when the file upload was attempted. Hence, the investigation should also include the OMS / Repository components.

An OMS-side Upload / Loader problem can have the following symptoms:

    • The Loader Backlog graph in the Grid Console Setup -> Management Services and Repository -> Overview page, is growling linearly. If the Loader backlog is high and Loader output is low, there is data pending load, which indicates a system bottleneck. 
    • The Grid Console Setup -> Management Services and Repository -> Management Services page shows a large number in the 'Files Pending Load' column for the OMS(s).
    • There are errors related to the LOADER component in the Grid Console Setup -> Management Services and Repository -> Errors page.
    • The <OMS_HOME>/sysman/log/emoms.trc or emoms.log show errors related to the XMLLoader thread.
    • The 'emctl upload' from the Agents on all the target machines will fail with errors. As indicated above, errors will also be seen in the <AGENT_HOME>/sysman/log/emagent.trc for the upload operation.
    • As the xml files being sent by the Agents are not getting into the Repository Database, the Grid Console shows stale data - Metric data is not updated, Target status is not current, no new Metric Alerts raised, etc

For troubleshooting steps, refer to Note 285384.1: How To Effectively Investigate & Diagnose 10g Oracle Management Service (OMS) Upload Problems into the Repository.

To find documents related to Upload problems from the OMS to the Repository Database, login to My Oracle Support portal and query the 'Knowledge' with the following keywords:

OMS Upload to Repository <symptom seen or error message>

Some examples:

OMS Upload to Repository Fails with ORA-14400 due to Failing DBMS_JOBS in the Repository Database 
OMS Upload to Repository Hangs due to Self deadlock in the Repository Database

  • Searching My Oracle Support Documents for EM Data Upload

    As the search is specific to Enterprise Manager Grid Control issues, we recommend that the search be performed only under the Grid Control section, using the following navigation:

    Login to My Oracle Support then Click Knowledge -> Enterprise Management -> Enterprise Manager Consoles - Packs - and Plugins -> Enterprise Manager Grid Control ->All of Enterprise Manager Grid Control.
  • Using RDA and EMDiagkit for troubleshooting EM Data Upload
    • RDA output generated with the GridControl profile - is useful for the following:
      • Collecting all data to check the communication between the OMS and Repository, which is needed for ensuring that the Loader component is able to connect to the repository database.
      • Collection of the OMS log/trace files which will show the upload-related errors.
      • Collection of the earlier run EMDiagkit outputs.
    • RDA output generated with the AGT profile - are useful for the following:
      • Collecting all data for check the communication between the Agent and OMS
      • Collection of the Agent log/trace files which will show the upload-related errors.
    • EMDiagkit output - is useful in obtaining a list of Loader-related problems in the Repository and Agents that are not uploading. This will help in checking if there is an upload problem only at the Agent side on certain machines or at the OMS / Repository database side.




********************************************************************************

Best Practices (Certification, Maintenance Activities, OCM, Healthcheck, CPU & PSU)

This section lists some of the best practices which will help prevent problems with EM Communication and Data Upload:

EM Certification Checker

It is strongly recommended that you always use a certified combination of OMS, Agent and Repository Database for managing Targets which are certified with this combination.
The Enterprise Manager certification details are available in:

Note 412431.1: Oracle Enterprise Manager 10g Grid Control Certification Checker

Maintenance Activities

  • Execute EMDiagKit at regular intervals (once per week or more frequently, depending on your setup) and check for any new problems that are reported.
  • When the Loader is performing well without any issues, take a dump of the loader using the EMDiag command: repvfy dump loader
    Keep the output as a baseline to compare the loader dump when the preformance goes bad, to quickly identify if it is related to reciever / upload / database related issue. As each Grid Control deployment has its own IO, cpu etc, the time taken to upload each file needs to be baselined using the EMDiagkit. 
  • Take valid backups of the Agent, OMS and Repository Database Homes at regular intervals, to restore back any configuration files that are deleted by accident.
    For a 10.2.0.5 OMS, the 'emctl exportconfig oms' command can be used to backup the necessary OMS configuration details. Refer to the details in
    Oracle Enterprise Manager Administration 10g Release 5 (10.2.0.5), Chapter - 9 Backup, Recovery, and Disaster Recovery. Topic : 
    OMS Backup and Recovery 
  • Before Installing any new Grid Control components or re-locating them to other machines, ensure that the communication between these machines are fine. Refer to
    Note 763844.1: How to Verify the Hostname/IP Address Resolution Between the 10g Enterprise Manager Grid Control Components? 
  • Regularly monitor the Loader backlog shown in the Grid Console Setup -> Management Services and Repository -> Overview page.
  • Plan to execute on a regular basis, the tasks described in the

    1. White Paper:
    Enterprise Manager Grid Control Performance Best Practices (page 12/30).
    2. The EM Documentation:
        For Grid Control 10.1.x.x.x Release, refer 
    Oracle Enterprise Manager Advanced Configuration
        For Grid Control 10.2.x.x.x Release, refer Oracle Enterprise Manager Advanced Configuration

  • If using 10.2.0.5 OMS, refer to Note 853691.1: ALERT: Important Upgrade Steps Required for Enterprise Manager Grid Control 10gR5 (10.2.0.5) Upgrades, for list of important patches that need to be applied to the OMS / Agent.



OCM

Oracle Configuration Manager (OCM) works with My Oracle Support to enable proactive support capability that helps you organize, collect and manage your Oracle configurations by providing Proactive configuration-specific notification of Security and General Alerts, HealthCheck recommendations based on Support Best practices when using configuration auto-collection, Simplified Service Request logging, tracking and reporting and Project cataloging of key milestones and contacts associated with your configurations.

  • Among these the following topics are related to the Enterprise Manager Components: 
    • 2.52 Oracle Enterprise Manager 10g Grid Control Management Agent:
    • 2.54 Oracle Enterprise Manager 10g Grid Control Management Service
    • 2.53 Oracle Enterprise Manager 10g Grid Control Management Repository
    • 2.72 Oracle Grid Control Repository (for oracle_emrep target)
    • 2.38 Oracle Agent Deployment Configuration (oracle_emd target)
    • 2.73 Oracle Home
    • 2.23 Host

Note: The above list is expected to be expanded as and when new collections are introduced in future.

  • It is also advisable to review the collections available for the Database instance, so that the Database hosting the repository can be monitored as well:
    • 2.10 Database Instance
    • 2.78 Oracle Listener

Healthcheck

Healthchecks are executed dynamically against the Oracle Configuration Manager uploaded configurations in My Oracle Support. These checks, based on Oracle Best practices, will proactively notify you of potential problems in your environment, and provide recommendations that help you improve system performance and avoid problems in your Oracle environment. 

  • If you are receiving any Healthcheck alerts in My Oracle support, then refer to the following document for the alert details and its corresponding document for resolving the same:

Note 868955.1: My Oracle Support Health Checks Catalog

  • For Healthchecks specific to the Enterprise Manager and Repository Database, refer to the sections titled:
    • Enterprise Manager (for the OMS)
    • Oracle Database (for the Database hosting the Repository)
  • Some examples of Enterprise Manager specific healthchecks which will alert potential Communication / Upload issues are:
    • job_queue_processes May Be Set Too Low
      To resolve, refer 
      Note 957560.1: Health Check Alert: JOB_QUEUE_PROCESSES is set to a value lower than 10 
    • UNDO_TABLESPACE is NULL
      To resolve, refer 
      Note 957569.1: Health Check Alert: UNDO_TABLESPACE is NULL




CPU and PSU

  • CPU

    Critical Patch Updates (CPU) is the primary means of releasing security fixes for Oracle products. They are released on the Tuesday closest to the 15th day of January, April, July and October. This page lists all the currently available Critical Patch Updates (CPUs) in chronological order and is updated whenever new Critical Patch is released. You can also subscribe to the CPU Email Alerts using the steps listed here.

    To obtain the latest CPU patch details for the Enterprise Manager Grid Control and its dependent products - Oracle Application Server and Oracle Database:

    - In the 
    page, click on the link shown for the latest CPU in the table under the 'Critical Patch Updates'.
    - The next page, lists all the products which have security fixes in the chosen CPU release. Scroll down to 'Patch Availability Table ..' topic and find the table with details for the Product Group and Patch Availability and Installation Information. 
    - In the table, find the row related to Product Group: 'Oracle Enterprise Manager' and pick up the document number given in the Patch Availability and Installation Information column. In the document, navigate to: 

                 "Critical Patch Update Availability for Oracle Products" and then to
                 "Oracle Enterprise Manager Grid Control"
  • PSU

    Patch Set Updates (PSU) are proactive cumulative patches containing recommended bug fixes that are released on a regular and predictable schedule. PSUs are on the same quarterly schedule as the Critical Patch Updates (CPU), specifically the Tuesday closest to the 15th of January, April, July, and October. The PSUs serve as a new baseline version for reporting issues to Oracle, hence it is always recommended to be on the latest PSU release.
    • For more details on PSU, refer Note 854428.1: Patch Set Updates for Oracle Products 
    • For Enterprise Manager specific PSU, refer Note 822485.1: Oracle Recommended Patches -- Oracle Enterprise Manager
  • Choosing between CPU / PSU patches 

    The PSU and CPU released each quarter contain the same security content. However, the patches employ different patching mechanisms, so customers need to choose wisely which patch satisfies their needs better:
    • A PSU can be applied on the CPU released at the same time or on an any earlier CPU for the base release version. A PSU can be applied on any earlier PSU or the base release version. CPUs are only created on the base release version. 
    • Once a PSU has been installed, the recommended way to get future security content is to apply subsequent PSUs. Reverting from PSU back to CPU, while possible, would require significant effort, and so is not advised. 
  • Getting CPU / PSU patch recommendations via OCM 

    OCM also collects and recommends the latest CPU and PSU patch that can be applied to a particular Oracle Home. These details can be seen in the My Oracle Support ->Patches and Updates -> Patch Recommendations section 
    - 'Security' patch recommendations include the CPU patches.
    - 'Other Recommendations' include the PSU patches.

References

NOTE:1081865.1 - Master Note for 10g Grid Control OMS Process Control (Start, Stop and Status) & Configuration
NOTE:1082009.1 - Master Note for 10g Grid Control Agent Process Control (Start, Stop & Status) & Configuration
NOTE:1087997.1 - Master Note for 10g Enterprise Manager Grid Control Agent Performance & Core Dump issues
NOTE:1092513.1 - Master Note for 10g Enterprise Manager Grid Control Security Framework
NOTE:1098262.1 - Master Note for Diagnostic Tools for 10g Enterprise Manager Grid Control Components
NOTE:1161003.1 - Master Note for 10g Grid Control OMS Performance Issues

Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

News and Troubleshooting tips for Oracle Database and Enterprise Manager

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today