Oracle Support Master Note for Target Maintenance Through 10g Enterprise Manager Grid Control

For most current information refer Master Note for Target Maintenance Through 10g Enterprise Manager Grid Control (Doc ID 1202453.1)

 

Master Note for Target Maintenance Through 10g Enterprise Manager Grid Control (Doc ID 1202453.1)

 

In this Document
  Purpose
  Scope and Application
  Master Note for Target Maintenance Through 10g Enterprise Manager Grid Control
     Concepts Related to Grid Control Targets
     Diagnostic Tools Available for Troubleshooting Target Maintenance Issues
     Troubleshooting Target Maintenance Issues
     Best Practices (Certification, Maintenance Activities, OCM, Healthcheck, CPU & PSU)
  References


Applies to:

Enterprise Manager Grid Control - Version: 10.1.0.2 to 10.2.0.5 - Release: 10.1 to 10.2
Information in this document applies to any platform.

Purpose

This Master Note helps understand 10g Grid Control Target Maintenance and provides assistance in using diagnostics effectively to debug/troubleshoot and resolve issues encountered.

Scope and Application

This document is intended to assist Enterprise Manager Grid Control Administrators effectively troubleshoot Target Maintenance Issues in the 10g Grid Control. This document covers the following topics:

1. Concepts related to Grid Control Targets
2. Diagnostic Tools Available for Troubleshooting Target Maintenance Issues
3. Troubleshooting Target Maintenance Issues
4. Best Practices (Maintenance Activities, OCM, Healthcheck, CPU & PSU and Certification)

Master Note for Target Maintenance Through 10g Enterprise Manager Grid Control

Concepts Related to Grid Control Targets

A Grid Control Target can be broadly defined as a single entity / component that can be monitored, managed and configured via the Enterprise Manager Grid Control. For this purpose an Management Agent must be installed on the host where the target resides. Examples of a target include:
- A Linux host computer
- An Oracle9i Application Server or an instance of Oracle HTTP Server
- A 10.2.0.5 Database
- A Web Application, etc

For a list of certified targets, refer to Note 412431.1: Oracle Enterprise Manager 10g Grid Control Certification Checker

1. Key Characteristics of a Target

  • Target Type: Represents a class of managed entities / Targets, which can be monitored by the Grid Control. Each of the supported target types have an internal name assigned to uniquely identify them. Some examples: host, oracle_database, oracle_ias, oracle_listener, etc.
  • Target Name: Name of the target as displayed in the Grid Console. In most cases, this is same as the name of the target identified by the Agent during the discovery but it is possible to provide a different target name during manual discovery of a target as described in Note 755630.1: Grid Control Target Maintenance: How to Change the Display Name of Targets in Grid Console?
  • TARGET_GUID: Based on the Target Name, the Repository assigns a unique identifier to the Target. This TARGET_GUID is used for identifying the target when performing any target-related operation from the Grid Console. For this purpose, no two targets in the Grid Control can have the same target name. The second target being added with the same name will be marked as a duplicate target.
  • Monitoring Configuration and Target Properties: Properties using which the Target can be identified and accessed from the Grid Console.
    • Monitoring Configuration includes details such as Target Name, Hostname, Connection properties using which the Agent can connect to the Target, ORACLE_HOME, etc. These details are saved by the Agent in the targets.xml file. For targets such as Databases, Application Server, Listener etc these details can also be viewed in the Grid Console by clicking on the 'Monitoring Configuration' link in the Target Homepage.A Host target is configured automatically, hence no properties can be modified.
    • User-defined Target Properties are optional descriptive attributes that can be associated with the target, which can be used for searching, filtering, notifications or classification purposes. For more details, refer to Note 750070.1: Grid Control Target Maintenance: How to Add and Populate Target Properties in the Grid Console Targets Pages
  • Metadata and Default Collections: Metadata is "Data that describes data". With respect to a Target, Metadata defines the target type and the methods used to monitor it. This includes:
    • Target-Type Metadata: is read-only definition of a Target Type. Each target type has its own metadata, which is defined in <target_type>.xml file under the <AGENT_HOME>/sysman/admin/metadata directory in each Agent installation. Each file defines properties of the target type, metrics for this target type and the method used to collect the metrics.
    • Target Specific Metadata (dynamic properties): The properties of the target itself, for example: DBName, VersionCategory, OpenMode etc for a Database Target. The method to collect these properties is also defined in the <target_type>.xml files under the <AGENT_HOME>/sysman/admin/metadata directory. The Target Manager component of the Agent computes these properties at the time of the Agent startup.
      Note 1101615.1: Description of Important Components / Threads in a 10g Enterprise Manager Grid Control Agent, Topic: Target Manager
    • Default Collections: Associated with the metadata of a target type and defines the collection schedule for the pre-defined list of metrics. These are defined in a <target_type>.xml file in the <AGENT_HOME>/sysman/admin/default_collection directory in each Agent installation.
  • Availability Rule: Defines how to calculate whether the target is Up / Down. The logic is mostly straight forward for a single target but will involve some calculations based on the status of the child-targets in case of a composite / aggregate target. This is not applicable to some composite targets such as Groups, System.
    The status of a Target could be one of the following: Up, Down, Under Blackout, Unknown (due to Metric Collection Error), Unknown due to Agent Unreachable (Agent Down or Host unreachable), Status Pending.

2. File: targets.xml

The list of targets discovered by an Agent is stored in the targets.xml file. This is located at <AGENT_HOME>/sysman/emd in case of a standalone Agent installation and in
<AGENT_HOME>/<nodename>/sysman/emd in case of a RAC Agent installation.

For details, refer to Note 234872.1: Understanding the Enterprise Manager 10g Grid Control Management Agent, Directory Structure and Key Configuration Files.

For a machine to be seen in the Grid Console, this file should at the minimum have entries for the Host and Agent (oracle_emd) targets. If the file has been lost and a valid backup does not exist, the Host and Agent entries can be added using the steps in
Note 303105.1: Grid Control Target Maintenance: How To Recreate the Targets.xml File in Grid Control 10.1.x
Note 365252.1: Grid Control Target Maintenance: How To Recreate the Targets.xml File in Grid Control 10.2.x Using 'agentca' Commands

Note:
- A valid backup of the targets.xml file should be maintained always.
- The targets.xml file should not be manually modified for any reason unless advised by EM Support / Development. Incorrect edition of this file can corrupt the file and cause the Agent to crash.

3. Target Discovery

A Target can be monitored from the Grid Console only if it has been discovered in the Grid Console. Hence, the Target discovery is necessary for important management operations such as fault detection, configuration and inventory tracking, real-time monitoring, and historical reporting. The discovery mechanisms are described in Note 239224.1: Grid Control Target Maintenance: Overview of Target Discovery in 10g Grid Control

If there are multiple Oracle Central Inventories in the machine, the Agent should be made aware of the required inventories for discovering the Oracle Homes. Steps are described in
Note 292084.1: How to Enable and Troubleshoot Multiple-Inventory Support for the Agent/ECM

4. Target Deletion / Removal

To understand the steps for deleting a Target from the Grid Console, refer to Note 271691.1: Grid Control Target Maintenance: Understanding the Target Deletion / Removal Process in the Enterprise Manager Grid Control

5. Other  Target Concepts

For more details related to a Grid Control Target, refer to Note 1214933.1: Grid Control Target Maintenance: Understanding Concepts Related to Enterprise Manager Grid Control Targets

Back to Top

********************************************************************************

Diagnostic Tools Available for Troubleshooting Target Maintenance Issues

  • EMDiagkit

The EMDiagkit is a diagnostic tool developed to assist in diagnosis and correction of Enterprise Manager 10g Framework issues. At present, the tool allows us to extract necessary troubleshooting data from the EM Repository Schema using the repvfy utility.

The details for installation, usage of EMDiagkit are available in

          Note 421053.1: EMDiagkit Download and Master Index

EMDiagkit (repvfy) also has options for obtaining the target details and its availability from the Repository. These options are described in:
Note 399899.1: Grid Control Target Maintenance: Troubleshooting Script for Target Availability in Enterprise Manager Grid Control
Note 1228803.1: Grid Control Target Maintenance: Script for Target Analysis (dump) in Enterprise Manager Grid Control

  • RDA

    The Remote Diagnostic Agent (RDA) can be executed specifically with the Grid Control / OMS profile name: GridControl, Agent profile name: AGT and the Database profile name: DB10g / DB11g in order to reduce the number of questions that need to be answered and also to collect all details of the OMS / Agent / Database Homes correctly.

    The steps to execute the RDA with GridControl and profiles are explained in:

               Note 1057051.1: How to Run the RDA against a Grid Control Installation

    It is highly recommended that the latest EMDiagkit is installed and executed in the OMS home, before running the RDA.
  • Other tools and options are described in Note 1098262.1: Master Note for Diagnostic Tools for 10g Enterprise Manager Grid Control Components

Back to Top

********************************************************************************

Troubleshooting Target Maintenance Issues

  • Problems During Target Discovery

    Discovery problems can be classified as:
    • Permissions / Formatting issues of the targets.xml file: The <AGENT_HOME>/sysman/log/emagent.trc shows errors such as

      ...oracle.sysman.emSDK.emd.comm.OperationException: Could not save target to targets.xml - Could not save target to targets.xml

      For resolving this error, refer to Note 745448.1:Grid Control Target Maintenance: Trying to Perform Target Discovery Fails with "Could not save target to targets.xml"

      The targets.xml file should not be modified manually unless advised by Support / Development. Incorrect modification of this file can corrupt the file. For example:

      Note 945002.1: Grid Control Target Maintenance: Trying to Manually Discover a Database Results in "LPX-00245: extra data after end of document"
    • Incorrect Format for the Hostname: The <hostname> of the machine is referenced for the entries of many other targets in the targets.xml file. If the hostname in various target entries do not match with each other, then trying to access the target details from the Grid Console can return errors such as:
      Error finding target <target name> from the repository. The target does not exist or you may not have access to the target.

      For details, refer to Note 752693.1: Grid Control Target Maintenance: Acessing the Target Reports "Error finding target from the repository. The target does not exist or you may not have the access to the target"
    • Multiple Agents on a Machine: If multiple Agents are installed on a machine, then the host target will be discovered by all the Agents and will be treated as a duplicate one in the Grid Console. For details, refer to Note 363062.1: Grid Control Target Maintenance: Host Target Does Not Appear in Grid Control After Agent Is Removed From Grid Control
    • Discovery of Targets in Oracle Homes, installed as different OS users (in Unix/Linux): If the Agent is installed as a OS User different from the OS user who has installed the other Oracle Home's in a machine, then the Agent should have sufficient permissions to read the required files in the other Oracle homes for the target discovery.

      Some of the issues face for certain target discoveries are described in the below documents:
      Note 437078.1: Problem iAS Discovery/Monitoring Fails If Agent Is Installed Under Another OS User Than iAS
      Note 787571.1: E-PSEM How to Setup PeopleSoft Plug-in With Installations Done by Different Users ?
      Note 371539.1: Grid Control Target Maintenance: Agent Does Not Discover Targets in ORACLE_HOME's Installed by Other OS Users
    • Re-discovery Failing due to Incomplete Deletion: If you remove a target and try to add it again with the same name before the deletion is complete, you will get error such as:
      java.sql.SQLException: ORA-20600: The specified target is in the process of being deleted.
      You must wait till the deletion is complete to re-use the same target name or can manually discover the target using the steps in Note 417690.1: Grid Control Target Maintenance: How To Manually Discover a Target in the Grid Console?
    • Adding Targets to Groups During Manual Discovery:

      A target can get added to a Group automatically at the time of manual discovery. This feature however adds the target to all available groups. This issue has been described in Note 846470.1: Grid Control Target Maintenance: Manual Discovery Automatically Adds Targets to all Groups, NO Option to Choose a Particular Group.

Back to Top

  • Problems During Target Deletion

    For diagnosing targets pending in deletion, refer to Note 271714.1: Grid Control Target Maintenance: How to Diagnose Target Removal issues in Enterprise Manager Grid Control
  • Troubleshooting Target Availability
    • 'Agent Unreachable' Status: A target will be shown in the 'Agent Unreachable' status if the Agent monitoring this target is not reachable. There are several reasons why the Management Agent will show a status of "Agent Unreachable":

          1. The Agent is not running.
          2. The Agent cannot resolve the OMS hostname after the initial successful heartbeat.
          3. The Agent is running and has files to upload but cannot upload files to the OMS.
          4. The OMS has been locked down to receive only HTTPS connections from the Agents
              but this particular Agent is not configured for HTTPS communications, etc.

      For troubleshooting such issues, refer to Note 271126.1: Grid Control Target Maintenance: Steps to Diagnose Issues Related to "Agent Unreachable" Status
    • Incorrect Target Status: If a target is running but is not shown as Up in the Grid Console, this will affect monitoring of this target from the console as none of the metrics will be collected by the Agent. To debug such issues, refer to Note 730757.1: Grid Control Target Maintenance: Troubleshooting Incorrect Target Status Issues in Grid Control

 

  • Host Preferred Credentials

    To troubleshoot errors when setting or using a feature that makes use of the Host Preferred credentials, refer to Note.757425.1: Troubleshooting Host Preferred Credentials
  • Searching My Oracle Support Documents for Target Maintenance Issues

    As the search is specific to Enterprise Manager Grid Control issues, we recommend that the search be performed only under the Grid Control section, using the following navigation:

    Login to My Oracle Support then Click Knowledge -> Enterprise Management -> Enterprise Manager Consoles - Packs - and Plugins -> Enterprise Manager Grid Control ->All of Enterprise Manager Grid Control.

    To find documents related to Target Maintenance in Grid Control, login to My Oracle Support portal and query the 'Knowledge' with the following keywords:

    Grid Control Target Maintenance: <actual issue or error message seen>

    Some examples:

    Grid Control Target Maintenance: Steps to Remove an Orphaned Target Using the EMDiag Kit
    Grid Control Target Maintenance: How to Manage Duplicate Targets in the Grid Console?
  • Using RDA and EMDiagkit for troubleshooting Target Maintenance Issues
    • The RDA output generated with the GridControl profile is very useful in obtaining all the configuration files and log/trace files together. These can assist in identifying if a problem is specific to the Console UI or other components like the OMS / Repository.
    •  The EMDiagkit (repvfy) output is very useful in diagnosing problems / mis-configurations with Grid Control Repository objects, which can result in errors when maintaining the Targets.
       In addition, repvfy provides options for dumping the target's configuration and availability details.


Note: It is highly recommended that the latest EMDiagkit (repvfy) is installed and executed in the OMS home, before running the RDA. This will ensure that the RDA picks up the latest data collected by the EMDiagkit.

Back to Top

********************************************************************************

Best Practices (Certification, Maintenance Activities, OCM, Healthcheck, CPU & PSU)

This section lists some of the best practices which will help prevent problems with Target Maintenance.

EM Certification Checker

It is strongly recommended that you always use a certified combination of OMS, Agent and Repository Database for managing Targets which are certified with this combination.
The Enterprise Manager certification details are available in:

Note 412431.1: Oracle Enterprise Manager 10g Grid Control Certification Checker

Maintenance Activities

  • Execute EMDiagKit at regular intervals (once per week or more frequently, depending on your setup) and check for any new problems that are reported for the OMS / Repository operations.
  • Always maintain a valid backup of the targets.xml file in the Agent machine.
  • The targets.xml file should not be manually modified for any reason unless advised by EM Support / Development. Incorrect edition of this file can corrupt the file and cause the Agent to crash.
  • The Discovery attempted by an Agent is always a "Best-effort approach" i.e the Agent tries all possible methods defined in the perl scripts to discover the targets in the machine. If a particular target is still not discovered, then it can be manually added to the Grid Console. Refer to
    Note 417690.1: Grid Control Target Maintenance: How To Manually Discover a Target in the Grid Console?
  • 'OMS and Repository' is a special target monitored by the Agent on the OMS machine. When using this Agent in cloning operations to install Agents on other machines, follow the steps in
    Note 579156.1: How to Clone a Management Agent And Known Issues
    Topic: Steps to clone Chained Management Agent (The Agent which is installed as part of the OMS installation is also known as the chained Agent).
    If the options mentioned in the above note are not used correctly, this will result in multiple 'OMS and Repository' targets in the Grid Console, which will affect the details shown for the Management Services and repository pages.
  • Take valid backups of the Agent, OMS and Repository Database Homes at regular intervals, to restore back any configuration files that are deleted by accident.
    For a 10.2.0.5 OMS, the 'emctl exportconfig oms' command can be used to backup the necessary OMS configuration details. Refer to the details in Oracle Enterprise Manager Administration 10g Release 5 (10.2.0.5), Chapter - 9 Backup, Recovery, and Disaster Recovery. Topic : OMS Backup and Recovery.

Back to Top


OCM

Oracle Configuration Manager (OCM) works with My Oracle Support to enable proactive support capability that helps you organize, collect and manage your Oracle configurations by providing Proactive configuration-specific notification of Security and General Alerts, HealthCheck recommendations based on Support Best practices when using configuration auto-collection, Simplified Service Request logging, tracking and reporting and Project cataloging of key milestones and contacts associated with your configurations.

  • Among these the following topics are related to the Enterprise Manager Components:
    • 2.52 Oracle Enterprise Manager 10g Grid Control Management Agent:
    • 2.54 Oracle Enterprise Manager 10g Grid Control Management Service
    • 2.53 Oracle Enterprise Manager 10g Grid Control Management Repository
    • 2.72 Oracle Grid Control Repository (for oracle_emrep target)
    • 2.38 Oracle Agent Deployment Configuration (oracle_emd target)
    • 2.73 Oracle Home
    • 2.23 Host

Note: The above list is expected to be expanded as and when new collections are introduced in future.

  • It is also advisable to review the collections available for the Database instance, so that the Database hosting the repository can be monitored as well:
    • 2.10 Database Instance
    • 2.78 Oracle Listener

Healthcheck

Healthchecks are executed dynamically against the Oracle Configuration Manager uploaded configurations in My Oracle Support. These checks, based on Oracle Best practices, will proactively notify you of potential problems in your environment, and provide recommendations that help you improve system performance and avoid problems in your Oracle environment.

  • If you are receiving any Healthcheck alerts in My Oracle support, then refer to the following document for the alert details and its corresponding document for resolving the same:

Note 868955.1: My Oracle Support Health Checks Catalog

  • For Healthchecks specific to the Enterprise Manager and Repository Database, refer to the sections titled:
    • Enterprise Manager (for the OMS)
    • Oracle Database (for the Database hosting the Repository)

Back to Top


CPU and PSU

  • CPU

    Critical Patch Updates (CPU) is the primary means of releasing security fixes for Oracle products. They are released on the Tuesday closest to the 15th day of January, April, July and October. This page lists all the currently available Critical Patch Updates (CPUs) in chronological order and is updated whenever new Critical Patch is released. You can also subscribe to the CPU Email Alerts using the steps listed here.

    To obtain the latest CPU patch details for the Enterprise Manager Grid Control and its dependent products - Oracle Application Server and Oracle Database:

    - In the page, click on the link shown for the latest CPU in the table under the 'Critical Patch Updates'.
    - The next page, lists all the products which have security fixes in the chosen CPU release. Scroll down to 'Patch Availability Table ..' topic and find the table with details for the Product Group and Patch Availability and Installation Information.
    - In the table, find the row related to Product Group: 'Oracle Enterprise Manager' and pick up the document number given in the Patch Availability and Installation Information column. In the document, navigate to:

                 "Critical Patch Update Availability for Oracle Products" and then to
                 "Oracle Enterprise Manager Grid Control"
  • PSU

    Patch Set Updates (PSU) are proactive cumulative patches containing recommended bug fixes that are released on a regular and predictable schedule. PSUs are on the same quarterly schedule as the Critical Patch Updates (CPU), specifically the Tuesday closest to the 15th of January, April, July, and October. The PSUs serve as a new baseline version for reporting issues to Oracle, hence it is always recommended to be on the latest PSU release.
    • For more details on PSU, refer Note 854428.1: Patch Set Updates for Oracle Products
    • For Enterprise Manager specific PSU, refer Note 822485.1: Oracle Recommended Patches -- Oracle Enterprise Manager
  • Choosing between CPU / PSU patches

    The PSU and CPU released each quarter contain the same security content. However, the patches employ different patching mechanisms, so customers need to choose wisely which patch satisfies their needs better:
    • A PSU can be applied on the CPU released at the same time or on an any earlier CPU for the base release version. A PSU can be applied on any earlier PSU or the base release version. CPUs are only created on the base release version. 
    • Once a PSU has been installed, the recommended way to get future security content is to apply subsequent PSUs. Reverting from PSU back to CPU, while possible, would require significant effort, and so is not advised.
  • Getting CPU / PSU patch recommendations via OCM

    OCM also collects and recommends the latest CPU and PSU patch that can be applied to a particular Oracle Home. These details can be seen in the My Oracle Support ->Patches and Updates -> Patch Recommendations section
    - 'Security' patch recommendations include the CPU patches.
    - 'Other Recommendations' include the PSU patches.

Back to Top

References

NOTE:1081865.1 - Master Note for 10g Grid Control OMS Process Control (Start, Stop and Status) & Configuration
NOTE:1082009.1 - Master Note for 10g Grid Control Agent Process Control (Start, Stop & Status) & Configuration
NOTE:1086343.1 - Master Note for 10g Grid Control Enterprise Manager Communication and Upload issues
NOTE:1087997.1 - Master Note for 10g Enterprise Manager Grid Control Agent Performance & Core Dump issues
NOTE:1092513.1 - Master Note for 10g Enterprise Manager Grid Control Security Framework
NOTE:1098262.1 - Master Note for Diagnostic Tools for 10g Enterprise Manager Grid Control Components
NOTE:1161003.1 - Master Note for 10g Grid Control OMS Performance Issues
NOTE:1190323.1 - Master Note for 10g Grid Console Browser / User-interface Issues

Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

News and Troubleshooting tips for Oracle Database and Enterprise Manager

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today