Best Practices, tips and news for managing Oracle’s Engineered Systems for both on-premise and cloud.

Oracle Enterprise Manager 13c and Always-On Monitoring

One of the common concerns of the Administrators of an Oracle Enterprise Manager (EM) environment is that when you have to take EM down for planned  maintenance, you are blind to the status of some of your company’s most critical assets. Well, as of EM 13c, Oracle now has a solution for you! Introducing Always-On Monitoring – or AOM for short. Always-On Monitoring provides the capability of monitoring all or a specific list of targets using the same EM agents already deployed to these targets during complete downtime of the EM environment. It is especially useful for monitoring critical targets during planned EM downtime such as patching and/or upgrades. AOM will synchronize with EM for data such as notification contacts and is able to take the alerts from the EM agents and send email notifications to the identified downtime contacts. AOM can be configured and left running all the time so that it is ready to “take over” the alert notification for critical targets once notifications have been enabled. This blog contains the first of a couple of posts on AOM starting with the setup and configuration of the tool.

The setup/configuration of the AOM tool consists of the following steps:

1.  AOM Installation - The AOM application is included with the EM 13c software distribution under the sysman/ems directory and is also available via the Self-Update function in EM. The installation of the application is as simple as unzipping the zip file in the location chosen for the AOM install. An AOM installation consists of a database to hold the AOM repository, and a server to run the AOM Instance. Chapter 12 of the Oracle Enterprise Manager Cloud Control Administrator’s Guide contains the details steps required for installing AOM. This server must have JAVA version 1.7 installed. At this time, the AOM instance/application must run on the EM OMS but the AOM repository database can be created on any Oracle database server.  With the EM 13.2 release, the AOM instance/application will be able to be installed on any host within the monitored environment.

2.  Configure AOM Communication to EM - The configuration is pretty simple and is all handled via the script called emsca found under the AOM installation home in the scripts directory. (Fun Fact: the name of this script and the command line utility all start with ems rather than aom due to a pre-release name change of the product). This script will handle all of the necessary steps to configure AOM including the creation of the user and schema for the AOM  repository. Once this configuration is complete, make sure that EM has an Email Server configured. At this time, AOM can only send email messages for the alerts and it uses the email server configuration that it gets from EM during synchronization. See "Configuring Email Servers in Enterprise Manager" for more details on how to setup Email Servers in EM.

3.  Configuration of downtime contacts - The downtime contacts are those that will receive the email notifications in the event of an alert on the identified  targets. Setting up the downtime contacts is done in your EM environment and can be done in a couple of different ways and is documented in the section
Configuring Downtime Contacts in Enterprise Manager”.

4.  Synchronizing AOM and EM - This synchronization copies over the notification configuration and downtime contacts for the EM targets over to the AOM application. This should be done before starting AOM for the first time. It is a simple command using the AOM command line utility called emsctl. Here is a sample of the command:

% $AOM_HOME/scripts/emsctl sync

5.  Note that AOM will run a synchronization job every 24 hours by default although this is configurable. It is a good practice to run a manual
sync after any changes to downtime contacts.

6.  Configure EM for AOM - The final step in the configuration process is to tell EM to include the upload URL for the AOM service. Once this is done, EM will  send this URL to all existing agents and any future agents. This makes this a one time step that will then be handled for you as your number of agents change in the future. To get the AOM upload URL, issue the command below (the upload URL is in bold below):

% $AOM_HOME/scripts/emsctl status

Oracle Enterprise Manager Cloud Control 13c Release 1

Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.


Always-On Monitoring Version:

Always-On Monitoring Home:/u01/app/oracle/aom

Started At: January 13, 2016 4:11:01 PM PST

Last Repository Sync: February 2, 2016 1:41:17 PM PST

Upload URL: https://aomserver.domain:8081/upload

Always-On Monitoring Process ID: 15399

Always-On Monitoring Repository: (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=aomRepo.domain)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=aom)))

Enterprise Manager Repository: (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=emServer.domain)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emdb)))

Notifications Enabled: false

Total Downtime Contacts Configured: 29

Then issue the following emctl command to set the property in EM (note this is emctl and not emsctl) – substituting the information in bold below with the correct upoad url and sysman password for your environment:

% emctl set property -name "oracle.sysman.core.events.ems.emsURL" -value "https://aomserver.domain:8081/upload" -sysman_pwd sysman

7.  Starting AOM and Enabling notifications - Starting AOM is done via the emsctl command and specifying the start option:

%$AOM_HOME/scripts/emsctl start

Even though the setup of AOM is complete and it has been started, you will not receive any alerts when EM is down unless you enable notifications. As you probably guessed by now, this is done via the emsctl command:

%$AOM_HOME/scripts/emsctl enable_notification

A sync with EM is done each time you enable_notification unless you use the option –nosync. To disable the notification, use the same sort of command with the disable_notification option:

%$AOM_HOME/scripts/emsctl disable_notification

It is a good practice to leave AOM started but make sure you disable the notifications. Only enable them just before you shutdown EM for planned  maintenance. REMEMBER that you will get more notification than you probably want if you leave this enabled all of the time because AOM has no knowledge of the notifications that EM is already doing.

To summarize, Always-On Monitoring offers a solution to the long-time problem of who monitors our critical targets when EM is down. The next blog will provide the details on how to configure AOM for High Availability and Disaster Recovery.

Join the discussion

Comments ( 4 )
  • Nishant Thursday, January 12, 2017

    Does OAM also monitors when EM repository database is down ?

  • Bethany Lapaglia Thursday, January 12, 2017

    Always On Monitoring will send agent alerts for any target that it is configured to monitor. If EM is not down, then you may get duplicate alerts from EM and AOM. AOM is configurable as to which targets it monitors. AOM is designed to provide monitoring of key targets in your environment when EM is down. The targets that it will monitor have to be configured by you.

  • Keyur Friday, June 16, 2017
    Does all agents upload data to both OEM and AOM all the time? Or only alert data are uploaded to AOM (and not any other data)?
    And how it work with already defined Incident Rule?
  • Bethany Lapaglia Friday, June 16, 2017
    The EM agents will forward alerts for all targets on that machine to AOM. AOM will send email notifications only for targets that have been configured with downtime contacts Be aware that if using a global email contact, target status alerts and metric alerts as well as metric collection errors will be sent for all targets. The alerts are still "detected" by the EM agent and uploaded to both EM and AOM. This means that the agent supports two upload URL's - one for EM and one for AOM. AOM does not know about the event rules in EM. AOM simply receives alerts that the agent is detecting based on the metrics defined in EM for that target. You can however configure your downtime contacts in AOM based on what you have defined in your EM event rules if you want. This is described in the documentation.

    When EM is up, keep in mind that you will get double alerts if you have notifications turned on for AOM. The best practice recommendation is to keep AOM running all the time but to turn on the notifications during EM downtime. This has the benefit of AOM staying in sync with EM as a synchronization is done on a scheduled basis.
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.