Friday Jul 22, 2016

Oracle Enterprise Manager 13c and Always-On Monitoring -Part 2

 
   
  

As discussed in an earlier blog post, Always-On Monitoring (AOM) provides the capability of monitoring all (or a specific list of) targets using the same EM agents already deployed to these targets during complete downtime of the EM environment. It is especially useful for monitoring critical targets during planned EM downtime such as patching and/or upgrades. In this previous blog post we reviewed the high-level steps for the setup and configuration of the AOM tool. In this blog post, we will discuss the following topics: configuring High Availability, configuring Disaster Recovery and troubleshooting AOM.

As with any other application, Always-On Monitoring should be configured for High Availability (HA) and Disaster Recovery (DR) according to the Maximum Availability Architecture standards. This will ensure that the identified list of targets that must be monitored during Enterprise Manger downtime will continue to be monitored even if a system failure occurs on one of the AOM instances.

High Availability

Setting up High Availability for AOM is as simple as setting up additional AOM instances. This helps ensure availability of monitoring but also provides load sharing. The incoming alerts from agents can be directed to another AOM instance if one goes down via a server load balancer (SLB). To keep events from a target in the proper order, a single target will send a message to one AOM instance only. If that AOM instance goes down, the application assigns the responsibility of the queues previously managed by the down AOM instance to another instance. Not only are the incoming agent alerts shared by multiple AOM instances, but the work of sending out the notifications can also be shared among the AOM instances.

Adding another AOM instance is as simple as running the following command from the new server: emsca add_ems
Below is a sample of the output from this command:

% $AOM_HOME/scripts/emsca add_ems

Oracle Enterprise Manager Cloud Control 13c Release 1

Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.

---------------------------------------------------------------

Always-On Monitoring Repository Connection String : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=aomRepo.domain)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=aom)))

Always-On Monitoring Repository Username [ems] :

Always-On Monitoring Repository Password [ems] :

Enterprise Manager Repository Connection String : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=emServer.domain)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emdb)))

Enterprise Manager Repository Username : sysman

Enterprise Manager Repository Password :

Connecting to Always-On Monitoring repository.

Enter Enterprise Manager Middleware Home : /u01/app/oracle/MWare

Registering Always-On Monitoring instance

Always-On Monitoring Upload URL: https://aomserver.domain:8081/upload

Oracle PKI Tool : Version 12.1.3.0.0

Copyright (c) 2004, 2014, Oracle and/or its affiliates. All rights reserved.

Certificate was added to keystore

removed the key from the repository:

Disaster Recovery

For those familiar with the Disaster Recovery setup for an Enterprise Manager 13c environment, the AOM Disaster Recovery setup is pretty much the same implementation. The AOM Disaster Recovery setup is an active/passive configuration. This means that if the primary AOM site goes down, the virtual IP will fail over to a standby node. Once the AOM instance is started, it will run exactly the same on the DR site as it did on the primary site. To support DR, the AOM repository needs to be available on the DR site and the AOM instance file systems need to be replicated to the DR site.

Troubleshooting

The following commands are helpful in troubleshooting AOM.

  • emsctl status

This command will return the status of the AOM service. Below is an example:

% $AOM_HOME/scripts/emsctl status

Oracle Enterprise Manager Cloud Control 13c Release 1

Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.

------------------------------------------------------------------

Always-On Monitoring Version : 13.1.0.0.0

Always-On Monitoring Home : /u01/app/oracle/aom

Started At : January 13, 2016 4:11:01 PM PST

Last Repository Sync : February 2, 2016 1:41:17 PM PST

Upload URL : https://aomserver.domain:8081/upload

Always-On Monitoring Process ID : 15399

Always-On Monitoring Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=aomRepo.domain)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=aom)))

Enterprise Manager Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=emServer.domain)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emdb)))

Notifications Enabled : false

Always-On Monitoring is up.

  • emsctl ping

Another way to check the status of the AOM service is to use the ping option as seen below:

Oracle Enterprise Manager Cloud Control 13c Release 1

Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.

------------------------------------------------------------------

Always-On Monitoring is running

  • emsctl list_agents

This command provides a count and list of URLs that have communicated with the AOM service in the past hour.

  • emsctl getstats

This command will display AOM performance statistics.

Log files are also helpful during AOM troubleshooting. AOM records events that occur during operation in log files located under the $AOM_HOME/logs directory and are listed below. The .log files are rotated once the primary file reaches the maximum size of 10M. Two copies of the files are kept, the primary and one previous (rotated) file.

emsca logs:

  • emsca.err (only errors)
  • emsca.log.0 (rotating log file that contains all output including errors)

ems logs:

  • ems.err (only errors)
  • ems.log.0 (rotating log file that contains all output including errors)

There are different log levels that determine the type of information recorded in these log files. The levels can be set at INFO, DEBUG, WARN(default setting), and ERROR. The log level can be changed by adding the logLevel property to the $AOM_HOME/conf/emsconfig.properties file. The AOM instance must be bounced for the change to take effect. This is done with the commands below:

emsctl stop

emsctl start

Below is a recommended list of steps to take when troubleshooting AOM:

1. Check the log file.

2. Verify that the URL is set in Enterprise Manager.

This is done using this command: emctl get property -name "oracle.sysman.core.events.ems.emsURL"

3. Verify that the URL is set on the Management Agent.

To do this, verify the proper setting for EMS_URL in the $AGENT_HOME/sysman/config/emd.properties file.

4. Verify that Always-On Monitoring is running and notifications are enabled.

This can be verified by running the emsctl status command

5. Verify that downtime contacts have been specified in Enterprise Manger and Always-On Monitoring. Verifying this is a manual process depending on how the downtime contacts were configured.

a. If a global downtime contact was specified, verify this setting using emcli get_oms_config_property -property_name='oracle.sysman.core.events.ems.downtimeContact'

b. If per target downtime contacts were specified, it can be verified by looking at the “Downtime Contact” target property for each of the targets.

To summarize, Always-On Monitoring offers a solution to the long-time problem of who monitors our critical targets when EM is down. To make sure you get the most from this solution, at a minimum, it is important to setup AOM with High Availability. If a DR/Standby site is setup for the EM environment, it is an MAA best practice to also setup a DR site for the AOM application. For more detailed information on Always-On Monitoring, refer to the Always-On Monitoring chapter in the Enterprise Manager Cloud Control Administrator’s Guide.

Monday May 23, 2016

Oracle Enterprise Manager 13c and Always-On Monitoring

 
   
  
One of the common concerns of the Administrators of an Oracle Enterprise Manager (EM) environment is that when you have to take EM down for planned maintenance, you are blind to the status of some of your company’s most critical assets. Well, as of EM 13c, Oracle now has a solution for you! Introducing Always-On Monitoring – or AOM for short. Always-On Monitoring provides the capability of monitoring all or a specific list of targets using the same EM agents already deployed to these targets during complete downtime of the EM environment. It is especially useful for monitoring critical targets during planned EM downtime such as patching and/or upgrades. AOM will synchronize with EM for data such as notification contacts and is able to take the alerts from the EM agents and send email notifications to the identified downtime contacts. AOM can be configured and left running all the time so that it is ready to “take over” the alert notification for critical targets once notifications have been enabled. This blog contains the first of a couple of posts on AOM starting with the setup and configuration of the tool.

The setup/configuration of the AOM tool consists of the following steps:

1.  AOM Installation - The AOM application is included with the EM 13c software distribution under the sysman/ems directory and is also available via the Self-Update function in EM. The installation of the application is as simple as unzipping the zip file in the location chosen for the AOM install. An AOM installation consists of a database to hold the AOM repository, and a server to run the AOM Instance. Chapter 12 of the Oracle Enterprise Manager Cloud Control Administrator’s Guide contains the details steps required for installing AOM. This server must have JAVA version 1.7 installed. At this time, the AOM instance/application must run on the EM OMS but the AOM repository database can be created on any Oracle database server.  With the EM 13.2 release, the AOM instance/application will be able to be installed on any host within the monitored environment.


2.  Configure AOM Communication to EM - The configuration is pretty simple and is all handled via the script called emsca found under the AOM installation home in the scripts directory. (Fun Fact: the name of this script and the command line utility all start with ems rather than aom due to a pre-release name change of the product). This script will handle all of the necessary steps to configure AOM including the creation of the user and schema for the AOM repository. Once this configuration is complete, make sure that EM has an Email Server configured. At this time, AOM can only send email messages for the alerts and it uses the email server configuration that it gets from EM during synchronization. See "Configuring Email Servers in Enterprise Manager" for more details on how to setup Email Servers in EM.


3.  Configuration of downtime contacts - The downtime contacts are those that will receive the email notifications in the event of an alert on the identified targets. Setting up the downtime contacts is done in your EM environment and can be done in a couple of different ways and is documented in the section “Configuring Downtime Contacts in Enterprise Manager”.

4.  Synchronizing AOM and EM - This synchronization copies over the notification configuration and downtime contacts for the EM targets over to the AOM application. This should be done before starting AOM for the first time. It is a simple command using the AOM command line utility called emsctl. Here is a sample of the command:

% $AOM_HOME/scripts/emsctl sync

5.  Note that AOM will run a synchronization job every 24 hours by default although this is configurable. It is a good practice to run a manual sync after any changes to downtime contacts.

6.  Configure EM for AOM - The final step in the configuration process is to tell EM to include the upload URL for the AOM service. Once this is done, EM will send this URL to all existing agents and any future agents. This makes this a one-time step that will then be handled for you as your number of agents change in the future. To get the AOM upload URL, issue the command below (the upload URL is in bold below):

% $AOM_HOME/scripts/emsctl status

Oracle Enterprise Manager Cloud Control 13c Release 1

Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.

------------------------------------------------------------------

Always-On Monitoring Version : 13.1.0.0.0

Always-On Monitoring Home : /u01/app/oracle/aom

Started At : January 13, 2016 4:11:01 PM PST

Last Repository Sync : February 2, 2016 1:41:17 PM PST

Upload URL : https://aomserver.domain:8081/upload

Always-On Monitoring Process ID : 15399

Always-On Monitoring Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=aomRepo.domain)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=aom)))

Enterprise Manager Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=emServer.domain)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emdb)))

Notifications Enabled : false

Total Downtime Contacts Configured : 29

Then issue the following emctl command to set the property in EM (note this is emctl and not emsctl) – substituting the information in bold below with the correct upoad url and sysman password for your environment:

% emctl set property -name "oracle.sysman.core.events.ems.emsURL" -value " https://aomserver.domain:8081/upload " -sysman_pwd sysman

7.  Starting AOM and Enabling notifications - Starting AOM is done via the emsctl command and specifying the start option:

% $AOM_HOME/scripts/emsctl start

Even though the setup of AOM is complete and it has been started, you will not receive any alerts when EM is down unless you enable notifications. As you probably guessed by now, this is done via the emsctl command:

% $AOM_HOME/scripts/emsctl enable_notification

A sync with EM is done each time you enable_notification unless you use the option –nosync. To disable the notification, use the same sort of command with the disable_notification option:

% $AOM_HOME/scripts/emsctl disable_notification

It is a good practice to leave AOM started but make sure you disable the notifications. Only enable them just before you shutdown EM for planned maintenance. REMEMBER that you will get more notification than you probably want if you leave this enabled all of the time because AOM has no knowledge of the notifications that EM is already doing.

To summarize, Always-On Monitoring offers a solution to the long-time problem of who monitors our critical targets when EM is down. The next blog will provide the details on how to configure AOM for High Availability and Disaster Recovery.

Tuesday Feb 11, 2014

Oracle Enterprise Manager Software Planned Maintenance

 
   
  

A critical component of managing an application includes both patching and maintaining the software. Applying patches is not only required for bug fixes, it is also a means of obtaining new and beneficial functionality. Thus it becomes an important  task to maintain a healthy and productive Enterprise Manager (EM) solution. The process of patching itself can present different challenges that can potentially increase the work and time involved in each patching exercise. Issues could arise such as patch conflicts, not meeting required  prerequisites and even unnecessary downtime. Spending the proper time to setup a patching strategy can save time and effort as well as possible errors and risk when patching a production EM environment.

The MAA team has recently published a new whitepaper which provides an overview of the recommended patching strategy for Oracle Enterprise Manager.  This information is intended to be a guideline for maintaining a patched and highly available EM environment and may need customization to accommodate requirements of an individual organization.

There are five main topics covered in this paper as outlined below:

  1. Enterprise Manager Components

  2. Defining Business Requirements

  3. Patching Strategy Overview

    1. Types of Patches

    2. Define Patch List and Steps

    3. Planning

    4. Preparation

    5. Testing

  4. Sample Patching Steps

  5. Optional Patching Strategy

http://www.oracle.com/technetwork/database/availability/em-patching-bp-2132261.pdf


About

bocadmin_ww

Search

Archives
« August 2016
SunMonTueWedThuFriSat
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
   
       
Today