Thursday Feb 27, 2014

Fast Recovery Area for Archive Destination


If you are using Fast Recovery Area (FRA) for the archive destination and the destination is set to USE_DB_RECOVERY_FILE_DEST, you may notice that the Archive Area % Used metric does not trigger anymore. Instead you will see the Recovery Area % Used metric trigger when it hits a Warning threshold of 85% full, and Critical of 97% full. Unfortunately, this metric is controlled by the database and the thresholds cannot be modified (see MOS Note 428473.1 for more information). Thresholds of 85/97 are not sufficient for some of the larger, busier databases. This may not give you enough time to kickoff a backup and clear enough logs before the archiver hangs. If you need different thresholds, you can easily accomplish this by creating a Metric Extension (ME) and setting thresholds to your desired values.  This blog will walk through an example of creating an ME to monitor archive on FRA destinations, for more information on ME's and how they can be used, refer to the Oracle Enterprise Manager Cloud Control Administrator's Guide

[Read More]

Monday Feb 24, 2014

Monitoring Archive Area % Used on Cluster Databases


One of the most critical events to monitor on an Oracle Database is your archive area. If the archive area fills up, your database will halt until it can continue to archive the redo logs. If your archive destination is set to a file system, then the Archive Area % Used metric is often the best way to go. This metric allows you to monitor a particular file system for the percentage space that has been used. However, there are a couple of things to be aware of for this critical metric.

Cluster Database vs. Database Instance

You will notice in EM 12c, the Archive Area metric exists on both the Cluster Database and the Database Instance targets. The majority of Cluster Databases (RAC) are built against database best practices which indicate that the Archive destination should be shared read/write between all instances. The purpose for this is that in case of recovery, any instance can perform the recovery and has all necessary archive logs to do so. Monitoring this destination for a Cluster Database at the instance level caused duplicate alerts and notifications, as both instances would hit the Warning/Critical threshold for Archive Area % Used within minutes of each other. To eliminate duplicate notifications, the Archive Area % Used metric for Cluster Databases was introduced. This allows the archive destination to be monitored at a database level, much like tablespaces are monitored in a RAC database.

In the Database Instance (RAC Instance) target, you will notice the Archive Area % Used metric collection schedule is set to Disabled.

If you have a RAC database and you do not share archive destinations between instances, you will want to Disable the Cluster Database metric, and enable the Database Instance metric to ensure that each destination is monitored individually.


Friday Jul 26, 2013

Operational Considerations and Troubleshooting Oracle Enterprise Manager 12c

Oracle Enterprise Manager (EM) 12c has become a valuable component in monitoring and administrating an enterprise environment. The more critical the application, servers and services that are monitored and maintained via EM, the more critical the EM environment becomes. Therefore, EM must be as available as the most critical target it manages.


There are many areas that need to be discussed when talking about managing Enterprise Manager in a data center. Some of these are as follows:

• Recommendations for staffing roles and responsibilities for EM administration

• Understanding the components that make up an EM environment

• Backing up and monitoring EM itself

• Maintaining a healthy EM system

• Patching the EM components

• Troubleshooting and diagnosing guidelines

The Operational Considerations and Troubleshooting Oracle Enterprise Manager 12c whitepaper available on the  Enterprise Manager Maximum Availability Architecture (MAA) site will help define administrator requirements and responsibilities.  It provides guidance in setting up the proper monitoring and maintenance activities to keep Oracle Enterprise Manager 12c healthy and to ensure that EM stays highly available.

Wednesday Apr 03, 2013

Using Advanced Notifications in Oracle Enterprise Manager 12c

When using an enterprise monitoring tool such as Oracle Enterprise Manager 12c, one of the most critical components is notification. Once an alert or issue has been identified, how do you tell the right people at the right time? Most enterprises use e-mail or open a trouble ticket. As you can imagine, no two enterprises are the same when it comes to their tools and processes. Many customers use one of the more common and well known trouble ticketing systems but quite a few use non-standard or custom (homegrown) trouble ticketing systems. Some customers have special routing requirements or corporate standards and have custom applications which handle all emailing functions instead of directly emailing using an SMTP server.

Oracle Enterprise Manager 12c can handle all of these situations by utilizing one of the various notification methods provided: E-mail, 3rd party connectors and advanced notification methods. There are three types of advanced notifications: SNMP, OS Command or PL/SQL. This blog will introduce you to the OS Command and PL/SQL notification methods available in EM 12c and provide an example of using a custom OS script for notifications.

Advanced Notification Methods: OS Command and PL/SQL

With the advanced notification methods, you can write a notification directly to a table or an OS log for further processing or push a notification to any trouble ticketing system using their command line tools providing the data and variables from EM 12c as input.  This method is used by some customers whose corporate standard requires that all alerts be written to a log file, which the ticketing system can poll on a regular basis for alerts. Additionally, advanced notifications allow you to call a procedure whose interface is PL/SQL or access additional data in the Enterprise Manager repository.  For example, for a database alert on % of Processes or Sessions Used, you could use PL/SQL to notify the proper application team (stored in a custom target property) that they have too many sessions.   To create and configure advanced notifications, the user must have Super Admin privileges.

Creating OS or PL/SQL Scripts

When creating an advanced notification the first step is to create the OS or PL/SQL script. It’s highly recommended to include debugging and logging information so you can fully understand what information EM 12c is passing and assist in troubleshooting.  When using an OS Command in a multi-OMS environment, the script needs to reside on all OMS servers or preferably in a shared location. If you plan to use PL/SQL, the procedure must be created in the repository database before configuring it as a notification method.   Of course, any custom objects created in the repository should be created under a separate schema and privileges granted to the SYSMAN user. 

The Oracle Enterprise Manager Cloud Administrator’s Guide chapter on Notifications has detailed examples of OS Scripts and PL/SQL that can be utilized in different situations.  This chapter also provides detailed information on using passing information to the OS or PL/SQL script and troubleshooting.

Creating Custom Notification Methods

After defining the OS Command or PL/SQL Script, you need to add it to EM 12c as a notification method.  In this example, we will create a simple OS script that logs events to a log file, which can be further processed by a custom ticketing system.

#!/bin/ksh
LOG_FILE=/tmp/event.log
 
if test -f $LOG_FILE
then
echo $TARGET_NAME $MESSAGE $EVENT_REPORTED_TIME >> $LOG_FILE
else
   exit 100
fi

To create a notification method login as a user with Super Admin privileges and select Setup / Notifications / Notification Methods.  Under the Scripts and SNMP Traps section, click the Add drop down box and select OS Command and click Go.


Enter a name and provide the fully qualified script location.  Use the Test button to validate and click Save.


Adding Notification Methods to Incident Rules

To receive notifications, you will need to create an Incident Rule set and select relevant targets and events to notify and then select the OS Command advanced notification method we created earlier. In this section we will go through the steps to create a simple incident rule.  For full details on how to configure your Incident Rules, see the Oracle Enterprise Manager Cloud Control Administrator’s Guide.

Go to Setup / Incidents / Incident Rules.   At this point you can either select a rule set to edit, or create a new rule set.  In this example we are going to create a new rule set by clicking Create


On the first screen, enter a user friendly name and description. Since this example is planning to use an advanced notification choose Enterprise


Since we only want notifications on our Production targets, we’re going to narrow down the list of targets. On Targets tab select Specific targets, select Groups in the drop down box and click Add. Search for the desired group and select.


Click on the Rules tab, and select Create.


Select the type of rule to create, in this example we are using Incoming events and updates to events.


Enter filter criteria, in this example we are filtering on all Metric Alerts in Severity Warning and Critical, click Next.


On the Add Actions screen you will define the action that you wish the notification to perform.  Click Add.


In the Create Incident or Update Incident section you can choose to create an Incident, assign Incidents to an administrator, set priority/status or escalate (if update is selected).  Under the Notifications section, you can select Basic Notifications to send an e-mail or page to a particular user or users, on top of any methods you might select in the Advanced Notifications section.  In Advanced Notifications you will see the method we previously created, select that method and click Continue.



Click Next.


Provide a user friendly name for your rule and a description, click Next



Review the details of your rule and click Continue


Note the rules are not saved until the Save button is clicked. Click OK.


Review your rule set and click Save.


Click OK.


Validating Incident Rules

Once you have configured your rule set, you should trigger a critical alert on one of the targets included in the rule set. Verify that the rule set triggers the notification method you created and logs the event in the log file.  It's helpful to keep the e-mail option turned on during testing so you can validate when the rule was triggered as well. 

Identify a target in the group you configured notifications for in the previous section. For this example, we will use a database instance and trigger a Process Limit Usage (%) event. From Oracle Database / MonitoringAll Metrics, I’ve identified a metric that I can lower thresholds on to trigger an event. Click on Process Limit Usage (%) to drill down to this metric. Notice our Real Time Value of Processes is 16.4.


Click on Modify Thresholds to set thresholds.


Set the Warning and Critical thresholds lower than the Real Time Value we made note of earlier. Ours was 16.4, so setting the Warning to 5, and Critical to 10. In the Occurrences before Alert field, you may wish to change the value to 1. Since the collection frequency is 10 minutes, if occurrences are set to 3, you will have to wait 30 minutes for it to trigger and again to clear. Click Save Thresholds.


Close the confirmation window.


Navigate to Oracle Database / Monitoring / Incident Manager. Since we have the out-of-box rule set enabled we get an Incident for every critical metric alert and we can see the incident for Process Limit in the Unacknowledged incidents view. If you have disabled the out-of-box rule set or don’t see the incident, check the Events without incidents view. Select the appropriate line and in the lower pane click on the Events tab.


On the Events tab click on the Message link to drill into the event details.


Here you’ll see the notification method was called under the Last Comment field. From here, click on Updates tab to see more details.


In the Updates tab, you will see the details of the alert and the notification method that was called to run our /tmp/event_log.sh script.


Finally, we can check the /tmp/event.log file and see the information that was reported.   Be sure to go back and set your threshold to its regular value to clear the false alert you triggered!


Summary

The options for notifications in Oracle Enterprise Manager 12c are very flexible. You can choose e-mail, one of the available connectors to integrate with a 3rd party ticketing system, or use one of the advanced notification methods (SNMP, OS Commands or PL/SQL). In this blog, we’ve shown you how to create an advanced OS Command notification method, how to configure an incident rule set to call that method and how to validate by triggering an alert. Once you are familiar with how to create advanced notification methods and rule sets, you can customize your notifications to suit your needs. 

For additional information on configuring your environment for enterprise monitoring, see the whitepaper Strategies for Scalable, Smarter Monitoring using Oracle Enterprise Manager Cloud Control 12c.  

Stay Connected with Oracle Enterprise Manager:

Twitter | Facebook | YouTube | Linkedin | Newsletter

Network Ports Used in Oracle Enterprise Manager 12c

When planning and configuring your Oracle Enterprise Manager 12c implementation, you will have many infrastructure considerations. One of the most often discussed pieces is the network ports that are used and how to configure load balancers, firewalls and ACLs for communication.

This blog post will help identify the typical default port and range for each component, how to identify it and how to modify the port usage.

To modify most ports during installation, select the Advanced Installation and set the appropriate ports on the Port Configuration Details screen.


Once the system is installed, you can use the following EMCTL or OMSVFY commands to validate components and port assignment:

$emctl status oms -details
$omsvfy show opmn
$omsvfy show ports

To verify if a port is free, run the following command:

On Unix:
$netstat -an | grep <port no>

On Microsoft Windows:
>netstat -an|findstr <port_no>

For more information on OMSVFY (part of the EMDIAG toolkit) see MOS Note 421053.1: EMDIAG Troubleshooting Kits Master Index

External Ports

These ports will be used in every Enterprise Manager 12c installation and will require firewall and/or ACL modifications if your network is restricted.  These are also the components that will be added to your load balancer configuration.

Default Port

Range

Component

Usage

Modify

4889

4889 – 4898

Enterprise Manager OHS Upload HTTP

Agent Communication to OMS (unsecure). Used in load balancer.

To modify after install follow notes 1381030.1 and 1385776.1. Requires changes on all Agents.

1159

1159, 4899 – 4908

Enterprise Manager OHS Upload HTTP SSL

Agent Communication to OMS (secure). Used in load balancer.

To modify after install follow notes 1381030.1 and 1385776.1. Requires changes on all Agents.

7788

7788 – 7798

Enterprise Manager OHS Central Console HTTP (Apache/UI)

Web browser connecting to Cloud Control Console (unsecure). Used in load balancer and for EM CLI.

To modify after install follow notes 1381030.1.

7799

7799 - 7809

Enterprise Manager OHS Central Console HTTP SSL (Apache/UI)

Web browser connecting to Cloud Control Console (secure). Used in load balancer and for EM CLI.

To modify after install follow note 1381030.1.

7101

7101 - 7200

EM Domain WebLogic Admin Server HTTP SSL Port

Cloud Control Admin Server.

To modify after install follow note 1109638.1.

3872

3872, 1830 – 1849

Cloud Control Agent

Only the OMS will connect to this port, to either report changes in the monitoring, submit jobs, or to request real-time statistics.

Port can be provided during Agent install.

If the agent port needs to be changed at a later date this can be done with the following command on the agent:
emctl setproperty agent -name EMD_URL -value https://hostname.domain:port/emd/main/

This will allow the agent to run on the new port, however the target does not get renamed so continues to show the original port.

1521*

Depends on Listener Configuration

Database Targets -  SQL*Net Listener

For Repository database, only the OMS will connect to store management data from

the agents. For all monitored target databases OMS will retrieve information requested by browser clients.

To modify this port for the repository database:

Change the listener.ora file for the EM repository. Restart the listener. Then for every OMS machine using that repository run the following:

emctl stop oms
emctl config oms -store_repos_details -repos_conndesc <connect descriptor of database> -repos_user sysman
emctl start oms
emctl config emrep -agent <agent name> -conn_desc <connect descriptor of database> 

To modify this port for monitored targets, change the listener configuration on the target, then update Monitoring Configuration in EM.

7101

7101 - 7200

FMW Targets – Admin Console

Outgoing from OMS, used for managing FMW targets.

To modify after install follow note 1109638.1.

NA

NA

ICMP

Outgoing from OMS to host servers if the Agent is unreachable. Validates if server is up or down.

NA

Internal Ports

These ports are required for internal Enterprise Manager communication and typically do not require additional firewall/ACL configuration.

Default Port

Range

Component

Usage

Modify

7201

7201 – 7300

EM Domain WebLogic Managed Server HTTP Port

Used for Fusion Middleware communication.

Configured during installation

7301

7301 – 7400

EM Domain WebLogic Managed Server HTTP SSL Port

Used for Fusion Middleware communication.

Configured during installation

7401

7401 – 7500

Node Manager HTTP SSL Port

Used for Fusion Middleware communication.

Configured during installation

6702

6100 - 6199

Oracle Notification Server (OPMN) Local

Ports used by OPMN  can be verified from <MW_HOME> /gc_inst/WebTierIH1

/config /OPMN/opmn/opmn.xml:


<debug comp="" rotation-size="1500000"/>
<notification-server interface="any">
<port local="6700" remote="6701"/>

Modify the opmn.xml to use free ports as below:

1. Stop OMS

2. Take a backup of the existing opmn.xml and ports.prop in the <MW_HOME>/ gc_inst/WebTierIH1/ config /OPMN/opmn directory.

3. Edit the opmn.xml file, under the <notification-server> element, modify the local / remote port, as necessary to the free port available and save the file.

4. Edit the ports.prop file and modify the remote / local port parameters as necessary and save the file.

5. Start the OMS

6703

6200 - 6201

Oracle Notification Server (OPMN) Remote

Ports used by OPMN  can be verified from <MW_HOME> /gc_inst/WebTierIH1

/config/OPMN/opmn/opmn.xml:


<debug comp="" rotation-size="1500000"/>
<notification-server interface="any">
<port local="6700" remote="6701"/>

Modify the opmn.xml to use free ports as below:

1. Stop OMS

2. Take a backup of the existing opmn.xml and ports.prop in the <MW_HOME> /gc_inst/WebTierIH1/ config/OPMN/opmn directory.

3. Edit the opmn.xml file, under the <notification-server> element, modify the local / remote port, as necessary to the free port available and save the file.

4. Edit the ports.prop file and modify the remote / local port parameters as necessary and save the file.

5. Start the OMS

Optional

These ports required only if certain components are to be used and firewall/ACL changes may be needed.

Default Port

Range

Component

Usage

Modify

443


Secure web connection (https - 443) to updates.oracle.com support.oracle.com

ccr.oracle.com

login.oracle.com

aru-akam.oracle.com

Outgoing from OMS used for online communication with Oracle for OCM, MOS, Patching, Self-Updates, ASR

Proxy settings defined via the UI (Setup -> Proxy Settings)
Do not use the OMS parameters!  

51099


Application Dependency and Performance RMI Registry Port

ADP

Configured during installation

55003


Application Dependency and Performance Java Provider Port

ADP

Configured during installation

55000


Application Dependency and Performance Remote Service Controller Port

ADP

Configured during installation

4210


Listen

ADP

Configured during installation

4211


SSL Listen Port

ADP

Configured during installation

3800


JVM Managed Server Listen

JVM

Configured during installation

3801


JVM Managed Server SSL Listen

JVM

Configured during installation

9701

9701-49152

BI Publisher HTTP

BI Publisher

During install can modify with configureBIP script.  Post-install can be modified per Note 1524248.1

9702

9701-49152

BI Publisher HTTP SSL Port

BI Publisher

During install can modify with configureBIP script.  Post-install can be modified per Note 1524248.1

Stay Connected with Oracle Enterprise Manager:

Twitter | Facebook | YouTube | Linkedin | Newsletter

About

bocadmin_ww

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today