Thursday Jun 27, 2013

Oracle Enterprise Manager 12c Configuration Best Practices (Part 3 of 3)

This is part 3 of a three-part blog series that summarizes the most commonly implemented configuration changes to improve performance and operation of a large Enterprise Manager 12c environment. A “large” environment is categorized by the number of agents, targets and users. See the Oracle Enterprise Manager Cloud Control Advanced Installation and Configuration Guide chapter on Sizing for more details on sizing your environment properly.

  • Part 1 of this series covered recommended configuration changes for the OMS and Repository
  • Part 2 covered recommended changes for the Weblogic server
  • Part 3 covers general configuration recommendations and a few known issues

The entire series can be found in the My Oracle Support note titled id=1553342.1">Oracle Enterprise Manager 12c Configuration Best Practices [1553342.1].

Configuration Recommendations

Configure E-Mail Notifications for EM related Alerts

In some environments, the notifications for events for different target types may be sent to different support teams (i.e. notifications on host targets may be sent to a platform support team). However, the EM application administrators should be well informed of any alerts or problems seen on the EM infrastructure components.


Recommendation: Create a new Incident rule for monitoring all EM components and setup the notifications to be sent to the EM administrator(s). The notification methods available can create or update an incident, send an email or forward to an event connector. To setup the incident rule set follow the steps below. Note that each individual rule in the rule set can have different actions configured.


1.  To create an incident rule for monitoring the EM components, click on Setup / Incidents / Incident Rules. On the All Enterprise Rules page, click on the out-of-box rule called “Incident management Ruleset for all targets” and then click on the Actions drop down list and select “Create Like Rule Set…”

2. For the rule set name, enter a name such as MTM Ruleset. Under the Targets tab, select “Specified targets” and select "Targets" from the Add drop down list.  Click on the green "+" sign.  Click on the drop down arrow for Target Type and deselect all target types except "EM Service" and “OMS and Repository".  Click "Search".  Select the targets returned and click "Select".


3. Click on the Rules tab. To edit a rule, click on the rule name and click on Edit as seen below

4. Modify the following rules (names for rules in 12.1.0.3 are in parentheses if they have changed):

a. Incident creation Rule for metric alerts (Create incident for critical metric alerts)

i. Leave the Type set as is but change the Severity to add Warning by clicking on the drop down list and selecting “Warning”. Click Next.

ii.  Add or modify the actions as required (i.e. add email notifications). Click Continue and then click Next.

iii. Leave the Name and description the same and click Next.

iv. Click Continue on the Review page.

b. Incident creation Rule for target unreachable.

i.   Leave the Type set as is but change the Target type to add EM Service and OMS and Repository by clicking on the drop down list selecting both "EM Service" and “OMS and Repository”. Click Next.

ii.  Add or modify the actions as required (i.e. add email notifications) Click Continue and then click Next.

iii. Leave the Name and description the same and click Next.

iv. Click Continue on the Review page.

5 Modify the actions for any other rule as required and be sure to click the “Save” push button to save the rule set or all changes will be lost.

Configure Out-of-Band Notifications for EM Agent

Out-of-Band notifications act as a backup when there’s a complete EM outage or a repository database issue. This is configured on the agent of the OMS server and can be used to send emails or execute another script that would create a trouble ticket. It will send notifications about the following issues:

? Repository Database down

? All OMS are down

? Repository side collection job that is broken or has an invalid schedule

? Notification job that is broken or has an invalid schedule

Recommendation: To setup Out-of-Band Notifications, refer to the MOS note “How To Setup Out Of Bound Email Notification In 12c” (Doc ID 1472854.1)

Modify the Performance Test for the EM Console Service

The EM Console Service has an out-of-box defined performance test that will be run to determine the status of this service. The test issues a request via an HTTP method to a specific URL. By default, the HTTP method used for this test is a GET but for performance reasons, should be changed to HEAD. The URL used for this request is set to point to a specific OMS server by default. If a multi-OMS system has been implemented and the OMS servers are behind a load balancer, then the URL used by EM as the URL in notifications and by this EM Service test must be modified to point to the load balancer name instead of a specific server name. If this is not done and a portion of the infrastructure is down then the EM Console Service will show down as this test will fail.

Recommendation: Modify the HTTP Method for the EM Console Service test and the URL if required following the detailed steps below.

Setting the Console URL if a multi-OMS system is implemented:

1.  Click on Setup / Manage Cloud Control / Health Overview

2.  Click on the "Add" push button next to Console URL as seen in the picture below.


3.  Type in the URL and click OK.


Modifying the HTTP Method for the EM Console Service test:

1.  To create an incident rule for monitoring the EM components, click on Targets / Services. From the list of services, click on the EM Console Service.

2. On the EM Console Service page, click on the Test Performance tab.

3.  At the bottom of the page, click on the Web Transaction test called EM Console Service Test

4.  Click on the Service Tests and Beacons breadcrumb near the top of the page.

5.  Under the Service Tests section, make sure the EM Console Service Test is selected and click on the Edit push button.

6.  Under the Transaction section, make sure the Access Logout page transaction is selected and click on the Edit push button

7) Under the Request section, change the HTTP Method from the default of GET to the recommended value of HEAD. The URL in this section should point to the load balancer name instead of a specific server name if multi-OMSes have been implemented and the Console URL was set according to the steps above.

Check for Known Issues

Job Purge Repository Job is Shown as Down

This issue is caused after upgrading EM from 12c to 12cR2. On the Repository page under Setup → Manage Cloud Control → Repository, the job called “Job Purge” is shown as down and the Next Scheduled Run is blank. Also, repvfy reports that this is a missing DBMS_SCHEDULER job.  NOTE:  this issue is fixed in version 12.1.0.3

Recommendation: In EM 12cR2, the apply_purge_policies have been moved from the MGMT_JOB_ENGINE package to the EM_JOB_PURGE package. To remove this error, execute the commands below:

$ repvfy verify core -test 2 -fix

To confirm that the issue resolved, execute

$ repvfy verify core -test 2

It can also be verified by refreshing the Job Service page in EM and check the status of the job, it should now be Up.

Configure the Listener Targets in EM with the Listener Password (where required)

EM will report this error every time it is encountered in the listener log file. In a RAC environment, typically the grid home and rdbms homes are owned by different OS users. The listener always runs from the grid home. Only the listener process owner can query or change the listener properties. The listener uses a password to allow other OS users (ex. the agent user) to query the listener process for parameters. EM has a default listener target metric that will query these properties. If the agent is not permitted to do this, the TNS incident (TNS-1190) will be logged in the listener’s log file. This means that the listener targets in EM also need to have this password set. Not doing so will cause many TNS incidents (TNS-1190). Below is a sample of this error from the listener log file:

Recommendation: Set a listener password and include it in the configuration of the listener targets in EM

For steps on setting the listener passwords, see MOS notes: 260986.1 , 427422.1

Thursday Jun 20, 2013

Oracle Enterprise Manager 12c Configuration Best Practices (Part 2 of 3)

This is part 2 of a three-part blog series that summarizes the most commonly implemented configuration changes to improve performance and operation of a large Enterprise Manager 12c environment. A “large” environment is categorized by the number of agents, targets and users. See the Oracle Enterprise Manager Cloud Control Advanced Installation and Configuration Guide chapter on Sizing for more details on sizing your environment properly.

  • Part 1 of this series covered recommended configuration changes for the OMS and Repository
  • Part 2 covers recommended changes for the Weblogic server
  • Part 3 will cover general configuration recommendations and a few known issues

The entire series can be found in the My Oracle Support note titled id=1553342.1">Oracle Enterprise Manager 12c Configuration Best Practices [1553342.1].

WebLogic Server Recommendations

Stuck Thread Max Time

By design WLS will ping applications and wait for a response for up to the value of Stuck Thread Max Time which is set to 600 seconds by default. This is a heartbeat to ensure that a particular thread is not stuck. EM on the other hand will keep threads running as long as there is work in the queue and they will not respond to a heartbeat. This is expected behavior for both EM and WebLogic Server however it will cause WLS to timeout and error which will create an incident within EM. If this parameter is not increased, the number of incidents created by this WLS error can be significant. Below is an example of the incident that may be seen. Please note, an enhancement bug has been created requesting that EM install out of the box with a higher value for this parameter.


Recommendation: To assist in reducing these errors, increase the stuck thread timeout in the Admin server as per the steps below. Note that this will reduce the number of above alerts but may not remove them completely.

1. Log onto the WLS Admin server.

2. Click on Environment in the top right side menu and expand Servers. Click on one of the OMS server names.

3. Click on the Tuning tab on in the middle window and then on the Lock and Edit under the Change Center (top left).

4. Change the value for Stuck Thread Max Time to 1800.


5. Save and Activate the change. This will require a restart of the OMS server for it to go into effect and will need to be repeated for all servers in the Admin Console (i.e. OMS servers and ADMINSERVER) but only needs to be done once per site/domain. If the environment contains standby OMS servers, repeat these steps for all standby OMS servers and the ADMINSERVER although a reboot is not required for the standby OMS servers as they are not running.

Modify Log Settings

The default severity setting for logging information in the WebLogic Server is set at a level that will create excessive logging data. These settings should be set to a higher severity level.

Recommendation: To modify these settings, follow the steps below:

1.  Log onto the WLS Admin server.

2.  Click on Environment in the top right side menu and expand Servers. Click on the first OMS server.


3. Click on the Logging tab in the middle window and then on the Lock and Edit under the Change Center (top left).




4. Expand the Advanced option at the bottom of the page.


5. Change the Minimum log severity from Info to Warning.


6. Change the Domain Log Broadcaster Severity Level from Notice to Error.


7. Save and Activate the change. This does not require a restart of the OMS server for it to go into effect but will need to be repeated for all servers in the Admin Console (i.e. OMS servers and ADMINSERVER. This change only needs to be done once per site/domain. If the environment contains standby OMS servers, repeat these steps for all standby OMS servers and the ADMINSERVER.

About

bocadmin_ww

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today