Tuesday May 12, 2015

Creating an Enterprise Manager Metric Extension to monitor Huge Page Allocation


Huge Pages enables the Linux kernel to manage large pages of memory in addition to the standard 4KB page size. If you have a system with more than 16GB of memory running Oracle databases with a total System Global Area (SGA) larger than 8GB, you should enable the HugePages feature to improve database performance.

If Huge Pages is enabled but the system isn't configured to allow for the requested huge page allocation excessive swapping will occur. This will result in degraded database performance.

To ensure this doesn't happen a Metric Extension can be created in Enterprise Manager to notify administrators if the system is incorrectly configured.

The following will describe the process of creating the Metric Extension in Enterprise Manager


Create the script that will be uploaded to the OMS:


1. On your local machine create a file called check_hugepage_usage.sh that contains the following

RECD_SIZE=0

CFG_TOTAL=`grep HugePages_Total /proc/meminfo | awk '{print $2}'`

SIZE=$((`grep  Hugepagesize /proc/meminfo | awk '{print $2}'` * 1024))

for SHMSEGS in `ipcs -m | awk '{print $5}' | sed -n '4,$p'`

do

    SEGSIZE=$(($SHMSEGS / $SIZE))

  if [ $SEGSIZE -gt 0 ]; then

   RECD_SIZE=$(($RECD_SIZE+$SEGSIZE+1))

   fi

done 

DIFF=$(($RECD_SIZE-$CFG_TOTAL))

echo $RECD_SIZE:$CFG_TOTAL:$DIFF


Next create the Metric Extension


1. Create the metric extension in the Enterprise Manager Console


Enterprise-->Monitoring-->Metric Extensions

Select the create button-->Metric Extension

2. On the General Properties Screen set the following


Target type select "Host"

Name "huge_page_monitoring"

Display Name "Huge Page Monitoring"

Adapter "OS Command - Multiple Columns"

Data Collection "Enabled"

Repeat Every "12 Hours"

Use of Metric Data "Alerting and Historical Trending"

Upload Interval "1 Collections"

Select the Next Button


3. On the  “Adapter" page enter the following


Command '\/bin\/ksh'

Select the upload button by the “Script” text box and upload the file created in step 1

Delimiter ":"

Select the upload button and select the file created in step 1

Select the Next button



4. On the "Create New : Columns" page Use the Add-->New metric column button create three columns with the add button


Column one should be setup as:

Name "Recommended_Value"

Display Name "Recommended_Value"

Column Type "Data Column"

Value Type "Number"

Metric Category "Capacity"


Column two should be setup as:

Name "Configured_Value"

Display Name "Configured Value"

Column Type "Data Column"

Value Type "Number"

Metric Category "Capacity"

Comparison Operator ">"

Critical "0"


Column three should be setup as:

Name "Huge_Page_Shortage"

Display Name "Configured Value"

Column Type "Data Column"

Value Type "Number"

Metric Category "Capacity"

Comparison Operator ">"

Critical "0"


After Setting up the three columns select the next button

5. On The Credentials Screen


Select the “Default Monitoring Credentials” radio button

Click the next button

6. On the “Create New : Test” page


Add a target to test with in the “Test Targets” section

Click the “Run Test” button and ensure that results are displayed properly in the “Test Results” box.

The results should be similar to below

Target Name Recommended Value Configured Value Huge Page Shortage 

Targetname 31745 36000 0


Click the Finish button


7. Next the Metric Extension must be saved as a deployable draft. This is accomplished on the main metric extension page. This allows the metric to be deployed to targets for testing. However at this stage only the developer has access to publish the metric. After satisfactory testing is completed the metric is then published. This is once again accomplished from the main metric extension page.


8. Once testing is complete and the Metric extension is deployable. It can be deployed to targets. This is accomplished from the metric extension page

Highlight the metric extension just created. From the Actions drop down box select “Deploy to Targets…”



To ensure that administrators are notified in the event the metric created fails an incident rule should be created.


1. To Begin navigate to the Incident Rules Home Page

         From the Setup button on the upper right hand corner of the Enterprise Overview Page

         Setup->Incidents->Incident Rules

         Now click the “Create Rule Set.” button

2. On the Create Rule Set screen enter the following information

        Name: Whatever the rule should be called. i.e. Metric Collection Error Rule

        Enabled Box: Should be checked

        Type: Enterprise

Applies To: Targets

Select the “All Targets of Type” radio button on the bottom of the screen followed by Host in the drop down box


3. Now select the “Rules” tab at the bottom of the screen


4. Chose the "Create.." button on the middle of the screen


5. On the “Select Type of Rule to Create” Popup box select the “Incoming events and updates to events” radio button. Click the continue button.


6. On the “Create New Rule: Select Events” screen check the type check box. In the drop down select “Metric Alert". Click the next button


7. Select the “Add” button 

On the “Add conditional Actions” page you can specify conditions that must occur for the action to apply, if an incident should be created and email notifications. Specify the desired properties and select the continue button.


8. On the Target Type drop down select “Host”

Select the search button and select the metric extension created above

Select the check box next to the desired metric

Select the desired severity status in the check boxes at the bottom of the screen 

Click the “OK” button


9. Click the “Next” button back on the  “Add Actions” page


10. Click next and select the desired option on the next pages and complete creating the rule set.


Wednesday Apr 22, 2015

April 2015 EM Recommended Patch List

 
   
  

As part of the recent April 2015 PSU Patch release for Enterprise Manager, a list of recommended patches has been compiled for EM 12c to be included in the Exadata Quarterly Full Stack Download patch.  This list contains recommended patches for the EM infrastructure including the OMS, WLS, Plugins, Repository and Agent. 

For more details and to review these EM recommended patches for Exadata monitoring, refer to My Oracle Support Note titled Patch Requirements for Setting up Monitoring and Administration for Exadata [1323298.1]

Also, the information in the note contains not only a recommended list of patches for both EM versions 12.1.0.4 and 12.1.0.3 but also provides a link to a My Oracle Support Note that provides the steps for applying these patches. This note is titled Applying Enterprise Manager 12c Recommended Patches [1664074.1].  The patch apply process in this note provides maximum availability for the EM environment by structuring the patch apply steps so that the primary OMS server is up as long as possible.

April 2015 Exadata Quarterly Full Stack Patch Availability

 
   
  

The latest Quarterly Full Stack patch was recently released for Exadata.  This patch contains all of the latest recommended patches for the Exadata Database Machine. 

For more details, refer to My Oracle Support Note titled Database Machine and Exadata Storage Server 11g Release 2 (11.2) Supported Versions [888828.1]

Note that the information in the note contains details for both Exadata Storage Server Software 12.1.1.1.2 and 12.1.2.1.1 releases.  This note is maintained on a regular basis so bookmark it and use it as reference for the latest Exadata Database Machine patching recommendations.

Thursday Nov 06, 2014

Tools For Generating Consistent Loads

It's finally ready. The new database machine you've spent months, planning and architecting. All those shiny new components perfectly aligned for extreme performance. Will it give you the results you expect? How can you baseline and compare your new and existing systems? Pointing your application at your new machine may take some time to setup and depending on the behavior of the application, may not stress test all hardware components as you might like. You need a set of easy to configure scripts for load testing and you need tools and procedures to compare old systems to new. 

This will be the first of three blog posts to help with that effort. In this post, I'll go over some tools you can use to generate various loads. The next two posts in the series I'll talk about using Enterprise Manager to evaluate and baseline loads, and strategies to compare systems using consistent loads.

Test Suite

My current test suite includes two methods to generate loads: one leverages Swingbench, which is a well known and popular load generating tool, and the other is a solution I cooked up myself. Both sets of scripts can be altered to tailor their load characteristics. I've also included a variable load script wrapper for each, which you can use to adjust the load over time. For example: you can have a load test that runs for a total of 5 hours and within that 5 hour window your load could fluctuate every 30 minutes from heavy to light. The custom scripts are also flexible enough to support altering their behavior if you have a specific set of SQL/PLSQL commands you would like to run.

For this article, my database is running on an Exadata X2-2 quarter rack.

Using Swingbench


Swingbench is a great tool for quickly generating loads on an Oracle database. It's easy to setup and has many configurable options. Although swingbench has a nice GUI interface for creating your test schemas and running your load, I really like the command line interface. With the CLI you can create scripts to interact with Swingbench and nohup loads on remote hosts so your load can run hours or days without needing to be logged in.

Setup

If you don't have it already, download a copy of Swingbench and unzip the files on your host machine. You can run Swingbench from your database host or a remote client. If you co-locate them on your database host, take this into account during load measurement. 

There are a few different types of schemas you can create with Swingbench, and each type has an associated XML wizard file in the bin directory to help with creating that schema. I tend to use the Order Entry (OE) schema the most as it's behavior is more representative of an OLTP system, so we will be using the oewizard.xml file for this example. Open up the XML file in your favorite editor and update the connection information for the system user that will create the schema, then run oewizard on the file like this...

oewizard -c oewizard.xml -cl -cs //<your_host_or_vip>/<service> -u <test_user_name> -p <test_user_pw> -ts <tablespace_name> -create -df <asm_disk_group> -part -scale 4 -debug

You can use -scale to adjust the size of your schema which will also increase the time it takes to build. A scale of 4 gives me about a 10G schema.  

Execution

When your schema is ready, edit the supplied swingconfig.xml file with your connection info and use charbench to verify your schema. 

charbench -c swingconfig.xml

With our schema ready, now we can define our load also using the swingconfig.xml file. There are a number of parameters you can adjust to define your load. Here are the ones I find affective.

  • <NumberOfUsers>40</NumberOfUsers>
  • <MinDelay>100</MinDelay>
  • <MaxDelay>1000</MaxDelay>
  • <LogonDelay>150</LogonDelay>
  • <WaitTillAllLogon>false</WaitTillAllLogon>

MinDelay and MaxDelay specify the wait time between transactions in milliseconds. A LogonDelay helps avoid connection storms (unless that's what you want to test) and I like setting WaitTillAllLogin to false so my load starts right away and there is a nice ramp up over time. If I want to push the system hard I set Min/MaxDelay low and increase the number of users.

Further down the swingconfig.xml file you will find descriptions of the actual transactions that will be executed. Each transaction type can be turned on/off and it's weighted value compared to other transactions can be adjusted. This section is were you will do most of your tweaking to get the load profile you want.

Example

Here's a Top Activity graph in Enterprise Manager showing two consecutive tests. The first test had 300 users with a Min/MaxDelay of 15/20. I decreased the Min/MaxDelay to 10/15 for an increased load which you can see below.  

Here's an example of a heavily overloaded system in which the application doesn't scale. I've setup Swingbench with 800 users connecting every 2 seconds for a slow buildup, Min/MaxDelay of 5/15, and I'm only using the "Order Products" transactions. These transactions perform single row inserts with no batching. Around 11:30am there are ~500 sessions and the system starts to overload. CPU has become saturated and other background processes like the database writers start to slowdown causing excessive Concurrency and Configuration waits in the buffer cache. Our SGA for this test was 10G.

overload example

Variable Loads With Swingbench


In order to generate variable swingbench loads over time, I've created a small wrapper script, variable_load.pl written in Perl that can be used to define how long your load should run and also the variation in that load. To adjust the load you define how many users will be connected. Here's a snippet of the script which describes each parameter.

### how long you want the test to run in hourse:minutes
$runtime = "00:30";

### your swingbench config file
$conf_file = 'swingconfig.xml';

### Adjust your vaiable loads here
###
### RunTime = how ling this load will run in hours:minutes
### UserCount = how many user connections
### SleepTime = how many seconds to sleep before running the load, if needed
###
###              RunTime  UserCount  SleepTime
@swing_list = ( ["00:02", 400,       120],
                ["00:05", 200,         0],
                ["00:05", 100,         0] ); 

With these settings here's what our load profile looks like.

variable load swingbench

Custom Load Generation


There have been times during my own performance testing in which I needed to generate a very specific type of load. Most recently, I needed to generate a heavy large block IO load, so I put together these scripts in response to that need. I tried to keep them easy to setup, run and alter if necessary. The load uses a single schema and creates a test table for each session that will be connected, so the load needs to be initialized based on the maximum number of sessions expected for testing.

Setup and Execution

  1. Download the package to your host and unzip/tar in an empty directory.
  2. Edit the load_env.sh file to setup variables for your test. This is where you will define the maximum number of test tables you will need.
  3. Run the init_load.sh script to setup your user test schema and test tables. You will be prompted for the SYSTEM user password.
  4. Run the start_load.sh script to begin your test. This script requires two parameters, the low and high values for the test tables to use and thus the number of sessions. This was done to allow running a small load and then ramping up and running additional loads as needed. Examples...
    • start_load.sh 1 10 : Will run ten sessions, using test tables 1 through 10.
    • start_load.sh 16 20 : Will run 5 sessions, using test tables 16 through 20.
    • start_load.sh 1 1 : Will run 1 session.
  5. Running stop_load.sh will kill all sessions and thus stop the load. 

Here's what a load of 20 users looks like. Lots of large block IO!

io load example

Custom variable loads can also be run using the variable_load.pl script found in the package. It has the same parameters to adjust as in the Swingbench script. Here's an example of a variable load that ramps up, overloads the system, then drops back down again.

variable load io

As the IO gets heavy we start seeing more contention in the log buffer. 

variable io waits

Customizations

It's possible to design your own custom loads with these scripts, as you may need to execute a particular PL/SQL package or perhaps test how well SQL will scale against a large partitioned table. This can be achieved by editing the init_object.sh and load_object.sh files.

init_object.sh : Edit this script to create or initialize any objects needed for your test. This script gets executed multiple times depending on how many sessions you plan to run concurrently. If you don't have a session specific setup, you can leave an empty code block.

load_object.sh : This is the code that gets executed for each session you define. If you had PL/SQL you wanted to test, this is where you would put it.

As an example, for this test I created some database links for each instance and altered the script to select from our test table using the database links, thus creating a heavy network load. I've included this example script load_object_network.sh in the package zip file as well.

network load

Ready!


With a set of tools to define consistent, predictable loads we are now ready to baseline our systems. Next in the series I will go over the tools available in Enterprise Manager which will help in that effort. 

Tuesday Aug 26, 2014

Exadata Health and Resource Usage Monitoring v2

A newly updated version of the Exadata  Health and Resource Usage monitoring has been released! This white paper documents an end to end approach to health and resource utilization monitoring for Oracle Exadata Environments. The document has been substantially modified to help Exadata administrators easily follow the troubleshooting methodology defined. Other additions include:

  • Exadata 12.1.0.6 plugin for Enterprise Manager new features
  • Enterprise Manager 12.1.0.4 updates
  • Updates to Include X4 environment

Download the white paper as the link below: 

http://www.oracle.com/technetwork/database/availability/exadata-health-resource-usage-2021227.pdf


Wednesday Apr 30, 2014

Latest Enterprise Manager Recommended Patch List

 
   
  

With the release of the latest quarterly patches, the list of recommended patches for managing exadata via Enterprise Manager has been updated.  The details for this patch recommendation can be found in My Oracle Support Note titled Patch Requirements for Setting up Monitoring and Administration for Exadata (Doc ID 1323298.1).

This note contains the recommended patches for the OMS servers, Repository database and the Agent.  It also contains the recommended plugin versions for the database, exadata and fusion middleware plugins as well as the latest bundle patches for these plugins. 

A My Oracle Support note has been created providing the patching steps for this list of recommended patches.  This note can be found here:  Applying Enterprise Manager 12c Recommended Patches (Doc ID 1664074.1)

Also, please refer to the following whitepaper for guidance in the Enterprise Manager patch application process: Oracle Enterprise Manager Software Planned Maintenance

Tuesday Apr 29, 2014

April 2014 Quarterly Full Stack Patch Availability

 
   
  

The latest Quarterly Full Stack patch was released in April for Exadata.  This patch contains all of the latest recommended patches for the Exadata Database Machine. 

For more details, refer to My Oracle Support Note titled Database Machine and Exadata Storage Server 11g Release 2 (11.2) Supported Versions [888828.1]

Note that the information in the note contains details for both Exadata Storage Server Software 11.2 and the new 12.1 release.  This note is maintained on a regular basis so bookmark it and use it as reference for the latest Exadata Database Machine patching recommendations.

Thursday Feb 27, 2014

Fast Recovery Area for Archive Destination


If you are using Fast Recovery Area (FRA) for the archive destination and the destination is set to USE_DB_RECOVERY_FILE_DEST, you may notice that the Archive Area % Used metric does not trigger anymore. Instead you will see the Recovery Area % Used metric trigger when it hits a Warning threshold of 85% full, and Critical of 97% full. Unfortunately, this metric is controlled by the database and the thresholds cannot be modified (see MOS Note 428473.1 for more information). Thresholds of 85/97 are not sufficient for some of the larger, busier databases. This may not give you enough time to kickoff a backup and clear enough logs before the archiver hangs. If you need different thresholds, you can easily accomplish this by creating a Metric Extension (ME) and setting thresholds to your desired values.  This blog will walk through an example of creating an ME to monitor archive on FRA destinations, for more information on ME's and how they can be used, refer to the Oracle Enterprise Manager Cloud Control Administrator's Guide

[Read More]

Monday Feb 24, 2014

Monitoring Archive Area % Used on Cluster Databases


One of the most critical events to monitor on an Oracle Database is your archive area. If the archive area fills up, your database will halt until it can continue to archive the redo logs. If your archive destination is set to a file system, then the Archive Area % Used metric is often the best way to go. This metric allows you to monitor a particular file system for the percentage space that has been used. However, there are a couple of things to be aware of for this critical metric.

Cluster Database vs. Database Instance

You will notice in EM 12c, the Archive Area metric exists on both the Cluster Database and the Database Instance targets. The majority of Cluster Databases (RAC) are built against database best practices which indicate that the Archive destination should be shared read/write between all instances. The purpose for this is that in case of recovery, any instance can perform the recovery and has all necessary archive logs to do so. Monitoring this destination for a Cluster Database at the instance level caused duplicate alerts and notifications, as both instances would hit the Warning/Critical threshold for Archive Area % Used within minutes of each other. To eliminate duplicate notifications, the Archive Area % Used metric for Cluster Databases was introduced. This allows the archive destination to be monitored at a database level, much like tablespaces are monitored in a RAC database.

In the Database Instance (RAC Instance) target, you will notice the Archive Area % Used metric collection schedule is set to Disabled.

If you have a RAC database and you do not share archive destinations between instances, you will want to Disable the Cluster Database metric, and enable the Database Instance metric to ensure that each destination is monitored individually.


Tuesday Feb 11, 2014

Oracle Enterprise Manager Software Planned Maintenance

 
   
  

A critical component of managing an application includes both patching and maintaining the software. Applying patches is not only required for bug fixes, it is also a means of obtaining new and beneficial functionality. Thus it becomes an important  task to maintain a healthy and productive Enterprise Manager (EM) solution. The process of patching itself can present different challenges that can potentially increase the work and time involved in each patching exercise. Issues could arise such as patch conflicts, not meeting required  prerequisites and even unnecessary downtime. Spending the proper time to setup a patching strategy can save time and effort as well as possible errors and risk when patching a production EM environment.

The MAA team has recently published a new whitepaper which provides an overview of the recommended patching strategy for Oracle Enterprise Manager.  This information is intended to be a guideline for maintaining a patched and highly available EM environment and may need customization to accommodate requirements of an individual organization.

There are five main topics covered in this paper as outlined below:

  1. Enterprise Manager Components

  2. Defining Business Requirements

  3. Patching Strategy Overview

    1. Types of Patches

    2. Define Patch List and Steps

    3. Planning

    4. Preparation

    5. Testing

  4. Sample Patching Steps

  5. Optional Patching Strategy

http://www.oracle.com/technetwork/database/availability/em-patching-bp-2132261.pdf


Friday Oct 25, 2013

Latest Exadata Quarterly Full Stack Patch

 
   
  

The latest Quarterly Full Stack patch was released on October 17th for Exadata.  This patch contains all of the latest recommended patches for the Exadata Database Machine. 

For more details, refer to My Oracle Support Note titled Database Machine and Exadata Storage Server 11g Release 2 (11.2) Supported Versions [888828.1]

Note that the information in the note applies only to Exadata Storage Server Software 11.2.  This note is maintained on a regular basis so bookmark it and use it as reference for the latest Exadata Database Machine patching recommendations.

Saturday Oct 12, 2013

Exadata Health and Resource Usage Monitoring

 
   
  

 MAA has recently published a new whitepaper documenting an end to end approach to health and resource utilization monitoring for Oracle Exadata Environments. In an addition to technical details a troubleshooting methodology will be explored that allows administrators to quickly identify and correct issues in an expeditious manner. 

The document takes a “rule out” approach in that components of the system will be verified as performing correctly to eliminate its role in the incident. There will be five areas of concentration in the overall system diagnosis 

1. Steps to take before problems occur that can assist in troubleshooting 

2. Changes made to the system 

3. Quick analysis 

4. Baseline comparison 

5. Advanced diagnostics

http://www.oracle.com/technetwork/database/availability/exadata-health-resource-usage-2021227.pdf


Friday Oct 04, 2013

Java Heap Size Settings For Enterprise Manager 12c

This blog is to provide an update to a previous blog (Oracle Enterprise Manager 12c Configuration Best Practices (Part 1 of 3)) on how to increase the java heap size for an OMS running release 12cR3.  The entire series can be found in the My Oracle Support note titled Oracle Enterprise Manager 12c Configuration Best Practices [1553342.1].

Increase JAVA Heap Size

For larger enterprises, there may be a need to increase the amount of memory used for the OMS.  One of the symptoms of this condition is a “sluggish” performance on the OMS.  If it is determined that the OMS needs more memory, it is done by increasing the JAVA heap size parameters.  However, it is very important to increase this parameter incrementally and be careful not to consume all of the memory on the server.  Also, java does not always perform better with more memory. 

Verify:  The parameters for the java heap size are stored in the following file:

<MW_HOME>/user_projects/domains/GCDomain/bin/startEMServer.sh

Recommendation:  If you have more than 250 agents, increase the -Xmx parameter which specifies the maximum size for the java heap to 2 gb.  As the number of agents grows, it can be incrementally increased.  Note:  Do not increase this larger than 4gb without contacting Oracle.  Change only the –Xmx value in the line containing USER_MEM_ARGS="-Xms256m –Xmx1740m …options…" as seen in the example below.   Do not change the Xms or MaxPermSize values. Note:  change both lines as seen below.  The second occurrence will be used if running in debug mode.

Steps to modify the Java setting for versions prior to 12cR3 (12.1.0.3)

Before

 if [ "${SERVER_NAME}" != "EMGC_ADMINSERVER" ] ; then
  USER_MEM_ARGS="-Xms256m -Xmx1740m
 -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing 
-XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+CMSClassUnloadingEnabled"
  if [ "${JAVA_VENDOR}" = "Sun" ] ; then
    if [ "${PRODUCTION_MODE}" = "" ] ; then
      USER_MEM_ARGS="-Xms256m -Xmx1740m
 -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing 
-XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+CMSClassUnloadingEnabled -XX:CompileThreshold=8000 
-XX:PermSize=128m"
    fi
  fi
  export USER_MEM_ARGS
fi

After

 if [ "${SERVER_NAME}" != "EMGC_ADMINSERVER" ] ; then
  USER_MEM_ARGS="-Xms256m -Xmx2560m -XX:MaxPermSize=768M
 -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing 
-XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+CMSClassUnloadingEnabled"
  if [ "${JAVA_VENDOR}" = "Sun" ] ; then
    if [ "${PRODUCTION_MODE}" = "" ] ; then
      USER_MEM_ARGS="-Xms256m –Xmx2560m
 -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing 
-XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+CMSClassUnloadingEnabled -XX:CompileThreshold=8000 
-XX:PermSize=128m"
    fi
  fi
  export USER_MEM_ARGS
fi

Steps to modify the Java setting for version 12.1.0.3

emctl set property -name JAVA_EM_MEM_ARGS -value "<value>"
emctl stop oms -all
emctl start oms

Please note that this value gets seeded inside emgc.properties and is used to start the OMS.  Please be careful setting this property as this would be the property used by the OMS to start and the oms can fail to start if it is not specified correctly.  Below is an example of the command:

emctl set property -name JAVA_EM_MEM_ARGS -value "-Xms256m -Xmx2048m -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing -XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled"

 
    

Friday Aug 23, 2013

Creating a disk monitoring metric extension for Exadata Compute Nodes


 It is highly desirable to monitor the Exadata Compute node disks for current failures or degraded performance. By using the Enterprise Manager metric extension functionality Compute nodes can be monitored for these conditions and an alert created in the event of an issue. The following steps will guide you through this process


1. First a root monitoring credential set must be created . Login into the OMS using emcli
$ ./emcli login -username=sysman
Enter password :
Login successful

2. Create the credential set:
$ ./emcli create_credential_set -set_name=root_monitoring -target_type=cluster -supported_cred_types=HostCreds -descript=root_monitoring monitoring
Create credential set for target type host

3. Next login to EM and go to the monitoring credentials page to setup credentials for a test target

Setup--> Security-->Monitoring Credentials
Select Cluster and the push the "Manage Monitoring Credentials" button
Find the target you want to test on with the credential set defined in step 2( In this case root_monitoring)
Highlight the credential set and push the "Set Credentials" button. Enter the credentials and use the test and save button to ensure they are correctly defined


4. Next create the metric extenstion

Enterprise-->Monitoring-->Metric Extensions

Select the create button

5. On the General Properties Screen set the following

Target type select "Host"
Name "Compute_Node_Disk_Monitoring"
Display Name "Compute Node Disk Monitoring"
Adapter "OS Command - Multiple Columns"
Data Collection "Enabled"
Repeat Every "5 Minutes"
Use of Metric Data "Alerting and Historical Trending"
Upload Interval "1 Collections"
Select the Next Button

6. Now create the script to run on the agent

On your local machine create a file called megaclicommand.sh that contains the following
/opt/MegaRAID/MegaCli/MegaCli64 AdpAllInfo -aALL | grep "Virtual Drives" -A 6 | grep -w 'Degraded\|Critical\|Offline\|Failed' | sed 's/Degraded/Virtual Drives Degrades/g' | sed 's/Offline/Virtual Drives Offline/g' | sed 's/Critical Disks/Critical Physical Disks/g' | sed 's/Failed Disks/Failed Physical Disks/g'

7. On the "Edit Storage Server Disk Status (ME$Storage_Server_Disk_Status) v1 : Adapter" page enter the following

Command "/bin/sh"
Script "%scriptsDir%/megaclicommand.sh"
Delimiter ":"

8. On the Upload Custom Files Section

Select the upload button and select the file created in step 6
Click okay and one back to the Create New:Adapter page select the "Next" button

9. On the "Create New : Columns" page create two columns

Column one should be setup as:
Name "Type"
Display Name "Type"
Column Type "Key Column"
Value Type "String"
Metric Category "Fault"
Column two should be setup as:
Name "Value"
Display Name "Value"
Column Type "Data Column"
Value Type "Number"
Metric Category "Fault"
Comparison Operator ">"
Critical "0"
After Setting up the two column select the next button

10. On The Credentials Screen

Select the “Specify Credential Set” radio button
In the drop down box select the credential set created in step 1
Click the next button
11. On the “Create New : Test” page

Add a target to test with in the “Test Targets” section
Click the “Run Test” button and ensure that results are displayed properly in the “Test Results” box.
The results should be similar to below
Type Value
Virtual Drives Degrades 0
Virtual Drives Offline 0
Critical Physical Disks 0
Failed Physical Disks 0

12. Next the the Metric Extension must be saved as a deployable draft. This is accomplished on the main metric extension page. This allows the metric to be deployed to targets for testing. However at this stage only the developer has access to publish the metric. After satisfactory testing is completed the metric is then published. This is once again accomplished from the main metric extension page.


To ensure that administrators are notified in the event the metric created fails an incident rule should be created.

1, To Begin navigate to the Incident Rules Home Page
         From the Setup button on the upper right hand corner of the Enterprise Overview Page
         Setup->Security->Incident Rules

         Now click the “Create Rule Set..” button

 2. On the Create Rule Set screen enter the following information

        Name: Whatever the rule should be called. i.e. Metric Collection Error Rule
        Enabled Box: Should be checked
        Type: Enterprise
        Applies To: Targets
        Select the “All Targets of Type” radio button on the bottom of the screen followed by Host in the drop down box

3. Now select the “Rules” tab at the bottom of the screen

4. Chose the "Create.." button on the middle of the screen

4. On the “Select Type of Rule to Create” Popup box select the “Incoming events and updates to events” radio button. Click the continue button.

5. On the “Create New Rule: Select Events” screen check the type check box. In the drop down select “Metric Extension Update”. Click the next button

6. On the “Add conditional Actions” page you can specify conditions that must occur for the action to apply, if an incident should be created and email notifications. Specify the desired properties and select the continue button.

7. If no additional rules are required select the next button on the “Create New Rule: Add Actions” page.

8. On the next screen either accept the default rule name or specify the desired name

9. For the “Create New Rule : Review” page, ensure everything looks correct and select the “continue button”

10. Lastly click the “Save” button to complete the setup

11. The metric can now be deployed to the desired target by selecting the “Deploy to Targets” option from the “Actions” drop down button on the Metric Extensions Page

Monday Aug 19, 2013

Simplified Agent and Plug-in Deployment

On your site of hundreds or thousands of hosts have you had to patch agents immediately as they get deployed?  For this reason I’ve always been a big fan of cloning an agent that has the required plug-ins and all the recommended core agent and plug-in patches, then using that clone for all new agent deployments. With Oracle Enterprise Manager 12c this got even easier as you can now clone the agent using the console “Add Host” method. You still have to rely on the EM users to use the clone. The one problem I have with cloning is that you have to have a reference target for each platform that you support. If you have a consolidated environment and only have Linux x64, this may not be a problem. If you are managing a typical data center with a mixture of platforms, it can become quite the maintenance nightmare just to maintain your golden images. You must update golden image agents whenever you get a new patch (generic or platform specific) for the agent or plug-in, and recreate the clone for each platform. Typically, I find people create a clone for their most common platforms, and forget about the rest. That means, maybe 80% of their agents meet their standard patch requirements and plug-ins upon deployment, but the other 20% have to be patched post-deploy, or worse – never get patched!

While deployed agents and plug-ins can be patched easily using EM Patches & Updates, but what about the agents still getting deployed or upgraded? Wouldn’t it be nice if they got patched as part of the deployment or upgrade? This article will show you two new features in EM 12.1.0.3 (EM 12cR3) that will help you deploy the most current agent and plug-in versions. Whether you have 100s or 1000s of agents to manage, reducing maintenance and keeping the agents up to date is an important task, and being able to deploy or upgrade to a fully patched agent will save you a lot of time and effort.

[Read More]
About

bocadmin_ww

Search

Archives
« May 2015
SunMonTueWedThuFriSat
     
1
2
3
4
5
6
7
8
9
10
11
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
      
Today