Monday Jul 27, 2015

July 2015 EM Recommended Patch List

 
   
  

As part of the recent July 2015 PSU Patch release for Enterprise Manager, a list of recommended patches has been compiled for EM 12c to be included in the Exadata Quarterly Full Stack Download patch.  This list contains recommended patches for the EM infrastructure including the OMS, WLS, Plugins, Repository and Agent. 

For more details and to review these EM recommended patches for Exadata monitoring, refer to My Oracle Support Note titled Patch Requirements for Setting up Monitoring and Administration for Exadata [1323298.1]

Also, the information in the note contains not only a recommended list of patches for both EM versions 12.1.0.4 and 12.1.0.3 but also provides a link to a My Oracle Support Note that provides the steps for applying these patches. This note is titled Applying Enterprise Manager 12c Recommended Patches [1664074.1].  The patch apply process in this note provides maximum availability for the EM environment by structuring the patch apply steps so that the primary OMS server is up as long as possible.

Friday Jun 05, 2015

Maintaining Trace and Log Files for an Enterprise Manager 12c Environment

Maintaining Trace and Log Files for an Enterprise Manager 12c Environment

In an EM 12c environment, the majority of the log and trace files for the OMS, WebLogic Server, HTTP Server, Agent and database are maintained automatically. However, there are a few exceptions. For these exceptions, the size/retention of the files must be maintained manually. Also, for most of the automated methods, the details such as file size and number of files to retain can also be controlled. This is an important item to understand to help ensure that log and trace files are maintained for the expected period of time and that the amount of disk space consumed by these type of files is also maintained. Below are some helpful tips for managing these files as well as pointers to MOS notes and Oracle documentation for further information.

Management Agent Log and Trace Files

Agent logs are segmented and have a limited overall size and hence need not be deleted or managed. The agent log files are located in the $AGENT_INST/sysman/log directory. The log files are segmented (archived) into a configurable number of segments of a configurable size. These settings are controlled by properties in emd.properties but should be modified via the EMCTL utility. The latest segment is always filename.log and the oldest is the filename.log.X where X is the highest number. For more details on the Management Agent log and trace files, refer to the Oracle Enterprise Manager Cloud Control Administrator’s Guide. For details on these emctl setproperty commands, refer to MOS note 12c Cloud Control Agent: How to Configure Logging for the 12.1.0.1 Management Agent? [1363757.1]

Key Considerations:

  • Control the size of the logs: The size of each of the individual agent log files is controlled by the property <handler>.totalSize. This property will specify the total size in MB for the file segments. Once the file reaches this size, it will be archived to a new file. The default setting is 100M. Here is an example of finding the value for the gcagent.log file in 12c: emctl getproperty agent –name “Logger.log.totalSize”. For more details on setting the max size of the agent log files, refer to the documentation and MOS links above.

  • Control when the agent writes to log files: The amount of data that an agent will log is controlled by the property <handler>.level where the possible levels are DEBUG, INFO,WARN, ERROR, and FATAL. By default, this is set to INFO which means that logs messages with the level of INFO and above (WARN, ERROR, and FATAL) will be logged. The recommendation is to keep the default setting unless debugging an agent issue. Then, set the logging level to DEBUG for specific modules only rather than changing the root logging level. For more details on setting the agent logging level, refer to the documentation and MOS links above.

  • Control when to purge log files: The number of archived log segments that will be maintained is specified by the property <handler>.segment.count. The default setting varies by log file. Here is an example of finding the value for the gcagent.log file in 12c: emctl getproperty agent –name “Logger.log.segment.count”. For more details on setting the number of log files to retain, refer to the documentation and MOS links above

OMS Trace and Log Files

The maintenance of trace and log files on the OMS servers can be handled automatically via properties set for the different components.

Oracle Management Service (OMS) Log and Trace Files

The OMS log and trace files are stored under the middleware_home/gc_inst/em/OMS_NAME/sysman/log directory. These files are segmented (archived) into a configurable number of segments of a configurable size so they do not need to be manually deleted or managed. These settings are controlled by properties in emomslogging.properties but should be modified via the EMCTL utility. For more details on the OMS log and trace files, refer to the Oracle Enterprise Manager Cloud Control Administrator’s Guide. For details on the location of the OMS log and trace files as well as the commands to modify the logging and tracing parameters, refer to MOS note EM 12c: Steps to Locate and Manage the Various Logs/Trace files in a 12c OMS Installation [1448308.1]

Key Considerations:

  • Control the size of the logs and trace files: the size of the individual log and trace files are controlled by the properties log4j.appender.emlogAppender.MaxFileSize and log4j.appender.emtrcAppender.MaxFileSize respectively. These properties will specify the total size in bytes for the file segments. Once a file reaches this size, it will be archived to a new file. Here is an example of finding the value for the gcagent.log file in 12c: emctl get property –name “log4j.appender.emlogAppender.MaxFileSize”. For more details on setting the max size of the OMS log files, refer to the documentation and MOS links above.

  • Control when the OMS writes to log files: The amount of data that an OMS will log is controlled by the property –value “<LEVEL>” where the possible levels are DEBUG, INFO,WARN, ERROR, and FATAL. The specific category or logging module name must also be specified. The recommendation is to keep the default setting unless debugging an issue. Then, set the logging level to DEBUG for specific modules only rather than changing the root logging level which can cause a lot of messages to be written to the trace files and potentially slowing down the system. For more details on setting the OMS logging level, refer to the documentation and MOS links above

  • Control when to purge log files: The number of archived log and trace files that will be maintained is specified by the properties log4j.appender.emlogAppender.MaxBackupIndex and log4j.appender.emtrcAppender.MaxBackupIndex respectively. Here is an example of finding the value for the gcagent.log file in 12c: emctl get property –name “log4j.appender.emlogAppender.MaxBackupIndex”. For more details on setting the max number of OMS log and trace files to retain, refer to the documentation and MOS links above.

Oracle WebLogic Server and HTTP Server Log Files

The different WebLogic Server components generate their own log files. These files are stored under different sub-directories in the middleware_home/gc_inst location. For the majority of these log files, the size and number of files can be maintained automatically. To maintain the size and number of backup files, the log files are segmented (archived) into a new segment of a configurable size based on a configurable rotation type. These settings can be set in the WebLogic Server Administration Console or via the WLST utility. Starting with the 12.1.0.2 OMS release onwards, the log file rotation and retention options are set out-of-the box for the GCDomain.log*, EMGC_ADMINSERVER.log* and access.log* files. For more details on the WebLogic and HTTP Server log files, refer to the Oracle Fusion Middleware Administrator's Guide. For the steps required to modify the rotation file size and retained limit for the GCDomain.log*, EMGC_ADMINSERVER.log* and access.log* files if required, refer to MOS note 12 Cloud Control: How to Enable Log Rotation Policy and Automatically Delete Older GCDomain.log, EMGC_ADMINSERVER.log and access.log Files? [1450535.1]

Key Considerations:

  • Control the size of the logs files: the size of the individual log files can be based on file size or time. If the Rotation type “By Size” is selected, the log file will be archived to a new segment once the file reaches a specified size. If the Rotation type “By Time” is selected, the log file will be archived to a new segment according to the specified Rotation interval (in hours). The default setting is a rotation type of “By Size” and a files size of 5M. These default settings should be sufficient for most EM installations. If needed, the settings can be modified by connecting to the Oracle WebLogic Server Administration Console. For more details on how to modify this setting, refer to the MOS note above.

  • Control when the WebLogic and HTTP servers write to log files: The amount of data that will be logged can be controlled by specifying the level for the log and different message destinations. Some of the different levels are Debug, Info, Notice, Warning, Trace, Error, Critical, Alert, Emergency and Off. The default setting is Warning for the log files and Error for the domain log broadcaster file. These settings should be sufficient for most EM installations. If needed, the settings can be modified by connecting to the Oracle WebLogic Server Administration Console.

  • Control when to purge log files: The maximum number of archived log files that will be maintained can be controlled by selecting the option to limit the number of retained files and then specifying the number of files to retain. This number does not include the current log file. The default setting is to retain 10 files and should be sufficient for most EM installations. If needed, this setting can be modified by connecting to the Oracle WebLogic Server Administration Console. For more details on how to modify this setting, refer to the MOS note above.

NOTE: The following log files need to be maintained and manually purged. One method for addressing these files would be to create a cronjob that would find all files older than a specific period of time and delete them. Here is an example crontab entry to remove all of the *.out* files under the OMS instance domain directory that are older than 30 days:

00 * * * * cd /u01/app/oracle/gc_inst/user_projects/domain;find . -name "*.out*" -mtime +30 -exec rm {} \;
  • All files under middleware_home/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs#/.

  • Log files under the admin server and emgc_oms server:

o middleware_home/gc_inst/user_projects/domains/<domain_name>/servers/EMGC_ADMINSERVER/logs/*.out*

o middleware_home/gc_inst/user_projects/domains/<domain_name>/servers/EMGC_OMS#/logs/*.out*

Database Trace and Log Files

Starting with release 11g of Oracle Database, all diagnostic data including the alert log is stored in the Automatic Diagnostic Repository (ADR). Each instance of each product stores diagnostic data underneath its own ADR home directory. The ADR homes are grouped together under the same directory referred to as the ADR Base. This location is set via the DIAGNOSTIC_DEST initialization parameter. The different ADR home directories that are known to the Automatic Diagnostic Repository Command Interpreter (ADRCI) utility can be seen by issuing a “show homes” command. To execute any commands such as purge or list, the ADRCI utility must be told which home to operate against via the “set home adr_home” command. Failure to do this will result in this error: DIA-48448: This command does not support multiple ADR homes.

Alert Log

The alert log is stored as an XML file in the ADR and can be viewed with Enterprise Manager and with the ADRCI utility. Oracle also provides a text-formatted version of the alert log for backward compatibility. For details on using the ADRCI PURGE command, refer to the Oracle Database Utilities 12c Release 1 (12.1) guide.

Key Considerations:

  • Control the size of the text based alert log: the size of the alert log must be controlled manually. It is safe to delete the alert log while the instance is running but it is recommended to create an archived copy first for possible future reference.

  • Control when to purge the XML based alert log file: The ADRCI utility can be used to purge the XML-formatted alert log file. The content is only automatically purged based on these purging policies. Therefore, data in the files that does not meet the time in the purging policy is maintained. Alert log content is subject to purging after one year (long-lived or LONGP_POLICY). To see the default purging policies for long-lived ADR content, issue the SHOW CONTROL command in ADRCI. The values are specified in hours. The default values should be sufficient for most database installations. NOTE: if you have multiple homes on the server, you must issue the SET HOMEPATH command to a specific home before issuing a SHOW CONTROL command. The content can also be purged manually. Here is an example of a command to purge the alert log content older than 30 days (note that the age is specified in minutes): purge -age 43200 -type alert. For further details on the ADRCI utility refer to the Oracle Database Utilities 12c Release 1 (12.1) guide.

Trace Files

Trace files are created by server and background processes, the SQL trace facility and also by enabling SQL tracing for a session or an instance. The file extension for trace files is .trc (i.e. orcl_ora_762.trc). Trace files are sometimes accompanied by a corresponding trace map file which contains structural information about trace files. These trace map files use the file extension .trm.

Key Considerations:

  • Control the size of the trace files: the maximum size of all trace files can be controlled using the initialization parameter MAX_FILE_SIZE. This will limit the size of trace files to a specific number of operating system blocks.

  • Control when the database writes to trace files: The only background process that allows this control is the ARCn process. This is controlled via the LOG_ARCHIVE_TRACE initialization parameter. There are multiple trace levels that can be set which control the amount of trace data written. For more details, refer to the Oracle Database Administrator’s Guide 11c Release 1 (12.1) guide.

  • Control when to purge trace files: Trace files can be purged from the ADR home based on purging policies. The content is only purged based on these purging policies. Therefore, data in the files that does not meet the time in the purging policy is maintained. Incidents and incident dump content are subject to purging after one year (long-lived or LONGP_POLICY) however the trace files, core dumps and incident packaging information are subject to purging after 30 days (short-lived or SHORTP_POLICY). Some Oracle products (Oracle Database) automatically purge diagnostic data at the end of its life cycle. The default values should be sufficient for most database installations. To see the default purging policies for short and long-lived ADR content, issue the SHOW CONTROL command in ADRCI. The values are specified in hours. NOTE: if you have multiple homes on the server, you must issue the SET HOMEPATH command to a specific home before issuing a SHOW CONTROL command. The content can also be purged manually. Here is an example of a command to purge the trace files (including dumps) older than 30 days (note that the age is specified in minutes): purge -age 43200 -type trace. For further details on the ADRCI utility refer to the Oracle Database Utilities 12c Release 1 (12.1) guide.

Listener Log Files

The listener log, much like the alert log, is stored in the ADR home for the listener in an xml file. It is also stored in a text-formatted version for backward compatibility.

Key Considerations:

  • Control the size of the trace files: When the XML-formatted listener file (log.xml) reaches 10MB in size, it will be archived into a file named log_1.xml, log_2.xml, etc. This only applies to the XML-formatted file.

  • Purge listener log files: Listener log files and information within a listener log are not purged via the automatic purge. Unlike the RDBMS product, the network products have not supported purging. Therefore, the maintenance of the listener log and trace files is manual. The XML-formatted files can be purged using the ADRCI utility as with the XML-formatted alert logs.

NOTE: I have found on an 11.2 RAC install that the ADR location for the LISTENER trace and log files is not the same as for the SCAN LISTENER log and trace files. When connecting into the ADRCI utility, it will show one ADR Base location only. This can be seen when issuing the “show base” command. To point to the ADR Base for the Oracle Grid install, issue the “set base” command. For example: set base /u01/app/11.2.0.4/grid/log. After issuing the set base command, then a “show homes” will show all of the diag locations that can be controlled. Be sure to issue the “set home” command before purging trace files. The locations are as follows:

  • ADR Base for LISTENER log and trace files - This is based on the DIAGNOSTIC_DEST initialization parameter.

  • ADR Base for the SCAN LISTENER log and trace files is located under the GRID_HOME/log directory regardless of the value for the DIAGNOSTIC_DEST initialization parameter in the ASM or database instances.

REFERENCES

· Master Note for 12c Cloud Control OMS [1957139.1]

· 12 Cloud Control: How to Enable Log Rotation Policy and Automatically Delete Older GCDomain.log, EMGC_ADMINSERVER.log and access.log Files? [1450535.1]

· EM 12c: Steps to Locate and Manage the Various Logs/Trace files in a 12c OMS Installation [1448308.1]

· 12c Cloud Control Agent: How to Configure Logging for the 12.1.0.1 Management Agent? [1363757.1]

· Oracle Database Utilities 12c Release 1 (12.1)

· Oracle Database Administrator’s Guide 11c Release 1 (12.1)

· Oracle Fusion Middleware Administrator's Guide

Tuesday May 12, 2015

Creating an Enterprise Manager Metric Extension to monitor Huge Page Allocation


Huge Pages enables the Linux kernel to manage large pages of memory in addition to the standard 4KB page size. If you have a system with more than 16GB of memory running Oracle databases with a total System Global Area (SGA) larger than 8GB, you should enable the HugePages feature to improve database performance.

If Huge Pages is enabled but the system isn't configured to allow for the requested huge page allocation excessive swapping will occur. This will result in degraded database performance.

To ensure this doesn't happen a Metric Extension can be created in Enterprise Manager to notify administrators if the system is incorrectly configured.

The following will describe the process of creating the Metric Extension in Enterprise Manager


Create the script that will be uploaded to the OMS:


1. On your local machine create a file called check_hugepage_usage.sh that contains the following

RECD_SIZE=0

CFG_TOTAL=`grep HugePages_Total /proc/meminfo | awk '{print $2}'`

SIZE=$((`grep  Hugepagesize /proc/meminfo | awk '{print $2}'` * 1024))

for SHMSEGS in `ipcs -m | awk '{print $5}' | sed -n '4,$p'`

do

    SEGSIZE=$(($SHMSEGS / $SIZE))

  if [ $SEGSIZE -gt 0 ]; then

   RECD_SIZE=$(($RECD_SIZE+$SEGSIZE+1))

   fi

done 

DIFF=$(($RECD_SIZE-$CFG_TOTAL))

echo $RECD_SIZE:$CFG_TOTAL:$DIFF


Next create the Metric Extension


1. Create the metric extension in the Enterprise Manager Console


Enterprise-->Monitoring-->Metric Extensions

Select the create button-->Metric Extension

2. On the General Properties Screen set the following


Target type select "Host"

Name "huge_page_monitoring"

Display Name "Huge Page Monitoring"

Adapter "OS Command - Multiple Columns"

Data Collection "Enabled"

Repeat Every "12 Hours"

Use of Metric Data "Alerting and Historical Trending"

Upload Interval "1 Collections"

Select the Next Button


3. On the  “Adapter" page enter the following


Command '\/bin\/ksh'

Select the upload button by the “Script�� text box and upload the file created in step 1

Delimiter ":"

Select the upload button and select the file created in step 1

Select the Next button



4. On the "Create New : Columns" page Use the Add-->New metric column button create three columns with the add button


Column one should be setup as:

Name "Recommended_Value"

Display Name "Recommended_Value"

Column Type "Data Column"

Value Type "Number"

Metric Category "Capacity"


Column two should be setup as:

Name "Configured_Value"

Display Name "Configured Value"

Column Type "Data Column"

Value Type "Number"

Metric Category "Capacity"

Comparison Operator ">"

Critical "0"


Column three should be setup as:

Name "Huge_Page_Shortage"

Display Name "Configured Value"

Column Type "Data Column"

Value Type "Number"

Metric Category "Capacity"

Comparison Operator ">"

Critical "0"


After Setting up the three columns select the next button

5. On The Credentials Screen


Select the “Default Monitoring Credentials” radio button

Click the next button

6. On the “Create New : Test” page


Add a target to test with in the “Test Targets” section

Click the “Run Test” button and ensure that results are displayed properly in the “Test Results” box.

The results should be similar to below

Target Name Recommended Value Configured Value Huge Page Shortage 

Targetname 31745 36000 0


Click the Finish button


7. Next the Metric Extension must be saved as a deployable draft. This is accomplished on the main metric extension page. This allows the metric to be deployed to targets for testing. However at this stage only the developer has access to publish the metric. After satisfactory testing is completed the metric is then published. This is once again accomplished from the main metric extension page.


8. Once testing is complete and the Metric extension is deployable. It can be deployed to targets. This is accomplished from the metric extension page

Highlight the metric extension just created. From the Actions drop down box select “Deploy to Targets…”



To ensure that administrators are notified in the event the metric created fails an incident rule should be created.


1. To Begin navigate to the Incident Rules Home Page

         From the Setup button on the upper right hand corner of the Enterprise Overview Page

         Setup->Incidents->Incident Rules

         Now click the “Create Rule Set.” button

2. On the Create Rule Set screen enter the following information

        Name: Whatever the rule should be called. i.e. Metric Collection Error Rule

        Enabled Box: Should be checked

        Type: Enterprise

Applies To: Targets

Select the “All Targets of Type” radio button on the bottom of the screen followed by Host in the drop down box


3. Now select the “Rules” tab at the bottom of the screen


4. Chose the "Create.." button on the middle of the screen


5. On the “Select Type of Rule to Create” Popup box select the “Incoming events and updates to events” radio button. Click the continue button.


6. On the “Create New Rule: Select Events” screen check the type check box. In the drop down select “Metric Alert". Click the next button


7. Select the “Add” button 

On the “Add conditional Actions” page you can specify conditions that must occur for the action to apply, if an incident should be created and email notifications. Specify the desired properties and select the continue button.


8. On the Target Type drop down select “Host”

Select the search button and select the metric extension created above

Select the check box next to the desired metric

Select the desired severity status in the check boxes at the bottom of the screen 

Click the “OK” button


9. Click the “Next” button back on the  “Add Actions” page


10. Click next and select the desired option on the next pages and complete creating the rule set.

 For Detailed Information on creating Metric Extensions please refer to:

http://docs.oracle.com/cd/E24628_01/doc.121/e24473/metric_extension.htm#EMADM10035 


Wednesday Apr 22, 2015

April 2015 EM Recommended Patch List

 
   
  

As part of the recent April 2015 PSU Patch release for Enterprise Manager, a list of recommended patches has been compiled for EM 12c to be included in the Exadata Quarterly Full Stack Download patch.  This list contains recommended patches for the EM infrastructure including the OMS, WLS, Plugins, Repository and Agent. 

For more details and to review these EM recommended patches for Exadata monitoring, refer to My Oracle Support Note titled Patch Requirements for Setting up Monitoring and Administration for Exadata [1323298.1]

Also, the information in the note contains not only a recommended list of patches for both EM versions 12.1.0.4 and 12.1.0.3 but also provides a link to a My Oracle Support Note that provides the steps for applying these patches. This note is titled Applying Enterprise Manager 12c Recommended Patches [1664074.1].  The patch apply process in this note provides maximum availability for the EM environment by structuring the patch apply steps so that the primary OMS server is up as long as possible.

April 2015 Exadata Quarterly Full Stack Patch Availability

 
   
  

The latest Quarterly Full Stack patch was recently released for Exadata.  This patch contains all of the latest recommended patches for the Exadata Database Machine. 

For more details, refer to My Oracle Support Note titled Database Machine and Exadata Storage Server 11g Release 2 (11.2) Supported Versions [888828.1]

Note that the information in the note contains details for both Exadata Storage Server Software 12.1.1.1.2 and 12.1.2.1.1 releases.  This note is maintained on a regular basis so bookmark it and use it as reference for the latest Exadata Database Machine patching recommendations.

Thursday Nov 06, 2014

Tools For Generating Consistent Loads

It's finally ready. The new database machine you've spent months, planning and architecting. All those shiny new components perfectly aligned for extreme performance. Will it give you the results you expect? How can you baseline and compare your new and existing systems? Pointing your application at your new machine may take some time to setup and depending on the behavior of the application, may not stress test all hardware components as you might like. You need a set of easy to configure scripts for load testing and you need tools and procedures to compare old systems to new. 

This will be the first of three blog posts to help with that effort. In this post, I'll go over some tools you can use to generate various loads. The next two posts in the series I'll talk about using Enterprise Manager to evaluate and baseline loads, and strategies to compare systems using consistent loads.

Test Suite

My current test suite includes two methods to generate loads: one leverages Swingbench, which is a well known and popular load generating tool, and the other is a solution I cooked up myself. Both sets of scripts can be altered to tailor their load characteristics. I've also included a variable load script wrapper for each, which you can use to adjust the load over time. For example: you can have a load test that runs for a total of 5 hours and within that 5 hour window your load could fluctuate every 30 minutes from heavy to light. The custom scripts are also flexible enough to support altering their behavior if you have a specific set of SQL/PLSQL commands you would like to run.

For this article, my database is running on an Exadata X2-2 quarter rack.

Using Swingbench


Swingbench is a great tool for quickly generating loads on an Oracle database. It's easy to setup and has many configurable options. Although swingbench has a nice GUI interface for creating your test schemas and running your load, I really like the command line interface. With the CLI you can create scripts to interact with Swingbench and nohup loads on remote hosts so your load can run hours or days without needing to be logged in.

Setup

If you don't have it already, download a copy of Swingbench and unzip the files on your host machine. You can run Swingbench from your database host or a remote client. If you co-locate them on your database host, take this into account during load measurement. 

There are a few different types of schemas you can create with Swingbench, and each type has an associated XML wizard file in the bin directory to help with creating that schema. I tend to use the Order Entry (OE) schema the most as it's behavior is more representative of an OLTP system, so we will be using the oewizard.xml file for this example. Open up the XML file in your favorite editor and update the connection information for the system user that will create the schema, then run oewizard on the file like this...

oewizard -c oewizard.xml -cl -cs //<your_host_or_vip>/<service> -u <test_user_name> -p <test_user_pw> -ts <tablespace_name> -create -df <asm_disk_group> -part -scale 4 -debug

You can use -scale to adjust the size of your schema which will also increase the time it takes to build. A scale of 4 gives me about a 10G schema.  

Execution

When your schema is ready, edit the supplied swingconfig.xml file with your connection info and use charbench to verify your schema. 

charbench -c swingconfig.xml

With our schema ready, now we can define our load also using the swingconfig.xml file. There are a number of parameters you can adjust to define your load. Here are the ones I find affective.

  • <NumberOfUsers>40</NumberOfUsers>
  • <MinDelay>100</MinDelay>
  • <MaxDelay>1000</MaxDelay>
  • <LogonDelay>150</LogonDelay>
  • <WaitTillAllLogon>false</WaitTillAllLogon>

MinDelay and MaxDelay specify the wait time between transactions in milliseconds. A LogonDelay helps avoid connection storms (unless that's what you want to test) and I like setting WaitTillAllLogin to false so my load starts right away and there is a nice ramp up over time. If I want to push the system hard I set Min/MaxDelay low and increase the number of users.

Further down the swingconfig.xml file you will find descriptions of the actual transactions that will be executed. Each transaction type can be turned on/off and it's weighted value compared to other transactions can be adjusted. This section is were you will do most of your tweaking to get the load profile you want.

Example

Here's a Top Activity graph in Enterprise Manager showing two consecutive tests. The first test had 300 users with a Min/MaxDelay of 15/20. I decreased the Min/MaxDelay to 10/15 for an increased load which you can see below.  

Here's an example of a heavily overloaded system in which the application doesn't scale. I've setup Swingbench with 800 users connecting every 2 seconds for a slow buildup, Min/MaxDelay of 5/15, and I'm only using the "Order Products" transactions. These transactions perform single row inserts with no batching. Around 11:30am there are ~500 sessions and the system starts to overload. CPU has become saturated and other background processes like the database writers start to slowdown causing excessive Concurrency and Configuration waits in the buffer cache. Our SGA for this test was 10G.

overload example

Variable Loads With Swingbench


In order to generate variable swingbench loads over time, I've created a small wrapper script, variable_load.pl written in Perl that can be used to define how long your load should run and also the variation in that load. To adjust the load you define how many users will be connected. Here's a snippet of the script which describes each parameter.

### how long you want the test to run in hourse:minutes
$runtime = "00:30";

### your swingbench config file
$conf_file = 'swingconfig.xml';

### Adjust your vaiable loads here
###
### RunTime = how ling this load will run in hours:minutes
### UserCount = how many user connections
### SleepTime = how many seconds to sleep before running the load, if needed
###
###              RunTime  UserCount  SleepTime
@swing_list = ( ["00:02", 400,       120],
                ["00:05", 200,         0],
                ["00:05", 100,         0] ); 

With these settings here's what our load profile looks like.

variable load swingbench

Custom Load Generation


There have been times during my own performance testing in which I needed to generate a very specific type of load. Most recently, I needed to generate a heavy large block IO load, so I put together these scripts in response to that need. I tried to keep them easy to setup, run and alter if necessary. The load uses a single schema and creates a test table for each session that will be connected, so the load needs to be initialized based on the maximum number of sessions expected for testing.

Setup and Execution

  1. Download the package to your host and unzip/tar in an empty directory.
  2. Edit the load_env.sh file to setup variables for your test. This is where you will define the maximum number of test tables you will need.
  3. Run the init_load.sh script to setup your user test schema and test tables. You will be prompted for the SYSTEM user password.
  4. Run the start_load.sh script to begin your test. This script requires two parameters, the low and high values for the test tables to use and thus the number of sessions. This was done to allow running a small load and then ramping up and running additional loads as needed. Examples...
    • start_load.sh 1 10 : Will run ten sessions, using test tables 1 through 10.
    • start_load.sh 16 20 : Will run 5 sessions, using test tables 16 through 20.
    • start_load.sh 1 1 : Will run 1 session.
  5. Running stop_load.sh will kill all sessions and thus stop the load. 

Here's what a load of 20 users looks like. Lots of large block IO!

io load example

Custom variable loads can also be run using the variable_load.pl script found in the package. It has the same parameters to adjust as in the Swingbench script. Here's an example of a variable load that ramps up, overloads the system, then drops back down again.

variable load io

As the IO gets heavy we start seeing more contention in the log buffer. 

variable io waits

Customizations

It's possible to design your own custom loads with these scripts, as you may need to execute a particular PL/SQL package or perhaps test how well SQL will scale against a large partitioned table. This can be achieved by editing the init_object.sh and load_object.sh files.

init_object.sh : Edit this script to create or initialize any objects needed for your test. This script gets executed multiple times depending on how many sessions you plan to run concurrently. If you don't have a session specific setup, you can leave an empty code block.

load_object.sh : This is the code that gets executed for each session you define. If you had PL/SQL you wanted to test, this is where you would put it.

As an example, for this test I created some database links for each instance and altered the script to select from our test table using the database links, thus creating a heavy network load. I've included this example script load_object_network.sh in the package zip file as well.

network load

Ready!


With a set of tools to define consistent, predictable loads we are now ready to baseline our systems. Next in the series I will go over the tools available in Enterprise Manager which will help in that effort. 

Tuesday Aug 26, 2014

Exadata Health and Resource Usage Monitoring v2

A newly updated version of the Exadata  Health and Resource Usage monitoring has been released! This white paper documents an end to end approach to health and resource utilization monitoring for Oracle Exadata Environments. The document has been substantially modified to help Exadata administrators easily follow the troubleshooting methodology defined. Other additions include:

  • Exadata 12.1.0.6 plugin for Enterprise Manager new features
  • Enterprise Manager 12.1.0.4 updates
  • Updates to Include X4 environment

Download the white paper as the link below: 

http://www.oracle.com/technetwork/database/availability/exadata-health-resource-usage-2021227.pdf


Wednesday Apr 30, 2014

Latest Enterprise Manager Recommended Patch List

 
   
  

With the release of the latest quarterly patches, the list of recommended patches for managing exadata via Enterprise Manager has been updated.  The details for this patch recommendation can be found in My Oracle Support Note titled Patch Requirements for Setting up Monitoring and Administration for Exadata (Doc ID 1323298.1).

This note contains the recommended patches for the OMS servers, Repository database and the Agent.  It also contains the recommended plugin versions for the database, exadata and fusion middleware plugins as well as the latest bundle patches for these plugins. 

A My Oracle Support note has been created providing the patching steps for this list of recommended patches.  This note can be found here:  Applying Enterprise Manager 12c Recommended Patches (Doc ID 1664074.1)

Also, please refer to the following whitepaper for guidance in the Enterprise Manager patch application process: Oracle Enterprise Manager Software Planned Maintenance

Tuesday Apr 29, 2014

April 2014 Quarterly Full Stack Patch Availability

 
   
  

The latest Quarterly Full Stack patch was released in April for Exadata.  This patch contains all of the latest recommended patches for the Exadata Database Machine. 

For more details, refer to My Oracle Support Note titled Database Machine and Exadata Storage Server 11g Release 2 (11.2) Supported Versions [888828.1]

Note that the information in the note contains details for both Exadata Storage Server Software 11.2 and the new 12.1 release.  This note is maintained on a regular basis so bookmark it and use it as reference for the latest Exadata Database Machine patching recommendations.

Thursday Feb 27, 2014

Fast Recovery Area for Archive Destination


If you are using Fast Recovery Area (FRA) for the archive destination and the destination is set to USE_DB_RECOVERY_FILE_DEST, you may notice that the Archive Area % Used metric does not trigger anymore. Instead you will see the Recovery Area % Used metric trigger when it hits a Warning threshold of 85% full, and Critical of 97% full. Unfortunately, this metric is controlled by the database and the thresholds cannot be modified (see MOS Note 428473.1 for more information). Thresholds of 85/97 are not sufficient for some of the larger, busier databases. This may not give you enough time to kickoff a backup and clear enough logs before the archiver hangs. If you need different thresholds, you can easily accomplish this by creating a Metric Extension (ME) and setting thresholds to your desired values.  This blog will walk through an example of creating an ME to monitor archive on FRA destinations, for more information on ME's and how they can be used, refer to the Oracle Enterprise Manager Cloud Control Administrator's Guide

[Read More]

Monday Feb 24, 2014

Monitoring Archive Area % Used on Cluster Databases


One of the most critical events to monitor on an Oracle Database is your archive area. If the archive area fills up, your database will halt until it can continue to archive the redo logs. If your archive destination is set to a file system, then the Archive Area % Used metric is often the best way to go. This metric allows you to monitor a particular file system for the percentage space that has been used. However, there are a couple of things to be aware of for this critical metric.

Cluster Database vs. Database Instance

You will notice in EM 12c, the Archive Area metric exists on both the Cluster Database and the Database Instance targets. The majority of Cluster Databases (RAC) are built against database best practices which indicate that the Archive destination should be shared read/write between all instances. The purpose for this is that in case of recovery, any instance can perform the recovery and has all necessary archive logs to do so. Monitoring this destination for a Cluster Database at the instance level caused duplicate alerts and notifications, as both instances would hit the Warning/Critical threshold for Archive Area % Used within minutes of each other. To eliminate duplicate notifications, the Archive Area % Used metric for Cluster Databases was introduced. This allows the archive destination to be monitored at a database level, much like tablespaces are monitored in a RAC database.

In the Database Instance (RAC Instance) target, you will notice the Archive Area % Used metric collection schedule is set to Disabled.

If you have a RAC database and you do not share archive destinations between instances, you will want to Disable the Cluster Database metric, and enable the Database Instance metric to ensure that each destination is monitored individually.


Tuesday Feb 11, 2014

Oracle Enterprise Manager Software Planned Maintenance

 
   
  

A critical component of managing an application includes both patching and maintaining the software. Applying patches is not only required for bug fixes, it is also a means of obtaining new and beneficial functionality. Thus it becomes an important  task to maintain a healthy and productive Enterprise Manager (EM) solution. The process of patching itself can present different challenges that can potentially increase the work and time involved in each patching exercise. Issues could arise such as patch conflicts, not meeting required  prerequisites and even unnecessary downtime. Spending the proper time to setup a patching strategy can save time and effort as well as possible errors and risk when patching a production EM environment.

The MAA team has recently published a new whitepaper which provides an overview of the recommended patching strategy for Oracle Enterprise Manager.  This information is intended to be a guideline for maintaining a patched and highly available EM environment and may need customization to accommodate requirements of an individual organization.

There are five main topics covered in this paper as outlined below:

  1. Enterprise Manager Components

  2. Defining Business Requirements

  3. Patching Strategy Overview

    1. Types of Patches

    2. Define Patch List and Steps

    3. Planning

    4. Preparation

    5. Testing

  4. Sample Patching Steps

  5. Optional Patching Strategy

http://www.oracle.com/technetwork/database/availability/em-patching-bp-2132261.pdf


Friday Oct 25, 2013

Latest Exadata Quarterly Full Stack Patch

 
   
  

The latest Quarterly Full Stack patch was released on October 17th for Exadata.  This patch contains all of the latest recommended patches for the Exadata Database Machine. 

For more details, refer to My Oracle Support Note titled Database Machine and Exadata Storage Server 11g Release 2 (11.2) Supported Versions [888828.1]

Note that the information in the note applies only to Exadata Storage Server Software 11.2.  This note is maintained on a regular basis so bookmark it and use it as reference for the latest Exadata Database Machine patching recommendations.

Saturday Oct 12, 2013

Exadata Health and Resource Usage Monitoring

 
   
  

 MAA has recently published a new whitepaper documenting an end to end approach to health and resource utilization monitoring for Oracle Exadata Environments. In an addition to technical details a troubleshooting methodology will be explored that allows administrators to quickly identify and correct issues in an expeditious manner. 

The document takes a “rule out” approach in that components of the system will be verified as performing correctly to eliminate its role in the incident. There will be five areas of concentration in the overall system diagnosis 

1. Steps to take before problems occur that can assist in troubleshooting 

2. Changes made to the system 

3. Quick analysis 

4. Baseline comparison 

5. Advanced diagnostics

http://www.oracle.com/technetwork/database/availability/exadata-health-resource-usage-2021227.pdf


Friday Oct 04, 2013

Java Heap Size Settings For Enterprise Manager 12c

This blog is to provide an update to a previous blog (Oracle Enterprise Manager 12c Configuration Best Practices (Part 1 of 3)) on how to increase the java heap size for an OMS running release 12cR3.  The entire series can be found in the My Oracle Support note titled Oracle Enterprise Manager 12c Configuration Best Practices [1553342.1].

Increase JAVA Heap Size

For larger enterprises, there may be a need to increase the amount of memory used for the OMS.  One of the symptoms of this condition is a “sluggish” performance on the OMS.  If it is determined that the OMS needs more memory, it is done by increasing the JAVA heap size parameters.  However, it is very important to increase this parameter incrementally and be careful not to consume all of the memory on the server.  Also, java does not always perform better with more memory. 

Verify:  The parameters for the java heap size are stored in the following file:

<MW_HOME>/user_projects/domains/GCDomain/bin/startEMServer.sh

Recommendation:  If you have more than 250 agents, increase the -Xmx parameter which specifies the maximum size for the java heap to 2 gb.  As the number of agents grows, it can be incrementally increased.  Note:  Do not increase this larger than 4gb without contacting Oracle.  Change only the –Xmx value in the line containing USER_MEM_ARGS="-Xms256m –Xmx1740m …options…" as seen in the example below.   Do not change the Xms or MaxPermSize values. Note:  change both lines as seen below.  The second occurrence will be used if running in debug mode.

Steps to modify the Java setting for versions prior to 12cR3 (12.1.0.3)

Before

 if [ "${SERVER_NAME}" != "EMGC_ADMINSERVER" ] ; then
  USER_MEM_ARGS="-Xms256m -Xmx1740m
 -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing 
-XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+CMSClassUnloadingEnabled"
  if [ "${JAVA_VENDOR}" = "Sun" ] ; then
    if [ "${PRODUCTION_MODE}" = "" ] ; then
      USER_MEM_ARGS="-Xms256m -Xmx1740m
 -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing 
-XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+CMSClassUnloadingEnabled -XX:CompileThreshold=8000 
-XX:PermSize=128m"
    fi
  fi
  export USER_MEM_ARGS
fi

After

 if [ "${SERVER_NAME}" != "EMGC_ADMINSERVER" ] ; then
  USER_MEM_ARGS="-Xms256m -Xmx2560m -XX:MaxPermSize=768M
 -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing 
-XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+CMSClassUnloadingEnabled"
  if [ "${JAVA_VENDOR}" = "Sun" ] ; then
    if [ "${PRODUCTION_MODE}" = "" ] ; then
      USER_MEM_ARGS="-Xms256m –Xmx2560m
 -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing 
-XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+CMSClassUnloadingEnabled -XX:CompileThreshold=8000 
-XX:PermSize=128m"
    fi
  fi
  export USER_MEM_ARGS
fi

Steps to modify the Java setting for version 12.1.0.3

emctl set property -name JAVA_EM_MEM_ARGS -value "<value>"
emctl stop oms -all
emctl start oms

Please note that this value gets seeded inside emgc.properties and is used to start the OMS.  Please be careful setting this property as this would be the property used by the OMS to start and the oms can fail to start if it is not specified correctly.  Below is an example of the command:

emctl set property -name JAVA_EM_MEM_ARGS -value "-Xms256m -Xmx2048m -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing -XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled"

 
    

About

bocadmin_ww

Search

Archives
« August 2015
SunMonTueWedThuFriSat
      
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
     
Today