Friday Jun 05, 2015

Maintaining Trace and Log Files for an Enterprise Manager 12c Environment

Maintaining Trace and Log Files for an Enterprise Manager 12c Environment

In an EM 12c environment, the majority of the log and trace files for the OMS, WebLogic Server, HTTP Server, Agent and database are maintained automatically. However, there are a few exceptions. For these exceptions, the size/retention of the files must be maintained manually. Also, for most of the automated methods, the details such as file size and number of files to retain can also be controlled. This is an important item to understand to help ensure that log and trace files are maintained for the expected period of time and that the amount of disk space consumed by these type of files is also maintained. Below are some helpful tips for managing these files as well as pointers to MOS notes and Oracle documentation for further information.

Management Agent Log and Trace Files

Agent logs are segmented and have a limited overall size and hence need not be deleted or managed. The agent log files are located in the $AGENT_INST/sysman/log directory. The log files are segmented (archived) into a configurable number of segments of a configurable size. These settings are controlled by properties in emd.properties but should be modified via the EMCTL utility. The latest segment is always filename.log and the oldest is the filename.log.X where X is the highest number. For more details on the Management Agent log and trace files, refer to the Oracle Enterprise Manager Cloud Control Administrator’s Guide. For details on these emctl setproperty commands, refer to MOS note 12c Cloud Control Agent: How to Configure Logging for the 12.1.0.1 Management Agent? [1363757.1]

Key Considerations:

  • Control the size of the logs: The size of each of the individual agent log files is controlled by the property <handler>.totalSize. This property will specify the total size in MB for the file segments. Once the file reaches this size, it will be archived to a new file. The default setting is 100M. Here is an example of finding the value for the gcagent.log file in 12c: emctl getproperty agent –name “Logger.log.totalSize”. For more details on setting the max size of the agent log files, refer to the documentation and MOS links above.

  • Control when the agent writes to log files: The amount of data that an agent will log is controlled by the property <handler>.level where the possible levels are DEBUG, INFO,WARN, ERROR, and FATAL. By default, this is set to INFO which means that logs messages with the level of INFO and above (WARN, ERROR, and FATAL) will be logged. The recommendation is to keep the default setting unless debugging an agent issue. Then, set the logging level to DEBUG for specific modules only rather than changing the root logging level. For more details on setting the agent logging level, refer to the documentation and MOS links above.

  • Control when to purge log files: The number of archived log segments that will be maintained is specified by the property <handler>.segment.count. The default setting varies by log file. Here is an example of finding the value for the gcagent.log file in 12c: emctl getproperty agent –name “Logger.log.segment.count”. For more details on setting the number of log files to retain, refer to the documentation and MOS links above

OMS Trace and Log Files

The maintenance of trace and log files on the OMS servers can be handled automatically via properties set for the different components.

Oracle Management Service (OMS) Log and Trace Files

The OMS log and trace files are stored under the middleware_home/gc_inst/em/OMS_NAME/sysman/log directory. These files are segmented (archived) into a configurable number of segments of a configurable size so they do not need to be manually deleted or managed. These settings are controlled by properties in emomslogging.properties but should be modified via the EMCTL utility. For more details on the OMS log and trace files, refer to the Oracle Enterprise Manager Cloud Control Administrator’s Guide. For details on the location of the OMS log and trace files as well as the commands to modify the logging and tracing parameters, refer to MOS note EM 12c: Steps to Locate and Manage the Various Logs/Trace files in a 12c OMS Installation [1448308.1]

Key Considerations:

  • Control the size of the logs and trace files: the size of the individual log and trace files are controlled by the properties log4j.appender.emlogAppender.MaxFileSize and log4j.appender.emtrcAppender.MaxFileSize respectively. These properties will specify the total size in bytes for the file segments. Once a file reaches this size, it will be archived to a new file. Here is an example of finding the value for the gcagent.log file in 12c: emctl get property –name “log4j.appender.emlogAppender.MaxFileSize”. For more details on setting the max size of the OMS log files, refer to the documentation and MOS links above.

  • Control when the OMS writes to log files: The amount of data that an OMS will log is controlled by the property –value “<LEVEL>” where the possible levels are DEBUG, INFO,WARN, ERROR, and FATAL. The specific category or logging module name must also be specified. The recommendation is to keep the default setting unless debugging an issue. Then, set the logging level to DEBUG for specific modules only rather than changing the root logging level which can cause a lot of messages to be written to the trace files and potentially slowing down the system. For more details on setting the OMS logging level, refer to the documentation and MOS links above

  • Control when to purge log files: The number of archived log and trace files that will be maintained is specified by the properties log4j.appender.emlogAppender.MaxBackupIndex and log4j.appender.emtrcAppender.MaxBackupIndex respectively. Here is an example of finding the value for the gcagent.log file in 12c: emctl get property –name “log4j.appender.emlogAppender.MaxBackupIndex”. For more details on setting the max number of OMS log and trace files to retain, refer to the documentation and MOS links above.

Oracle WebLogic Server and HTTP Server Log Files

The different WebLogic Server components generate their own log files. These files are stored under different sub-directories in the middleware_home/gc_inst location. For the majority of these log files, the size and number of files can be maintained automatically. To maintain the size and number of backup files, the log files are segmented (archived) into a new segment of a configurable size based on a configurable rotation type. These settings can be set in the WebLogic Server Administration Console or via the WLST utility. Starting with the 12.1.0.2 OMS release onwards, the log file rotation and retention options are set out-of-the box for the GCDomain.log*, EMGC_ADMINSERVER.log* and access.log* files. For more details on the WebLogic and HTTP Server log files, refer to the Oracle Fusion Middleware Administrator's Guide. For the steps required to modify the rotation file size and retained limit for the GCDomain.log*, EMGC_ADMINSERVER.log* and access.log* files if required, refer to MOS note 12 Cloud Control: How to Enable Log Rotation Policy and Automatically Delete Older GCDomain.log, EMGC_ADMINSERVER.log and access.log Files? [1450535.1]

Key Considerations:

  • Control the size of the logs files: the size of the individual log files can be based on file size or time. If the Rotation type “By Size” is selected, the log file will be archived to a new segment once the file reaches a specified size. If the Rotation type “By Time” is selected, the log file will be archived to a new segment according to the specified Rotation interval (in hours). The default setting is a rotation type of “By Size” and a files size of 5M. These default settings should be sufficient for most EM installations. If needed, the settings can be modified by connecting to the Oracle WebLogic Server Administration Console. For more details on how to modify this setting, refer to the MOS note above.

  • Control when the WebLogic and HTTP servers write to log files: The amount of data that will be logged can be controlled by specifying the level for the log and different message destinations. Some of the different levels are Debug, Info, Notice, Warning, Trace, Error, Critical, Alert, Emergency and Off. The default setting is Warning for the log files and Error for the domain log broadcaster file. These settings should be sufficient for most EM installations. If needed, the settings can be modified by connecting to the Oracle WebLogic Server Administration Console.

  • Control when to purge log files: The maximum number of archived log files that will be maintained can be controlled by selecting the option to limit the number of retained files and then specifying the number of files to retain. This number does not include the current log file. The default setting is to retain 10 files and should be sufficient for most EM installations. If needed, this setting can be modified by connecting to the Oracle WebLogic Server Administration Console. For more details on how to modify this setting, refer to the MOS note above.

NOTE: The following log files need to be maintained and manually purged. One method for addressing these files would be to create a cronjob that would find all files older than a specific period of time and delete them. Here is an example crontab entry to remove all of the *.out* files under the OMS instance domain directory that are older than 30 days:

00 * * * * cd /u01/app/oracle/gc_inst/user_projects/domain;find . -name "*.out*" -mtime +30 -exec rm {} \;
  • All files under middleware_home/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs#/.

  • Log files under the admin server and emgc_oms server:

o middleware_home/gc_inst/user_projects/domains/<domain_name>/servers/EMGC_ADMINSERVER/logs/*.out*

o middleware_home/gc_inst/user_projects/domains/<domain_name>/servers/EMGC_OMS#/logs/*.out*

Database Trace and Log Files

Starting with release 11g of Oracle Database, all diagnostic data including the alert log is stored in the Automatic Diagnostic Repository (ADR). Each instance of each product stores diagnostic data underneath its own ADR home directory. The ADR homes are grouped together under the same directory referred to as the ADR Base. This location is set via the DIAGNOSTIC_DEST initialization parameter. The different ADR home directories that are known to the Automatic Diagnostic Repository Command Interpreter (ADRCI) utility can be seen by issuing a “show homes” command. To execute any commands such as purge or list, the ADRCI utility must be told which home to operate against via the “set home adr_home” command. Failure to do this will result in this error: DIA-48448: This command does not support multiple ADR homes.

Alert Log

The alert log is stored as an XML file in the ADR and can be viewed with Enterprise Manager and with the ADRCI utility. Oracle also provides a text-formatted version of the alert log for backward compatibility. For details on using the ADRCI PURGE command, refer to the Oracle Database Utilities 12c Release 1 (12.1) guide.

Key Considerations:

  • Control the size of the text based alert log: the size of the alert log must be controlled manually. It is safe to delete the alert log while the instance is running but it is recommended to create an archived copy first for possible future reference.

  • Control when to purge the XML based alert log file: The ADRCI utility can be used to purge the XML-formatted alert log file. The content is only automatically purged based on these purging policies. Therefore, data in the files that does not meet the time in the purging policy is maintained. Alert log content is subject to purging after one year (long-lived or LONGP_POLICY). To see the default purging policies for long-lived ADR content, issue the SHOW CONTROL command in ADRCI. The values are specified in hours. The default values should be sufficient for most database installations. NOTE: if you have multiple homes on the server, you must issue the SET HOMEPATH command to a specific home before issuing a SHOW CONTROL command. The content can also be purged manually. Here is an example of a command to purge the alert log content older than 30 days (note that the age is specified in minutes): purge -age 43200 -type alert. For further details on the ADRCI utility refer to the Oracle Database Utilities 12c Release 1 (12.1) guide.

Trace Files

Trace files are created by server and background processes, the SQL trace facility and also by enabling SQL tracing for a session or an instance. The file extension for trace files is .trc (i.e. orcl_ora_762.trc). Trace files are sometimes accompanied by a corresponding trace map file which contains structural information about trace files. These trace map files use the file extension .trm.

Key Considerations:

  • Control the size of the trace files: the maximum size of all trace files can be controlled using the initialization parameter MAX_FILE_SIZE. This will limit the size of trace files to a specific number of operating system blocks.

  • Control when the database writes to trace files: The only background process that allows this control is the ARCn process. This is controlled via the LOG_ARCHIVE_TRACE initialization parameter. There are multiple trace levels that can be set which control the amount of trace data written. For more details, refer to the Oracle Database Administrator’s Guide 11c Release 1 (12.1) guide.

  • Control when to purge trace files: Trace files can be purged from the ADR home based on purging policies. The content is only purged based on these purging policies. Therefore, data in the files that does not meet the time in the purging policy is maintained. Incidents and incident dump content are subject to purging after one year (long-lived or LONGP_POLICY) however the trace files, core dumps and incident packaging information are subject to purging after 30 days (short-lived or SHORTP_POLICY). Some Oracle products (Oracle Database) automatically purge diagnostic data at the end of its life cycle. The default values should be sufficient for most database installations. To see the default purging policies for short and long-lived ADR content, issue the SHOW CONTROL command in ADRCI. The values are specified in hours. NOTE: if you have multiple homes on the server, you must issue the SET HOMEPATH command to a specific home before issuing a SHOW CONTROL command. The content can also be purged manually. Here is an example of a command to purge the trace files (including dumps) older than 30 days (note that the age is specified in minutes): purge -age 43200 -type trace. For further details on the ADRCI utility refer to the Oracle Database Utilities 12c Release 1 (12.1) guide.

Listener Log Files

The listener log, much like the alert log, is stored in the ADR home for the listener in an xml file. It is also stored in a text-formatted version for backward compatibility.

Key Considerations:

  • Control the size of the trace files: When the XML-formatted listener file (log.xml) reaches 10MB in size, it will be archived into a file named log_1.xml, log_2.xml, etc. This only applies to the XML-formatted file.

  • Purge listener log files: Listener log files and information within a listener log are not purged via the automatic purge. Unlike the RDBMS product, the network products have not supported purging. Therefore, the maintenance of the listener log and trace files is manual. The XML-formatted files can be purged using the ADRCI utility as with the XML-formatted alert logs.

NOTE: I have found on an 11.2 RAC install that the ADR location for the LISTENER trace and log files is not the same as for the SCAN LISTENER log and trace files. When connecting into the ADRCI utility, it will show one ADR Base location only. This can be seen when issuing the “show base” command. To point to the ADR Base for the Oracle Grid install, issue the “set base” command. For example: set base /u01/app/11.2.0.4/grid/log. After issuing the set base command, then a “show homes” will show all of the diag locations that can be controlled. Be sure to issue the “set home” command before purging trace files. The locations are as follows:

  • ADR Base for LISTENER log and trace files - This is based on the DIAGNOSTIC_DEST initialization parameter.

  • ADR Base for the SCAN LISTENER log and trace files is located under the GRID_HOME/log directory regardless of the value for the DIAGNOSTIC_DEST initialization parameter in the ASM or database instances.

REFERENCES

· Master Note for 12c Cloud Control OMS [1957139.1]

· 12 Cloud Control: How to Enable Log Rotation Policy and Automatically Delete Older GCDomain.log, EMGC_ADMINSERVER.log and access.log Files? [1450535.1]

· EM 12c: Steps to Locate and Manage the Various Logs/Trace files in a 12c OMS Installation [1448308.1]

· 12c Cloud Control Agent: How to Configure Logging for the 12.1.0.1 Management Agent? [1363757.1]

· Oracle Database Utilities 12c Release 1 (12.1)

· Oracle Database Administrator’s Guide 11c Release 1 (12.1)

· Oracle Fusion Middleware Administrator's Guide

Friday Oct 04, 2013

Java Heap Size Settings For Enterprise Manager 12c

This blog is to provide an update to a previous blog (Oracle Enterprise Manager 12c Configuration Best Practices (Part 1 of 3)) on how to increase the java heap size for an OMS running release 12cR3.  The entire series can be found in the My Oracle Support note titled Oracle Enterprise Manager 12c Configuration Best Practices [1553342.1].

Increase JAVA Heap Size

For larger enterprises, there may be a need to increase the amount of memory used for the OMS.  One of the symptoms of this condition is a “sluggish” performance on the OMS.  If it is determined that the OMS needs more memory, it is done by increasing the JAVA heap size parameters.  However, it is very important to increase this parameter incrementally and be careful not to consume all of the memory on the server.  Also, java does not always perform better with more memory. 

Verify:  The parameters for the java heap size are stored in the following file:

<MW_HOME>/user_projects/domains/GCDomain/bin/startEMServer.sh

Recommendation:  If you have more than 250 agents, increase the -Xmx parameter which specifies the maximum size for the java heap to 2 gb.  As the number of agents grows, it can be incrementally increased.  Note:  Do not increase this larger than 4gb without contacting Oracle.  Change only the –Xmx value in the line containing USER_MEM_ARGS="-Xms256m –Xmx1740m …options…" as seen in the example below.   Do not change the Xms or MaxPermSize values. Note:  change both lines as seen below.  The second occurrence will be used if running in debug mode.

Steps to modify the Java setting for versions prior to 12cR3 (12.1.0.3)

Before

 if [ "${SERVER_NAME}" != "EMGC_ADMINSERVER" ] ; then
  USER_MEM_ARGS="-Xms256m -Xmx1740m
 -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing 
-XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+CMSClassUnloadingEnabled"
  if [ "${JAVA_VENDOR}" = "Sun" ] ; then
    if [ "${PRODUCTION_MODE}" = "" ] ; then
      USER_MEM_ARGS="-Xms256m -Xmx1740m
 -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing 
-XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+CMSClassUnloadingEnabled -XX:CompileThreshold=8000 
-XX:PermSize=128m"
    fi
  fi
  export USER_MEM_ARGS
fi

After

 if [ "${SERVER_NAME}" != "EMGC_ADMINSERVER" ] ; then
  USER_MEM_ARGS="-Xms256m -Xmx2560m -XX:MaxPermSize=768M
 -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing 
-XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+CMSClassUnloadingEnabled"
  if [ "${JAVA_VENDOR}" = "Sun" ] ; then
    if [ "${PRODUCTION_MODE}" = "" ] ; then
      USER_MEM_ARGS="-Xms256m –Xmx2560m
 -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing 
-XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+CMSClassUnloadingEnabled -XX:CompileThreshold=8000 
-XX:PermSize=128m"
    fi
  fi
  export USER_MEM_ARGS
fi

Steps to modify the Java setting for version 12.1.0.3

emctl set property -name JAVA_EM_MEM_ARGS -value "<value>"
emctl stop oms -all
emctl start oms

Please note that this value gets seeded inside emgc.properties and is used to start the OMS.  Please be careful setting this property as this would be the property used by the OMS to start and the oms can fail to start if it is not specified correctly.  Below is an example of the command:

emctl set property -name JAVA_EM_MEM_ARGS -value "-Xms256m -Xmx2048m -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing -XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled"

 
    

About

bocadmin_ww

Search

Archives
« September 2015
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today