Tuesday Nov 27, 2012

Using Oracle Enterprise Manager Ops Center to Update Solaris via Live Upgrade

Introduction: LU99_AD

This Oracle Enterprise Manager Ops Center blog entry provides tips for using Ops Center to update Solaris using Live Upgrade on Solaris 10 and Boot Environments on Solaris 11.

Why use Live Upgrade?

  • Live Upgrade (LU) can significantly reduce downtime associated with patching
  • Live Upgrade avoids dropping to single-user mode for long periods of time during patching
  • Live Upgrade relies on an Alternate Boot Environment (ABE)/(BE), which is patched while in multi-user mode; thereby allowing normal system operations to continue with the active BE, while the alternate BE is being patched
  • Activating a newly patched (A)BE is essentially a reboot; therefore the downtime is ~= reboot
  • Admins can easily revert to the prior Boot Environment (BE) as a safeguard / fallback.

Why use Ops Center to patch via Live Upgrade, Alternate Boot Environments, and Solaris 11 equivalents?

  • All the benefits of Ops Center's extensive patch and package knowledge base can be leveraged on top of Live Upgrade
  • Ops Center can orchestrate patching based on Live Upgrade and Solaris 11 features, which all works together to minimize downtime
  • Ops Center's advanced inventory and reporting features assure that each OS is updated to a verifiable, consistent standard, rather than relying on ad-hoc (error prone) procedures and scripts
  • Ops Center gives admins control over the boot environment specifications or they can let Ops Center decide when a BE is necessary, thereby reducing complexity and lowering the opportunity for user error

Preparing to use Live Upgrade-like features in Solaris 11

Requirements and information you should know:

  • Global Zone Root file-systems must be separate from Solaris Container / Zone filesystems
  • Solaris 11 has features which are similar in concept to Live Upgrade on Solaris 10, but differ greatly in implementation
    Important distinctions:
    • Solaris 11 assumes ZFS root
    • Solaris 11 adds Boot Environments (BE's) as an integrated feature (see beadm)
    • Solaris 11 BE's avoid single-user patching (vs. Solaris 10 w/ ZFS snapshot=ABE).
    • Solaris 11 Image Packaging System (IPS) has hooks for BE creation, as needed
      • Solaris 11 allows pkgs to be installed + upgraded in alternate BE (e.g. instead of the live system) but it is controlled on a per-pkg basis
      • Boot Environments are activated across a reboot; instead of spending long periods installing + upgrading packages in single user mode.
    • Fallback to a prior BE is a function of the BE infrastructure (a la beadm).
    • (Generally) Reboot + BE activation can be much much faster on Solaris 11

Preparing to use Live Upgrade on Solaris 10

Requirements and information you should know:

  • Global Zone Root file-systems must be separate from Solaris Container / Zone filesystems
  • Live Upgrade Pre-requisite patches must be applied before the first Live Upgrade Alternate Boot Environments are created (see "Pre-requisite Patches" section, below...)
  • Solaris 10 Update 6 or newer on ZFS root is the practical starting point for Live Upgrade
    • Live Upgrade with ZFS root is far more straight-forward than any scheme based on Alternative Boot Environments in slices or temporarily breaking mirrors
    • Use Solaris best practices to upgrade the OS to at least Solaris 10 Update 4 (outside of Ops Center)
    • UFS root can (technically) be used, but it is significantly more involved (e.g. discouraged) -- there are many reasons to move to ZFS while going through the process to update to Solaris 10 Update 6 or newer (out side of Ops Center)
  • Recommendation: Start with Solaris 10 Update 6 or newer on ZFS root
  • Recommendation: Start with Ops Center 12c or newer
    • Ops Center 12c can automatically create your ABE's for you, without the need for custom scripts
    • Ops Center 12c Update 2 avoids kernel panic on unpatched Solaris 10 update 9 (and older) -- unrelated to Live Upgrade, but more on the issue, below.
NOTE: There is no magic!  If you have systems running Solaris 10 Update 5 or older on UFS root, and you don't know how to get them updated to Solaris 10 on ZFS root, then there are services available from Oracle Advanced Customer Support (ACS), which specialize in this area.

Live Upgrade Pre-requisite Patches (Solaris 10)

Certain Live Upgrade related patches must be present before the first Live Upgrade ABE's are created on Solaris 10.
Use the following MOS Search String to find the “living document” that outlines the required patch minimums, which are necessary before using any Live Upgrade features:

Solaris Live Upgrade Software Patch Requirements

(Click above – the link is valid as of this writing, but search in MOS for the same "Solaris Live Upgrade Software Patch Requirements" string if necessary)

It is a very good idea to check the document periodically and adapt to its contents, accordingly.

IMPORTANT:  In case it wasn't clear in the above document, some direct patching of the active OS, including a reboot, may be required before Live Upgrade can be successfully used the first time.

HINT: You can use Ops Center to determine what to expect for a given system, and to schedule the “pre-patching” during a maintenance window if necessary.

Preparing to use Ops Center

  • Discover + Manage (Install + Configure the Ops Center agent in) each Global Zone
  • Recommendation:  Begin by using OCDoctor --agent-prereq to determine whether OS meets OC prerequisites (resolve any issues)
  • See prior requirements and recommendations w.r.t. starting with Solaris 10 Update 6 or newer on ZFS (or at least Solaris 10 Update 4 on UFS, with caveats)

  • WARNING: Systems running unpatched Solaris 10 update 9 (or older) should run the Ops Center 12c Update 2 agent to avoid a potential kernel panic
    • The 12c Update 2 agent will check patch minimums and disable certain process accounting features if the kernel is not sufficiently patched to avoid the panic
      • SPARC: 142900-05 Obsoleted by: 142900-06 SunOS 5.10: kernel patch 10 Oracle Solaris on SPARC (32-bit)
      • X64: 142901-05 Obsoleted by: 142901-06 SunOS 5.10_x86: kernel patch 10 Oracle Solaris on x86 (32-bit)
    • OR
      • SPARC: 142909-17 SunOS 5.10: kernel patch 10 Oracle Solaris on SPARC (32-bit)
      • X64: 142910-17 SunOS 5.10_x86: kernel patch 10 Oracle Solaris on x86 (32-bit)
    • Ops Center 12c (initial release) and 12c Update 1 agent can also be safely used with a workaround (to be performed BEFORE installing the agent):
# mkdir -p /etc/opt/sun/oc
# echo "zstat_exacct_allowed=false" > /etc/opt/sun/oc/zstat.conf
# chmod 755 /etc/opt/sun /etc/opt/sun/oc
# chmod 644 /etc/opt/sun/oc/zstat.conf
# chown -Rh root:sys /etc/opt/sun/oc

NOTE: Remove the above after patching the OS sufficiently, or after upgrading to the 12c Update 2 agent

Using Ops Center to apply Live Upgrade-related Pre-Patches (Solaris 10)
Overview:

  • Create an OS Update Profile containing the minimum LU-related pre-patches, based on the Solaris Live Upgrade Software Patch Requirements, previously mentioned.
  • SIMULATE the deployment of the LU-related pre-patches
  • Observe whether any of the LU-related pre-patches will require a reboot
    • The job details for each Global Zone will advise whether a reboot step will be required
  • ACTUALLY deploy the LU-related pre-patches, according to your change control process (e.g. if no reboot, maybe okay to do now; vs. must do later because of the reboot).
    • You can schedule the job to occur later, during a maintenance window
  • Check the job status for each node, resolving any issues found
  • Once the LU-related pre-patches are applied, you can Ops Center to patch using Live Upgrade on Solaris 10

Using Ops Center to patch Solaris 10 with LU/ABE's -- the GOODS!
(this is the heart of the tip):

  • Create an OS Update Profile containing the patches that make up your standard build
    • Use Solaris Baselines when possible
    • Add other individual patches as needed
  • ACTUALLY deploy the OS Update Profile
    • Specify the appropriate Live Upgrade options, e.g.
      • Synchronize the active BE to the alternate BE before patching
      • Do not activate the BE after patching
  • Check the job status for each node, resolving any issues found
  • Activate the newly patched BE according to your change control process
    • Activate = Reboot to the ABE, making the ABE the new active BE
      • Ops Center does not separate LU activate from reboot, so expect a reboot!
  • Check the job status for each node, resolving any issues found


Examples (w/Screenshots)

Solaris 10 and Live Upgrade: Auto-Create the Alternate Boot Environment (ZFS root only)

ABE to be created on ZFS with name S10_12_07REC (Example)

Uses built in feature to call “lucreate -n S10_12_07REC” behind scenes if not already present

NOTE: Leave “lucreate” params blank (if you do specify options, the will be appended after -n $ABEName)

LU01_S10_AutoCreate_BE

Solaris 10 and Live Upgrade: Alternate Boot Environment Creation via Operational Profile (script)

The Alternate Boot Environment is to be created via custom, user-supplied script, which does whatever is needed for the system where Live Upgrade will be used.

Operational Profile, which provides the script to create an ABE:

  • Very similar to the automatic case, but with a Script (Operational Profile), which is used to create the ABE
  • Relies on user-supplied script in the form of an Operational Profile
  • Could be used to prepare an ABE based on a UFS root in a slice, or on a separate device (e.g. by breaking a mirror first) – it is up to the script author to do the right thing!
  • EXAMPLE: Same result as the ZFS case, but illustrating the Operational Profile (e.g. script) approach to call:
# lucreate -n S10_1207REC

LU02_S10_Create_BE_Contents

NOTE: OC special variable is $ABEName


Boot Environment Profile, which references the Operational Profile

  • Script = Operational Profile on this screen
  • Refers to Operational Profile shown in the previous section
  • The user-supplied S10_Create_BE Operational Profile will be run
  • The Operational Profile must send a non-zero exit code if there is a problem (so that the OS Update job will not proceed)

LU03_S10_Create_BE

Solaris 10 OS Update Profile (to provide the actual patch specifications)

Solaris 10 Baseline “Recommended” chosen for “Install”

LU04_S10_OSUpProf

Solaris 10 OS Update Plan (two-steps in this case)

“Create a Boot Environment” + “Update OS” are chosen.

LU05_S10_OSUpPlan

Using Ops Center to patch Solaris 11 with Boot Environments (as needed)

  • Create a Solaris 11 OS Update Profile containing the packages that make up your standard build
  • ACTUALLY deploy the Solaris 11 OS Update Profile
    • BE will be created if needed (or you can stipulate no BE)
    • BE name will be auto-generated (if needed), or you may specify a BE name
  • Check the job status for each node, resolving any issues found
  • Check if a BE was created; if so, activate the new BE
    • Activate = Reboot to the BE, making the new BE the active BE
      • Ops Center does not separate BE activate from reboot
  • NOTE: Not every Solaris 11 OS Update will require a new BE, so a reboot may not be necessary.


Solaris 11: Auto BE Create (as Needed -- let Ops Center decide)

  • BE to be created as needed
  • BE to be named automatically
  • Reboot (if necessary) deferred to separate step

LU06_S11_AutoCreateBE

Solaris 11: OS Profile

Solaris 11 “entire” chosen for a particular SRU

LU07_S11_OSUpProf

Solaris 11: OS Update Plan (w/BE)

 “Create a Boot Environment” + “Update OS” are chosen.

LU08_S11_OSUpPlan

Summary:

Solaris 10 Live Upgrade, Alternate Boot Environments, and their equivalents on Solaris 11 can be very powerful tools to help minimize the downtime associated with updating your servers.  For very old Solaris, there are some important prerequisites to adhere to, but once the initial preparation is complete, Live Upgrade can be used going forward.  For Solaris 11, the built-in Boot Environment handling is leveraged directly by the Image Packaging System, and the result is a much more straight forward way to patch, and far fewer prerequisites to satisfy in getting there.  Ops Center simplifies using either approach, and helps you improve consistency from system to system, which ultimately helps you improve the overall up-time across all the Solaris systems in your environment.

Please let us know what you think?  Until next time...
\Leon
--

Leon Shaner | Senior IT/Product Architect
Systems Management | Ops Center Engineering @ Oracle

The views expressed on this [blog; Web site] are my own and do not necessarily reflect the views of Oracle.



For more information, please go to Oracle Enterprise Manager  web page or  follow us at : 

Twitter | Facebook | YouTube | Linkedin | Newsletter

Tuesday Nov 20, 2012

Upgrading to Oracle Enterprise Manager 12c Release 2: Top Tips One Must Know

Recently Oracle announced incremental release of Enterprise Manager 12c called Enterprise Manager 12c Release 2 (EM12c R2) which includes several new exciting features (Press announcement). Right before the official release, we upgraded an internal production site from EM 12c R1 to EM 12c R2 and had an extremely pleasant experience. Let me share few key takeaways as well as few tips from this upgrade exercise.

I - Why Should You Upgrade To Enterprise Manager 12c Release 2

While an upgrade is usually recommended primarily to take benefit of the latest features (which is valid for this upgrade as well), I found several other compelling reasons purely from deployment perspective.

  1. Standardize your EM deployment:  Enterprise Manager comprises of several different components (OMS, agents, plug-ins, etc) and it might be possible that these are at varied patch levels in your environment. For instance, in case of an environment containing Bundle Patch 1 (customer announcement), there is a good chance that you may not have all the components up-to-date. There are two possible reasons.
    • Bundle Patch 1 involved patching different components (OMS, agents, plug-ins) with multiple one-off patches which may not have been applied to all components yet.
    • Bundle Patch 1 for different platforms were not released together. Which means you may not have got the chance to patch all the components on different platforms.

    Note: BP1 patches are not mandatory to upgrade to EM12c R2 release

    EM 12c R2 provides an excellent opportunity to standardize your Cloud Control environment (OMS, repository and agents) and plug-ins to latest versions in single shot.

  2. All platform releases are made available simultaneously: For the very first time in the history of EM release, all the platforms were released on day one itself, which means you do not need to wait for platform specific binaries for EM OMS or Agent to perform install or upgrades in a heterogeneous environment.
  3. Highly refined and automated process – Upgrade process is by far the smoothest and the cleanest as compared to previous releases of Enterprise manager. Following are the ones that stand out.
    • Automatic Plug-in management – Plug-in upgrade along with new plug-in deployment is supported in upgrade installer wizard which means bulk of the updates to OMS and repository can be done in the same workflow. Saves time and minimizes user inputs.
    • Plug-in Upgrade or Migrate

    • Auto Update: While doing the OMS and repository upgrade, you can use Auto Update screen in Oracle Universal Installer to check for any updates/patches. That will help you to avoid the know issues and will make sure that your upgrade is successful.
    • Allows mass upgrade of EM Agents – A new dedicated menu has been added in the EM console for agent upgrade. Agent upgrade workflow is extremely simple that requires agent name as the only input.
    • ADM / JVMD Manager/Agent upgrade – complete process is supported via UI screens.
    • EM12c R2 Upgrade Guide is much simpler to follow as compared to those for earlier releases. This is attributed to the simpler upgrade process.
  4. Robust and Performing Platform: EM12c R2 release not only includes several new features, but also provides a more stable platform which incorporates several fixes and enhancements in the Enterprise Manager framework.

II - Few Tips To Remember

In my last post (blog link) I shared few tips and tricks from my experience applying the Bundle Patch. Recently I upgraded the same site to EM 12c R2 and found few points that you must take note of, while planning this upgrade. The tips below are also applicable to EM 12c R1 environments that do not have Bundle Patch 1 patches applied.

  1. Verify the monitored application certification – Specific targets like E-Business Suite have not yet been certified as managed target in EM 12c R2. Therefore make sure to recheck the Enterprise Manager certification Matrix on My Oracle Support before planning the upgrade.
  2. Plan downtime – Because EM 12c R2 is an incremental release of EM 12c, for EM 12c R1 to EM 12c R2 upgrade supports only 1-system upgrade approach, which mean there will be downtime.
  3. OMS name change after upgrade – In case of multi OMS environments, additional OMS is renamed after upgrade, which has few implications when you upgrade JVMD and ADP agents on OMS. This is well documented in upgrade guide but make sure you read through all the notes.
  4. Upgrading BI Publisher– EM12c R2 is certified with BI Publisher 11.1.1.6.0 only. Therefore in case you are using EM 12c R1 which is integrated with BI Publisher 11.1.1.5.0, you must upgrade the BI Publisher to 11.1.1.6.0. Follow the steps from Advanced Installation and Configuration Guide here.
  5. Perform Post upgrade Tasks – Make sure to perform post upgrade steps mentioned in documentation here. These include critical changes that must be done right after upgrade to get the right configuration. For instance Database plug-in should be upgraded to Revision 3 (12.1.0.2.0 [u120804]).
  6. Delete old OMS Home – EM12c R1 to EM12c R2 is an out of place upgrade, which means it creates a new oracle home for OMS, plug-ins, etc. Therefore please ensure that
    • You have sufficient extra space for new OMS before starting the upgrade process.
    • You clean up the old OMS home after the upgrade process. Steps are available here.
    • DO NOT remove the agent home on OMS host, because agent is upgraded in-place.
  7. If you have standby OMS setup then do look into the steps to upgrade the standby OMS from the upgrade guide before going ahead.
  8. Read the right documentation – Make sure to follow the Upgrade guide which provides the most comprehensive information on EM12c R2 upgrade process. Additionally you can refer other resources to get familiar with upgrade concepts.

We are very excited about this latest release and will look forward to hear back any feedback from your upgrade experience!

Stay Connected:

Twitter |  Facebook |  YouTube |  Linkedin |  Newsletter

Thursday Nov 15, 2012

Ops Center Solaris 11 IPS Repository Management: Using ISO Images

A recording of this community call is now available here:

https://oracleconferencing.webex.com/oracleconferencing/ldr.php?AT=pb&SP=MC&rID=71862997&rKey=76be5a666dc12c90

With Enterprise Manager Ops Center 12c, you can provision, patch, monitor and manage Oracle Solaris 11 instances. To do this, Ops Center creates and maintains a Solaris 11 Image Packaging System (IPS) repository on the Enterprise Controller. During the Enterprise Controller configuration, you can load repository content directly from Oracle's Support Web site and subsequently synchronize the repository as new content becomes available.

Of course, you can also use Solaris 11 ISO images to create and update your Ops Center repository. There are a few excellent reasons for doing this:

  1. You're running Ops Center in disconnected mode, and don't have Internet access on your Enterprise Controller
  2. You'd rather avoid the bandwidth associated with live synchronization of a Solaris 11 package repository

This demo will show you how to use Solaris 11 ISO images to set up and update your Ops Center repository.


Prerequisites

This tip assumes that you've already installed the Enterprise Controller on a Solaris 11 OS instance and that you're ready for post-install configuration.

In addition, there are specific Ops Center and OS version requirements depending on which version of Solaris 11 you plan to install.You can get full details about the requirements in the Release Notes for Ops Center 12c update 2.

Additional information is available in the Ops Center update 2 Readme document.


Part 1: Using a Solaris 11 ISO Image to Create an Ops Center Repository

Step 1 – Download the Solaris 11 Repository Image

The Oracle Web site provides a number of download links for official Solaris 11 images. Among those links is a two-part downloadable repository image, which provides repository content for Solaris 11 SPARC and X86 architectures. In this case, I used the Solaris 11 11/11 image.

First, navigate to the Oracle Web site and accept the OTN License agreement:

http://www.oracle.com/technetwork/server-storage/solaris11/downloads/index.html

Next, download both parts of the Solaris 11 repository image. I used the Solaris 11 11/11 image, and have provided the URLs here:

http://download.oracle.com/otn/solaris/11/sol-11-1111-repo-full.iso-a
http://download.oracle.com/otn/solaris/11/sol-11-1111-repo-full.iso-b

Finally, use the cat command to generate an ISO image you can use to create your repository:
# cat sol-11-1111-repo-full.iso-a sol-11-1111-repo-full.iso-b > sol-11-1111-repo-full.iso

The process is very similar if you plan to set up a Solaris 11.1 release in Ops Center. In that case, navigate to the Solaris 11 download page, accept the license agreement and download both parts of the Solaris 11.1 repository image. Use the cat command to create a single ISO image for Solaris 11.1

Step 2 – Mount the Solaris 11 ISO Image in your Local Filesystem

Once you have created the Solaris 11 ISO file, use the mount command to attach it to your local filesystem. After the image has been mounted, you can browse the repository from the ./repo subdirectory, and use the pkgrepo command to verify that Solaris 11 recognizes the content:


Step 3 – Use the Image to Create your Ops Center Repository

When you have confirmed the repository is available, you can use the image to create the Enterprise Controller repository. The operation will be slightly different depending on whether you configure Ops Center for Connected or Disconnected Mode operation.

For connected mode operation, specify the mounted ./repo directory in step 4.1 of the configuration wizard, replacing the default Web-based URL. Since you're synchronizing from an OS repository image, you don't need to specify a key or certificate for the operation.


For disconnected mode configuration, specify the Solaris 11 directory along with the path to the disconnected mode bundle downloaded by running the Ops Center harvester script:

Ops Center will run a job to import package content from the mounted ISO image. A synchronization job can take several hours to run – in my case, the job ran for 3 hours, 22 minutes on a SunFire X4200 M2 server.


During the job, Ops Center performs three important tasks:

  1. Synchronizes all content from the image and refreshes the repository
  2. Updates the IPS publisher information
  3. Creates OS Provisioning profiles and policies based on the content

When the job is complete, you can unmount the ISO image from your Enterprise Controller. At that time, you can view the repository contents in your Ops Center Solaris 11 library. For the Solaris 11 11/11 release, you should see 8,668 packages and patches in the contents.


You should also see default deployment plans for Solaris 11 provisioning. As part of the repository import, Ops Center generates plans and profiles for desktop, small and large servers for the SPARC and X86 architecture.



Part 2: Using a Solaris 11 SRU to update an Ops Center Repository

It's possible to use the same approach to upgrade your Ops Center repository to a Solaris 11 Support Repository Update, or SRU. Each SRU provides packages and updates to Solaris 11 - for example, SRU 8.5 provided the packaged for Oracle VM Server for SPARC 2.2

SRUs are available for download as ISO images from My Oracle Support, under document ID 1372094.1. The document provides download links for all SRUs which have been released by Oracle for Solaris 11. SRUs are cumulative, so later versions include the packages from earlier SRUs.


After downloading an ISO image for an SRU, you can mount it to your local filesystem using a mount command similar to the one shown for Solaris 11 11/11.

When the ISO image is mounted to the file system, you can perform the Add Content action from the Solaris 11 Library to synchronize packages and patches from the mounted image. I used the same mount point, so the repository URL was file://mnt/repo once again:


After the synchronization of an SRU is complete, you can verify its content in the Solaris 11 library using the search function. The version pattern is 0.175.0.#, where the # is the same value as the SRU.

In this example, I upgraded to SRU 1. The update job ran in just under 8 minutes, and a quick search shows that 22 software components were added to the repository:


It's also possible to search for "Support Repository Update" to confirm the SRU was successfully added to the repository. Details on any of the update content are available by clicking the "View Details" button under the Packages/Patches entry.


Wednesday Nov 07, 2012

Keeping track of File System Utilization in Ops Center 12c

Enterprise Manager Ops Center 12c provides significant monitoring capabilities, combined with very flexible incident management. These capabilities even extend to monitoring the file systems associated with Solaris or Linux assets. Depending on your needs you can monitor and manage incidents, or you can fine tune alert monitoring rules to specific file systems.

This article will show you how to use Ops Center 12c to

  • Track file system utilization
  • Adjust file system monitoring rules
  • Disable file system rules
  • Create custom monitoring rules

A recording of this community call is now available here:

https://oracleconferencing.webex.com/oracleconferencing/ldr.php?AT=pb&SP=MC&rID=71648742&rKey=a8b3b199a3d9a997

Monitoring File Systems for OS Assets

The Libraries tab provides basic, device-level information about the storage associated with an OS instance. This tab shows you the local file system associated with the instance and any shared storage libraries mounted by Ops Center.

More detailed information about file system storage is available under the Analytics tab under the sub-tab named Charts. Here, you can select and display the individual mount points of an OS, and export the utilization data if desired:


In this example, the OS instance has a basic root file partition and several NFS directories. Each file system mount point can be independently chosen for display in the Ops Center chart.

File Systems and Incident  Reporting

Every asset managed by Ops Center has a "monitoring policy", which determines what represents a reportable issue with the asset. The policy is made up of a bunch of monitoring rules, where each rule describes

  • An attribute to monitor
  • The conditions which represent an issue
  • The level or levels of severity for the issue

When the conditions are met, Ops Center sends a notification and creates an incident.

By default, OS instances have three monitoring rules associated with file systems:

  • File System Reachability: Triggers an incident if a file system is not reachable
  • NAS Library Status: Triggers an incident for a value of "WARNING" or "DEGRADED" for a NAS-based file system
  • File System Used Space Percentage: Triggers an incident when file system utilization grows beyond defined thresholds

You can view these rules in the Monitoring tab for an OS:

Of course, the default monitoring rules is that they apply to every file system associated with an OS instance. As a result, any issue with NAS accessibility or disk utilization will trigger an incident. This can cause incidents for file systems to be reported multiple times if the same shared storage is used by many assets, as shown in this screen shot:


Depending on the level of control you'd like, there are a number of ways to fine tune incident reporting.

Note that any changes to an asset's monitoring policy will detach it from the default, creating a new monitoring policy for the asset. If you'd like, you can extract a monitoring policy from an asset, which allows you to save it and apply the customized monitoring profile to other OS assets.


Solution #1: Modify the Reporting Thresholds

In some cases, you may want to modify the basic conditions for incident reporting in your file system. The changes you make to a default monitoring rule will apply to all of the file systems associated with your operating system. Selecting the File Systems Used Space Percentage entry and clicking the "Edit Alert Monitoring Rule Parameters" button opens a pop-up dialog which allows you to modify the rule.

The first screen lets you decide when you will check for file system usage, and how long you will wait before opening an incident in Ops Center. By default, Ops Center monitors continuously and reports disk utilization issues which exist for more than 15 minutes.

The second screen lets you define actual threshold values. By default, Ops Center opens a Warning level incident is utilization rises above 80%, and a Critical level incident for utilization above 95%

Solution #2: Disable Incident Reporting for File System

If you'd rather not report file system incidents, you can disable the monitoring rules altogether. In this case, you can select the monitoring rules and click the "Disable Alert Monitoring Rule(s)" button to open the pop-up confirmation dialog.

Like the first solution, this option affects all file system monitoring. It allows you to completely disable incident reporting for NAS library status or file system space consumption.

Solution #3: Create New Monitoring Rules for Specific File Systems

If you'd like to have the greatest flexibility when monitoring file systems, you can create entirely new rules. Clicking the "Add Alert Monitoring Rule" (the icon with the green plus sign) opens a wizard which allows you to define a new rule.

 This rule will be based on a threshold, and will be used to monitor operating system assets. We'd like to add a rule to track disk utilization for a specific file system - the /nfs-guest directory. To do this, we specify the following attribute

FileSystemUsages.name=/nfs-guest.usedSpacePercentage

The value of name in the attribute allows us to define a specific NFS shared directory or file system... in the case of this OS, we could have chosen any of the values shown in the File Systems Utilization chart at the beginning of this article.

usedSpacePercentage lets us define a threshold based on the percentage of total disk space used. There are a number of other values that we could use for threshold-based monitoring of FileSystemUsages, including

  • freeSpace
  • freeSpacePercentage
  • totalSpace
  • usedSpace
  • usedSpacePercentage

The final sections of the screen allow us to determine when to monitor for disk usage, and how long to wait after utilization reaches a threshold before creating an incident. The next screen lets us define the threshold values and severity levels for the monitoring rule:

If historical data is available, Ops Center will display it in the screen. Clicking the Apply button will create the new monitoring rule and active it in your monitoring policy.


If you combine this with one of the previous solutions, you can precisely define which file systems will generate incidents and notifications. For example, this monitoring policy has the default "File System Used Space Percentage" rule disabled, but the new rule reports ONLY on utilization for the /nfs-guest directory.


Stay Connected:

Twitter |  Facebook |  YouTube |  Linkedin |  Newsletter

About

Latest information and perspectives on Oracle Enterprise Manager.

Related Blogs




Search

Archives
« November 2012 »
SunMonTueWedThuFriSat
    
1
2
3
4
5
8
9
10
11
13
14
16
17
18
19
21
22
23
24
25
26
28
29
30
 
       
Today