Thursday May 02, 2013

How to Monitor Systems in Enterprise Manager using Ops Center

In this post, we'll use Ops Center to add hardware monitoring to Enterprise Manager. We'll discuss the existing capabilities of Host targets, show how to create an Infrastructure Stack and demonstrate some of the features it provides to Enterprise Manager.



A recording of this community call is now available here:

WebEx Recording: Ops Center integration with Cloud Control
          


Prerequisites

This blog post uses both the Enterprise Manager 12c Cloud Control and Ops Center products. The following list describes the initial setup state and provides links to Oracle documents you can use to install and configure both products:

  • Enterprise Manager 12c
    • Install and configure an Oracle Management Server and Repository (OMS and OMR)
    • Install the Oracle Management Agent (OMA) on a target OS instance
  • Ops Center 12c
    • Install and Configure the Enterprise and Proxy Controller (EC and PC)
    • Discover and manage the system and its associated OS (installing the Ops Center agent on the OS instance)

I've configured the environment for this post in the following way:

  • Enterprise Manager 12c: OMS and OMR running separately
  • Ops Center 12c: EC and PC running co-located
  • Two sample systems, both running Solaris 11
    • An Oracle SPARC T4-2 server
    • An Oracle SunFire X4200 M2 server

Host Capabilities in Enterprise Manager

By default, installing an Enterprise Manager agent on an OS instance creates an associated Host target. A Host provides a lot of useful data about the platform that hosts the OMA, such as

  • CPU and memory utilization
  • File system size and utilization
  • Network interfaces and activity
  • Program and process resources
  • User activity

The Host does not provide sensor data associated with the server and cannot report issues with the underlying hardware. Fortunately Ops Center can do both of these things, and you can incorporate server data, monitoring thresholds and alerts into your Enterprise Manager environment.

Setting Up an Infrastructure Stack

Ops Center: Configure the connection to Enterprise Manager

To connect Ops Center with Enterprise Manager, select the left-hand Navigation link Administration > Enterprise Manager Cloud Control. Click on the right-hand Action link Configure/Connect and open a pop-up dialog box. The dialog lets you configure the OMS and OMR settings. The screen shot below shows both steps from the wizard.

Enterprise Manager: Download and deploy the Ops Center Plug-In

Enterprise Manager 12c provides a deployable plug-in to manage an Infrastructure Stack. For Enterprise Manager installations running in online mode, you can download it from the Extensibility > Plug-Ins menu from the Servers, Storage and Network category.

Download and deploy the plug-in to the OMS. You can immediately deploy it to the OMA instances you want to integrate with Ops Center, or wait until you create an infrastructure stack. (Enterprise Manager will automatically install the plug-in to the OMA if it is not already present)

Enterprise Manager: Create an Infrastructure Stack Target

An Infrastructure Stack associates data from a system in Ops Center with targets in Enterprise Manager. To create one, select the menu option Setup > Add Targets > Add Targets Manually. From the wizard, select Infrastructure Stack from the pull-down menu and identify the Monitoring Agent that will be used.

The subsequent configuration screen allows you to define the name for the target, to identify the Enterprise Controller that will provide the data for the server, and to specify the Ops Center login credentials. Any user account defined in Ops Center is suitable for the target.

Infrastructure Stack Capabilities

What benefits does an Infrastructure Stack provide? As the consolidation point for server-related information, it enables you to perform three principal tasks:

  • Monitoring, with metrics and thresholds for the server
  • Reporting, using a set of standard Information Publisher reports
  • Incident Management, with Ops Center hardware alerts

Let's look at some examples for each.

Monitoring

An Infrastructure Stack provides a wealth of information about a server, including identification information, state, capabilities and sensor data. The home page provides a summary of current values for power consumption, temperature, fan speed, and reported incidents.

The metrics section provides more detailed data such as sensor values, thresholds, installed firmware and server capabilities.

Note that the reported values ultimately depend on what data is available for a specific type of server. Some earlier models don't report temperature data, for instance.

Reporting

Enterprise Manager provides three standard Information Publisher reports:

  • Infrastructure Stack Topology, showing the server and OS associated with the stack
  • Infrastructure Stack Configuration, providing a tabular summary of key data
  • Hardware Sensors, showing current values and thresholds for monitored server data

The following screen shots provide sample report output for the infrastructure stack configuration and hardware sensor data.

Incident Management

Incident reporting is an optional capability for an Infrastructure Stack. Enabling the feature causes Ops Center to forward hardware alarms, allowing you to consolidate problem management in Enterprise Manager.

To enable incident reporting, navigate to the Infrastructure Stack and select the menu option Monitoring > Metric and Collection Settings. Select the link Collection Schedule for Infrastructure Stack Alarms to edit the settings:

If you toggle the collection schedule to Enabled, Enterprise Manager will activate incident reporting based on hardware alarms. By default, the data refresh frequency is once every five minutes, with warning or critical alarms being reported.

In this example, we simulated an hardware alarm using the IPMItool utility from the Hardware Management Pack. Ops Center forwarded the event and all associated data, which generated an actionable incident in Enterprise Manager,

Summary

In this post, we have demonstrated how to integrate Ops Center data into Enterprise Manager, and described the features available for an Infrastructure Stack. If you would like to learn more, please join us for the WebEx demo on May 9th.


Tuesday Nov 27, 2012

Using Oracle Enterprise Manager Ops Center to Update Solaris via Live Upgrade

Introduction: LU99_AD

This Oracle Enterprise Manager Ops Center blog entry provides tips for using Ops Center to update Solaris using Live Upgrade on Solaris 10 and Boot Environments on Solaris 11.

Why use Live Upgrade?

  • Live Upgrade (LU) can significantly reduce downtime associated with patching
  • Live Upgrade avoids dropping to single-user mode for long periods of time during patching
  • Live Upgrade relies on an Alternate Boot Environment (ABE)/(BE), which is patched while in multi-user mode; thereby allowing normal system operations to continue with the active BE, while the alternate BE is being patched
  • Activating a newly patched (A)BE is essentially a reboot; therefore the downtime is ~= reboot
  • Admins can easily revert to the prior Boot Environment (BE) as a safeguard / fallback.

Why use Ops Center to patch via Live Upgrade, Alternate Boot Environments, and Solaris 11 equivalents?

  • All the benefits of Ops Center's extensive patch and package knowledge base can be leveraged on top of Live Upgrade
  • Ops Center can orchestrate patching based on Live Upgrade and Solaris 11 features, which all works together to minimize downtime
  • Ops Center's advanced inventory and reporting features assure that each OS is updated to a verifiable, consistent standard, rather than relying on ad-hoc (error prone) procedures and scripts
  • Ops Center gives admins control over the boot environment specifications or they can let Ops Center decide when a BE is necessary, thereby reducing complexity and lowering the opportunity for user error

Preparing to use Live Upgrade-like features in Solaris 11

Requirements and information you should know:

  • Global Zone Root file-systems must be separate from Solaris Container / Zone filesystems
  • Solaris 11 has features which are similar in concept to Live Upgrade on Solaris 10, but differ greatly in implementation
    Important distinctions:
    • Solaris 11 assumes ZFS root
    • Solaris 11 adds Boot Environments (BE's) as an integrated feature (see beadm)
    • Solaris 11 BE's avoid single-user patching (vs. Solaris 10 w/ ZFS snapshot=ABE).
    • Solaris 11 Image Packaging System (IPS) has hooks for BE creation, as needed
      • Solaris 11 allows pkgs to be installed + upgraded in alternate BE (e.g. instead of the live system) but it is controlled on a per-pkg basis
      • Boot Environments are activated across a reboot; instead of spending long periods installing + upgrading packages in single user mode.
    • Fallback to a prior BE is a function of the BE infrastructure (a la beadm).
    • (Generally) Reboot + BE activation can be much much faster on Solaris 11

Preparing to use Live Upgrade on Solaris 10

Requirements and information you should know:

  • Global Zone Root file-systems must be separate from Solaris Container / Zone filesystems
  • Live Upgrade Pre-requisite patches must be applied before the first Live Upgrade Alternate Boot Environments are created (see "Pre-requisite Patches" section, below...)
  • Solaris 10 Update 6 or newer on ZFS root is the practical starting point for Live Upgrade
    • Live Upgrade with ZFS root is far more straight-forward than any scheme based on Alternative Boot Environments in slices or temporarily breaking mirrors
    • Use Solaris best practices to upgrade the OS to at least Solaris 10 Update 4 (outside of Ops Center)
    • UFS root can (technically) be used, but it is significantly more involved (e.g. discouraged) -- there are many reasons to move to ZFS while going through the process to update to Solaris 10 Update 6 or newer (out side of Ops Center)
  • Recommendation: Start with Solaris 10 Update 6 or newer on ZFS root
  • Recommendation: Start with Ops Center 12c or newer
    • Ops Center 12c can automatically create your ABE's for you, without the need for custom scripts
    • Ops Center 12c Update 2 avoids kernel panic on unpatched Solaris 10 update 9 (and older) -- unrelated to Live Upgrade, but more on the issue, below.
NOTE: There is no magic!  If you have systems running Solaris 10 Update 5 or older on UFS root, and you don't know how to get them updated to Solaris 10 on ZFS root, then there are services available from Oracle Advanced Customer Support (ACS), which specialize in this area.

Live Upgrade Pre-requisite Patches (Solaris 10)

Certain Live Upgrade related patches must be present before the first Live Upgrade ABE's are created on Solaris 10.
Use the following MOS Search String to find the “living document” that outlines the required patch minimums, which are necessary before using any Live Upgrade features:

Solaris Live Upgrade Software Patch Requirements

(Click above – the link is valid as of this writing, but search in MOS for the same "Solaris Live Upgrade Software Patch Requirements" string if necessary)

It is a very good idea to check the document periodically and adapt to its contents, accordingly.

IMPORTANT:  In case it wasn't clear in the above document, some direct patching of the active OS, including a reboot, may be required before Live Upgrade can be successfully used the first time.

HINT: You can use Ops Center to determine what to expect for a given system, and to schedule the “pre-patching” during a maintenance window if necessary.

Preparing to use Ops Center

  • Discover + Manage (Install + Configure the Ops Center agent in) each Global Zone
  • Recommendation:  Begin by using OCDoctor --agent-prereq to determine whether OS meets OC prerequisites (resolve any issues)
  • See prior requirements and recommendations w.r.t. starting with Solaris 10 Update 6 or newer on ZFS (or at least Solaris 10 Update 4 on UFS, with caveats)

  • WARNING: Systems running unpatched Solaris 10 update 9 (or older) should run the Ops Center 12c Update 2 agent to avoid a potential kernel panic
    • The 12c Update 2 agent will check patch minimums and disable certain process accounting features if the kernel is not sufficiently patched to avoid the panic
      • SPARC: 142900-05 Obsoleted by: 142900-06 SunOS 5.10: kernel patch 10 Oracle Solaris on SPARC (32-bit)
      • X64: 142901-05 Obsoleted by: 142901-06 SunOS 5.10_x86: kernel patch 10 Oracle Solaris on x86 (32-bit)
    • OR
      • SPARC: 142909-17 SunOS 5.10: kernel patch 10 Oracle Solaris on SPARC (32-bit)
      • X64: 142910-17 SunOS 5.10_x86: kernel patch 10 Oracle Solaris on x86 (32-bit)
    • Ops Center 12c (initial release) and 12c Update 1 agent can also be safely used with a workaround (to be performed BEFORE installing the agent):
# mkdir -p /etc/opt/sun/oc
# echo "zstat_exacct_allowed=false" > /etc/opt/sun/oc/zstat.conf
# chmod 755 /etc/opt/sun /etc/opt/sun/oc
# chmod 644 /etc/opt/sun/oc/zstat.conf
# chown -Rh root:sys /etc/opt/sun/oc

NOTE: Remove the above after patching the OS sufficiently, or after upgrading to the 12c Update 2 agent

Using Ops Center to apply Live Upgrade-related Pre-Patches (Solaris 10)
Overview:

  • Create an OS Update Profile containing the minimum LU-related pre-patches, based on the Solaris Live Upgrade Software Patch Requirements, previously mentioned.
  • SIMULATE the deployment of the LU-related pre-patches
  • Observe whether any of the LU-related pre-patches will require a reboot
    • The job details for each Global Zone will advise whether a reboot step will be required
  • ACTUALLY deploy the LU-related pre-patches, according to your change control process (e.g. if no reboot, maybe okay to do now; vs. must do later because of the reboot).
    • You can schedule the job to occur later, during a maintenance window
  • Check the job status for each node, resolving any issues found
  • Once the LU-related pre-patches are applied, you can Ops Center to patch using Live Upgrade on Solaris 10

Using Ops Center to patch Solaris 10 with LU/ABE's -- the GOODS!
(this is the heart of the tip):

  • Create an OS Update Profile containing the patches that make up your standard build
    • Use Solaris Baselines when possible
    • Add other individual patches as needed
  • ACTUALLY deploy the OS Update Profile
    • Specify the appropriate Live Upgrade options, e.g.
      • Synchronize the active BE to the alternate BE before patching
      • Do not activate the BE after patching
  • Check the job status for each node, resolving any issues found
  • Activate the newly patched BE according to your change control process
    • Activate = Reboot to the ABE, making the ABE the new active BE
      • Ops Center does not separate LU activate from reboot, so expect a reboot!
  • Check the job status for each node, resolving any issues found


Examples (w/Screenshots)

Solaris 10 and Live Upgrade: Auto-Create the Alternate Boot Environment (ZFS root only)

ABE to be created on ZFS with name S10_12_07REC (Example)

Uses built in feature to call “lucreate -n S10_12_07REC” behind scenes if not already present

NOTE: Leave “lucreate” params blank (if you do specify options, the will be appended after -n $ABEName)

LU01_S10_AutoCreate_BE

Solaris 10 and Live Upgrade: Alternate Boot Environment Creation via Operational Profile (script)

The Alternate Boot Environment is to be created via custom, user-supplied script, which does whatever is needed for the system where Live Upgrade will be used.

Operational Profile, which provides the script to create an ABE:

  • Very similar to the automatic case, but with a Script (Operational Profile), which is used to create the ABE
  • Relies on user-supplied script in the form of an Operational Profile
  • Could be used to prepare an ABE based on a UFS root in a slice, or on a separate device (e.g. by breaking a mirror first) – it is up to the script author to do the right thing!
  • EXAMPLE: Same result as the ZFS case, but illustrating the Operational Profile (e.g. script) approach to call:
# lucreate -n S10_1207REC

LU02_S10_Create_BE_Contents

NOTE: OC special variable is $ABEName


Boot Environment Profile, which references the Operational Profile

  • Script = Operational Profile on this screen
  • Refers to Operational Profile shown in the previous section
  • The user-supplied S10_Create_BE Operational Profile will be run
  • The Operational Profile must send a non-zero exit code if there is a problem (so that the OS Update job will not proceed)

LU03_S10_Create_BE

Solaris 10 OS Update Profile (to provide the actual patch specifications)

Solaris 10 Baseline “Recommended” chosen for “Install”

LU04_S10_OSUpProf

Solaris 10 OS Update Plan (two-steps in this case)

“Create a Boot Environment” + “Update OS” are chosen.

LU05_S10_OSUpPlan

Using Ops Center to patch Solaris 11 with Boot Environments (as needed)

  • Create a Solaris 11 OS Update Profile containing the packages that make up your standard build
  • ACTUALLY deploy the Solaris 11 OS Update Profile
    • BE will be created if needed (or you can stipulate no BE)
    • BE name will be auto-generated (if needed), or you may specify a BE name
  • Check the job status for each node, resolving any issues found
  • Check if a BE was created; if so, activate the new BE
    • Activate = Reboot to the BE, making the new BE the active BE
      • Ops Center does not separate BE activate from reboot
  • NOTE: Not every Solaris 11 OS Update will require a new BE, so a reboot may not be necessary.


Solaris 11: Auto BE Create (as Needed -- let Ops Center decide)

  • BE to be created as needed
  • BE to be named automatically
  • Reboot (if necessary) deferred to separate step

LU06_S11_AutoCreateBE

Solaris 11: OS Profile

Solaris 11 “entire” chosen for a particular SRU

LU07_S11_OSUpProf

Solaris 11: OS Update Plan (w/BE)

 “Create a Boot Environment” + “Update OS” are chosen.

LU08_S11_OSUpPlan

Summary:

Solaris 10 Live Upgrade, Alternate Boot Environments, and their equivalents on Solaris 11 can be very powerful tools to help minimize the downtime associated with updating your servers.  For very old Solaris, there are some important prerequisites to adhere to, but once the initial preparation is complete, Live Upgrade can be used going forward.  For Solaris 11, the built-in Boot Environment handling is leveraged directly by the Image Packaging System, and the result is a much more straight forward way to patch, and far fewer prerequisites to satisfy in getting there.  Ops Center simplifies using either approach, and helps you improve consistency from system to system, which ultimately helps you improve the overall up-time across all the Solaris systems in your environment.

Please let us know what you think?  Until next time...
\Leon
--

Leon Shaner | Senior IT/Product Architect
Systems Management | Ops Center Engineering @ Oracle

The views expressed on this [blog; Web site] are my own and do not necessarily reflect the views of Oracle.



For more information, please go to Oracle Enterprise Manager  web page or  follow us at : 

Twitter | Facebook | YouTube | Linkedin | Newsletter

Thursday Nov 15, 2012

Ops Center Solaris 11 IPS Repository Management: Using ISO Images

A recording of this community call is now available here:

https://oracleconferencing.webex.com/oracleconferencing/ldr.php?AT=pb&SP=MC&rID=71862997&rKey=76be5a666dc12c90

With Enterprise Manager Ops Center 12c, you can provision, patch, monitor and manage Oracle Solaris 11 instances. To do this, Ops Center creates and maintains a Solaris 11 Image Packaging System (IPS) repository on the Enterprise Controller. During the Enterprise Controller configuration, you can load repository content directly from Oracle's Support Web site and subsequently synchronize the repository as new content becomes available.

Of course, you can also use Solaris 11 ISO images to create and update your Ops Center repository. There are a few excellent reasons for doing this:

  1. You're running Ops Center in disconnected mode, and don't have Internet access on your Enterprise Controller
  2. You'd rather avoid the bandwidth associated with live synchronization of a Solaris 11 package repository

This demo will show you how to use Solaris 11 ISO images to set up and update your Ops Center repository.


Prerequisites

This tip assumes that you've already installed the Enterprise Controller on a Solaris 11 OS instance and that you're ready for post-install configuration.

In addition, there are specific Ops Center and OS version requirements depending on which version of Solaris 11 you plan to install.You can get full details about the requirements in the Release Notes for Ops Center 12c update 2.

Additional information is available in the Ops Center update 2 Readme document.


Part 1: Using a Solaris 11 ISO Image to Create an Ops Center Repository

Step 1 – Download the Solaris 11 Repository Image

The Oracle Web site provides a number of download links for official Solaris 11 images. Among those links is a two-part downloadable repository image, which provides repository content for Solaris 11 SPARC and X86 architectures. In this case, I used the Solaris 11 11/11 image.

First, navigate to the Oracle Web site and accept the OTN License agreement:

http://www.oracle.com/technetwork/server-storage/solaris11/downloads/index.html

Next, download both parts of the Solaris 11 repository image. I used the Solaris 11 11/11 image, and have provided the URLs here:

http://download.oracle.com/otn/solaris/11/sol-11-1111-repo-full.iso-a
http://download.oracle.com/otn/solaris/11/sol-11-1111-repo-full.iso-b

Finally, use the cat command to generate an ISO image you can use to create your repository:
# cat sol-11-1111-repo-full.iso-a sol-11-1111-repo-full.iso-b > sol-11-1111-repo-full.iso

The process is very similar if you plan to set up a Solaris 11.1 release in Ops Center. In that case, navigate to the Solaris 11 download page, accept the license agreement and download both parts of the Solaris 11.1 repository image. Use the cat command to create a single ISO image for Solaris 11.1

Step 2 – Mount the Solaris 11 ISO Image in your Local Filesystem

Once you have created the Solaris 11 ISO file, use the mount command to attach it to your local filesystem. After the image has been mounted, you can browse the repository from the ./repo subdirectory, and use the pkgrepo command to verify that Solaris 11 recognizes the content:


Step 3 – Use the Image to Create your Ops Center Repository

When you have confirmed the repository is available, you can use the image to create the Enterprise Controller repository. The operation will be slightly different depending on whether you configure Ops Center for Connected or Disconnected Mode operation.

For connected mode operation, specify the mounted ./repo directory in step 4.1 of the configuration wizard, replacing the default Web-based URL. Since you're synchronizing from an OS repository image, you don't need to specify a key or certificate for the operation.


For disconnected mode configuration, specify the Solaris 11 directory along with the path to the disconnected mode bundle downloaded by running the Ops Center harvester script:

Ops Center will run a job to import package content from the mounted ISO image. A synchronization job can take several hours to run – in my case, the job ran for 3 hours, 22 minutes on a SunFire X4200 M2 server.


During the job, Ops Center performs three important tasks:

  1. Synchronizes all content from the image and refreshes the repository
  2. Updates the IPS publisher information
  3. Creates OS Provisioning profiles and policies based on the content

When the job is complete, you can unmount the ISO image from your Enterprise Controller. At that time, you can view the repository contents in your Ops Center Solaris 11 library. For the Solaris 11 11/11 release, you should see 8,668 packages and patches in the contents.


You should also see default deployment plans for Solaris 11 provisioning. As part of the repository import, Ops Center generates plans and profiles for desktop, small and large servers for the SPARC and X86 architecture.



Part 2: Using a Solaris 11 SRU to update an Ops Center Repository

It's possible to use the same approach to upgrade your Ops Center repository to a Solaris 11 Support Repository Update, or SRU. Each SRU provides packages and updates to Solaris 11 - for example, SRU 8.5 provided the packaged for Oracle VM Server for SPARC 2.2

SRUs are available for download as ISO images from My Oracle Support, under document ID 1372094.1. The document provides download links for all SRUs which have been released by Oracle for Solaris 11. SRUs are cumulative, so later versions include the packages from earlier SRUs.


After downloading an ISO image for an SRU, you can mount it to your local filesystem using a mount command similar to the one shown for Solaris 11 11/11.

When the ISO image is mounted to the file system, you can perform the Add Content action from the Solaris 11 Library to synchronize packages and patches from the mounted image. I used the same mount point, so the repository URL was file://mnt/repo once again:


After the synchronization of an SRU is complete, you can verify its content in the Solaris 11 library using the search function. The version pattern is 0.175.0.#, where the # is the same value as the SRU.

In this example, I upgraded to SRU 1. The update job ran in just under 8 minutes, and a quick search shows that 22 software components were added to the repository:


It's also possible to search for "Support Repository Update" to confirm the SRU was successfully added to the repository. Details on any of the update content are available by clicking the "View Details" button under the Packages/Patches entry.


Wednesday Nov 07, 2012

Keeping track of File System Utilization in Ops Center 12c

Enterprise Manager Ops Center 12c provides significant monitoring capabilities, combined with very flexible incident management. These capabilities even extend to monitoring the file systems associated with Solaris or Linux assets. Depending on your needs you can monitor and manage incidents, or you can fine tune alert monitoring rules to specific file systems.

This article will show you how to use Ops Center 12c to

  • Track file system utilization
  • Adjust file system monitoring rules
  • Disable file system rules
  • Create custom monitoring rules

A recording of this community call is now available here:

https://oracleconferencing.webex.com/oracleconferencing/ldr.php?AT=pb&SP=MC&rID=71648742&rKey=a8b3b199a3d9a997

Monitoring File Systems for OS Assets

The Libraries tab provides basic, device-level information about the storage associated with an OS instance. This tab shows you the local file system associated with the instance and any shared storage libraries mounted by Ops Center.

More detailed information about file system storage is available under the Analytics tab under the sub-tab named Charts. Here, you can select and display the individual mount points of an OS, and export the utilization data if desired:


In this example, the OS instance has a basic root file partition and several NFS directories. Each file system mount point can be independently chosen for display in the Ops Center chart.

File Systems and Incident  Reporting

Every asset managed by Ops Center has a "monitoring policy", which determines what represents a reportable issue with the asset. The policy is made up of a bunch of monitoring rules, where each rule describes

  • An attribute to monitor
  • The conditions which represent an issue
  • The level or levels of severity for the issue

When the conditions are met, Ops Center sends a notification and creates an incident.

By default, OS instances have three monitoring rules associated with file systems:

  • File System Reachability: Triggers an incident if a file system is not reachable
  • NAS Library Status: Triggers an incident for a value of "WARNING" or "DEGRADED" for a NAS-based file system
  • File System Used Space Percentage: Triggers an incident when file system utilization grows beyond defined thresholds

You can view these rules in the Monitoring tab for an OS:

Of course, the default monitoring rules is that they apply to every file system associated with an OS instance. As a result, any issue with NAS accessibility or disk utilization will trigger an incident. This can cause incidents for file systems to be reported multiple times if the same shared storage is used by many assets, as shown in this screen shot:


Depending on the level of control you'd like, there are a number of ways to fine tune incident reporting.

Note that any changes to an asset's monitoring policy will detach it from the default, creating a new monitoring policy for the asset. If you'd like, you can extract a monitoring policy from an asset, which allows you to save it and apply the customized monitoring profile to other OS assets.


Solution #1: Modify the Reporting Thresholds

In some cases, you may want to modify the basic conditions for incident reporting in your file system. The changes you make to a default monitoring rule will apply to all of the file systems associated with your operating system. Selecting the File Systems Used Space Percentage entry and clicking the "Edit Alert Monitoring Rule Parameters" button opens a pop-up dialog which allows you to modify the rule.

The first screen lets you decide when you will check for file system usage, and how long you will wait before opening an incident in Ops Center. By default, Ops Center monitors continuously and reports disk utilization issues which exist for more than 15 minutes.

The second screen lets you define actual threshold values. By default, Ops Center opens a Warning level incident is utilization rises above 80%, and a Critical level incident for utilization above 95%

Solution #2: Disable Incident Reporting for File System

If you'd rather not report file system incidents, you can disable the monitoring rules altogether. In this case, you can select the monitoring rules and click the "Disable Alert Monitoring Rule(s)" button to open the pop-up confirmation dialog.

Like the first solution, this option affects all file system monitoring. It allows you to completely disable incident reporting for NAS library status or file system space consumption.

Solution #3: Create New Monitoring Rules for Specific File Systems

If you'd like to have the greatest flexibility when monitoring file systems, you can create entirely new rules. Clicking the "Add Alert Monitoring Rule" (the icon with the green plus sign) opens a wizard which allows you to define a new rule.

 This rule will be based on a threshold, and will be used to monitor operating system assets. We'd like to add a rule to track disk utilization for a specific file system - the /nfs-guest directory. To do this, we specify the following attribute

FileSystemUsages.name=/nfs-guest.usedSpacePercentage

The value of name in the attribute allows us to define a specific NFS shared directory or file system... in the case of this OS, we could have chosen any of the values shown in the File Systems Utilization chart at the beginning of this article.

usedSpacePercentage lets us define a threshold based on the percentage of total disk space used. There are a number of other values that we could use for threshold-based monitoring of FileSystemUsages, including

  • freeSpace
  • freeSpacePercentage
  • totalSpace
  • usedSpace
  • usedSpacePercentage

The final sections of the screen allow us to determine when to monitor for disk usage, and how long to wait after utilization reaches a threshold before creating an incident. The next screen lets us define the threshold values and severity levels for the monitoring rule:

If historical data is available, Ops Center will display it in the screen. Clicking the Apply button will create the new monitoring rule and active it in your monitoring policy.


If you combine this with one of the previous solutions, you can precisely define which file systems will generate incidents and notifications. For example, this monitoring policy has the default "File System Used Space Percentage" rule disabled, but the new rule reports ONLY on utilization for the /nfs-guest directory.


Stay Connected:

Twitter |  Facebook |  YouTube |  Linkedin |  Newsletter

Tuesday Oct 09, 2012

Oracle Enterprise Manager Ops Center : Using Operational Profiles to Install Packages and other Content

Oracle Enterprise Manager Ops Center provides numerous ways to deploy content, such as through OS Update Profiles, or as part of an OS Provisioning plan or combinations of those and other "Install Software" capabilities of Deployment Plans.  This short "how-to" blog will highlight an alternative way to deploy content using Operational Profiles.

Usually we think of Operational Profiles as a way to execute a simple "one-time" script to perform a basic system administration function, which can optionally be based on user input; however, Operational Profiles can be much more powerful than that.  There is often more to performing an action than merely running a script -- sometimes configuration files, packages, binaries, and other scripts, etc. are needed to perform the action, and sometimes the user would like to leave such content on the system for later use.

For shell scripts and other content written to be generic enough to work on any flavor of UNIX, converting the same scripts and configuration files into Solaris 10 SVR4 package, Solaris 11 IPS package, and/or a Linux RPM's might be seen as three times the work, for little appreciable gain.   That is where using an Operational Profile to deploy simple scripts and other generic content can be very helpful.  The approach is so powerful, that pretty much any kind of content can be deployed using an Operational Profile, provided the files involved are not overly large, and it is not necessary to convert the content into UNIX variant-specific formats.

The basic formula for deploying content with an Operational Profile is as follows:

  • Begin with a traditional script header, which is a UNIX shell script that will be responsible for decoding and extracting content, copying files into the right places, and executing any other scripts and commands needed to install and configure that content.
  • Include steps to make the script platform-aware, to do the right thing for a given UNIX variant, or a "sorry" message if the operator has somehow tried to run the Operational Profile on a system where the script is not designed to run.  Ops Center can constrain execution by target type, so such checks at this level are an added safeguard, but also useful with the generic target type of "Operating System" where the admin wants the script to "do the right thing," whatever the UNIX variant.
  • Include helpful output to show script progress, and any other informational messages that can help the admin determine what has gone wrong in the case of a problem in script execution.  Such messages will be shown in the job execution log.
  • Include necessary "clean up" steps for normal and error exit conditions
  • Set non-zero exit codes when appropriate -- a non-zero exit code will cause an Operational Profile job to be marked failed, which is the admin's cue to look into the job details for diagnostic messages in the output from the script.

That first bullet deserves some explanation.  If Operational Profiles are usually simple "one-time" scripts and binary content is not allowed, then how does the actual content, packages, binaries, and other scripts get delivered along with the script?  More specifically, how does one include such content without needing to first create some kind of traditional package?   All that is required is to simply encode the content and append it to the end of the Operational Profile.  The header portion of the Operational Profile will need to contain the commands to decode the embedded content that has been appended to the bottom of the script.  The header code can do whatever else is needed, and finally clean up any intermediate files that were created during the decoding and extraction of the content.

One way to encode binary and other content for inclusion in a script is to use the "uuencode" utility to convert the content into simple base64 ASCII text -- a form that is suitable to be appended to an Operational Profile.   The behavior of the "uudecode" utility is such that it will skip over any parts of the input that do not fit the uuencoded "begin" and "end" clauses.  For that reason, your header script will be skipped over, and uudecode will find your embedded content, that you will uuencode and paste at the end of the Operational Profile.  You can have as many "begin" / "end" clauses as you need -- just separate each embedded file by an empty line between "begin" and "end" clauses.

Example:  Install SUNWsneep and set the system serial number

Script:  deploySUNWsneep.sh ( <- right-click / save to download)

Highlights:

#!/bin/sh

# Required variables:
OC_SERIAL="$OC_SERIAL" # The user-supplied serial number for the asset
...

Above is a good practice, showing right up front what kind of input the Operational Profile will require.   The right-hand side where $OC_SERIAL appears in this example will be filled in by Ops Center based on the user input at deployment time.

The script goes on to restrict the use of the program to the intended OS type (Solaris 10 or older, in this example, but other content might be suitable for Solaris 11, or Linux -- it depends on the content and the script that will handle it).

A temporary working directory is created, and then we have the command that decodes the embedded content from "self" which in scripting terms is $0 (a variable that expands to the name of the currently executing script):

# Pass myself through uudecode, which will extract content to the current dir
uudecode $0

At that point, whatever content was appended in uuencoded form at the end of the script has been written out to the current directory.  In this example that yields a file, SUNWsneep.7.0.zip, which the rest of the script proceeds to unzip, and pkgadd, followed by running "/opt/SUNWsneep/bin/sneep -s $OC_SERIAL" which is the command that stores the system serial for future use by other programs such as Explorer.   Don't get hung up on the example having used a pkgadd command.  The content started as a zip file and it could have been a tar.gz, or any other file.  This approach simply decodes the file.  The header portion of the script has to make sense of the file and do the right thing (e.g. it's up to you).

The script goes on to clean up after itself, whether or not the above was successful.  Errors are echo'd by the script and a non-zero exit code is set where appropriate.

Second to last, we have:

# just in case, exit explicitly, so that uuencoded content will not cause error
OPCleanUP
exit

# The rest of the script is ignored, except by uudecode

#
# UUencoded content follows
#
# e.g. for each file needed, 
#  $ uuencode -m {source} {source} > {target}.uu5
# then paste the {target}.uu5 files below
# they will be extracted into the workding dir at $TDIR
#

The commentary above also describes how to encode the content.

Finally we have the uuencoded content:

begin-base64 444 SUNWsneep.7.0.zip
UEsDBBQAAAAIAPsRy0Di3vnukAAAAMcAAAAKABUAcmVhZG1lLnR4dFVUCQADOqnVT7up
...
VXgAAFBLBQYAAAAAAgACAJEAAADTNwEAAAA=
====
That last line of "====" is the base64 uuencode equivalent of a blank line, followed by "end" and as mentioned you can have as many begin/end clauses as you need.  Just separate each embedded file by a blank line after each ==== and before each begin-base64.

Deploying the example Operational Profile looks like this (where I have pasted the system serial number into the required field):

OpProfBlogImage1

OpProfBlogImage2b

The job succeeded, but here is an example of the kind of diagnostic messages that the example script produces, and how Ops Center displays them in the job details:

OpProfBlogImage3b

This same general approach could be used to deploy Explorer, and other useful utilities and scripts.

Please let us know what you think?  Until next time...
\Leon
--

Leon Shaner | Senior IT/Product Architect
Systems Management | Ops Center Engineering @ Oracle

The views expressed on this [blog; Web site] are my own and do not necessarily reflect the views of Oracle.



For more information, please go to Oracle Enterprise Manager  web page or  follow us at : 

Twitter | Facebook | YouTube | Linkedin | Newsletter

Monday May 21, 2012

Benefits of using Ops Center to deploy and manage Solaris 11

One of the more significant new features in Oracle Enterprise Manager Ops Center 12c is the ability to install Ops Center on Oracle Solaris 11, and to deploy and manage systems running Solaris 11.  The Solaris 11 capabilities are in addition to the analogous features for Solaris 10 and Linux, which can all be handled from the same Ops Center infrastructure.

When the Ops Center Enterprise Controller (EC) is installed on a system running Solaris 11, the EC can create a Solaris 11 Software Update Library containing Oracle Solaris Image Packaging System (IPS) content that is synchronized with the main Oracle repository at pkg.oracle.com. The Ops Center managed Solaris 11 repository becomes the package (pkg) publisher for downstream Solaris 11 deployments and updates on all Solaris 11 systems being managed by Ops Center.

Ops Center provides the ability to define Solaris 11 OS and Software Profiles comprised of Oracle Solaris 11 packages, user-supplied custom packages, scripts, and other files.  Such Software Profiles profiles can then be used to install and update software on systems already running Solaris 11 in a structured and consistent way.  Ops Center not only caches the main Oracle Solaris IPS repository, but more importantly it gives admins the ability to define their own preferred collection of packages so that systems can easily be kept in sync with each other, running a well-defined, life-cycle-managed Standard Operating Environment (SOE), instead of just whatever the latest content is at pkg.oracle.com.

Ops Center 12c also adds Solaris 11 features for bare-metal OS Provisioning, based on the Solaris 11 Auto Install (AI) facility.  Ops Center configures the Solaris 11 AI in a way that shields admins from needing to write custom AI manifests or custom "first boot" packages.  Solaris 11 deployments using Ops Center follow similar profile-based patterns as for Solaris 10 or Linux, all of which can all be deployed from the same Ops Center infrastructure running on Solaris 11.  The gory details of all these different times of bare-metal OS Provisioning are handled automatically for the user so that he or she does not need to put time and resources into manually creating and maintaining infrastructure for deploying different OS's natively -- Solaris 11 with AI, Solaris 10 JumpStart with JET, or Linux with Kickstart or AutoYast.  All of those OS's are handled by Ops Center under the covers, based on whatever network boot capability the OS requires (PXE/DHCP, WANBOOT, or AI), and all from the same Ops Center infrastructure running on Solaris 11.

Specific to Solaris 11 OS Provisioning (via AI), Ops Center provides its own "first-boot" package+scripts to customize the Solaris 11 deployment, and in particular this approach automatically installs the Ops Center agent.  With the Ops Center Agent in place right from the start, it is easy to handle post-install steps using the Ops Center features for handling Solaris 11 OS and Software Profiles containing additional packages, scripts, and content from the Solaris 11 Software Library, described above.

Tying bare-metal Solaris 11 deployment and post-install customization together is a key way that Ops Center simplifies the overall life-cycle management for Solaris 11 (in addition to Solaris 10 and Linux).   For example, a top-level plan based on "Configure and Install Logical Domains" can create numerous Logical Domains into an Oracle VM Server for SPARC "Server Pool" and provision Solaris 11 into each LDom Guest based on a powerful multi-step "Install Server" plan.  Such a plan can cover the end-to-end steps for installing and updating the OS, running scripts and adjusting monitoring parameters, etc.

Here is an example of the kinds of activities that can be performed in conjunction an OS Deployment, or separately, depending on the need:

Oracle Enterprise Manager Ops Center 12c Deployment Plan

NOTE: In the above example, the last step "Create Guests" can be used to create one or more Solaris Containers within the LDom Guest, rounding out the end-to-end deployment all the way from LDom Guest to Solaris 11 Global Zone to multiple Solaris Containers, if so desired.

One of the nicest aspect of deploying and managing Solaris 11 using Ops Center Plans and Profiles is that the same content can be applied as updates to existing Solaris 11 systems, aligned to the same content as chained off a bare-metal OS Provisioning.  It is up to the user which steps they want to include in a deployment plan -- whether they are updating Software Profiles on systems deployed 6 months ago to match the latest standard, or they are deploying new systems based on that same standard, Ops Center provides the means to insure that the outcome is consistent.

Here is an example of the kinds of activities that can be performed on an existing Solaris 11 OS -- either ad hoc, or immediately after a Solaris 11 OS Provisioning step, so that whether the life-cycle started with a new system, or the intent is to update a system deployed six months ago, the outcome can be the same:

S11 Update Multi-Step Plan

In short, Ops Center running on Solaris 11 can manage Solaris 10 and Linux systems, all from a common infrastructure, and all based on a simplified, consistent, profile- and plan-based way to do the OS and Software deployments and updates.  The net effect is an easy to use way to managing the life-cycle of heterogeneous systems, in a very consistent way through automation and re-use.

Please let us know what you think?  Until next time...
\Leon
--

Leon Shaner | Senior IT/Product Architect
Systems Management | Ops Center Engineering @ Oracle

The views expressed on this [blog; Web site] are my own and do not necessarily reflect the views of Oracle.



For more information, please go to Oracle Enterprise Manager  web page or  follow us at : 

Twitter | Facebook | YouTube | Linkedin | Newsletter

About

Latest information and perspectives on Oracle Enterprise Manager.

Related Blogs




Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
3
5
6
7
9
10
11
12
13
14
15
17
18
19
20
23
24
25
26
27
28
29
30
   
       
Today