Tuesday Aug 26, 2014

Q&A: Oracle's Andrew Sutherland on Managing the Entire Oracle Stack with Oracle Enterprise Manager 12c

As senior vice president of middleware for Oracle in Europe, Dr. Andrew Sutherland has more than 20 years’ experience in emerging technologies and their application to business problems. Currently, he manages a team of architects, business development managers, and technical specialists who help customers make the best use of their investments in Oracle technologies.

Given his breadth and depth of experience, we decided to ask Sutherland how Oracle Enterprise Manager 12c Release 4 is helping the Oracle customers he works with.

Q. What makes Oracle Enterprise Manager 12c different from competitors' offerings?
A. Oracle Enterprise Manager's approach is unique in that it manages across the entire Oracle stack, from applications, middleware, and the database all the way down to servers and storage. That means it can truly unlock the value of the entire Oracle stack.

Q. What is the payoff for organizations that adopt such a comprehensive approach?
A. Our customers are able to manage the entire Oracle estate in the most cost-effective way possible by automating many of their day-to-day tasks. To give you an idea of its scope, many of our customers have made sure that Oracle Enterprise Manager 12c’s dashboard is available to their senior IT management team. They use it to ensure that all parts of their IT stack are delivering what they should be delivering, when they should be delivering it.

Perhaps most important of all, Oracle Enterprise Manager 12c enables organizations to move beyond the old paradigm of multiple independent IT stacks to offer infrastructure as a service and platform as a service.

Q. As someone who helps customers make the most of their investment in Oracle technology, what do you find most promising about Oracle Enterprise Manager 12c Release 4?
A. There are three key areas that are especially exciting. First, it provides an accelerated path to the cloud. Whether you are building a small, medium, or large private cloud within your organization, it provides the tools you need to make it happen, from designing the cloud to provisioning and testing.

Secondly, this release provides monitoring and management tools that go both deeper into the stack and wider across components within the stack. That means an even more comprehensive dashboard.

Finally Oracle Enterprise Manager 12c Release 4 offers true enterprise-grade management. With the growth of social and mobile connectivity, the need for a highly performant and robust stack is more prominent than ever. And Oracle Enterprise Manager 12c is there to do exactly that: manage true, enterprise-grade IT deployments.

Q. What should Oracle customers do if they want to learn more about the latest release of Oracle Enterprise Manager 12c?
A. First, speak to your Oracle contact, whether it is a partner or Oracle representative, to get more complete information. Also consider coming to an Oracle Day event in your area, especially if you can attend one dedicated to cloud computing. And in the meantime, you can always head to the Oracle Enterprise Manager pages on oracle.com to get started.

Find out more about Oracle Enterprise Manager 12c Release 4.

Watch a short video featuring Sutherland.

Stay Connected:
Twitter |
Facebook | YouTube | Linkedin | Newsletter
Download the Oracle Enterprise Manager 12c Mobile app

Monday Aug 25, 2014

Aramark Streamlines IT with Oracle Enterprise Manager

Aramark, a global leader in the food services industry, facilities management, and uniform services business, delivers world-class application SLAs consistently using Oracle Enterprise Manager.

Stay Connected:
Twitter |
Facebook | YouTube | Linkedin | Newsletter
Download the Oracle Enterprise Manager 12c Mobile app

Wednesday Aug 20, 2014

Oracle OpenWorld 2014 Preview: Don't-Miss Sessions, Hands-on Labs, and More

Check out all the latest Oracle Enterprise Manager 12c sessions at this year's Oracle OpenWorld. Organizers of the event, taking place in San Francisco from September 28 to October 2, expect heavy turnout at sessions, hands-on labs, and customer panels devoted to Oracle Enterprise Manager 12c. Find out who is participating and which sessions are most recommended by the Oracle Enterprise Manager team.
Read More

Stay Connected:
Twitter |
Facebook | YouTube | Linkedin | Newsletter
Download the Oracle Enterprise Manager 12c Mobile app

Wednesday Aug 13, 2014

Oracle Enterprise Manager 12c Wins 2 DBTA Readers’ Choice Awards

In the first annual 2014 Readers’ Choice Awards, Database Trends and Applications (DBTA) magazine announced that Oracle Enterprise Manager 12c has won two awards for "Best Database Administration Solution" and "Best Database Performance Solution."

More than 22,000 votes were cast across 31 different database categories, from Best Relational Database to Best Database Appliance, DBTA called the contest a tight race and was "often neck and neck" between nominees.

"Oracle has a longstanding commitment to superior manageability for our products. Oracle Enterprise Manager's continued success and innovations in providing the leading solution for managing Oracle Database is a prime example of this commitment. These awards are a further testament to the importance customers place on Oracle Enterprise Manager." says Moe Fardoost, Senior Director, Product Marketing for Oracle Enterprise Manager.

You can see the complete list of winners here: Database Trends and Applications magazine—2014 Readers’ Choice Awards

Stay Connected:
Twitter |
Facebook | YouTube | Linkedin | Newsletter
Download the Oracle Enterprise Manager 12c Mobile app

Tuesday Aug 05, 2014

EM12c Release 4: Using Repository-Side Metric Extensions

A few weeks ago, someone asked on the OTN forums how to alert on some of the JVM metrics such as ‘JVM Threads – Threads Started (since startup)’ using Enterprise Manager 12c (EM).    This is one of those few metrics that EM collects, but does not allow custom thresholds.    Let’s take a look at the metrics that EM collects on the WebLogic Server target.

Read more... 

Friday Aug 01, 2014

Organizing Your Enterprise Manager Targets

If you’re monitoring more than a handful of servers or databases in your Enterprise Manager 12c (EM), you have probably started creating groups to manage many targets together.   If you haven’t, this is one of the most critical aspects of setting up your EM to properly monitor and manage targets.  There are several use cases where you will want to perform a single action on multiple targets.

  • Setting monitoring thresholds
  • Granting privileges
  • Sending notifications
  • Applying compliance rules
  • Viewing dashboards
  • Running jobs, upgrades, backups
  • Creating reports

The easiest way to perform a single action against multiple targets is to use groups.  Oracle Enterprise Manager (EM) 12c has 4 types of groups that we can take advantage of.  


Wednesday Jul 30, 2014

Reducing Downtime While Patching Multi-OMS Environments

Enterprise Manager has now been released for a few weeks, as well as the OMS Bundle patches (also known as System patches). If you plan to apply these bundle patches to your OMS, and you are concerned about the downtime, then, you can reduce the downtime by referring to this whitepaper that contains patching instructions to reduce downtime. 

This whitepaper covers various Enterprise Manager High Availability (EM HA)  usecases (level 1, 2, 3, 4), and contains instructions on how to reduce downtime while applying patches to each of these usecases. It also clearly defines the steps that require downtime and those that do not.

If you have a multi-OMS setup, you can also refer to this whitepaper which covers script creation using the opatchauto command, which automates the substeps and further reduces downtime.During our internal testing of this whitepaper on an EM HA setup, we have noticed a significant reduction in downtime. 

If your customer plans to do an Enterprise Manager Upgrade to, then as a post upgrade recommendation, they should patch their OMS with the latest bundle patches by following the instructions outlined in this whitepaper.

White paper on OTN:

MOS note for the latest Bundle Patches:
Enterprise Manager (PS3) Master Bundle Patch List (Doc ID 1900943.1)

Stay Connected:

Twitter |  Face book |  You Tube |  Linked in |  Google+ Newsletter

Wednesday Jul 23, 2014

Enterprise Manager Ops Center 12c R2 U1 Released

We are happy and excited to announce that on July 20, 2014, Oracle Enterprise Manager Ops Center 12c Release 2 Update 1 was released for all platforms including Oracle Solaris SPARC/x86 and Linux.

Ops Center 12cR2 PSU1 is an update release containing improvements and enhancements in the following areas: performance, new hardware support and general quality improvements.

In the performance area we have made improvements in core Ops Center components such as the Enterprise Controller, Proxy Controller and Virtualization Agent. We have reduced the Enterprise Controller memory footprint and enhanced start up times for the Enterprise Controller and agents. We have looked at areas such as deployment wizards and the management of LUN's and made improvements in the performance of these areas.

For new hardware we support the discovery, monitoring and provisioning of both OS and Firmware for: X4-4, X4-8, M4000 and M10. We also made improvements for firmware management of the X4-2 and introduced enhanced support for add / modify hardware configurations for Oracle SuperCluster. 

In the general quality area we improved security, refined logging, made improvements to OS provisioning and enhanced areas such as LDAP and the UI.

For customers with a support contract historically any new updates to the Ops Center components would automatically appear in the Download window of the UI. However, we have noticed a bug which prevents this new version appearing. A fix / IDR is in place and is described in the release notes for this version available here. There is also a MOS note 1908726.1 which describes the IDR and procedure to enable the download.

Software downloads outside of this automatic process will be available on the OTN and edelivery in the coming days.

Before upgrading your Enterprise Controller please check the upgrade guide available here.

There is also an excellent blog which gives advice on pre upgrade tasks to ensure a smooth and successful upgrade experience here.

EM12c Release 4: Upgrading Agents with Ease

Now that Enterprise Manager 12cR4 has been out for a little while, more people are getting around to upgrading their agents.  Since the monthly Patch bundles were released we already have a few Agent side patches that we want to apply to our newly upgraded agents.  I’ve written about simplifying your agent patching before, but this feature still seems to fly under the radar.  It’s days like these that I miss running a massive Enterprise Manager with thousands of databases, because this is one of the things that would have made me dance in my cubicle.

Let's say, you have 100 agents (50 with Database plug-in, 50 with Middleware plug-in).  In my previous blog on EM patches, I explained the different types of patches available for EM, so I’m not going to go into detail here.   What I'm going to illustrate is how we can upgrade those 100 agents, and patch them with the following patches in one step (current as of today):

[Read More]

Tuesday Jul 22, 2014

Upgrading to Enterprise Manager Ops Center

Hi all,

this is just a quick note for those looking to upgrade to the new relase of Ops Center (

It is avalable for download:

  • From OTN - after July 25, 2014
  • From OSDC - after July 31, 2014
  • From  the Ops Center BUI - after July 20, 2014

    Before you can see the new version in an existing 12.2.0 (or earlier) version of Ops Center you will need to apply a quick patch to to make the new version visable to downlaod. Details of how to access this IDR and how to apply it can be found in the MOS Note (Doc ID 1908726.1) 

    Happy Upgrading



  • Sunday Jul 20, 2014

    Pre-Upgrade Checks Enterprise Manager Ops Center

    With the release of Enterprise Manager Ops Center 12.2.1, it is time to go through the upgrade cycle. I thought I would share the pre-upgrade checks I go through when I upgrade to a new Ops Center build. As part of the development team, I get involved in pre-release Quality Assurance testing, which means I end up doing hundreds of upgrades as part of the testing process.

    Update releases come out regularly and contain enhancements and bug fixes. As with any other application in your environment, you should upgrade Ops Center to the current release/update in a timely and controlled manner. For those of you who are long time sys-admins, there is no rocket science here. It is the same sort of planning you would do for any other Enterprise level application.

     In my test environments, I have my Enterprise Controller (EC) and Proxy Controllers (PC) inside Solaris Zones (Solaris 11), so I have a couple of extra checks I do, but the process as a whole is still valid if your EC/PC are on their own separate hardware.

    1) Read the Release Notes

    Yes, those release notes/README files are important and you should spend the time reading them. They will contain the latest information about the update and any known issues and workarounds.

    2) Check Free Disk Space

    Confirm that there is enough disk space to unpack and install the upgrade. How much is enough space is the ultimate question. It will vary with each different upgrade and will depend on how you have configured your underlying filesystems and your actual environment. Here are some guidelines. Please note that the numbers I quote tend to be a little generous as it is always better to have more free space than not enough.

    • There should always be a few GB of space free in the root partition (it is just good sys-admin practice - below 90 % would be ideal).
    • The filesystem that holds /var/tmp  will need space for the DB backup that is run as part of the installer. The size of this will depend on the size of your DB. So check how much a "ecadm backup" takes on your system.
    • The filesystem that holds /var/tmp is also the temporary location where we unpack the upgrade bundle.
    • The filesystem that holds /var/opt/sun/xvm will have the majority of the upgrade code installed into it as well as a copy of the installer under the update-saved-state directory.
    So what does that mean for size requirements?
    • You need about 5 times the upgrade bundle. The current upgrade bundle is 3.8GB unpacked, so that would be 20GB.
    • The DB backup will take about 10% of the actual DB size.
    root@ec:/ec_backup# du -hs *
     1.3G   sat-backup-pre-12.2.1-upgrade.20140702
    root@ec:/ec_backup# du -hs /var/opt/sun/xvm/oracle/oradata/OCDB
    14G   /var/opt/sun/xvm/oracle/oradata/OCDB

     Although more space is actually used during the backup before it is packed up, I would allow for about 4 GB of space.

    • So for my environment, I would look for about 25GB (rounding up) free space (your number may vary). I am sure I could scrimp and save and get this number down, but the idea is to have plenty of free space to allow for the upgrade to go through without incident.

    3) Backups Backups Backups

    Before commencing any upgrade, you should make sure you can roll back if something goes horribly wrong. Years of history in administration and support have made me a paranoid person. I believe you can never have too many backups, so I do the following:

    • Confirm you have a successful database backup using "ecadm backup". (You should already be doing this on a weekly basis)
    root@ec:/# /opt/SUNWxvmoc/bin/ecadm backup -d pre-12.2.1-upgrade -o
    ecadm: using logFile =
    ecadm: *** PreBackup Phase 
    ecadm: *** Backup Phase
    ecadm: *** PostBackup Phase 
    ecadm: *** Backup complete
    ecadm: *** Output in /ec_backup/sat-backup-pre-12.2.1-upgrade.20140702 
    ecadm: *** Log in /var/opt/sun/xvm/logs/sat-backup-2014-07-02-11:52:16.log
    Of course, copy the generated backup file to somewhere safe on another system.
    • Confirm you have a successful filesystem backup using your Enterprise backup software. (You should already be doing this on a weekly basis.) I would recommend full filesystem backups and having a separate backup of the /var/opt/sun/xvm directory and any of your Ops Center software libraries if you did not put them in the default location (/var/opt/sun/xvm/locallib/swlib[0-2]).
    • Take a ZFS snapshot (recursive) of the full zone (rpool and any other zpool that are part of the zone). This is normally your easiest and fastest roll back method should you need it. NOTE: Make sure you know how to recover/rollback a zone. "zfs snapshot -r rpool" recursively snapshots all underlying filesystems, but "zfs rollback -r rpool" will only rollback a single filesystem. You need to rollback each filesystem separately. If you are not sure, practice it on a test zone first.
    ### Take a zfs snapshot ###
        root@ec:/# zfs list
        NAME                              USED  AVAIL  REFER  MOUNTPOINT
        rpool                             156G  41.1G    31K  /rpool
        rpool/ROOT                        134G  41.1G    31K  legacy
        rpool/ROOT/solaris                134G  41.1G  24.6G  /
        rpool/ROOT/solaris-backup-1       174K  41.1G  1.37G  /
        rpool/ROOT/solaris-backup-1/var   110K  41.1G  27.9G  /var
        rpool/ROOT/solaris-backup-2       296K  41.1G  24.2G  /
        rpool/ROOT/solaris-backup-2/var   232K  41.1G  48.4G  /var
        rpool/ROOT/solaris/var            109G  41.1G  77.2G  /var
        rpool/VARSHARE                     88K  41.1G  66.5K  /var/share
        rpool/ec_backup                  1.29G  41.1G  1.29G  /ec_backup
        rpool/export                      161K  41.1G    32K  /export
        rpool/export/home                 111K  41.1G    32K  /export/home
        rpool/export/home/ocadmin          61K  41.1G  40.5K 
        rpool/oracle                     20.7G  41.1G  20.7G 
        root@ec:/# zfs snapshot -r rpool@pre-OC-12.2.1-install.20140702

    4) Check for any failed services

    It is good practice to clear/enable/disable any broken SMF services, but there are a few key ones to check.

    • Make sure all the Ops Center services that should be running are running and the ones that should not are not. A classic example here is when you have an EC running without a collocated PC. The PC shows as disabled, but still shows in a "svcs -xv" output.

        root@ec:/var/tmp/downloads# svcs -xv
        (Cacao, a common Java container for JDMK/JMX based management
         State: disabled since June 12, 2014 08:07:08 AM EST 
        Reason: Disabled by an administrator.
           See: http://support.oracle.com/msg/SMF-8000-05
           See: man -M /usr/share/man -s 1M cacaoadm 
           See: man -M /usr/share/man -s 5 cacao
           Impact: 1 dependent service is not running: 

    In this case, our EC did not have a collocated PC, so we should ensure that these services are really disabled and don't try to start-up during the upgrade process.

        root@ec:/var/tmp/downloads# svcadm disable
        root@ec:/var/tmp/downloads# svcadm disable
    •  If you are using zones either on the system where the EC is installed in the GZ or your EC/PC run in a NGZ, you also need to check that the IPS proxies are running to allow the Solaris 11 packaging system to work correctly.
      • In a Global Zone (GZ) check that zones-proxyd is online.
    root@t4-1-syd04-b:~# svcs svc:/application/pkg/zones-proxyd:default
    STATE          STIME    FMRI
    online         Jul_02   svc:/application/pkg/zones-proxyd:default
      • In a Non Global Zone (NGZ)check that the zones-proxy-client is online.
    root@ec:~# svcs svc:/application/pkg/zones-proxy-client:default
    STATE          STIME    FMRI
    online          8:54:47 svc:/application/pkg/zones-proxy-client:default
    • What you are looking for is a clean bill of health from "svcs -xv" command.
        root@ec:/var/tmp/downloads# svcs -xv

    5) Check the pkg publishers

    To be able to do a successful upgrade, you need the pkg publisher for a system to be working. In a zones environment, that means the publishers in the GZ and all the NGZ should be working. Publishers that don't resolve when a package links into a zone will cause the whole upgrade to stop.

    So here are a couple things to look for when you are using an EC in a zone.

    • If this was a test environment where you had multiple EC/PC in different zones, either those EC/PC should be running or the publishers that point to a NON running EC/PC should be cleared. This can be done by issuing a
    # pkg unset-publisher Publisher-Name 

    The aim here is to clear all the local publishers in the zone and just use the proxied publishers in the GZ.

    • If you have the GZ pointing to a PC that points to the EC that is being upgraded, where the EC is in a NGZ under the GZ (yes this is the whole chicken and egg problem), you have a slightly different problem. During the upgrade, parts of the EC will be shutdown which will stop the remote PC from proxying access to EC's IPS repository. So you need to set the publishers to point to an IPS repository that they can reach. Luckily, the actual IPS repository on the EC does still keep running on port 11000 throughout the upgrade.
    root@t4-1-syd04-b:~# pkg publisher
    PUBLISHER                   TYPE     STATUS P LOCATION
    solaris                     origin   online F https://oracle-oem-oc-mgmt-pc217:8002/IPS/
    cacao                       origin   online F https://oracle-oem-oc-mgmt-pc217:8002/IPS/
    mp-re          (non-sticky) origin   online F https://oracle-oem-oc-mgmt-pc217:8002/IPS/
    opscenter                   origin   online F https://oracle-oem-oc-mgmt-pc217:8002/IPS/
    root@t4-1-syd04-b:~# pkg unset-publisher opscenter
    root@t4-1-syd04-b:~# pkg unset-publisher mp-re
    root@t4-1-syd04-b:~# pkg unset-publisher cacao
    root@t4-1-syd04-b:~# pkg set-publisher -G '*' -g http://ec:11000/ solaris
    root@t4-1-syd04-b:~# pkg publisher
    PUBLISHER                   TYPE     STATUS P LOCATION
    solaris                     origin   online F http://ec:11000/
    • You can reset to their original state, all the publishers that were set by Ops Center, by rebooting the system or running the install_ips_ac.sh script in each zone.
    # /var/opt/sun/xvm/utils/install_ips_ac.sh -P PC_IP_Address
    Use as the IP address for the EC/PC when it is pointing too itself

    6) Run OCDoctor troubleshoot

    Run the OCDoctor troubleshoot script over your EC and PC's before an upgrade. It is a good sanity check to look for and fix underlying problems before you start the upgrade process. If you are in connected mode, your EC should already have the latest version of OCDoctor downloaded. Otherwise, you can update it by running "OCDoctor.sh --update" or downloading from https://java.net/projects/oc-doctor/downloads/download/OCDoctor-4.36.zip

    Note: The error "'root' should not be a role" can be safely ignored as it was only required for earlier versions of Ops Center.

        root@ec:/var/tmp/downloads# /var/opt/sun/xvm/OCDoctor/OCDoctor.sh -t
        Ops Center Doctor 4.34  [OC,SunOS11] [Read only]
        [02-Jul-2014 11:25AM EST]
        ======================== Checking Enterprise Controller...==============================
        OK: Total number of OSes: 12  Total LDOMs:7  Total Zones: 
        ERROR: User 'root' should not be a role. You should convert it to a
        normal user before the installation.
               This can be done by running:
        # rolemod -K type=normal root
        OK: Files in /var/opt/sun/xvm/images/agent/ have the right permissions
        OK: Files in /var/opt/sun/xvm/osp/web/pub/pkgs/ have the right  permissions
        OK: both pvalue and pdefault in systemproperty are equal to false (at id 114)
        OK: Found only 285 OCDB*.aud files in oracle/admin/OCDB/adump folder
        OK: Found no ocdb*.aud files in oracle/admin/OCDB/adump folder
        OK: No auth.cgi was found in cgi-bin
        OK: User 'oracleoc' home folder points to the right location
        OK: User 'allstart' home folder points to the right location
        OK: Apache logs are smaller than 2 GB
        OK: n1gc folder has the right permissions
        OK: All agent packages are installed properly
        OK: All Enterprise Controller packages are installed properly
        OK: Enterprise Controller status is online
        OK: the version is the latest one (
        OK: satadm timeouts were increased
        OK: tar command was properly adjusted in satadm
        OK: stclient command works properly
        OK: Colocated proxy status is 'disabled'
        OK: Local Database used space is 19%, 6G out of 32G (local DB, using 1 files)
        OK: Debug is disabled in .uce.rc
        OK: Debug is disabled for cacao instance oem-ec
        OK: no 'conn_properties_file_name' value in .uce.rc
        OK: 30G available in /
        OK: 30G available in /var
        OK: 30G available in /var/tmp
        OK: 30G available in /var/opt/sun/xvm
        OK: 30G available in /opt
        OK: DNS does not unexpectedly resolve hostname '_default_'
        OK: Found the server .uce.rc at /var/opt/sun/xvm/uce/opt/server/cgi-bin/.uce.rc
        OK: Server .uce.rc has the correct file permissions
        OK: Server .uce.rc has the correct ownership
        OK: Connectivity to the KB public servers works properly (using download_large.cgi)
        OK: Grouplock file doesn't exist
        OK: package hmp-tools@2.2.1 is not installed
        OK: package driver/x11/xsvc is not installed
        OK: Cacao facet is set to False
        OK: All Solaris 11 agent bundles in /var/opt/sun/xvm/images/agent are imported properly to the repository
        OK: Disconnected mode is not configured
        OK: Locales are OK ("en_US.UTF-8")
        OK: No need to check for Solaris 11 agent bundle issue as this EC is newer than Update 1
        OK: No partially installed packages
        OK: UCE 'private' folder exists
        OK: No http_proxy is set in the user profile files
        OK: 'public' folder has the right ownership
        OK: 'public' folder is writable for uce-sds
        OK: 'private' folder has the right ownership
        OK: 'private' folder is writable for uce-sds
        OK: '/var/tmp' folder is writable for uce-sds
        OK: No old jobs rerun (CR 6990675)
        OK: No need to adjust SEQ_COUNT (MAXID:2986 SEQCOUNT:2986)
        OK: no row with ssh.tunnel.info found in DB table
        NOTICE: Can't perform cryptoadm test inside a zone.
                Run --troubelshoot from the global zone as well to test the crypto services.
        OK: System time is not in the past
        OK: User uce-sds is part of all the proper groups
        OK: oracleoc user ulimit -Sn is 1024
        OK: oracleoc user ulimit -Hn is 65536
        OK: FC Libraries do not contain duplicate LUNs
        OK: 'update-saved-state' folder exists and has the right permissions
        OK: verify-db does not return 'Invalid pad value' message
        OK: No credential issues found 
        =========== Proxy controller is installed but not configured, skipping ==================
        =========== Agent controller is installed but not configured, skipping ==================

    Now do the upgrade

    Choose whichever upgrade method you like. Both the BUI and CLI methods will give you the same end result. The Ops Center upgrade is not a difficult upgrade and following some simple pre-work checks will maximize your chance of a straightforward and successful upgrade.


    Rodney Lindner


    Thursday Jul 17, 2014

    Patching 101 - The User Friendly Guide to Understanding EM Patches

    There was a conversation on twitter last week about available patches for Enterprise Manager (EM), and it got a little deeper than 140 characters will allow.  I've written this blog to give a quick Patching 101 on the types of EM patches you might see and the details around how they can be applied.

    OMS Patches

    The core Enterprise Manager system is typically patched with the quarterly PSU patches (released Jan, Apr, July, Oct) or a one-off when directed by support for a critical issue.  PSU patches will be cumulative, so you need not apply each of them, just apply the latest.  The OMSes must be shutdown during patching, however some patches are being released with rolling patch instructions for multi-OMS systems.  These patches must be applied at the host level, and cannot be automated via EM.   ALWAYS read the readme, yes every time.  The patching steps can change from patch to patch so it's critical to read the readme. OPatch or OPatchauto will be used to apply these patches.  Did I mention to read the readme for every patch?  It's also important to note that there may be additional steps when patching in a multi-OMS or standby environment, so read the output of OPatchauto carefully.

    Always download the latest OPatch release for the appropriate version.  If you read the readme, you already know this!   Download patch 6880880 for 11.1 (the OPatch version used by EM) and unzip into the $ORACLE_HOME.  Most errors in patching are related to not updating OPatch. 

    For more information on PSU Patches and patching EM:
    Oracle Enterprise Manager Cloud Control Administrators Guide - Chapter 16 Patching Oracle Management Server and the Repository
    EM 12c Cloud Control: List of Available Patch Set Updates PSU (Doc ID 1605609.1)
    How to Determine the List of Patch Set Update(PSU) Applied to the Enterprise Manager OMS and Agent Oracle Homes? (Doc ID 1358092.1)

    Each plug-in has binaries that will require patches as well.  Same downtime requirements apply for plug-in patches as the quarterly PSUs.  Starting in, the plug-in patches are being released as a monthly bundle.  This means that if you have 6 plug-ins, you may have 6 OMS side patches to apply - 1 for each plug-in.  Bundles are not always released for every plug-in every month.  They are cumulative, so pick the latest.

    Starting with, the individual OMS-side plug-in bundles are being grouped into a System  Patch each month. So for example, in June 2014 the System patch includes MOS, Cloud, DB, FA, FMW, SMF, and Siebel plug-ins.  Non-required patches will be skipped.

    For more information on the EM Patch Bundles and Patching EM:
    Enterprise Manager (PS3) Master Bundle Patch List (Doc ID 1900943.1)
    Enterprise Manager Bundle Patch Master Note (Doc ID 1572022.1)

    Agent Patches

    Agent patches are applied to each agent.  They can be applied via EM using the MOS patch plans, which makes it a lot easier when you have 100s or 1000s of Agents to patch!  The Patch Plans will start a blackout, validate prerequisites, check for conflicts, and update OPatch for you.  If you don't use the Patch Plan you can patch manually with OPatch, don't forget to read the readme!  The Agent must be shutdown during the patch application.  There are 4 main types of Agent patches you will see:

    • Core Agent - Starting with the core Agent will have monthly patch bundles .  These are also cumulative, so my recommendation is to apply the latest one.  
    • Agent-side Discovery Plug-in - This is the lightweight piece of the plug-in used for target discovery.  Discovery plug-in patches are cumulative with other discovery plug-in patches for that component. 
    • Agent-side Monitoring Plug-in - This is the more detailed monitoring side of the plug-ins for the required components.  Monitoring plug-in patches are cumulative with other monitoring plug-in patches for that component.   So if there's a Discovery and Monitoring patch available for the DB Plug-in, you need to apply both of them.  
    • JDBC patches for the Agent will be JDBC version  These patches do get applied to the Agent, and can be applied via the Patch Plans.  

    You can apply the latest Agent bundle, JDBC patch and the plug-in bundles in one patch plan.   If there's a conflict, you'll be notified.   If the Agents you've selected don't have specified plug-ins, you'll also receive notice during the analyze step.  As of now, for my agents, I would apply the patch (18873338) and the two available plug-in agent patches DB monitoring (19002534) and FMW monitoring (18953219) and the latest JDBC patches (18502187,18721761) all in one patch plan.

    I discovered a new feature in while testing this.  Normally you had to have Normal Oracle Home preferred credentials set for all Agent targets to patch, or select Override and specify the Normal Oracle Home credentials.   In, the Agent uses it's internal credentials to Patch itself, so setting preferred credentials or specifying at run-time is not required.  The user patching would require the Manage Target Patch and Patch Plan privileges.  

    For more details on Agent patching:
    Oracle Enterprise Manager Cloud Control Administrators Guide - Chapter 17 Patching Enterprise Manager Agents 
    Simplified Agent and Plug-in Patching


    The OMS and Agent are the key components, and my main focus here.  However it's important to keep the infrastructure stack up to date as well.  This includes the Oracle Fusion Middleware and Oracle Database that are used for EM.   The recommendation is to follow the best practices for each of these components, and regularly update with the PSU patches available.   The following reference notes will help in identifying the current PSU patches.   The WebLogic Server version used by EM 12c is 10.3.6. 

    Oracle Recommended Patches -- Oracle Database (MOS 756671.1)
    Master Note on WebLogic Server Patch Set Updates (PSUs) (MOS 1470197.1)


    Hopefully this will help you understand the various types of components involved with keeping EM up to date.   Obviously, you may not want to patch each month and maybe not every quarter, but the patches are available to keep the software up to date and make things easier to apply in bundles.  You'll want to setup a plan for planned software maintenance in your environment.  There's a whitepaper Oracle Enterprise Manager Software Planned Maintenance that will help guide you through the best practices.  

    Tuesday Jul 01, 2014

    Limit Self Service User Access to Database Self Service Portal

    When implementing database as a service and/or snap clone, a common request was for a way to hide the other service types like IaaS, MWaaS, etc from the self service portal for the end users. Before EM12c R4, there was no way to restrict the portal view. Essentially, any user with the EM_SSA_USER role would be directed to the self service portal and would then be able to see all service types supported by EM12c.

    Of course, you could always set Database as your default self service portal from the 'My Preferences' pop up, but this only helps with their post-login experience. The end user still gets to see all the options as shown in screen above.

    In EM12c R4, a new out of the box role called EM_SSA_USER_BASE has been introduced. This role, by default, does not give access to any portal, that is an explicit selection. Here is how you use this role:

    1. Create a custom role and add the EM_SSA_USER_BASE role to it.

     2. Now in the Resource Privileges step, select the Resource Type 'Cloud Self Service Portal for Database', and edit it

    3. Check the 'Access the Cloud Self Service Portal for Database.' privilege. Finish the rest of the wizard.

     Now, when a user with this custom role accesses the self service portal, they can only do so for databases and nothing else.

    While the EM_SSA_USER role will continue to work, we recommend you start using the new EM_SSA_USER_BASE role. For more details on DBaaS or Snap Clone roles, refer to the cloud admin guide chapter on roles and users.

    -- Adeesh Fulay (@AdeeshF)

    Friday Jun 27, 2014

    Convert Crontab to Enterprise Manager Jobs

    Surprisingly, a popular question posted on our internal forum is about the possibility of using the Enterprise Manager (EM) Job System to replace customer’s numerous cron jobs. The answer is obviously YES! I say surprisingly because the EM Job system has been in existence for around 10 years (I believe since EM, and my hope was that, by now, customers would have moved to using more enterprise class job schedulers instead of cron. So, here is a quick post on how to get started with this conversion from cron to EM Jobs for some of our new users.

    Benefits of EM Job System

     Before we learn about the how, let’s look at the why. The EM job system is:

    • Free - (Yes, I said free) It is included with the base EM at no cost.
    • Flexible - It supports multiple options for scheduling, notification, authentication, etc
    • Infinitely scalable - the job system seamlessly scales to every new Oracle Management Server (OMS). In fact, in case of OMS failures, the job steps are automatically picked up by the next available OMS without affecting the job execution.
    • General purpose - General purpose since it provides numerous out-of-the-box job types like run OS command, start/stop, backup, SQL Script, patch, etc that span multiple target types. As of today, there are over 50 job types available in the product.
    • Enterprise grade - It allows users to automate multiple administrative tasks like backup, patching, cloning, etc across multiple targets. Customers have not only converted their cron jobs to EM, but have also replaced other enterprise tools like Autosys and migrated 1000s of jobs to EM Job System.
    • APIs - Jobs can be scheduled and managed from the UI and using EMCLI (the command line interface).

    Now back to our topic.

    The Conversion Process

    Let’s start with a sample crontab that we want to convert.

    Sample Crontab

    A cron expression consists of 6 fields, where the first 5 fields represent the schedule, while the last field represents the command or script to run.

     Field Name
    Mandatory?  Allowed Values
     Allowed special characters
     Minutes  Yes 0-59  * / , -
     Hours  Yes  0-23  * / , -
     Day of month
    Yes  1-31 * / , - ? L W
     Month Yes  1-12 or JAN-DEC
    * / , -
     Day of week
    Yes  0-6 or SUN-SAT
    * / , - ? L #

    Cron jobs run on the operating system, often using the native shell or other tools installed on the operating system. The equivalent of this capability in Enterprise Manager is the ‘OS Command’ job type. Here are the steps required to convert the first entry in the crontab to an EM job:

    1. Navigate to the Job Activity page
    Job activity menu

    2. Select the ‘OS Command’ job and click Go
    OS Command

    A 5-tab wizard will appear. Let’s step through this one by one.

    3. Select the first tab called ‘General’. Here provide a meaningful name and description for the job. Since this job will be run on the Host target, keep the target type selection as ‘Host’. Next, select all host targets in EM that you wish to run this script against.

    While cron jobs are defined on a per host bases, in EM a job definition can be run and managed across multiple hosts or groups of hosts. This avoids having to maintain the same crontab across multiple hosts.


    4. Select the ‘Parameters’ tab. Here enter the command or script as specified in the last field of the crontab entry. When constructing the command, you can make use of the various target properties.
    Parameters tab

    5. Next select ‘Credentials’. Here we provide the credentials required to connect to the host and execute the required commands or scripts. Three options are presented to the user:

    • Preferred – default credential set for the host
    • Named - Credentials are stored within Enterprise Manager as "named" entities. Named credentials can be a username/password, or a public key-private key pair. Here we choose pre-created named credentials
    • New – This allows us to create and use new named credential

    Note: If your OS user does not have the required privileges to execute the set command, Named Credentials also support use of sudo, powerbroker, sesu, etc.

    Credentials tab

    6. Next, we set the schedule and this is where it gets interesting. As discussed before, crontab uses a textual representation for the schedule, while EM Job system has a graphical representation for the schedule.

    Our sample schedule in the crontab is ‘00 0 * * Sun’. This translates to a weekly job at 12 midnight on every Sunday. To set this in EM, choose the ‘Repeating’ schedule type. The screenshot below shows all the other selections.
    Schedule tab

     The key here is to select the correct ‘Frequency Type’, the rest of the selections are quite obvious. This also lets you choose the desired timezone for the schedule. Your options are to either start the job w.r.t a fixed timezone, or start it in individual target's timezone. The latter is very popular, for example, I want to start a job at 2 AM local time in every region around the world.

    Another selection of note is that for ‘Grace Period’. This is an extremely powerful feature, but often not used by many customers. Typically, we expect jobs to be started within a few seconds or minutes (based on the load on the system and number of jobs scheduled) of the start time, but a job might not start on time for many reasons. The most common reasons are the Agent being down or due to a blackout. The grace period controls the latest start time for the job in case the job is delayed, else its is marked as skipped. By default, jobs are scheduled with indefinite grace periods, but I highly recommend setting a value for it. In the sample above, I set a 3 hr limit which may seem large but given the weekly nature of the job seems reasonable. So the job system will wait until 3 am (the job start time is 12 am) to start the job, after which the iteration will be skipped. For repeating schedules, the grace period should always be less than the repeat interval. If the job starts on time, the grace period is ignored.

    7. Finally, we navigate to the ‘Access’ tab. This tab has two parts:

    • Privilege assignment to roles and users: this allows you to control job level access for other users
    • Email notifications for the Job owner: this allows you to control the events for which you wish to receive notifications. Note, this only sets notification for the job owner, the other users can subscribe to emails by setting up notification and/or incident rules.

    To prevent EM from sending deluge of emails, I recommend the following settings in the notifications region:

    • Match status and severity: Both
    • Select severity of status: Critical
    • Select status: Problems & Action Required

           You can always come back and modify these settings to suit your needs.

    Access tab

    Not all cron jobs need to be converted to OS command. For example, if you are taking Oracle database backups using cron, then you probably want to use the out-of-the-box  job type for RMAN scripts. Just provide the RMAN script, list of databases to run this against, and the credentials required to connect to the database. Similarly, if you run sqls on numerous databases, you can leverage the SQL Script job type for this purpose. There are over 50 job types available in EM12c, all available for use from the UI and EMCLI.

    Finally, the best way to learn more about the EM Job System is to actually play with it. I also recommend blogs from Maaz, Kellyn, and other users on this topic. Good luck!!


    Maaz Anjum: http://maazanjum.com/2013/12/30/create-a-simple-job-for-a-host-target-in-em12c/
    Kellyn Pot'vin: http://dbakevlar.com/

    -- Adeesh Fulay (@adeeshf)

    Thursday Jun 26, 2014

    Creating A Secondary I/O domain with Ops Center 12.2

    Contributed by Juergen Fleischer and Mahesh Sharma.

    The purpose of this blog is to show you how to create a Secondary I/O domain. The First I/O domain is commonly known as Control Domain (CDOM). There are various terms that are used for a Secondary domain, like alternative I/O domain or redundant I/O domain.

    The secondary I/O domain will have been assigned some physical I/O devices, which may be a PCIe bus root complex, a PCI device, or a SR-IOV (Single Root I/O Virtualization) virtual function.

    Within Ops Center when creating a Secondary Domain we also use the terms Physical I/O Domain and Root Domain. The Physical I/O Domain maps PCI device end points, and the Root Domain maps PCIe buses, which also has an option to create SR-IOV functions.

    In this blog we will show you how to create a Root Domain by assigning PCIe buses, so that we have a redundant I/O domain which will enable us to shutdown the CDOM without affecting any of our guests.

    Our host a T5-2 (we’re using the same host as in the previous blogs) has two free PCIe buses that have not been assigned to domains (pci_2 and pci_3). So let’s create a Secondary I/O domain (Root Domain) with these buses and give the domain two whole cores with 4 GB of Memory.

    We’ll start by creating a Logical Domain Profile. This is created from:

    Navigation -> Plan Management -> Profiles and Polices, then click on Logical Domain. On the right under Actions select Create Profiles

    From the Identify Profile screen, give the Profile a name (Secondary-I/O in our case) and select Root Domain in the Sub-type, as shown above. Click Next.

    Step 2: is where we provide a name for our Secondary I/O domain, we called ours secondary and click Next to continue.

    The next screen, show below, we entered the amount of cores and memory.

     Step 4: is where we specify how many PCIe buses we will assign to this secondary domain. In our cases we have specified two for  pci_2 and pci_3.

    The next few steps are optional and are not required for this example.

    Step 7: Is the Summary, if everything looks correct click Finish.

    Note: The Metadata is on the local disk (file://guest). This is fine for the Secondary Domain as it will not be migrated. It’s just the Logical domains and their Guests that will get migrated, therefore making it mandatory to have the metadata on shared storage for these – if you want migration to succeed!

    Now that we have created the Profile, we will create our Secondary domain (Root domain).

    This is done from Navigation -> Plan Management -> Deployment Plans – Create Logical Domain and select the plan we have just created. From the Actions panel on the right select "Apply Deployment Plan".

    From the Select Target Asset pop-up, select the CDOM and move it to the Target list. Select Next.

    Complete Step 1, by specifying the secondary name.

    Step 2, we pick the PCIe Buses that will be assigned to the secondary domain.

    In step 3: we kept the default Virtual Disk Sever (vds) name.

    The next few screens are not required in this example.

    From the Summary screen click Finish. This will create the Secondary Domain.

    Once the Secondary Domain has been created we can check if it’s built as we specified. We can check via the Ops Center BUI and also command line.

    Let’s do both:

    And from the command line