By Scott Mcneil-Oracle on Aug 25, 2014
A few weeks ago, someone asked on the OTN forums how to alert on some of the JVM metrics such as ‘JVM Threads – Threads Started (since startup)’ using Enterprise Manager 12c (EM). This is one of those few metrics that EM collects, but does not allow custom thresholds. Let’s take a look at the metrics that EM collects on the WebLogic Server target.
Enterprise Manager 18.104.22.168 has now been released for a few weeks, as well as the 22.214.171.124 OMS Bundle patches (also known as System patches). If you plan to apply these bundle patches to your 126.96.36.199 OMS, and you are concerned about the downtime, then, you can reduce the downtime by referring to this whitepaper that contains patching instructions to reduce downtime.
Now that Enterprise Manager 12cR4 has been out for a little while, more people are getting around to upgrading their agents. Since the monthly Patch bundles were released we already have a few Agent side patches that we want to apply to our newly upgraded agents. I’ve written about simplifying your agent patching before, but this feature still seems to fly under the radar. It’s days like these that I miss running a massive Enterprise Manager with thousands of databases, because this is one of the things that would have made me dance in my cubicle.Let's say, you have 100 188.8.131.52.0 agents (50 with Database plug-in, 50 with Middleware plug-in). In my previous blog on EM patches, I explained the different types of patches available for EM, so I’m not going to go into detail here. What I'm going to illustrate is how we can upgrade those 100 agents, and patch them with the following patches in one step (current as of today):
With the release of Enterprise Manager Ops Center 12.2.1, it is time to go through the upgrade cycle. I thought I would share the pre-upgrade checks I go through when I upgrade to a new Ops Center build. As part of the development team, I get involved in pre-release Quality Assurance testing, which means I end up doing hundreds of upgrades as part of the testing process.
Update releases come out regularly and contain enhancements and bug fixes. As with any other application in your environment, you should upgrade Ops Center to the current release/update in a timely and controlled manner. For those of you who are long time sys-admins, there is no rocket science here. It is the same sort of planning you would do for any other Enterprise level application.
In my test environments, I have my Enterprise Controller (EC) and Proxy Controllers (PC) inside Solaris Zones (Solaris 11), so I have a couple of extra checks I do, but the process as a whole is still valid if your EC/PC are on their own separate hardware.
Yes, those release notes/README files are important and you should spend the time reading them. They will contain the latest information about the update and any known issues and workarounds.
Confirm that there is enough disk space to unpack and install the upgrade. How much is enough space is the ultimate question. It will vary with each different upgrade and will depend on how you have configured your underlying filesystems and your actual environment. Here are some guidelines. Please note that the numbers I quote tend to be a little generous as it is always better to have more free space than not enough.
root@ec:/ec_backup# du -hs * 1.3G sat-backup-pre-12.2.1-upgrade.20140702root@ec:/ec_backup# du -hs /var/opt/sun/xvm/oracle/oradata/OCDB 14G /var/opt/sun/xvm/oracle/oradata/OCDB root@ec:/ec_backup#
Although more space is actually used during the backup before it is packed up, I would allow for about 4 GB of space.
Before commencing any upgrade, you should make sure you can roll back if something goes horribly wrong. Years of history in administration and support have made me a paranoid person. I believe you can never have too many backups, so I do the following:
Of course, copy the generated backup file to somewhere safe on another system.root@ec:/# /opt/SUNWxvmoc/bin/ecadm backup -d pre-12.2.1-upgrade -o /ec_backup/sat-backup-pre-12.2.1-upgrade.20140702 ecadm: using logFile = /var/opt/sun/xvm/logs/sat-backup-2014-07-02-11:52:16.log ecadm: *** PreBackup Phase ecadm: *** Backup Phase ecadm: *** PostBackup Phase ecadm: *** Backup complete ecadm: *** Output in /ec_backup/sat-backup-pre-12.2.1-upgrade.20140702 ecadm: *** Log in /var/opt/sun/xvm/logs/sat-backup-2014-07-02-11:52:16.log root@ec:/#
### Take a zfs snapshot ### root@ec:/# zfs list NAME USED AVAIL REFER MOUNTPOINT rpool 156G 41.1G 31K /rpool rpool/ROOT 134G 41.1G 31K legacy rpool/ROOT/solaris 134G 41.1G 24.6G / rpool/ROOT/solaris-backup-1 174K 41.1G 1.37G / rpool/ROOT/solaris-backup-1/var 110K 41.1G 27.9G /var rpool/ROOT/solaris-backup-2 296K 41.1G 24.2G / rpool/ROOT/solaris-backup-2/var 232K 41.1G 48.4G /var rpool/ROOT/solaris/var 109G 41.1G 77.2G /var rpool/VARSHARE 88K 41.1G 66.5K /var/share rpool/ec_backup 1.29G 41.1G 1.29G /ec_backup rpool/export 161K 41.1G 32K /export rpool/export/home 111K 41.1G 32K /export/home rpool/export/home/ocadmin 61K 41.1G 40.5K /export/home/ocadmin rpool/oracle 20.7G 41.1G 20.7G /var/opt/sun/xvm/oracle root@ec:/# root@ec:/# zfs snapshot -r rpool@pre-OC-12.2.1-install.20140702 root@ec:/#
It is good practice to clear/enable/disable any broken SMF services, but there are a few key ones to check.
Make sure all the Ops Center services that should be running are running and the ones that should not are not. A classic example here is when you have an EC running without a collocated PC. The PC shows as disabled, but still shows in a "svcs -xv" output.
root@ec:/var/tmp/downloads# svcs -xvsvc:/application/management/common-agent-container-1:scn-proxy (Cacao, a common Java container for JDMK/JMX based management solution) State: disabled since June 12, 2014 08:07:08 AM ESTReason: Disabled by an administrator. See: http://support.oracle.com/msg/SMF-8000-05 See: man -M /usr/share/man -s 1M cacaoadm See: man -M /usr/share/man -s 5 cacao Impact: 1 dependent service is not running: svc:/application/scn/proxy-available:default root@ec:/var/tmp/downloads#
In this case, our EC did not have a collocated PC, so we should ensure that these services are really disabled and don't try to start-up during the upgrade process.
root@ec:/var/tmp/downloads# svcadm disable svc:/application/scn/proxy-available:defaultroot@ec:/var/tmp/downloads# svcadm disable svc:/application/management/common-agent-container-1:scn-proxy
root@t4-1-syd04-b:~# svcs svc:/application/pkg/zones-proxyd:default STATE STIME FMRI online Jul_02 svc:/application/pkg/zones-proxyd:default root@t4-1-syd04-b:~#
root@ec:~# svcs svc:/application/pkg/zones-proxy-client:default STATE STIME FMRI online 8:54:47 svc:/application/pkg/zones-proxy-client:default root@ec:~#
root@ec:/var/tmp/downloads# svcs -xvroot@ec:/var/tmp/downloads#
To be able to do a successful upgrade, you need the pkg publisher for a system to be working. In a zones environment, that means the publishers in the GZ and all the NGZ should be working. Publishers that don't resolve when a package links into a zone will cause the whole upgrade to stop.
So here are a couple things to look for when you are using an EC in a zone.
# pkg unset-publisher Publisher-Name
The aim here is to clear all the local publishers in the zone and just use the proxied publishers in the GZ.
root@t4-1-syd04-b:~# pkg publisher
PUBLISHER TYPE STATUS P LOCATION
solaris origin online F https://oracle-oem-oc-mgmt-pc217:8002/IPS/
cacao origin online F https://oracle-oem-oc-mgmt-pc217:8002/IPS/
mp-re (non-sticky) origin online F https://oracle-oem-oc-mgmt-pc217:8002/IPS/
opscenter origin online F https://oracle-oem-oc-mgmt-pc217:8002/IPS/
root@t4-1-syd04-b:~# pkg unset-publisher opscenter
root@t4-1-syd04-b:~# pkg unset-publisher mp-re
root@t4-1-syd04-b:~# pkg unset-publisher cacao
root@t4-1-syd04-b:~# pkg set-publisher -G '*' -g http://ec:11000/ solaris
root@t4-1-syd04-b:~# pkg publisher
PUBLISHER TYPE STATUS P LOCATION
solaris origin online F http://ec:11000/
# /var/opt/sun/xvm/utils/install_ips_ac.sh -P PC_IP_AddressUse 127.0.0.1 as the IP address for the EC/PC when it is pointing too itself
Run the OCDoctor troubleshoot script over your EC and PC's before an upgrade. It is a good sanity check to look for and fix underlying problems before you start the upgrade process. If you are in connected mode, your EC should already have the latest version of OCDoctor downloaded. Otherwise, you can update it by running "OCDoctor.sh --update" or downloading from https://java.net/projects/oc-doctor/downloads/download/OCDoctor-4.36.zip
Note: The error "'root' should not be a role" can be safely ignored as it was only required for earlier versions of Ops Center.
root@ec:/var/tmp/downloads# /var/opt/sun/xvm/OCDoctor/OCDoctor.sh -t Ops Center Doctor 4.34 [OC 184.108.40.20663,SunOS11] [Read only] [02-Jul-2014 11:25AM EST] ======================== Checking Enterprise Controller...============================== OK: Total number of OSes: 12 Total LDOMs:7 Total Zones: ERROR: User 'root' should not be a role. You should convert it to a normal user before the installation. This can be done by running: # rolemod -K type=normal root OK: Files in /var/opt/sun/xvm/images/agent/ have the right permissions OK: Files in /var/opt/sun/xvm/osp/web/pub/pkgs/ have the right permissions OK: both pvalue and pdefault in systemproperty are equal to false (at id 114) OK: Found only 285 OCDB*.aud files in oracle/admin/OCDB/adump folder OK: Found no ocdb*.aud files in oracle/admin/OCDB/adump folder OK: No auth.cgi was found in cgi-bin OK: User 'oracleoc' home folder points to the right location OK: User 'allstart' home folder points to the right location OK: Apache logs are smaller than 2 GB OK: n1gc folder has the right permissions OK: All agent packages are installed properly OK: All Enterprise Controller packages are installed properly OK: Enterprise Controller status is online OK: the version is the latest one (220.127.116.1163) OK: satadm timeouts were increased OK: tar command was properly adjusted in satadm OK: stclient command works properly OK: Colocated proxy status is 'disabled' OK: Local Database used space is 19%, 6G out of 32G (local DB, using 1 files) OK: Debug is disabled in .uce.rc OK: Debug is disabled for cacao instance oem-ec OK: no 'conn_properties_file_name' value in .uce.rc OK: 30G available in / OK: 30G available in /var OK: 30G available in /var/tmp OK: 30G available in /var/opt/sun/xvm OK: 30G available in /opt OK: DNS does not unexpectedly resolve hostname '_default_' OK: Found the server .uce.rc at /var/opt/sun/xvm/uce/opt/server/cgi-bin/.uce.rc OK: Server .uce.rc has the correct file permissions OK: Server .uce.rc has the correct ownership OK: Connectivity to the KB public servers works properly (using download_large.cgi) OK: Grouplock file doesn't exist OK: package email@example.com is not installed OK: package driver/x11/xsvc is not installed OK: Cacao facet is set to False OK: All Solaris 11 agent bundles in /var/opt/sun/xvm/images/agent are imported properly to the repository OK: Disconnected mode is not configured OK: Locales are OK ("en_US.UTF-8") OK: No need to check for Solaris 11 agent bundle issue as this EC is newer than Update 1 OK: No partially installed packages OK: UCE 'private' folder exists OK: No http_proxy is set in the user profile files OK: 'public' folder has the right ownership OK: 'public' folder is writable for uce-sds OK: 'private' folder has the right ownership OK: 'private' folder is writable for uce-sds OK: '/var/tmp' folder is writable for uce-sds OK: No old jobs rerun (CR 6990675) OK: No need to adjust SEQ_COUNT (MAXID:2986 SEQCOUNT:2986) OK: no row with ssh.tunnel.info found in DB table HD_RESOURCE_PARAMETER NOTICE: Can't perform cryptoadm test inside a zone. Run --troubelshoot from the global zone as well to test the crypto services. OK: System time is not in the past OK: User uce-sds is part of all the proper groups OK: oracleoc user ulimit -Sn is 1024 OK: oracleoc user ulimit -Hn is 65536 OK: FC Libraries do not contain duplicate LUNs OK: 'update-saved-state' folder exists and has the right permissions OK: verify-db does not return 'Invalid pad value' message OK: No credential issues found =========== Proxy controller is installed but not configured, skipping ================== =========== Agent controller is installed but not configured, skipping ================== root@ec:/var/tmp/downloads#
Choose whichever upgrade method you like. Both the BUI and CLI methods will give you the same end result. The Ops Center upgrade is not a difficult upgrade and following some simple pre-work checks will maximize your chance of a straightforward and successful upgrade.
The core Enterprise Manager system is typically patched with the quarterly PSU patches (released Jan, Apr, July, Oct) or a one-off when directed by support for a critical issue. PSU patches will be cumulative, so you need not apply each of them, just apply the latest. The OMSes must be shutdown during patching, however some patches are being released with rolling patch instructions for multi-OMS systems. These patches must be applied at the host level, and cannot be automated via EM. ALWAYS read the readme, yes every time. The patching steps can change from patch to patch so it's critical to read the readme. OPatch or OPatchauto will be used to apply these patches. Did I mention to read the readme for every patch? It's also important to note that there may be additional steps when patching in a multi-OMS or standby environment, so read the output of OPatchauto carefully.
Always download the latest OPatch release for the appropriate version. If you read the readme, you already know this! Download patch 6880880 for 11.1 (the OPatch version used by EM) and unzip into the $ORACLE_HOME. Most errors in patching are related to not updating OPatch.
For more information on PSU Patches and patching EM:
Oracle Enterprise Manager Cloud Control Administrators Guide - Chapter 16 Patching Oracle Management Server and the Repository
EM 12c Cloud Control: List of Available Patch Set Updates PSU (Doc ID 1605609.1)
How to Determine the List of Patch Set Update(PSU) Applied to the Enterprise Manager OMS and Agent Oracle Homes? (Doc ID 1358092.1)
Each plug-in has binaries that will require patches as well. Same downtime requirements apply for plug-in patches as the quarterly PSUs. Starting in 18.104.22.168, the plug-in patches are being released as a monthly bundle. This means that if you have 6 plug-ins, you may have 6 OMS side patches to apply - 1 for each plug-in. Bundles are not always released for every plug-in every month. They are cumulative, so pick the latest.
Starting with 22.214.171.124, the individual OMS-side plug-in bundles are being grouped into a System Patch each month. So for example, in June 2014 the System patch includes MOS, Cloud, DB, FA, FMW, SMF, and Siebel plug-ins. Non-required patches will be skipped.
For more information on the EM Patch Bundles and Patching EM:
Enterprise Manager 126.96.36.199.0 (PS3) Master Bundle Patch List (Doc ID 1900943.1)
Enterprise Manager 188.8.131.52 Bundle Patch Master Note (Doc ID 1572022.1)
Agent patches are applied to each agent. They can be applied via EM using the MOS patch plans, which makes it a lot easier when you have 100s or 1000s of Agents to patch! The Patch Plans will start a blackout, validate prerequisites, check for conflicts, and update OPatch for you. If you don't use the Patch Plan you can patch manually with OPatch, don't forget to read the readme! The Agent must be shutdown during the patch application. There are 4 main types of Agent patches you will see:
You can apply the latest Agent bundle, JDBC patch and the plug-in bundles in one patch plan. If there's a conflict, you'll be notified. If the Agents you've selected don't have specified plug-ins, you'll also receive notice during the analyze step. As of now, for my 184.108.40.206 agents, I would apply the 220.127.116.11.1 patch (18873338) and the two available plug-in agent patches DB monitoring (19002534) and FMW monitoring (18953219) and the latest JDBC patches (18502187,18721761) all in one patch plan.
I discovered a new feature in 18.104.22.168 while testing this. Normally you had to have Normal Oracle Home preferred credentials set for all Agent targets to patch, or select Override and specify the Normal Oracle Home credentials. In 22.214.171.124, the Agent uses it's internal credentials to Patch itself, so setting preferred credentials or specifying at run-time is not required. The user patching would require the Manage Target Patch and Patch Plan privileges.
The OMS and Agent are the key components, and my main focus here. However it's important to keep the infrastructure stack up to date as well. This includes the Oracle Fusion Middleware and Oracle Database that are used for EM. The recommendation is to follow the best practices for each of these components, and regularly update with the PSU patches available. The following reference notes will help in identifying the current PSU patches. The WebLogic Server version used by EM 12c is 10.3.6.
Hopefully this will help you understand the various types of components involved with keeping EM up to date. Obviously, you may not want to patch each month and maybe not every quarter, but the patches are available to keep the software up to date and make things easier to apply in bundles. You'll want to setup a plan for planned software maintenance in your environment. There's a whitepaper Oracle Enterprise Manager Software Planned Maintenance that will help guide you through the best practices.
When implementing database as a service and/or snap clone, a common request was for a way to hide the other service types like IaaS, MWaaS, etc from the self service portal for the end users. Before EM12c R4, there was no way to restrict the portal view. Essentially, any user with the EM_SSA_USER role would be directed to the self service portal and would then be able to see all service types supported by EM12c.
Of course, you could always set Database as your default self service portal from the 'My Preferences' pop up, but this only helps with their post-login experience. The end user still gets to see all the options as shown in screen above.
In EM12c R4, a new out of the box role called EM_SSA_USER_BASE has been introduced. This role, by default, does not give access to any portal, that is an explicit selection. Here is how you use this role:
1. Create a custom role and add the EM_SSA_USER_BASE role to it.
2. Now in the Resource Privileges step, select the Resource Type 'Cloud Self Service Portal for Database', and edit it
3. Check the 'Access the Cloud Self Service Portal for Database.' privilege. Finish the rest of the wizard.
Now, when a user with this custom role accesses the self service portal, they can only do so for databases and nothing else.
While the EM_SSA_USER role will continue to work, we recommend you start using the new EM_SSA_USER_BASE role. For more details on DBaaS or Snap Clone roles, refer to the cloud admin guide chapter on roles and users.
-- Adeesh Fulay (@AdeeshF)
With the latest release of Enterprise Manager 12c, Release 4 (126.96.36.199) the EM development team has added new functionality to assist the EM Administrator to monitor the health of the EM infrastructure. Taking feedback delivered from customers directly and through customer advisory boards some nice enhancements have been made to the “Manage Cloud Control” sections of the UI, commonly known in the EM community as “the MTM pages” (MTM stands for Monitor the Monitor). This part of the EM Cloud Control UI is viewed by many as the mission control for EM Administrators.
In this post we’ll highlight some of the new information that’s on display in these redesigned pages and explain how the information they present can help EM administrators identify potential bottlenecks or issues with the EM infrastructure. The first page we’ll take a look at is the newly designed Repository information page. You can get to this from the main Setup menu, through Manage Cloud Control, then Repository.
Most user accounts these days have a password profile on them that automatically expires the password after a set number of days. Depending on your company’s security requirements, this may be as little as 30 days or as long as 365 days, although typically it falls between 60-90 days. For a normal user, this can cause a small interruption in your day as you have to go get your password reset by an admin. When this happens to privileged accounts, such as the DBSNMP account that is responsible for monitoring database availability, it can cause bigger problems.
In Oracle Enterprise Manager 12c you may notice the error message “ORA-28002: the password will expire within 5 days” when you connect to a target, or worse you may get “ORA-28001: the password has expired". If you wait too long, your monitoring will fail because the password is locked out. Wouldn’t it be nice if we could get an alert 10 days before our DBSNMP password expired? Thanks to Oracle Enterprise Manager12c Metric Extensions (ME), you can! See the Oracle Enterprise Manager Cloud Control Administrator’s Guide for more information on Metric Extensions.[Read More]
Contributing Author: Eunjoo Lee, Principal Product Manager, Oracle
The Oracle Enterprise Manager Applications Management team is pleased to announce the release of their latest whitepaper (available here) and associated screenwatch (available here) on configuration management for Siebel.
The task of managing configuration parameters in a dynamic, multi-target environment can be extremely challenging. A large, production scale environment can have dozens of unique targets, and each target can have several hundred different configuration parameters.
Oracle Enterprise Manager 12c, with the Siebel Plug-in, provides advanced, out-of-the-box tools to manage the complexity of Siebel configuration management. Oracle Enterprise Manager has advanced configuration capture, change detection, and comparison reporting capabilities that can dramatically reduce the amount of time that System Administrators spend on configuration management.
The purpose of the whitepaper and screenwatch is to provide customers with details on Oracle’s Best Practices for managing Siebel Configurations. It addresses several key areas including;
The benefits of using configuration templates to manage Siebel applications are substantial, and include:
We hope this white paper and screenwatch help to provide you with a good starting point to manage your Siebel Application configurations more efficiently and pro-actively.
Do you still maintain a spreadsheet with Database or Server contact or business unit ownership? In Oracle Enterprise Manager 12c (EM) Target Properties allow you to store descriptive target information, such as Contact or Location, which can then be used in dynamic/administration group definition, reports, incident rules and notifications. This blog will show you how you can better leverage the features of EM to store your configuration data and utilize it to the fullest extent.
The out-of-the-box target properties include:
Here is a great article from resident Oracle ACE, Arup Nanda, who details insight into predicting the impact of consolidating separate database workloads into one. The article outlines a typical consolidation scenario and explains how Oracle Real Application Testing's Consolidated Database Replay capabilities can help measure the impact of the workload consolidation. A must read for those considering a consolidation project in the near future. Read the article.
10 — 7:00 a.m. PT | 10:00 a.m. ET | 4:00 p.m. CET
Since the demands from the business for IT services is non-stop, creating copies of production databases in order to develop, test and deploy new applications can be labor intensive and time consuming. Users may also need to preserve private copies of the database, so that they can go back to a point prior to when a change was made in order to diagnose potential issues. Using Snap Clone, users can create multiple snapshots of the database and “time travel” across these snapshots to access data from any point in time.
Join us for an in-depth technical webcast and learn how Oracle Cloud Management Pack for Oracle Database's capability called Snap Clone, can fundamentally improve the efficiency and agility of administrators and QA Engineers while saving CAPEX on storage. Benefits include:
Latest information and perspectives on Oracle Enterprise Manager.