Wednesday Feb 25, 2015

Editing EM12c Jobs in Bulk

I recently received requests for suggestions on how to edit EM12c jobs in bulk. The two examples that were presented to me were:

  1. Change the start time of 70+ jobs
  2. Change the oracle home path in 80+ RMAN script jobs

So how do we do this?

Read More

Friday Jun 27, 2014

Convert Crontab to Enterprise Manager Jobs

Surprisingly, a popular question posted on our internal forum is about the possibility of using the Enterprise Manager (EM) Job System to replace customer’s numerous cron jobs. The answer is obviously YES! I say surprisingly because the EM Job system has been in existence for around 10 years (I believe since EM 10.2.0.1), and my hope was that, by now, customers would have moved to using more enterprise class job schedulers instead of cron. So, here is a quick post on how to get started with this conversion from cron to EM Jobs for some of our new users.

Benefits of EM Job System

 Before we learn about the how, let’s look at the why. The EM job system is:

  • Free - (Yes, I said free) It is included with the base EM at no cost.
  • Flexible - It supports multiple options for scheduling, notification, authentication, etc
  • Infinitely scalable - the job system seamlessly scales to every new Oracle Management Server (OMS). In fact, in case of OMS failures, the job steps are automatically picked up by the next available OMS without affecting the job execution.
  • General purpose - General purpose since it provides numerous out-of-the-box job types like run OS command, start/stop, backup, SQL Script, patch, etc that span multiple target types. As of today, there are over 50 job types available in the product.
  • Enterprise grade - It allows users to automate multiple administrative tasks like backup, patching, cloning, etc across multiple targets. Customers have not only converted their cron jobs to EM, but have also replaced other enterprise tools like Autosys and migrated 1000s of jobs to EM Job System.
  • APIs - Jobs can be scheduled and managed from the UI and using EMCLI (the command line interface).

Now back to our topic.

The Conversion Process

Let’s start with a sample crontab that we want to convert.

Sample Crontab

A cron expression consists of 6 fields, where the first 5 fields represent the schedule, while the last field represents the command or script to run.

 Field Name
Mandatory?  Allowed Values
 Allowed special characters
 Minutes  Yes 0-59  * / , -
 Hours  Yes  0-23  * / , -
 Day of month
Yes  1-31 * / , - ? L W
 Month Yes  1-12 or JAN-DEC
* / , -
 Day of week
Yes  0-6 or SUN-SAT
* / , - ? L #

Cron jobs run on the operating system, often using the native shell or other tools installed on the operating system. The equivalent of this capability in Enterprise Manager is the ‘OS Command’ job type. Here are the steps required to convert the first entry in the crontab to an EM job:

1. Navigate to the Job Activity page
Job activity menu

2. Select the ‘OS Command’ job and click Go
OS Command

A 5-tab wizard will appear. Let’s step through this one by one.

3. Select the first tab called ‘General’. Here provide a meaningful name and description for the job. Since this job will be run on the Host target, keep the target type selection as ‘Host’. Next, select all host targets in EM that you wish to run this script against.

While cron jobs are defined on a per host bases, in EM a job definition can be run and managed across multiple hosts or groups of hosts. This avoids having to maintain the same crontab across multiple hosts.

General

4. Select the ‘Parameters’ tab. Here enter the command or script as specified in the last field of the crontab entry. When constructing the command, you can make use of the various target properties.
Parameters tab

5. Next select ‘Credentials’. Here we provide the credentials required to connect to the host and execute the required commands or scripts. Three options are presented to the user:

  • Preferred – default credential set for the host
  • Named - Credentials are stored within Enterprise Manager as "named" entities. Named credentials can be a username/password, or a public key-private key pair. Here we choose pre-created named credentials
  • New – This allows us to create and use new named credential

Note: If your OS user does not have the required privileges to execute the set command, Named Credentials also support use of sudo, powerbroker, sesu, etc.

Credentials tab

6. Next, we set the schedule and this is where it gets interesting. As discussed before, crontab uses a textual representation for the schedule, while EM Job system has a graphical representation for the schedule.

Our sample schedule in the crontab is ‘00 0 * * Sun’. This translates to a weekly job at 12 midnight on every Sunday. To set this in EM, choose the ‘Repeating’ schedule type. The screenshot below shows all the other selections.
Schedule tab

 The key here is to select the correct ‘Frequency Type’, the rest of the selections are quite obvious. This also lets you choose the desired timezone for the schedule. Your options are to either start the job w.r.t a fixed timezone, or start it in individual target's timezone. The latter is very popular, for example, I want to start a job at 2 AM local time in every region around the world.

Another selection of note is that for ‘Grace Period’. This is an extremely powerful feature, but often not used by many customers. Typically, we expect jobs to be started within a few seconds or minutes (based on the load on the system and number of jobs scheduled) of the start time, but a job might not start on time for many reasons. The most common reasons are the Agent being down or due to a blackout. The grace period controls the latest start time for the job in case the job is delayed, else its is marked as skipped. By default, jobs are scheduled with indefinite grace periods, but I highly recommend setting a value for it. In the sample above, I set a 3 hr limit which may seem large but given the weekly nature of the job seems reasonable. So the job system will wait until 3 am (the job start time is 12 am) to start the job, after which the iteration will be skipped. For repeating schedules, the grace period should always be less than the repeat interval. If the job starts on time, the grace period is ignored.

7. Finally, we navigate to the ‘Access’ tab. This tab has two parts:

  • Privilege assignment to roles and users: this allows you to control job level access for other users
  • Email notifications for the Job owner: this allows you to control the events for which you wish to receive notifications. Note, this only sets notification for the job owner, the other users can subscribe to emails by setting up notification and/or incident rules.

To prevent EM from sending deluge of emails, I recommend the following settings in the notifications region:

  • Match status and severity: Both
  • Select severity of status: Critical
  • Select status: Problems & Action Required

       You can always come back and modify these settings to suit your needs.

Access tab

Not all cron jobs need to be converted to OS command. For example, if you are taking Oracle database backups using cron, then you probably want to use the out-of-the-box  job type for RMAN scripts. Just provide the RMAN script, list of databases to run this against, and the credentials required to connect to the database. Similarly, if you run sqls on numerous databases, you can leverage the SQL Script job type for this purpose. There are over 50 job types available in EM12c, all available for use from the UI and EMCLI.

Finally, the best way to learn more about the EM Job System is to actually play with it. I also recommend blogs from Maaz, Kellyn, and other users on this topic. Good luck!!

References

Maaz Anjum: http://maazanjum.com/2013/12/30/create-a-simple-job-for-a-host-target-in-em12c/
Kellyn Pot'vin: http://dbakevlar.com/

-- Adeesh Fulay (@adeeshf)

Tuesday Jun 17, 2014

EM12c Release 4: Job System Easter Eggs - Part 2

This is part 2 of my two part blog post on Job System Easter Eggs (hidden features). The two features being covered are:

  1. New Job progress tracking UI
  2. Import/Export of job definitions

In the previous post, I talked about the new UI while in this post I will cover import and export for job definitions. 


2.  Import/Export of job definitions

The ability to export and import job definitions across EM environments has been a popular enhancement request for quite some time. There are 3 primary drivers:

  • Test to Production: Moving active and library job definitions from test site to production
  • Migration: Move job definitions across EM versions. This is often used as an alternative to upgrade.
  • As Failsafe: For that extra protection, in addition to repository backups. This can be useful for selective restoration of job definition.

In the first iteration, we are exposing these functions as emcli verbs.  Since these emcli verbs are not formally supported in the release, no documentation is included in the EMCLI Reference Guide. So to learn more about them, we will have to look at emcli help. The two verbs are called export_jobs, and import_jobs.

>> ./emcli help | grep _jobs


Some salient points about the import & export verbs are.

Export:

  • Rich filter criteria to form the list of jobs to export
  • Active and Library jobs can be exported
  • Preview mode, to view results before exporting them
  • Job definitions are exported as zip files for ease of transfer
  • Contextual information like targets, credentials, access settings, etc is exported, but not imported. In future, we may be able to support import for this information as well.
  • System jobs and nested jobs are not exported

Import:

  • Special Preview mode to view contents of the exported file, and to determine possible conflicts during import
  • Rich filter criteria to selectively import job definitions from the exported file
  • Job definitions are imported to the library ONLY. This is true even if an active job was exported
  • Two failure modes - skip on error, and rollback on error
  • Import only works on same or higher version of EM12c. The patch set number matters.


Export Job Definitions

Let's walk through an example.

1. We start with export_jobs. The help for this verb will show you all the available options. Most of the filters are self explanatory, so i will skip the explanation. The most important option of all is the -preview flag. This when used in conjunction with other filters, will show results without exporting the job definitions.

>> ./emcli help export_jobs

emcli export_jobs
   [-name="job name1;job name2;..."]
   [-type="job type1;job type2;..."]
   [-targets="tname1:ttype1;tname2:ttype2;..."]
   [-owner="owner1;owner2;..."]
   [-preview]
  -export_file=<Zip filename that will be created>"
   -libraryjobs

2.  Now lets play with this verb. If we invoke the verb with just the -preview flag, it will list all job definitions that can be exported both that are active, and from the library. Note: system jobs and nested jobs are skipped from this output.

>> ./emcli export_jobs -preview

Not all job types are exportable. To determine the list of job types supported via the import and export verbs, use the get_job_types verb.

>>  ./emcli get_job_types

Currently, there are over 50 job types supported, and this list will continue to grow with every release.

3. From the list above, I am primarily interested in jobs that are created by the user AFULAY. So I apply the -owner filter.

>> ./emcli export_jobs -owner=AFULAY -preview


In this output i see 2 jobs, 'Library Job' which is a simple OS command job stored in the job library, while the 'Test Job' is an active OS command job scheduled against a bunch of targets.

Note: if multiple options are specified, like -name and -owner, then these are ANDed together.

4. Since i am the lazy kind, i would rather export every thing and then later decide what i wish to import. So here it goes. The -export_file option takes the location and file name for the export file. Note the actual output file is an xml that contains job metadata, but the export file is always archived and stored in a zip format. At this time, I am sure most would instinctively unzip the file and start perusing through its contents, but doing so would be analogous to removing the warranty void sticker of your new TV or Blue Ray player. Basically, attempts to manually modify the contents of the exported zip file is highly discouraged.

>> ./emcli export_jobs -export_file=/tmp/afulay_alljobs.zip


Note how the status column reports success or failure for each job being exported. With the file exported, we now move on to the import verb.

.

Import Job Definitions

In the previous section, we exported a file with all job definitions in it. Now lets say we share this file with bunch of other admins and ask them to import whatever job definitions that make sense or are relevant to their environment.

1. To understand the capabilities of the import verb, we take a look at the help content.

>> ./emcli help import_jobs

emcli import_jobs
   [-name="job name1;job name2;..."]
   [-type="job type1;job type2;..."]
   [-targets="tname1:ttype1;tname2:ttype2;..."]
   [-owner="owner1;owner2;..."]
   [-preview]
   [-force]
   [-stoponerror]
   -import_file=<Zip filename that contains job definitions>"

 Most of the options are quite similar to export_jobs barring a few. The -force flag allows the admin to update an existing job definition. Typically, you will run into these situations when a conflicting job is found in the new environment, and you want to either update its definition with the new version in the import file or overwrite the localized changes. The -stoponerror flag, when specified, will stop the import process on first encountered failure, and then rollback all jobs imported in the session. We will likely change this label to rollbackonerror to correctly represent its behavior. The default behavior is to skip failed jobs and continue importing others.

2. Before our admins import job definitions, they first need to view the contents of the exported file. This again can be done using the -preview option.

>> ./emcli import_jobs -preview -import_file=/tmp/afulay_alljobs.zip


The -preview option in the import verb is very special. It not only just lists the contents of the exported file, but also connects to the new EM environment and looks for potential conflicts during import. So this is a deep validation test. As seen in the above screenshot, there are two sections in the output, first is just a listing of all job definitions from the import file, while the second is a list of all conflicts. Note: for demo sake, i am exporting and importing to the same EM site, and hence every job shows up as a conflict. To address this issue, i will eventually delete the 'Library Job' from my job library, and import it from the import file.

Disclaimer:

In the interest of full disclosure, i should mention that there are few known bugs for the import verb, hence the rationale for not releasing these verbs formally with EM12c R4. Some bugs i ran into when writing this blog were:

  • you cannot export an active job, delete it, and import it back to the same EM environment, this currently is only possible with library jobs.  This is an obscure case though.
  • The -force flag is a little flaky, so sometimes it wouldn't force import even if you want it to
  • The -owner flag does not work on the import file, it instead will throw an exception

That said, when the job does get imported, it does so properly, so there is never any risk of metadata corruption.


3. If i try to import the 'Library Job', the verb will fail and give me an error message.

>> ./emcli import_jobs -name='LIBRARY JOB' -import_file=/tmp/afulay_alljobs.zip


The Status column reports Error, while the Notes column gives the reason as 'job already exists'.

4. Now lets delete the library job and try to import it.

>> ./emcli delete_library_job -name='LIBRARY JOB'

>> ./emcli import_jobs -name='LIBRARY JOB' -import_file=/tmp/afulay_alljobs.zip


Success!!We were able to delete the library job, and import it back from the import file.


In summary, there two very useful enhancements made in EM12c R4. Unfortunately, due to time constraints and our inability to meet the set quality standards, we decided to ship these features in disabled state. This ensures that production sites and users are not impacted, while still giving the few brave souls the opportunity to test these features. In my assessment, i have found the new UI to be fairly robust as i have been using this exclusively for a while. On the other hand, there are few known bugs with the import and export emcli verbs, so use these with caution.

-- Adeesh Fulay (@adeeshf)  

Monday Jun 16, 2014

EM12c Release 4: Job System Easter Eggs - Part 1

So you just installed a new EM12c R4 environment or upgraded your existing EM environment to EM12c R4. Post upgrade you go to the Job System activity page (via Enterprise->Job->Activity menu) and view the progress details  of a job. Well nothing seems to have changed, its the same UI, the same multi-page drill down to view step output, same no. of clicks, etc. Wrong! In this two part blog post, i talk about two Job System Easter Eggs (hidden features) that most of you will find interesting. These are:

  1. New Job progress tracking UI
  2. Import/Export of job definitions

So before i go any further, let me address the issue of why are these features hidden? As we were building these features, we realized that we would not be ready to ship the desired quality of code by the set dates. Hence, instead of removing the code, it was decided to ship it in a disabled state so as not to impact customers, but still allowing a brave few to experiment with it and provide valuable feedback.

1.  New Job Progress Tracking UI

The job system UI hasn't changed much since its introduction almost 10 years ago. It is a daunting task to update all the job system related UIs in a single release, and hence we decided to take a piecemeal approach instead. In the first installment, we have revamped the job progress tracking page.

Old Job Progress Tracking UI

The current UI, as shown above, while being very functional, is also very laborious. Multiple clicks and drill downs are required to view the step output for a particular target. Also, any click leads to complete page refresh, which leads to wastage of time and resources. The new UI tries to address all these concerns. It is a single page UI, which means no matter where you click, you never leave the page and thus never lose context of the target or step you where in. It also significantly reduces the no. of clicks required to complete the same task as in the current UI. So lets take a look at this new UI.

 First, as i mentioned earlier, you need to enable this UI. To do this, you need to run the following emctl command from any of the OMS:

./emctl set property -name oracle.sysman.core.jobs.ui.useAdfExecutionUi -value true

 This command will prompt for the sysman password, and then will enable the new UI.

NOTE: This command does not require a restart of the OMS. Once run, the new UI will be enabled for all user across all OMSes.

EMCTL Output

Now revisit the job progress tracking page from before. You will be directed to the new UI.

New Job Progress Tracking UI

There are in all 6 key regions on this new single page job progress tracking UI. Starting from top left, these are:

  1. Job Run Actions - These are actions that can be performed on the job run like suspend resume, retry, stop, edit, etc.
  2. Executions - This region displays all the executions in the job run. An execution, in most cases, represents a single target and hence runs independently from other executions. This region thus shows the progress and status of all executions in a single view. The best part of this region is the column titled 'Execution Time'. The cigar chart in this column represents two things - one, the duration of the execution, and two, the difference in start times. The visual representation helps in identifying runaway executions, or just compare execution times across different targets. The Actions menu allows various options like start, stop, debug, delete, etc.
  3. Execution Summary - Clicking on an execution in the above region, paints the area on the right. This specific region shows execution summary with information like status, start & end date, execution id, command, etc
  4. Execution Steps - This region lists the steps that make up the execution.
  5. Step Output - Clicking on a step from the above region, paints this region. This shows the details of the step. This includes the step output and the ability to download it to a text file.
  6. Page Options - We imagine that learning any new UI takes time, and hence this final region provides the option to switch between the new and the classic view. Additionally, this also allows you to set the auto refresh rate for the page.

Essentially, considering that jobs have two levels - executions and steps, we have experimented with a multi-master style layout. EM has never used such a layout and hence there were concerns raised when we chose to do so.

Master 1 (region 2) -> Detail 1 (regions 3, 4, & 5)

Master 2 (region 4) -> Detail 2 (region 5)


In summary, with this new UI, we have been able to significantly reduce the no. of clicks required to track job progress and drill into details. We have also been able to show all relevant information in a single page, thus avoiding unnecessary page redirection and reloads. I would love to hear from you if this experiment has paid off and if you find this new UI useful.

In the next part of this blog i talk about the new emcli verbs to import and export job definitions across EM environments. This has been a long standing enhancement request, and we are quite excited about our efforts.

-- Adeesh Fulay (@adeeshf)  

About

Latest information and perspectives on Oracle Enterprise Manager.

Related Blogs




Search

Archives
« September 2015
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today