Thursday Feb 13, 2014

Steps to Fast Track your Database Cloud implementation on Exadata

Oracle Exadata Database Machine is the ideal consolidation platform for Enterprise Database Cloud and Oracle Enterprise Manager provides the most optimized and comprehensive solution to rapidly setup, manage and deliver Enterprise Clouds. Clearly, very significant innovations have been delivered via Exadata X4, Enterprise Manager 12c and Database 12c in Cloud Computing space and customers can start realizing benefits from this combination of most powerful and unique enterprise database cloud solution in industry.

As per OracleVoice blog on Forbes.com:  "Why Database As A Service (DBaaS) Will Be The Breakaway Technology of 2014":

"Database as a Service (DBaaS) is arguably the next big thing in IT. Indeed, the market analysis firm 451 Research projects an astounding 86% cumulative annual growth rate, with annual revenues from DBaaS providers rising from $150 million in 2012 to $1.8 billion by 2016."

In this blog post, I will walk through the steps aiming to simplify DBaaS Setup on Exadata and also describe automation kits available to achieve the following rapidly -

  • Setup Monitoring and Management of Exadata Database Machine platform in EM 12c
  • Setup and Deliver DBaaS on Exadata using EM 12c
  • Manage and Optimize Exadata and EM 12c powered DBaaS cloud platform on an ongoing basis



There are 2 separate automation kits that are provided with EM 12c, first kit is for enabling rapid monitoring and management setup of Exadata stack in EM 12c and second kit is for rapid setup of DBaaS -

1) Deploy EM 12c site or use existing site - If you do not have existing EM 12c R3 setup, you can use EM Automation Kit for Exadata for installing EM 12c R3 Plug-in update 1. This kit is available via patch 17036016 on My Oracle Support(MOS) and can be used to deploy EM 12c latest release. Refer to Readme of patch and MOS note "Obtaining the Oracle Enterprise Manager Setup Automation kit for Exadata (Doc ID 1440951.1)" for additional details. Please note that this will setup EM12c Oracle Management Service along with Management Repository. It can be deployed on a single m/c or OMS and OMR can be setup on different machines.

2) Deploy EM 12cR3 agents and required plug-ins on Exadata Machine - Agent kit is also part of the same EM Automation Kit for Exadata and can be used for deploying agents and plug-ins on Exadata stack. Refer to MOS note "Obtaining the Oracle Enterprise Manager Setup Automation kit for Exadata (Doc ID 1440951.1)" for additional details. Best practice is to use most recent version of Agent kit and also deploy latest plug-ins. Patch details for respective platform are described in the MOS note.

Agent kit script will require Java 1.6.0_43 or greater version on database node where this script is being run. Agent kit script will need to be run as root OS user on Exadata db node, however JAVA_HOME and PATH with JAVA_HOME/bin should be set up as agent OS owner, so these OS env variables need to setup in profile of agent OS owner.

Agent Automation kit helps with achieving following -

  • EMCLI setup on Exadata Server
  • EM 12c R3 site compatibility checks
  • Setup and remove SSH between Exadata nodes to test SSH setup
  • Deploy EM 12c Agent and required Plugins on all DB Nodes of Exadata Machine
  • Confirm Exachk tool availability and run Exachk tool
  • Run Exadata Discovery Prerequisites
  • Discover Targets Cluster, Grid Infrastructure, RAC database and listener targets

Note - In case of Exadata X4, ensure you have the EM 12cR3 latest Bundle patch(released in January 2014). Refer to following MOS notes -
Enterprise Manager 12.1.0.3 Bundle Patch Master Note (Doc ID 1572022.1)
Enterprise Manager for Exadata Plug-in 12cR3 Bundle Patch Bug List (Doc ID 1613177.1)

3) Discover Grid Infrastructure and RAC targets – Above setup script will discover Targets Cluster, Grid Infrastructure, RAC database and listener targets. Discover Grid Infrastructure, ASM and RAC targets manually if required.

4) Please note that this setup script will not discover Oracle Exadata Database machine target in EM 12c. You need to discover the machine using following steps

  • From the Setup menu, select Add Targets, then select “Add targets Manually”.
  • In the “Add Targets Manually” page, select 'Add Targets Using Guided Process (Also Adds Related Targets)' and Target Type as Oracle Exadata Database Machine.
  • Click Add Using Guided Discovery and follow the wizard.


5) Setup Database Cloud Using Rapid Start Kit - Once you have setup Exadata management in EM 12c, next step is to setup database cloud. Refer to Rapid Start Kit for setting up cloud for both DBaaS and Pluggable DBaaS/PDBaaS. This kit will help achieve the following -

  • Create Cloud Admin, SSA Admin and SSA User custom roles
  • Create Cloud Admin, SSA Admin and SSA Users
  • Grant Quota to SSA User custom roles
  • Setup Zones with Placement Policy Constraints
  • Setup Pools with Placement Constraints
  • Setup Service Template/Catalog and grant it SSA User custom roles.

Here are brief steps for setting up Database Cloud using Rapid start Kit, available in EM Agent Kit 12.1.0.3.0, after login to Exadata machine first DB node as EM 12c agent owner

  • Change to <location where Agent kit is unzipped>/cloudsetup directory.
  • Review the input files under config directory and customize the dbaas_cloud_input.xml for configuring DBaaS cloud and pdbaas_cloud_input.xml for configuring Pluggable Database as a Service.
  • Run the following command to setup DBaaS on Exadata Machine.
<EMCLI Home>/emcli login -username=sysman
<EMCLI Home>/emcli @exadata_cloud_setup.py –dbaas
Above command will use dbaas_cloud_input.xml (under cloudsetup/config) as input file for configuring DBaaS. 
  • To setup PDBaaS on Exadata, please use following command.
<EMCLI Home>/emcli @exadata_cloud_setup.py –pdbaas
Above command will use pdbaas_cloud_input.xml (under cloudsetup/config) as input file for configuring PDBaaS

Note: Currently Rapid Start kit for DBaaS makes use of 11.2.0.3.0 Database "Exadata Data Warehouse" Profile available out-of-box. However you can create your own DBCA based Profiles and customize the dbaas_cloud_input.xml. Also if you need to use RMAN backup based or Snap clone based profile, you can to login to EM12c SSA Portal as SSA Administrator, to create the profile and setup service template. 

At this stage, you will be able to manage and deliver your Exadata powered enterprise database cloud using EM 12c.

Additional References:

Obtaining the Oracle Enterprise Manager Setup Automation kit for Exadata (Doc ID 1440951.1)

Oracle® Enterprise Manager Cloud Administration Guide12c Release 3 (12.1.0.3)

Oracle® Enterprise Manager Exadata Management Getting Started Guide Release 12.1.0.5.0


Tuesday Aug 06, 2013

Secure, Subset and Self-Service Database provisioning in Cloud


One-step inline data masking and data subsetting is a very innovative solution in Oracle Enterprise Manager 12c that enables enterprises to provision secure and reduced size test systems directly from the production or standby database without the need for a full production database copy. So you are able to create reduced size copies of production database keeping the referential integrity of data set intact saving on storage costs and more importantly also ensure your sensitive database never creeps into any of your development or test systems in compliance with data privacy policies.

With Oracle Enterprise Manager 12c Release 3 customers have now started adopting this one-step on-the-fly masking and subsetting solution along with Database-as-a-Service(DBaaS) self-service provisioning capabilities and the outcome is one of the most comprehensive and unique solution to realize self-service provisioning of secured, subset copies of production like databases for all of testing and development needs in a private cloud model.




In this blog post I will walk through the steps required to achieve this comprehensive solution for test and development systems provisioning in enterprise private cloud.


1) Prepare source database

One of the ideal option in case customer has physical standby is to convert physical standby into a snapshot standby for one-step data subsetting and masking operation. A snapshot standby receives and archives but does not apply redo from primary until converted back to physical standby. So once you have created subsetted and secure dump from snapshot standby you can convert snapshot standby back to physical standby using Oracle Enterprise Manager. Refer to Oracle Data Guard guide for considerations in using snapshot standby.  In-line data subsetting and masking can also be performed on production database in case there is no physical standby.

2) Perform Inline data subsetting and masking workflow

Here are the steps :

a) Create Application Data Model(ADM)
ADM is a knowledge base entity in Oracle Enterprise Manager that captures application metadata, referential relationships, sensitive columns and is used by both test data management and data masking. There are pre-defined drivers to capture data relationships from application metadata tables for Oracle Applications such as Fusion Apps and eBusiness Suite. Security Administrator requires EM_ALL_OPERATOR privilege to create ADM and Data Masking and Subsetting definition.

b) Create Masking and Subsetting definition

As a best practice, Security Administrator can create masking formats for all regulated information in the enterprise. There are out-of-box Oracle supplied default masking formats also that can be used from format library. Next step is to create masking definition that includes information regarding table columns and the format for each column. You can choose which columns to mask. Data Masking workflow can be referred here.
Next step is to create Data subset definition and use the same ADM and database used in creating the masking definition. In this step you can define the table rules,rule parameters, include 'Ancestor and Descendant tables' or 'Ancestor tables only' option this ensuring referential integrity is maintained. At this step Space estimates can also be reviewed and depending on the result, you can modify or add new rules and review back on space estimates value. Pre/Post subset scripts can also be included in definition.

c) Generate subset using export option

Using the definition subset can be created by writing the subset data to export file. Specify directory where to save the export dump and schedule the subset job. Once the job is complete, the subset Data pump file of production data with sensitive data masked is ready. The overall flow is described here
Some key tips -
- Parallel degree can be used for faster export. Start with twice the number of CPUs and tune from there
- %U and max file size parameter can be used to ensure optimum use of parallelism
- Rule parameter in where clause can be used for actual subset generation with different values
- Column rules can be used to set large-sized columns to null or fix value to reduce database size further

3) Prepare secure test/reference database

Once the Data pump export file is generated it can be imported in a test/staging database . For details refer here.This test database can be used as a reference for all development and testing copies and hosted in Self-Service portal for end users to request from service catalog.

4) Self-Service Portal Setup

Following two options are available to cloud-enable this secure reference database for self-service provisioning of future copies of this database via Oracle Enterprise Manager 12c Database-as-a-Service (DBaaS) solution :

a) Snap Clone option that leverages storage level copy-on-write technologies for cloning and is suited best for functional testing requirements and for short lived databases.With this option you can clone terabytes of data in few minutes and storage saving is enormous with this option. you need to review the supported storage options for this method. As of time of writing the post Netapp and ZFS is supported though there are plans to support more storage options in near future.

b)  RMAN based cloning from backup is one of the favorite option for DBAs. The restore process is completely automated with EM 12c DBaaS and this option is best suited for performance and load testing, development requirements and for databases used for significant updates.
Once you decide on the option, steps to follow are documented in one of my recent blog post around Planning Database as a Service Implementation Project

Refer to Private Cloud Setup and Administration guide here for details on DBaaS Setup and Snap Clone, RMAN Profile options. The secure database prepared in previous step will be used as reference target by SSA Administrator while creating database provisioning profile using RMAN backup or Snapshot option. Please also ensure you have reviewed this MOS Note for DBaaS related patches -
"Enterprise Manager Cloud Control 12c Recommended Plug-Ins and Patches for Database as a Service (DBaaS) (Doc ID 1549855.1)"




5) Self-Service database provisioning in Cloud

End users can now request for secure subset copies of production database via Self-service portal for all kind of functional QA, load and performance testing, development requirements. All the databases provisioned from this approach are also enabled by default with EM 12c powered monitoring and diagnostics, lifecycle and cloud management capabilities.




Solution References

1) Oracle Enterprise Manager 12c Packs
Oracle Test Data Management Pack
Oracle Data Masking Pack
Oracle Database Lifecycle Management Pack
Oracle Database Cloud Management Pack

2) Oracle Enterprise Manager 12c Platform and Plug-ins
Enterprise Manager Cloud Control 12c Release 3 Base Platform (12.1.0.3)
Enterprise Manager for Oracle Database (DB) plug-in 12.1.0.4
Enterprise Manager for Oracle Virtualization (VT) plug-in 12.1.0.5
Enterprise Manager Storage Management Framework (SMF) plug-in 12.1.0.2
Enterprise Manager for Oracle Cloud (SSA) plug-in 12.1.0.6


Additional information

Stay Connected:

Twitter |  Face book |  You Tube |  Linked in |  Newsletter

Tuesday Jul 02, 2013

Oracle Enterprise Manager 12c R3 introduces advancements in cloud lifecycle and operations management

Oracle Enterprise Manager 12c Release 3 (R3) was announced ( Press Release ) earlier today. It is now available for download at  OTN .

This latest release features improvements in several areas, including:

  • Improvements to Private Cloud and Engineered Systems Management
  • Expanded Middleware and Application Management Capabilities
  • Efficiency Gains for Enterprise manager Users in EM’s Enterprise-Ready Framework

You can learn more about what's new in the Oracle Enterprise Manager 12c R3 in the Enterprise Manager 12c documentation . You will see more blogs and details about the new features during the next few weeks. Please let us what

On July 18th, you can join us at a webcast to hear Thomas Kurian, EVP of Product Development on what Oracle Engineering has achieved with Oracle Enterprise Manager 12c Release 3 to address these challenges. Later, during this webcast, Oracle experts will discuss the latest capabilities in Oracle Enterprise Manager 12c Release 3 for cloud lifecycle and operations management.

The presentation will be followed by a live Q&A session with Oracle experts. You can also join us online on Twitter to get your specific questions answered. Please use hash tag #em12c to join the conversation.

Register Now for the Webcast!

Stay Connected:

Twitter |  Face book |  You Tube |  Linked in |  Newsletter

Tuesday Jun 25, 2013

Planning Database as a Service Implementation Project

People, process and planning are the three key elements to success in a private cloud journey. Some common questions i hear from field/customers often relates to tasks involved in setting up Database-as-a-Service(DBaaS) using Oracle Enterprise Manager 12c from scratch and how these tasks are mapped to current IT roles and responsibilities - overall list of activities that needs to be performed to setup and achieve DBaaS and who does what during the implementation process.

In terms of roles, responsibilities and users, you will map to these categories

  • Project Manager is responsible for working with all stakeholders, project team and coordinating in planning and delivering DBaaS implementation project
  • EM Admin sets ups Cloud management infrastructure and is responsible for overall EM 12c installation/configuration and Administration
  • Cloud Admins are responsible for managing the Cloud infrastructure(Servers, Network, Storage) and related Administration
  • SSA Admins or Cloud DBAs are responsible for Self Service portal setup, managing quotas, policies, target DBs S/W provisioning/ongoing Administration
  • SSA Users or End users of Cloud who actually requests for DB instances to be rapidly provisioned on-demand from Self Service portal


Here is a sample Work-breakdown structure(WBS) listing down tasks involved and with individual owners mapped. Note - we have all type of use cases covered in the plan (DBCA, RMAN, Schema as a Service, Snap Clone) and you can pick up any or mix or all of these tasks in your plan depending on your use cases and requirements for implementing DBaaS -


 Activity Description
 Owner
A. Initiation and Startup

Determine scope of implementation
Determine and document scope of DBaaS Cloud implementation together with customer and end users. Determine requirements and goals. Introduction meeting implementation team. Project Manager
Coordination and quality management Coordination and quality management on throughout all phases of the implementation Project Manager
Define High availability Needs
Determine the level of High availability required Project Manager
Technical intake Technical intake meeting; Infrastructure requirements (Hardware, OS, network/firewall, s/w requirement) for cloud setup Project Manager
Functional intake Functional intake meeting; Monitoring /Reporting/Alerting requirement. User/Roles/Groups/Rules model Project Manager
Deliver implementation plan Document requirements and goals. Document required functional and technical configuration. Build architecture Diagram. Update estimate planning based on implementation plan. Project Manager
B. Infrastructure Setup

Install hardware systems Plan EM12c compatible hardware systems EM Admin
Operating system installation
Installation of certified OS platform [Refer Certify on MOS] EM Admin
Network configuration of hardware systems General network configuration of hardware systems I.e. TCP/IP, routing, NTP, firewall access, SSH, HTTP and HTTPS access. Set up network access between systems and work place of end users. Optionally implementation of remote access EM Admin
C. Production EM 12c Cloud Control Installation

Install of Oracle Database 11gR2 Installation of Oracle Database 11gR2 for Repository EM Admin
Installation of EM 12c Cloud Control Release 2 Update 1 and latest PSU Installation of EM 12c Cloud Control OMS/Agents EM Admin
Review HA, Security, Sizing Best practices Setup as per EM 12c Admin guide EM Admin
Review recommended patches Review latest PSU and DBaaS patches as per MOS Note 1549855.1 EM Admin
D. Setting up High Availability

Configure Repository For High Availability Enable Repository as per database high availability best practices.
Configure Standby database @ DR site for EM cloud control repository.
Configure repository database as RAC database with 2 or more nodes
Configure OMS for RAC/standby repository
EM Admin
Configure OMS for High Availability See Enterprise Manager Cloud Control Administrator's Guide for Configuring OMS for high availability EM Admin
Configure Agents for High availability See Enterprise Manager Cloud Control Administrator's Guide for Configuring Agents for high availability EM Admin