Wednesday Mar 06, 2013

Snap Clone: Instant, self-serviced database-on-demand

Snap Clone: Introduction
Oracle just released Enterprise Manager 12c Release 2 plugin Update 1 in February, 2013. This release has several new cloud management features that, such as Schema as a Service and Snap Clone. While the relevance of Schema as a Service is in the context of new database services, Snap Clone is useful in performing functional testing on pre-existing data.

One big consumer group of cloud is QA Engineers or Testers. They perform User Acceptance Tests (UAT) for various applications. To perform an UAT, they need to create copies of the production database. For intense testing, such as in pre-upgrade scenarios, they need a full updateable copy of the production data. There are other situations, such as in functional testing, they need to perform minimal updates to the data, but at the same time, need multiple functional copies. Enterprise Manager 12c supports both the scenarios. In the former case, it leverages RMAN backups to clone the data. In the latter case, it leverages the “Copy on Write” technology at the storage layer to perform Enterprise Manager 12c Snap Clone (or just Snap Clone). Currently, NAS technologies viz. Netapp and ZFS Storage Appliance are supported for Snap Clone. By using this technology, the entire data does not need to be cloned, but the new database can physically point to the source blocks within the same filer and only needs to allocate new blocks if there are updates to the cloned copy.

Underlying “Copy on Write” technology
To cover the underlying technology, let us look at the Netapp  and Sun ZFS storage technologies. First of all, Netapp supports pooling of storage resources and creating volumes on top of those. These volumes are called Flexvols. NetApp FlexClone technology enables true data cloning - instant replication of the Flexvols without requiring additional storage space at the time of creation.  Each cloned volume is a transparent, virtual copy that can be used for a wide range of operations such as product/system development testing, bug fixing, upgrade checks, data set simulations, etc. FlexClone volumes have all the capabilities of a FlexVol volume, including growing, shrinking, and being the source of a snapshot copy or even another FlexClone volume. Data ONTAP makes it happen by Copy on Write technology. When a volume is cloned, ONTAP does not allocate any new physical space but simply updates the metadata to point to the old blocks of the parent volume. NetApp filers use a Write Anywhere File Layout (WAFL) to manage disk storage. When a file is changed, the snapshot copy still points to the disk blocks where the file existed before it was modified, and only the changes (deltas) are written to new disk blocks. A block in WAFL currently can have a maximum of 255 pointers to it. This means that a single FlexVol volume can be cloned upto 255 times. All the metadata updates are just pointer changes, and the filer takes advantage of locality of reference, NVRAM, and RAID technology to keep everything fast and reliable. I found this documentation on the Netapp site specially useful to understand the concept. The following picture provides a graphical illustration of how this works.

Oracle  ZFS employs a similar copy-on-write methodology that creates clones that point to the source block of data. When one needs to modify the block, data is never overwritten in place. Oracle Solaris ZFS then creates new pointers to the new data and a new master block (uberblock) that points to the modified data tree. Only then does it move to using the new uberblock and tree. In addition to providing data integrity, having new and previous versions of the data on disk allows for services such as snapshots to be implemented very efficiently.

The best way to think of storage snapshots is that it is a point-in-time view of the data. It’s a time machine, letting you look into the past. Because it’s all just pointers, you can actually look at the snapshot as if it was the active filesystem. It’s read-only, because you can’t change the past, but you can actually look at it and read the data. NetApp and SunZFS snapshots just write the new information to a special bit of disk reserved for storing these changes, called the SnapReserve. Then, the pointers that tell the system where to find the data get updated to point to the new data in the SnapReserve.

Space efficiency: Since we are only recording the deltas, you get the disk savings of copy-on-write snapshots (typically a few hundred kilobytes for a 1 terabyte database).

Time efficiency: Because the snapshot is just pointers, to restore data (using SnapRestore), we simply update the pointers to point to the original data again. This is faster than copying all the data back. So taking a snapshot completes in seconds, even for really large volumes (like, terabytes) and so do restores. A typical terabyte database therefore takes only a couple of minutes to clone, backup and restore.

So, what is the additional benefit of Enterprise Manager Snap Clone over storage level cloning?

Snap Clone is complementary to the copy-on-write technologies described above. It leverages the technologies mentioned above;  however it provides additional value in:

  1. Automated registration and association with Test Master database: Registering the storage with Enterprise Manager in context of the Test Master database. For example, it queries the filer to find the storage volumes and then associates those with the volumes that the datafiles are associated with. It provides granular control to the admins to make a database clonable, since there could be databases that DBAs do not want cloned off.
  2. Database as a Service using a self-service paradigm: Provides a self-service user (typically a functional tester) to provision a clone based on the Test Master. The self-service capability has administrator side feature like setting up the pool of servers which will host the databases, creating a zone, creating service templates for provisioning and setting access controls for the users both at the zone level and the service template level.
  3. Time travel: Functional testers often need to go back to an earlier incarnation of a database. Enterprise Manager provides the self service users to take multiple snapshots of the database as backups. The users can then easily restore from an earlier snapshot. Since the snapshot is only a thin copy, the backup and restore are almost instantaneous, typically a couple of minutes. During restore a large part of that is spent in actually starting the database, for example and discovering its state in Enterprise manager and not in the actual restore.
  4. Manageability: Finally, Enterprise Manager provides the complete manageability of these clones. This includes performance management, lifecycle management, etc. For example, when cloning at a storage volume level, sysadmin tools have little idea on the databases and applications that are consuming those volumes. From an inventory management, capacity planning and compliance it is important to track the storage association and lineage of the clones at the database level. Enterprise Manager provides that rich set of manageability features.

So how does this work in Enterprise Manager 12c?

In order to understand the Snap Clone feature of Enterprise Manager and its relevance to DBaaS, it is important to understand the sequence of steps that enable the feature and the DbaaS.

Step 1: Setting up the DbaaS Pool
First of all the Sysadmin has to designate few servers (which become Enterprise Manager hosts when the agent is deployed on them) to constitute the PaaS Infrastructure Zone. Each of these servers should have the connectivity to be able to mount the volumes participating in the Snap Clone process.

The DBA intrinsically knows the exact versions and flavor of databases being used within each LoB along with the operating system version compatibility. As the next level of streamlining he/she can add each unique type of the database configuration to a single place called Pool. For example, single Instance, cluster database …etc.

A database pool contains a set of homogeneous resources that can be used to provision a database instance within a PaaS Infrastructure Zone. For Snap Clone in particular, the administrator needs to pre-provision the same version of Oracle Homes either on standalone hosts or in a RAC cluster, which should be a part of the PaaS Infrastructure Zone.

Step 2: Setting up the Test Master
In the first step the administrator has to set up a Test Master as a clone of the production. Sometimes, the administrator has to create another copy of production at the source itself for masking and subsetting. The solution would vary depending on the customer's specific need and infrastructure. One can use one of RMAN, Dataguard, Golden Gate or even storage technologies such as Netapp Snapmirror, but usually our customers have figured out one way or other to do it. If the customer wishes to use EM for this, they can also use the Database Clone feature to clone the data (this leverages RMAN behind the scenes) or even use data synchronization feature of the Change Manager (part of Database Lifecycle Management Pack) to keep production and Test Master consistent. There is no unique way of accomplishing this; it all depends on the specific use case. There can be cases where the customer may need to mask or subset the data at source for which they can use those EM features as well.

The test master has to be created on ZFS Storage Appliance or Netapp Filer. Currently, the versions supported are:
·    ZFS Storage Appliance models  7410 and 7420
·    Any Netapp storage model where Version ONTAP® or above of Netapp is supported.  The Netapp interoperability matrix is available here

Here’s a sample of database files on a Netapp filer that could constitute the Test Master database.
·    /vol/oradata (datafiles and indexes): [8-16 luns]
·    /vol/oralog (redologs only): [2-4 luns]
·    /vol/orarch  (archived redo logs ):[2-4 luns]
·    /vol/controlfiles (small vol for controlfiles):[2-4 luns]
·    /vol/oratemp (temp tablespace):[4-8 luns]

Step 3: Register the storage and designate the Test Master
Once the Test Master database has been created, one has to
1.    Discover the Test Master database as an EM target
2.    Register the storage with Enterprise Manager. Enterprise Manager uses an agent installed on Linux x86-64 bit to communicate with the filer. For NetApp storage, the connection is over http or https. For Sun ZFS storage, the connection is over ssh.

 Enterprise Manager associates a database with a filer by deriving the volumes  from the data files and then associating the volumes with those seen by the filer. For a database to participate in Snap Clone, it should be wholly located on flexvols or shares with Copy on Write enabled. Enterprise Manager performs the necessary validations for that.

Step 4: Creating the service template using the Profile
Finally, the Test Master needs to be exposed as a source of cloning to functional clones to self-service users. This is done by creating a provisioning profile. Provisioning Profile, in general, is an Enterprise Manager concept that denotes a gold image-whether in the form of a “tarball” archive or an RMAN backup or a Test Master.  The concept of profile makes the process repeatable by several users, such as QA testing different parts of the application.

The profile is exposed to the service catalog via a service template which also includes the provisioning procedure, pre and post scripts for deploying the image.

Finally, comes the user side experience
. Enterprise Manager supports a self-service model where users can provision databases without being gated by DBA. The self-service user can pick a service template (which indirectly via the provisioning profile links to the Test Master) , specify the zone where to deploy and the database gets provisioned.  This new database is actually a "thin clone" of the Test Master and new blocks will get allocated only when the data is updated. The user can also take backup the cloned database, which are essentially read-only snapshots of the database. If the user needs to restore the database the latest incarnation of the database is simply pointed to the snapshot, so that the restore is instantaneous. This literally enables the self-service user to go back in time, in a "time travel" fashion. In addition to provisioning and backup, self-service users can also monitor the databases-check their statuses, look at session statistics, etc.

Before concluding this blog entry, let me point to a bunch of collateral related to DBaaS that we recently published. Check out the new whitepaper, demos, and presentation. We will soon publish a technical whitepaper on performing E-Business Suite testing using Snap Clone. Till then...

Stay Connected:
Twitter |
Facebook | YouTube | Linkedin | Newsletter

Friday Mar 01, 2013

Oracle's TaaS: Spend More Time Testing (and Less on the Other Stuff)

Recently we asked a group of testers what percentage of their testing time was spent on peripheral activities such as: 
  • procuring hardware  
  • deploying the application under test  
  • deploying a test tool  
  • find/detect/log issues/bugs  and/or 
  • patching an application
Two thirds of those testers indicated they spent between 40% to 70% of their time on these peripheral activities. 

When asked what the right solution could be to solve that enormous time consumption, the majority selected testing cloud solutions that combine capabilities for Infrastructure as a Service (IaaS) and Software as a Service. Let's look at each one :

Infrastructure as a Service ( IaaS ) based testing cloud : This solution is typically based on provisioning of virtual machines on provided infrastructure . This will only resolve the provisioning of applications under test and test tools which is only 10-15% of the total solution.

  • Software as a Service ( SaaS ) based testing cloud : This is Software ( for test automation ) as a service solution. This only addresses the test execution and ( possibly ) issue/bottleneck identification. It does not offer provisioning for applications under test . In many cases, it does not offer monitoring of internal applications.

  • The brand new Oracle Enterprise Manager 12c Testing as a Service (TaaS) solution (.pdf) offers that combination and helps software development and QA organizations to spend more time on actual testing and less on peripheral activities, while significantly enhancing testing efficiency and reducing the duration of testing projects.

    Oracle's TaaS is a platform for delivering automated application testing services. It is a self-service solution designed for private clouds that orchestrates the testing process end-to-end by: 
    • Automating the provisioning of test labs including application under test and test tools. 
    • Executing load and/or functional test scripts against the application. 
    • Providing rich application monitoring and diagnostics data for analysis.
    • Sophisticated chargeback facility for metering and charging the usage of the testing cloud by end-users. 
    It's built with semantic understanding of testing artifacts like testing tools, applications, test scripts, it is not just VMs.

    All this information and more was discussed in a recent webcast we ran in cooperation with the online community. We uploaded the replay on Youtube for you to watch at your convenience.

    Happy testing!

    Stay Connected:
    Twitter |
    Facebook | YouTube | Linkedin | Newsletter

    Sunday Feb 24, 2013

    Oracle Enterprise Manager Introduces Key Enhancements for Deploying and Managing Clouds

    It has been a little more than a year, since we released our first cloud management features as a part of Oracle Enterprise Manager 12c. As customers started adopting our solution for a broad spectrum of workloads including production, QA, and development needs, we gained invaluable experience into the various use cases and requirements. Based on that experience, Oracle has released new and updated Management Plug-ins that precisely address the above requirements, ultimately leading to faster time-to-market for IT services delivery. In addition to providing enhanced cloud management support, the plug-ins extend Enterprise Manager's capabilities for Database as a Service (DBaaS) and Infrastructure as a Service (Iaas), as well as introduce new features for Testing as a Service (TaaS). See "New Plug-in Features" below for additional details.


    + Database as a Service (DBaaS):

        A sophisticated Database as a Service solution needs to cater to a variety of use cases:
    •    A developer or a project owner requiring a new database service  with or without seed data
    •    QA requiring a full database refresh for intense load testing
    •    QA requiring to create multiple clones for functional testing on subset of data

    The current release of Oracle Enterprise Manager Cloud Control 12c offers new capabilities and support for managing database cloud services in all the above cases. It comes with out-of-box capabilities such as schema-as-a-service for extreme database consolidation and database cloning through Snap Clone or RMAN Backups. These capabilities provide an optimum utilization of development and database resources, giving customers more flexibility and control in managing the database lifecycle.

    While some applications need dedicated databases, small home-grown applications can often share the database instance with other applications. Schema as a Service allows DBAs to consolidate multiple applications in the same database and offer logical slices of database to the end-users such as developers, thereby preventing database sprawl. They can enforce certain performance guarantee to these services by leveraging database Resource Manager, which prevents a particular user over-consuming the underlying compute resources.  End-users can therefore request schema services from the self-service application without stepping on one another. Schema as a Service significantly reduces the administrative and maintenance overhead since only a limited number of such databases need to be managed and patched.

    Cloning databases got significantly easier with the new plug-in. A new feature called “Snap Clone” really makes it snappy (terabytes of data can be cloned in a matter of minutes). The feature helps clone a database by leveraging the underlying “Copy on Write” technology offered by storage technologies, such as Netapp and ZFS Storage Appliance. DBAs can set up a “test master” database by refreshing a production database and mark the “test master” as the source of functional clones. Self-service users can create multiple copies  of the test-master in minutes without consuming additional space beyond what’s needed to make localized updates. This can be really beneficial for applications where the testing is primarily read-only and is limited to a subset of the data. Users can also take backups (snapshots) of their database and “time travel” across snapshots to restore the database to an earlier incarnation in a matter of minutes.

    For more involved testing, such as testing production loads with lots of updates, Oracle Enterprise Manager also supports a full clone using the RMAN technology.

    The following picture highlights the various use cases of DBaaS that Oracle Enterprise Manager addresses. While each of these have specific applicability, Oracle Enterprise Manager handles all these use cases comprehensively.

    + Infrastructure as a Service (IaaS): Oracle Enterprise Manager Cloud Control 12c has added support for building and managing Infrastructure as a Service clouds based on the latest Oracle VM 3.2.1 release. The updated kernel and enhanced storage management features in Oracle VM 3.2.1 make it an ideal foundation for building a secure, scalable, enterprise-class infrastructure cloud. The newly released Oracle Enterprise Manager Management Plug-ins for Oracle Virtualization and Oracle Cloud provide comprehensive support for setting up the cloud infrastructure, deploying Oracle virtual assemblies and Oracle VM templates, and monitoring and managing the health of the cloud.

    + Testing as a Service (TaaS): Any enterprise application needs extensive testing before changes are rolled into production. Testing is a time-consuming process, can take weeks owing to delays in provisioning the full application stack to run the test on. QA is often left waiting on IT to provide the infrastructure and platform and sometimes this wait itself can account for 50% of the total testing cycle. Currently, no integrated solution exists that handle the provisioning and testing phases of the overall testing process leading to significant process delays and poor efficiency.

    Oracle Enterprise Manager 12c introduces Testing as a Service, a new cloud platform for delivering automated application testing services. A self-service solution designed for private clouds, TaaS orchestrates the testing process end-to-end by automating provisioning of complete test labs (applications, test tools and assets comprising scripts and scenarios), execution of load and functional tests and rich application monitoring and diagnostics. It also includes a sophisticated chargeback facility and the ability to perform deeper diagnostics in context of testing.

    TaaS makes it possible to significantly reduce testing time and costs without compromising quality, and enables organizations to be more agile in delivering critical business applications. One can find more details on TaaS here.

    Oracle Enterprise Manager’s cloud management features are also enabled through a set of powerful RESTful APIs, which are summarized in this presentation. These APIs can be consumed from custom or 3rd party orchestration frameworks. In the new release, one can also orchestrate these APIs using an Oracle Enterprise manager provided framework called Blueprints. Using Blueprints, one can orchestrate the provisioning of a multi-layered application (such as the one shown below). One can download the documentation and sample code for the blueprints from here.

    + New Management Plug-Ins:  The following new and updated plug-Ins are now available as part of this release. In addition to providing new and enhanced functionality as mentioned above, the plug-ins incorporate numerous bug fixes.

    Plug-In Name / Version
    *Enterprise Manager for Oracle Database (DB) (new)
    *Enterprise Manager for Oracle Virtualization (VT) (new)
    *Enterprise Manager Storage Management Framework (SMF) (new)
    *Enterprise Manager for Oracle Cloud (SSA) (new)

    Read More:

    * New book: Building and Managing a Cloud Using Oracle Enterprise Manager 12c (Oracle Press)

    *Cloud Management page on OTN

    *Enterprise Manager 12c: Cloud Management Pack for DB Datasheet

    *Enterprise Manager 12c: The Nerve Center of Oracle Cloud Technical White Paper

    Stay Connected:
    Twitter |
    Facebook | YouTube | Linkedin | Newsletter


    Latest information and perspectives on Oracle Enterprise Manager.

    Related Blogs


    « April 2014