Friday Nov 06, 2015

Oracle Data Integrator 12.2.1 - Managing Versions in Apache Subversion

The latest Oracle Data Integrator (ODI) release, 12.2.1, came out with brand new lifecycle management features that allows you to use Apache Subversion as the external Version Control System (VCS) for creating and managing ODI objects versions. It does not just stop there but also facilitates the creation of tags, manage branches for parallel development, and create Deployment Archives for promoting ODI objects from Development to Production.

In this article I will talk about various version management operations available in ODI Studio for maintaining, viewing, comparing and restoring versions from Apache Subversion.

How does it work?

ODI Studio remains the sole User Interface for performing any such versioning operations. It directly communicates with the external Version Control System and presents all the needed information within ODI Studio user interface. For example to create a version, an object is exported into XML file which is then checked in into Version Control System. Conversely to restore an object, its XML file is first checked out from Version Control System and then imported back into ODI repository.


Configuring Apache Subversion

In order to use Subversion, we first need to setup its connectivity in ODI Studio in just a couple of steps.


  • Edit Connection - This screen allows to setup subversion URL and user credentials as per the selected authentication types. There are a number of authentication types supported for setting up such connectivity – HTTP, SSH, SSL etc.

  • Configure - This screen allows you select the subversion project and the Trunk or Branch under it for maintaining ODI objects. If you don’t have the subversion project or branch already created then you can create them right from this screen at the time of configuration.

There are a couple of other useful options configured here

  1. Auto version – Set this if you want to make ODI to automatically create version of a versioned object whenever it is saved. By enabling you can make sure that the subversion copy is always in sync with your repository copy of a versioned object. But it may impact your save performance due to underlying export and checkin.
  2. VCS Key – Configure the VCS key that is used to encrypt the sensitive content while exporting the object in to XML file. For example the passwords in a Data Server definition should not be exported in plain text when the Data Server is exported into XML. Such passwords are encrypted and decrypted using the VCS Key configured here.

Managing Versions

The operations for managing versions are available at two levels.

  • At Studio menu level - This menu lists the subversion operations that are not specific to a particular object and can be performed on a group of objects.

  • In the object context menu (Right click menu) - These options allow you to perform various subversion operations on that particular object.

Adding objects to Apache Subversion

Once you decide what objects need to be version controlled, then you can add them to subversion individually or in group.

Adding Single object

 It is performed through object's context menu


There are couple of convenience features available here

  1. All the necessary parent hierarchy is automatically added with child object so that developer need not do it individually. For example when you add a mapping for the first time the corresponding folder and project that contains this mapping are also added automatically.
  2. You can add selected or all children objects along with the parent object. For example when you add a folder to subversion then the packages, procedures, mappings etc present under the folder can also be added along with it in the same operation.

Adding group of objects

The “Add Non-versioned Objects to VCS” option in ODI Studio and object context menu presents you the list of objects that are not yet added to subversion and allows you to add them to subversion in group. At studio menu level it list all such objects in the entire repository whereas at object context menu level it lists all the non-versioned objects under that object and is applicable for only the container objects such as projects, folder etc.


Indicator for Version Controlled Objects

A version controlled object can be easily identified by the subversion indicator presence next to its icon. Notice the indicator next to “Version Control” folder in below screenshot indicating that folder is version controlled whereas the other folder is not yet added to subversion.


Creating Object Versions

New versions of a version controlled object can be created through the context menu option – "Create VCS Version"


When a version is created at the container object level – such as project, folder, model etc – then following operations are also performed along with creating version of the the container object.

  • Versions are created for all version controlled children that are modified since last version. Note: It does not affect any non-versioned child.
  • Any child object deletion, rename or move is also pushed to subversion.

Viewing Version History

You can view version history for an object within the ODI Studio itself. It pulls object's version history from subversion and presents it in a couple of formats. Each of the formats provides different filtering/search options to easily locate a particular version.

  • Tabular format – through context menu option “Version History”
  • Tree format – through context menu option “Version Tree”

Comparing Versions

You can compare a version in object’s version history with another version or with the current copy present in repository. The difference between the two is presented side by side in a special window. There are toolbar buttons provided for easy navigation through the changes. The changes are color coded to easily differentiate the added, deleted or modified properties.


Restoring Objects from Subversion

You can restore an object's version from subversion into ODI repository. There are a couple of different situations in which you want to perform such restore

  • Restoring deleted object – You deleted an object but later realized that it is needed and want to restore it. You can view the list of deleted objects and restore them through the studio menu option Team->Subversion->Restore Deleted Object

  • Restoring an older version – If you messed up the latest copy of the object in the repository and wants to restore an older version then you can do it from objects context menu or from version history dialog.

The “Restore with Merge” check-box allows overwriting or merging the version changes into the existing repository copy. For a container object such merging can be applied to the child objects as well.

Conclusion

ODI 12.2.1 provides seamless integration with Apache Subversion. All the important version management operations in subversion are now directly available from the ODI studio. Developers can create, view, compare and restore versions from Apache Subversion with the same ease as they can do any other ODI operations.

Stay tuned for the upcoming articles covering Branch Management and Release Management using ODI 12.2.1 lifecycle features.

Tuesday Nov 03, 2015

Oracle Data Integrator 12.2.1 and Big Data

Oracle Data Integrator 12.2.1 brings a number of Big Data features such as Spark and Pig language generation support, executing ODI jobs in the Oozie workflow engine, and all new knowledge modules with improved Hive support. 

We provided a first release of most Big Data features as a patch 12.1.3.0.1 on top of the 12.1.3 release, and have included it now as part of the 12.2.1 release. In addition, we provided further improvements to the initial feature set, such as new Oozie Execution modes. You can now choose between Task and Session execution modes for Oozie workflow generation. The new Session mode allows support for transactions, scripting, and loops in packages and will run a session in a single Oozie action, whereas the Task mode uses a separate Oozie action for each ODI task. ODI will automatically choose the correct mode based on the executed object or the mode can also be manually selected.

ODI provides developer productivity and can future-proof your investment by overcoming the need to manually code Hadoop transformations to a particular language.  You can logically design your mapping and then choose the implementation that best suits your use case. 

Friday Oct 23, 2015

Announcing Oracle Data Integrator (ODI) 12.2.1 – Focus on Big Data and Lifecycle Management Innovations

Oracle Data Integrator 12.2.1 makes its debut today, with new and enhanced functionality in many aspects of its core capabilities. Take a look here for the latest details – and read on for some highlights:

Big Data

Back in April we proudly announced the availability of Oracle Data Integrator for Big Data. The release brought together a series of advanced Big Data updates and features that Oracle Data Integration is rolling out for customers to help take their Hadoop projects to the next level with support for Apache Spark, Apache Pig, and orchestration using Oozie. Today, we continue to enhance this functionality by allowing users to choose between Task and Session execution modes for Oozie workflow generation.


Lifecycle Management

We all realize that the process for managing system software through its lifecycle from the initial development to the implementation is important! In Oracle Data Integrator 12.2.1 release management capabilities are introduced to provide a distinction between those development and deployment environments and artifacts.


ODI Exchange

Ever wanted a more community feel for ODI objects – including Global Knowledge Modules? That is now possible through the ODI Exchange which allows browsing, download, and install of global ODI objects made available by Oracle, or other ODI users through Official of Third-Party Update Centers.


Better Oracle Connectivity

Oracle Data Integrator’s repertoire of Knowledge Modules is continually improved. This release features a Knowledge Module to perform Partition Exchange Loading is available to allow users to swap partitions as needed. For more on that very subject view the A-Team Blog post: Using Oracle Partition Exchange with Oracle Data Integrator (ODI). Improvements have also been made to the Loading Knowledge Modules using External Tables, and the Data Pump Knowledge Modules.

More Data Integration Pillar Integration – Enterprise Data Quality (EDQ)

This release adds a new Oracle Enterprise Data Quality (EDQ) technology – meaning that it is directly available in Topology and allows the creation of data servers, physical schemas, and logical schemas for EDQ. This helps to tightly couple the various tools when in use together in any project. In addition, the OdiEnterpriseDataQuality tool was improved to support EDQ data servers through Contexts and Logical Schemas.

There are many other operational enhancements that have been made, for the entire list of improved functionality, please click here for a detailed list of what is new in Oracle Data Integrator (ODI) 12.2.1 – many are not listed here! Try it out and tell us what you think!