Monday Feb 13, 2017

Introducing Oracle Data Integrator Cloud Service (ODICS)!

We are pleased to announce and welcome Oracle Data Integrator Cloud Service (ODICS)

Read the press release:  Oracle Launches Cloud Service to Help Organizations Integrate Disparate Data and Drive Real-Time Analytics.

Overview

Oracle Data Integrator Cloud Service (ODICS) delivers high-performance data movement and transformation capabilities with its open and integrated E-LT architecture and extended support for Cloud and Big Data solutions. Oracle Data Integrator Cloud Service provides all of the functionality included in Oracle Data Integrator Enterprise Edition in a single heterogeneous Cloud Service integrated with the Oracle Public Cloud. Providing an easy-to-use user interface combined with a rich extensibility framework, Oracle Data Integrator Cloud Service improves productivity, reduces development costs and lowers total cost of ownership among data-centric architectures. Oracle Data Integrator Cloud Service is fully integrated with Oracle Platform as a Service (PaaS) offerings such Oracle Database Cloud Service, Oracle Database Exadata Cloud Service and/or Oracle Big Data Cloud Service to put data and value at the center of the enterprise. Oracle Data Integrator Cloud Service is open and standards-based such that it can work with 3rd party systems as well as Oracle’s solutions.


Cloud E-LT Architecture for High Performance

Oracle Data Integrator Cloud Service's E-LT architecture leverages disparate relational database management systems (RDBMS) or Big Data engines to process and transform the data. This approach optimizes performance and scalability and lowers overall solution costs. Instead of relying on a separate, conventional ETL transformation server, Oracle Data Integrator Cloud Service’s E-LT architecture generates native code for disparate RDBMS or big data engines (SQL, HiveQL, or bulk loader scripts, for example). The E-LT architecture extracts data from the disparate sources, loads it into a target, and executes transformations using the power of the database or Hadoop. By leveraging existing databases and big data infrastructures, Oracle Data Integrator Cloud Service provides unparalleled efficiency and lower cost of ownership. By reducing network traffic and transforming data in the server containing the target data, the E-LT architecture delivers the highest possible performance for Cloud environments.

Heterogeneous Cloud Support

Oracle Data Integrator Cloud Service provides heterogeneous support for 3rd party platforms, data-sources, data warehousing appliances and Big Data systems. While Oracle Data Integrator Cloud Service leverages optimizations for Oracle Database and Big Data Cloud Services to perform E-LT data movement, transformation, data quality and standardization operations, Oracle Data Integrator Cloud Service is fully optimized for mixed technologies including: sources, targets and applications, etc.


Knowledge Modules Provide Flexibility and Extensibility

Knowledge Modules are at the core of the Oracle Data Integrator Cloud Service’s architecture. They make all Oracle Data Integrator processes modular, flexible, and extensible. Knowledge Modules implement the actual data flows and define the templates for generating code across the multiple systems involved in each data integration process. Knowledge Modules are generic, because they allow data flows to be generated regardless of the transformation rules. At the same time, they are highly specific, because the code they generate and the integration strategy they implement are explicitly tuned for a given technology. Oracle Data Integrator Cloud Service provides a comprehensive library of Knowledge Modules, which can be tailored to implement existing best practices ranging from leveraging heterogeneous source and/or target systems, to methodologies for highest performance, for adhering to corporate standards, or for specific vertical know-how. By helping companies capture and reuse technical expertise and best practices, Oracle Data Integrator Cloud Service’s Knowledge Module framework reduces the cost of ownership. It also enables metadata-driven extensibility of product functionality to meet the most demanding data integration challenges.

Oracle’s Data Integration solutions provide continuous access to timely, trusted, and heterogeneous data across the enterprise to support both analytical and operational data integration. We look forward to hearing how you might use Oracle Data Integrator Cloud Service within your enterprise.

Monday Feb 06, 2017

The Briefing Room: Oracle Data Integration Solutions & Big Data

Join Oracle Data Integration's Jeff Pollock in The Briefing Room for a radio broadcast on February 28 at 4pm Eastern Time!

Power of the Platform: How to Fast-Track the Value of Big Data

Tuesday, February 28, 2017 at 4pm Eastern Time

Big Data offers big advantages, but getting to the starting line can be problematic. Many projects run into hurdles just trying to access and load the data that will fuel key insights. There is a significant impedance mismatch between traditional data practices and the new systems that manage Big Data. But under the covers, many of the same principles still apply, including the importance of a metadata-driven approach to help navigate the complexity of Big Data technologies. Oracle's de-coupling of logical design and physical implementation layers greatly facilitates the rapid orchestration of Big Data solutions that provide business value.

We hope you can listen in!  Remember to REGISTER HERE

Monday Jan 30, 2017

Git Versioning Support in Oracle Data Integrator (ODI) 12.2.1.2.6

Oracle Data Integrator (ODI) 12.2.1.2.6 now supports Git as an external Version Control System (VCS). Now you can use either Apache Subversion or Git for source controlling ODI objects. Regardless of which technology is used, the user experience for an ODI user will be the same for any versioning operation and the underlying differences between the two systems are transparent to ODI users. This is consistent with the ODI core benefits of keeping users shielded from learning underlying systems and providing a seamless experience across technologies.

In addition, there are numerous new features added to increase productivity and address various use cases. Now you can have a consolidated view and manage all out-of-sync versioned objects from a single screen. The ODI Populate Repository option is also enhanced to allow populating a repository from a tag so that you can restore objects state from it. You can create version for all dependent object as well, along with versioning of the base object. There are options provided to regenerate scenarios during tag or deployment archive creation to ensure that the scenario corresponds to the current state of the corresponding object in VCS. The automatic merge process is made smarter to perform three way merge with change detection which reduces conflicts during branch merges.

In this article we are going to explore the version control related features and capabilities. I will cover the smart merge capabilities in a later article so stay tuned for that.

Configuring Git

The administrator needs to first enable Git as version control system and configure the repository connectivity.


Selecting Git as version control system will enable all the Git related configuration menu options. You can switch anytime between the version control systems so an ODI repository, previously configured with Subversion, can be switched to Git-based versioning. ODI however, does not migrate the version history during such switch so you need to migrate any such history directly though the tools provided by those systems.

Selecting Git for versioning enables the VCS settings options in the studio menu.

You need to perform all three settings configuration in the order of their appearance. The Edit Connection dialog configures Git connectivity details such as authentication type (protocol), URL, password etc. After connection setup, you can create and configure local Git repository through Clone Remote Repository operation. Then select the Git branch through the Configure menu option. Once successfully configured, you will notice that the Git indicator in the bottom right corner of ODI Studio turns green from grey indicating successful Git configuration. The indicator also displays the configured Git branch name – for example master branch in below screenshot.

Managing Versions

All the Lifecycle Management functionalities that existed in previous ODI releases for creating and managing versions in Subversion are now available for Git as well. Some of the operations are as follows

  1. Adding one or more objects to VCS
  2. Creating versions for modified objects that are out of sync from VCS
  3. Viewing version history – Hierarchical view and Tabular view
  4. Comparing difference between two object versions in VCS or comparing the VCS version with repository object
  5. Restoring an object version from VCS
  6. Restoring a deleted object from VCS

Since these functionalities are the same as noted earlier, I am not going to cover their details here. Instead, I will focus here on the new options provided in the latest release to make these more user friendly. If you are interested in details on above-mentioned operations, please refer to my previous post Oracle Data Integrator 12.2.1 - Managing Versions in Apache Subversion.

New options while creating versions

There are a couple of useful options available when adding a new object to VCS or creating new version of an already versioned object. These advanced options are available at the bottom of all the versioning dialogs. By default, they are not selected to give you the existing behavior from previous releases.

  1. Include Dependencies
  2. Regenerate and Version Scenarios

Include Dependencies

This option allows you to ensure that all the dependencies of an object are also versioned along with the object. For example if you are versioning a mapping which uses two data sources which in turn depends upon some technology or logical schema, then using this option you can version all these objects in a single operation along with the mapping itself.

If a dependent object is not yet versioned then it adds it to Git or Subversion or if a dependent object is out-of-sync with VCS systems then it creates a new version. If the dependent object is already versioned then it will not do anything for that object.

This option is particularly useful in keeping consistency in the versioned objects in VCS, which is key for continuous integration. It removes any chances of missing to create a version for a dependent object so that the current copies of all the relevant objects are present in VCS.

Regenerate and Version Scenarios

This option takes care of regenerating the scenario before creating a version for it in VCS. If you also select Include Dependencies then it regenerate a scenario for any of the selected objects and their dependent objects.

This option will be useful when you want to ensure that the scenarios present in your VCS correspond to the current copy of the corresponding object in VCS. Such requirement could be crucial if you have an automated build and testing process, which takes the scenarios from VCS and validates them.

Pending Changes

The newly introduced Pending Changes dialog allows you to manage objects that are out-of-sync from a single place. You can access it from ODI Studio Team menu option.

The Pending Changes dialog shows the list of all the versioned objects from the ODI repository that are out-of-sync. If you are looking for a particular object, you can directly reach to it through the search field. This dialog allows you to perform following operations on the selected objects.

  1. Create version for the selected object(s) here if the selected object is a deleted object from ODI Repository then you can push the deletion to VCS.
  2. Restore a deleted object from VCS. This option is applicable only when you select all the deleted objects.
  3. Compare the highlighted object with the VCS version.

New options while creation Tag

There are a couple of useful options provided during a Tag creation allowing you to control this:

  1. Option to add only the versioned objects to the tag. This provides flexibility of creating Full Tag with only versioned object is a very useful flexibility that allows you to push all the versioned objects along with dependencies to VCS in a single operation.
  2. Regenerate and version scenarios so that the scenarios referred by a Tag always corresponds to the relevant object in the Tag.

New options while Populating Repository from VCS


The enhanced Populate ODI repository from VCS dialog provides a number of flexibility.

  1. Populate from branch or Tag: Now not only you can populate ODI repository contents from the currently configured branch but you can also restore the objects from a particular Tag created on the current branch.
  2. It now gives you the flexibility to select either all the objects or a subset of objects to be populated from the selected Tag or branch. Such selective populating will be useful if you want to break down the process into smaller chunks or if you are interested only in a selected subset of objects from the branch or Tag. However, this flexibility comes with a pitfall that you may miss to import some of the dependent objects affecting the repository objects consistency. So this selective population process should be used with caution.
  3. Deleting existing work repository objects from ODI repository before populating objects from VCS. This will be useful in ensuring that there are no remnant of old work repository and after populate the ODI work repository objects are in sync with VCS contents.

SDK APIs for Continuous Integration

One of the needs for Continuous Integration is to automate the Tag Creation and Deployment Archive build process. There are a couple of services available in ODI SDK APIs that allows you to automate your entire build and testing process. VersionManagementService provides the necessary APIs to configure the VCS system, create Tags, and populate an ODI repository from the VCS contents from a Tag. DeploymentService provides APIs for creating Deployment Archive and applying it to target ODI Repository.

Conclusion

The enhancements added in Oracle Data Integrator (ODI) 12.2.1.2.6 around Lifecycle Management capabilities provides broader support for the leading Version Control Systems, improves developer’s productivity, and addresses the needs for automating Continuous Integration and build process. Yet again ODI differentiates itself by providing Lifecycle Management capabilities not available in any other competing products.

Monday Jan 23, 2017

Oracle Data Integrator (ODI) to Load Business Intelligence Cloud Service (BICS)

Learn more about using Oracle Data Integrator (ODI) to load Business Intelligence Cloud Service (BICS) from Richard Williams of the A-Team...

It is not new to hear that ODI can load data into the Business Intelligence Cloud Service (BICS) environments.  (These use Database as a Service (DBaaS) as the underlying database.)

The recent 12.2.1.2.6 release of ODI (back in December 2016) however, added the ability to load data into BICS environments based on a Schema Service Database. ODI does this by using the BICS REST API.

The article written by Richard walks through the following steps to set up ODI to load data into the BICS schema service database:

  • Downloading latest version of ODI
  • Configuring the physical and logical connection to BICS in ODI
  • Loading BICS knowledge modules
  • Reverse engineering BICS model
  • Creating a simple mapping
  • Importing the BICS certificate into the trust store for the standalone agent

READ the article:  Using Oracle Data Integrator (ODI) to Load BI Cloud Service (BICS).

Additionally, for other A-Team articles about BICS - read here.

Thanks Richard!

Using Loading Knowledge Modules for both On-Premise & Cloud Computing - Oracle Data Integrator (ODI) Best Practices

More from the A-Team... Thank you Benjamin!

Are you curious about using Loading Knowledge Modules for both on-premise and cloud computing in Oracle Data Integrator (ODI)?  Benjamin has put together some best practices for selecting and using the Oracle Data Integrator (ODI) Loading Knowledge Modules (LKMs) for both on-premise and on cloud computing. 

The Loading Knowledge Modules (LKMs) are one of seven categories of Knowledge Modules (KMs) found in ODI.  Other categories of KMs include: Reverse-Engineering, Check, Integration, Extract, Journalizing, and Service.  This particular article focuses on the selection and use of LKMs.  You can learn more about other categories of KMs, by going to Oracle Data Integrator (ODI) Knowledge Modules (KMs).

LKMs are code templates in ODI that can perform data upload operations from on-premise data servers to cloud services, between cloud services, or between on-premise data servers.  ODI supports a variety of technologies such as SQL databases, Big Data, Files, Java Messaging Systems (JMSs), and many other technologies.  Most of these technologies are now available on both on-premise and on data cloud services.  For each of these technologies, a variety of LKMs are available.  For instance, ODI offers LKMs for SQL databases such as Oracle, Teradata, MySQL, MS SQL Server, among others.  For Big Data, ODI offers LKMs for Spark, Hive, Sqoop, Kafka, and Pig, among others.

Read Benjamin's post:  Oracle Data Integrator Best Practices: Using Loading Knowledge Modules on both On-Premises and Cloud Computing.

Friday Jan 06, 2017

Oracle Data Integrator Best Practices from the A-Team: Using Reverse-Engineering on the Cloud and on Premise

Thanks to Benjamin Perez-Goytia of the A-Team – find out more about using the reverse engineering features of Oracle Data Integrator (ODI) through this ARTICLE.

Do you need step by step instructions? This might just do the trick!

There are three sections to this article. The first section covers the various options available in ODI to reverse-engineer metadata from a data server. The second section discusses performance considerations when running and executing reverse-engineering tasks.  The last section of the article discusses the ODI reverse-engineering best practices overall.

Happy reading!

Monday Dec 12, 2016

Oracle Data Integrator (ODI) 12.2.1.2.6 Now Available

We are pleased to announce a new release to Oracle Data Integrator (ODI): 12.2.1.2.6.

Oracle Data Integrator (ODI) provides a flexible and heterogeneous data integration platform for data processing that enables you to transform, enrich, and govern data for faster, more informed decision-making. In this release of Oracle Data Integrator, we have further extended our capabilities in 4 major areas: Big Data, Cloud, Lifecycle Management and Developer Productivity. Cloud and Big Data remain key investment areas and ensure that Oracle Data Integrator will continue to accompany customers throughout their technological transformation. Please visit the following two links for more information on Oracle Data Integrator (OD): ODI OTN Page, O.com Data Integration Page.

Big Data investments include Spark Streaming support, Apache Kafka and Apache Cassandra support, enhanced support around Hadoop Complex Types and Storage Formats in addition to enhancements to ODI’s Big Data Configuration Wizard. Through its unique decoupling of the Logical and Physical design of Mappings, Oracle Data Integrator is the only Data Integration tool on the market giving developers the flexibility to design Mappings with a generic business logic and then generate code for as many data processing technologies (Hive, Spark, Spark Streaming etc.) as they want. This provides a great platform for the ever changing and improved world around Big Data.

Cloud investments include RESTful Service support, where Oracle Data Integrator can now invoke RESTful Service. Data chunking and pagination are also supported for uploading or downloading larger payloads. Additionally, Business Intelligence Cloud Service (BICS) Knowledge Modules is now supported out of the box in Oracle Data Integrator. You can define Business Intelligence Cloud Service connectivity in Topology, reverse engineer metadata and load data into it just like any other target data server.

Lifecycle Management investments include Git Support as an external version control system. Other improvements to the overall lifecycle management functionality have been made, such as the enhanced merge capability with auto-merging of changes and simplified conflict resolution.


Developer Productivity investments include a superior Knowledge Module Framework that helps maximize flexibility and minimize maintenance. You can now inherit steps from a Knowledge Module into another Knowledge module and override steps like in object oriented programming languages. There are brand new template languages and syntaxes introduced providing greater control over the generated code.

For the full review of new functionality, please view the following What’s New Whitepaper.

Tuesday Nov 08, 2016

Upcoming Webcast: Laying a Foundation for Big Data Environments with Flexible Data Integration

Laying a Foundation for Big Data Environments with Flexible Data Integration

As big data environments take a more prominent role in analytics solutions, customers are looking for flexible and cost-efficient yet robust solutions. Oracle’s data integration offering combines the powers of the cloud, big data, and machine learning aided by slick user experience-optimized interfaces.

During this webcast, our experts will explore how to:

          · Build a future-ready solution that supports open source for both speed and batch processing

          · Upgrade your big data analytics in the cloud to include self-service capabilities for data ingest and preparation

          · Use cloud-based data integration to improve efficiency while enjoying cutting edge capabilities

 Join us Thursday, November 10

 10:00am PT / 1:00pm ET

Register HERE!