Friday Mar 07, 2014

Using the Unpivot Component in ODI 12c

As we’ve seen last week the latest ODI 12c patches added several new Mapping Components such as Pivot or Unpivot. In this blog post we will walk you through an example of how to use the new Unpivot Component. 

You can use the following SQL statements to recreate this example in your environment. It will create the source (UNPIVOT_TEST) and target (TRG_UNPIVOT_TEST) tables used in this article in your database then you can reverse engineer them in ODI.

CREATE TABLE unpivot_test (
year NUMBER,
q1_sales NUMBER,
q2_sales NUMBER,
q3_sales NUMBER,
q4_sales NUMBER
);
insert into unpivot_test values (2012, 10.5, 11.4, 9.5, 8.7);
insert into unpivot_test values (2013, 9.5, 10.5, 10.3, 7.6);
commit;
CREATE TABLE trg_unpivot_test (
year NUMBER,
quarter VARCHAR2(255),
sales NUMBER
);

Our goal is to unpivot the data on the Quarter columns when going from UNPIVOT_TEST into TRG_UNPIVOT_TEST as shown below


Follow these steps to add and configure an Unpivot Component in a Mapping:

  1. First add the Source table UNPIVOT_TEST into your Mapping, to do so drag and drop the datastore from the Models into the Mapping
  2. Next add an Unpivot component into the Mapping. This is done by clicking on the Unpivot Component in the Component palette and then clicking on the Mapping diagram. A new UNPIVOT component will appear in the Mapping:

  3. Drag and drop the YEAR column from UNPIVOT_TEST into the UNPIVOT component. There is no need to add the remaining source attributes yet, we will be using them in the Unpivot Transforms section later.

  4. Then click on the UNPIVOT component and select the Attributes panel. We will add 2 new attributes representing the row and value locator attributes: QUARTER (VARCHAR) and SALES (NUMERIC).

  5. Click on the General panel and select QUARTER in the Row Locator drop-down list. We will be performing the unpivot operation using this column.

  6. Next add the transformations to unpivot the data. Click on the Unpivot Transforms panel and create a transformation rule for each of the 4 quarters and match the appropriate QUARTER and SALES values as shown below:

  7. Finally add the Target datastore TRG_UNPIVOT_TEST and link the UNPIVOT component to it. Leave the default settings in the Attributes Matching panel and click OK

  8. In this example you can use the default Physical settings for your Mapping. Integration Type is set to Control Append by default and the IKM Oracle Insert is used
  9. Finally click on Run to execute the Mapping, 8 inserts are performed and you should see the following data in your target table:

  10. If you review the generated code you will notice that ODI leverages the UNPIVOT function on Oracle to perform such operation. The Unpivot component supports Oracle as well as any database

You can recreate the following example using the ODI 12c Getting Started VirtualBox image which is available on OTN: http://www.oracle.com/technetwork/middleware/data-integrator/odi-demo-2032565.html

Monday Dec 16, 2013

ODI 12c - Pivoting Data and Correlated subqueries

Here are more data transformation illustrations using the ODI 12c SDK demonstrating pivoting/unpivoting data and correlated subqueries. The SDK examples are on the shared java.net site and are in addition to the existing examples I posted on the blog entry on Mapping Ins and Outs here.

You can download these ODI_12c_Mappings_SDK Examples here unzip and inspect the groovy and the images of the mappings within. There are examples from the SQL documentation on correlated subqueries and examples previously posted for OWB here on pivot/unpivot.

The mappings I have covered include (click on hyperlink for an image of each);

  1. unpivot data from a table with YEAR,Q1_sales,Q2_sales,Q3_sales and Q4_sales columns to YEAR,QTR,SALES (UNPIVOT component).
  2. pivot data from a table with YEAR,QTR,SALES columns to YEAR,Q1_sales,Q2_sales,Q3_sales and Q4_sales columns (PIVOT component).
  3. perform a subquery to get the employees who are in a department that exists in the subquery (SUBQUERY_FILTER component).
  4. perform a subquery to get the employees who have a salary greater than the average salary for employees in their department (SUBQUERY_FILTER component).

These examples all use the mapping SDK to build data integration flows, the existing ODI interface SDK is still supported for now. You can see the mapping built from the SDK to perform unpivot of data as an illustration. There are now 14 examples of components in the download illustrating each one, they can operate on arbitrary data and have generic KMs as well as specific Oracle implementations.

Plenty more to show including some of the nitty gritty details on the physical deployment specification side to help illustrate how to build flows using the SDK and backup the javadoc. A lot of great transformation components and plenty more to explore and discover!

Friday Nov 15, 2013

ODI 12c - OWB Migration, Integration Plus Components

Today is another big day for ODI and OWB! Get your hands on the much anticipated functionality covering everything from new components added into the ODI 12c mapping designer, to the ETL design migration utility (announced here) to move designs from OWB 11.2.0.4 into ODI 12c which all complement the integration and auditing of OWB jobs from ODI which was included in 12c.

The patches for ODI and OWB are great news on many fronts, not only in being a major point in the roadmap. As well as the OWB migration this is great for the ODI the product, the new components provide a lot of great capabilities (and will help with a lot of those 'how do I do this' questions).

Included in the ODI patch you'll find a component bundle including transformation components from pivot/unpivot through subquery filtering on to table functions and the like. All great examples of the product's extensibility - a fusion of declarative design and technology specific knowledge modules in harmony. Lots of good stuff that you will hear much more of.

There are multiple patches - if you are only interested in ODI features and enhancements then all you need is the ODI patch below;

  • ODI 12.1.2.0.0 installed (ODI 12.1.2.0.0 plus patch number 17053768)

If you want the OWB migration utility then you will need the following patch to OWB as well as the ODI patch above;

  • OWB 11.2.0.4 installed (OWB 11.2.0.4 plus patch number 17547241)

If you install the ODI patch you will find some new components in the palette, the components let you express many more transformations - these were transformations which could be designed in OWB, now in ODI as well as the components we have knowledge modules to support their code generation for heterogenous environments.

The ODI 12c release included a new ODI Tool and support for monitoring OWB jobs - jobs can be initiated using the tool OdiStartOwbJob tool which is available for use from within packages, procedures, KMs or command line. The jobs can be monitored from ODI, this includes OWB mappings and process flows. The patch provides the migration utility plus components.

In subsequent postings we will deep dive into these areas. 

About

Learn the latest trends, use cases, product updates, and customer success examples for Oracle's data integration products-- including Oracle Data Integrator, Oracle GoldenGate and Oracle Enterprise Data Quality

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
2
3
5
6
7
8
9
10
12
13
14
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today