ODI - File transformation tool

ODI 11g introduced the complex file technology which supported the ability to define a parser for data files and plug it into ODI. I have created an ODI Open Tool (ODIFileTransformation) which utilizes part of this capability just to do native file to XML file transformation - so just file to file based on the nXSD parser definition. The native input file (such as COBOL or whatever) is parsed based on the nXSD definition and the resultant XML file streamed directly (to disk, a pipe or wherever). Btw...its very easy to extend ODI in a number of ways, adding a tool is an easy way to integrate existing capabilities.

Why? It gives us different opportunities for consuming the parsed data and it is now in a convenient form (XML). As an illustration I processed a 200Mb data file and the resultant XML file was generated in 120 seconds whereas using the complex file technology directly in ODI and staging the same native data file in a database took 9000 seconds. There is no staging with the tool, it simply reads the data, transforms using the nXSD and produces the generated XML. So.... you could write the resultant file on to a disk such as HDFS then process it using Hadoop.

The tool has the following signature and is useful for running from the command line for performing file to file transformation or from within a package. This tool was based on a test tool that was blogged here and I changed to remove the XML DOM nature so that it streams to the result file.

The tool is on the java.net site, to install, you will need the java mail JAR (download here) and the tool itself (download zip containing JAR and source here). The JARs get copied to the ODI userlib directory and then you restart the client and register the open tool. The tool can have more options added including debatching into chunked files. Hopefully some useful bits and pieces here. 

Comments:

David,
It seems the download link is not open to the public.
Thanks

Posted by guest on August 27, 2013 at 06:53 AM PDT #

Hi

You have to register with java.net in order to download, there is a free signup page here;
https://java.net/people/new

There are a bunch of other useful downloads in the oracledi project within it including KMs, procedures and tools.
Cheers
David

Posted by David on August 27, 2013 at 07:47 AM PDT #

David,
I do have an account there. I can browse all sections (Wiki, Issue tracking, etc.) of the oracledi project BUT the Download section.
Got "You are not allowed to do that (read, or write, or update, or delete downloads)" error.

BTW, I really like some of the KMs you have there. Hope I will be able to access those too.

As always, very good article David.

Alex

Posted by Alex To on August 27, 2013 at 08:34 AM PDT #

Hi Alex

So if you use the following URL you cannot see the tools listed?
https://java.net/projects/oracledi/downloads/directory/ODI/Open%20Tools

I will check and see if there are some issues with its setup.
Cheers
David

Posted by guest on August 27, 2013 at 08:42 AM PDT #

I confirm. I cannot also access or browse the project.

By accessing this page,
https://java.net/projects/oracledi/
I get this message:
You are not allowed to do that (read, or write, or update, or delete downloads)

Cheers
Nico

Posted by GERARD Nicolas on August 28, 2013 at 10:01 AM PDT #

Hi Nico

You have to click on 'Join This Project', can you try that?

Cheers
David

Posted by David on August 28, 2013 at 10:03 AM PDT #

Works perfect after joining the project. Thanks.

Posted by Nicolas GERARD on August 28, 2013 at 11:53 AM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Learn the latest trends, use cases, product updates, and customer success examples for Oracle's data integration products-- including Oracle Data Integrator, Oracle GoldenGate and Oracle Enterprise Data Quality

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
2
3
5
6
7
8
9
10
12
13
14
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today