ODI 11g - Loading More Files Faster
By David Allan-Oracle on Dec 20, 2012
This post shows you how to leverage more of your Oracle system, you can load more data and faster using these kind of best practices. The LKM here, lets you leverage the Oracle database's preprocessor to load MORE data faster. The KM also has the capability to leverage a regular expression to identify the file or files so that you can load much much more, it also sorts the files based on size to comply with the most efficient way of loading into Oracle via the external table. The preprocessor can basically do anything, but is commonly used to uncompress data on the fly - so you can eliminate transferring large uncompressed files around or needlessly uncompressing them! There are various creative blog posts around showing you what some customers have done - some even include the transfer in the preprocessing phase.
I posted the LKM on the java.net site here.
You can see the LKM configured below, I have used the preprocessor and a regular expression to load all gzipped files in the directory;
When the code is generated below you can see the PREPROCESSOR directive used, plus you can see that the file names are going to be generated from a Java API call where the regular expression and directory are passed.
My gunzipdb.bat script has the following definition, wrappering the gzip utility (note the use of the -d -c directives);
- @echo off
- c:\ext_tab_test\gzip -d -c %1
Check this creative use of the PREPROCESSOR directive here (External table to load compressed data residing on remote locations) which does some creative work in the preprocessing script. This could do pretty much anything you want including transforming from some complex file to project through an external table.