Good Data Quality Read on Rittman-Mead's Blog, and validating input with data rules

There's a good read over on the Rittman-Mead Consulting blog on End-to-End data quality. OWB would be a good tool to implement parts of the solution they have in mind-- to do an initial DQ assessment, and to do cleansing during load. And DQ reporting on both the sources and the target system is certainly doable, using data auditors and your favorite BI application, to prove to the powers above that the problem isn't yours.

Input validation is where OWB's answer is a bit less obvious, though. Enforcing the OWB DQ rules during input validation is doable, I think, but you have to be creative. Here's how I'd go about it...
OWB creates PL/SQL packages for each PL/SQL mapping, including mappings with data rules. One way of coming at this would be to create a mapping for each set of rules you need to enforce at input time, and that returns the required validation status as an output. Then expose the PL/SQL package for the mapping as a database web service as described in the Warehouse Builder SDK.

Invoke this web service from your source application at the time of input, and reject bad entries and report problems based on your output.

Note that this also gives you consistency between your data rules as used in ETL w/DQ and your input validation at the front end; if you change your data rules, then re-generate and re-deploy, your validation logic is updated as well.


Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

ETL, CDC, Real-Time DI and Data Quality for the Oracle Database from the inside.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today