Oracle Warehouse Builder (OWB) Weblog

Catch 22 that is what we sometimes suffer from. How do we know what to build when we don't quite know what you really use day-to-day. Sure we can look at our bugs (not that we have many ;-) ), but would it not be great if you could just tell us what is really important for your project?

So fill out the feature survey and let us know what you use in your projects, and as important here, add your feedback to the enhancement request section at the end and drive the new Warehouse Builder release with your requests...!

Please download the feature survey here and send us the results (press the big nice button).

Thanks in advance for any and all feedback!

We all know what a KPI is, but what on earth is KQI? Well I just dreamed that up... but hang in there. This might get fun after all...

Common data quality tooling these days has you do the data profiling. That is good! But you really don't go off and profile your data every day. So between profiling runs you are operating in the dark. First part good, second part not so.

The trick to profile data once than always have your torch is the use of something called a data auditor in OWB. If you read more on the OWB solution in DQ, you will see that the product does profiling, but that the product also allows you to create data rules. Now these are important when we want to look at how our data is doing in between profiling runs.

A data rule is like a KPI definition. It is an indicator, so you should create data rules that are important things to measure with regards your data quality. One of these "choose wisely things"...

Now rather than not knowing what is going on in your data loads, you turn these key quality indicators (KQIs) into the points you measure every day. Not by running the profile again, no by turning them into a data auditor. The data auditor does nothing other than run a select on your incoming data (or on the data after it is fixed - more later) and tell you the value. It then translates that value into a Pass or Fail grade, or in KPI terms, the traffic light goes green or red. You now can have a dashboard of KQIs next to your KPIs and run a system that not only tells you how the business is doing, but also tells you how good your data is that drives the KPIs...

A little less pie in sky, let's say you are loading a warehouse and you have this construct where if the data is bad you can roll it all back out via an ID. Would it not be nice if you would agree on SLAs with the customer community and you would measure the SLAs beforehand? That would save you the time on loading the data. It also would prevent incorrect reporting on the data that gets backed out (imagine running the sales report in the morning and paying bonusses only to learn it is not correct!).

Now back to profiling, your data auditors give you the daily value of the KQI. With these KQIs in place, you can rerun data profiling to do two things:

1) (Re-)verify your data rules or KQIs - this can be done in profiling with a simple check box and allows you to quickly assess KQIs on new data (like a new source)
2) Discover new values that you should start adding to the KQIs for either existing system and or new systems

Based on the new profiling results you can now either regenerate your rules to cater for new data situations or you can tweak the thresholds.

You can also create KQI differentials - e.g. improvement through your process. As we said before, you can profile incoming data as well as data that has been fixed. In OWB, adding a data rule to the correction process gives you data rules automatically attached to the correct objects. You run the auditor on both sides and now you can derive the differential. This is a great way to quantify the amount of money you saved people by fixing the data issues.

Back to KQIs. KPIs are well accepted, but we are still all on the fence about data quality. Now if we accept that we can calculate KQIs, how do we use them with our business users? One of the things is to simply expose them to the KQIs. Show the business users what the rules are, what the measurements are. Then discuss both and start to work on finding the real cause of spikes on KQIs or the real cause for the low results. And yes, one cause might be the incorrect rule! Accept that as well and move on to improve rules and data! The goal is after all to deliver great data to the business users.