Rob Reynolds' BI & EPM Blog

With the announcement of the Exadata storage array and the Oracle-HP Database Machine at Openworld, we now have a great solution for customers looking for a DW Appliance. And unlike other DW appliances out there, ours runs the worlds best database too.

The response to this announcement has been incredible. I never expected the amount of customer interest and how quickly customers are looking to migrate from their current appliance solutions to the Oracle DBM. We are working with many customers to help determine if the Oracle DBM is the right solution for them and we are using our Data Warehouse Extreme Performance Assessment offering as the platform for assessing their needs.

Follow the link for more information on the offering or contact your Oracle representive for more information on engaging Oracle Consulting to help assess how your current environment can benefit from the Exadata storage solution or an Oracle Database Machine.

Dealing with data quality is a common task when implementing a BI Solution or Data Warehouse. Often times we find that data quality is really a complex, intertwined set of issues resulting from operational systems and processes that often cannot be changed to meet the needs of the BI Solution.

Avoiding data quality issues requires thinking a bit differently from considering "quality" in the traditional manufacturing sense. Discovering and correcting poor material in a manufacturing process is a binary decision. The material is right, or it isn't. The key mindset for data quality is to focus on data "usefulness," or data "effectiveness." Every application of information does not require the same levels of accuracy. If the information can be used to make the intended decisions, then it is considered effective. It is quite likely that a certain percentage of your information can meet many requirements without being perfect (i.e. marketing data). It is also likely that certain key information must be correct in every way (i.e. financial data).

First and foremost, BI & DW solutions are for making decisions. In many organizations, thresholds of acceptance can be created and data can be "good enough" to make certain decisions, but perhaps not "perfect." Data effectiveness for decision making can be accomplished at a lower level of "quality." In my next post, I will explore techniques for defining accuracy threshold and on-going monitoring of data quality against those thresholds.

As discussed in my previous post, sometimes data doesn’t necessarily have to be 100% accurate to be effective or actionable. So if there are thresholds to when information is “good enough” for decision making, how do we determine them?

The most common technique is to examine the decisions that the information will be supporting. Are the decisions time critical? Forward looking leading indicators that have impact on future results often do not require 100% accuracy due to their predictive nature. An example would be a sudden increase in sales pipeline that would trigger the need to increase production capacity.

Are the decisions based on information that is statistical in nature or broad enough where a wide set of information will be rolled up to a large order of magnitude? Measures that appear on a KPI/Balanced Scorecard can actually be allowed to be in error a few decimal points, because rounding can cover up a host of sins. Other measures may need to be very precise and the data supplying the algorithms must be accurate, such as creating financial reports.

Capturing the requirements for effectiveness of information verses timeliness we can then start to have conversations with business leaders about thresholds for data accuracy. Once thresholds are established, standard deviations can be used to mange the on-going effectiveness of information. Periodic review of data quality deviations is required to avoid gradual erosion in accuracy and effectiveness.

The key principle is to define and measure the degree to which the BI/DW system is becoming a vital source of reliable information. As long as the information creates decisions that add value, the information is "good enough." And most importantly BI team must ensure that the information being produced is "actionable."