Data Integration and Management

There’s been some interesting discussion around what’s next for data quality and the fascinating challenges of cleaning data for data warehouses and business intelligence applications. I am always intrigued by blogs that discuss the challenges of data management and applying cleansing principles for complex data-centric applications across the enterprise. However, I’m dismayed by discussions that quickly jump to the conclusion of out-sourcing data quality as a software-as-a -service model. For example David Rosenberg writes on his blog:

Look for the emergence of third party B2B integration and commerce management service providers that support data entry and validation for all trading partners. Integrated suites of direct system-to-system integration and Web portal services will be supplemented with combined e-mail and smart-form technologies solving the data quality problem associated with paper-based exchanges with small and occasional trading partners.

While it sounds good on paper, I think this is more marketing spin than a realistic use case. I think we can meet the goal of achieving clean, trusted authoritative data without going off-premise. When we ask companies to deliver their most critical asset into 3rd party hands, it’s going to lead to more challenges that aren’t easily solved. Terabytes of data aren’t easily moved like sacks of dirty laundry. here are 5 reasons why the business model of outsourced data quality is ahead of its time:

1) Moving Data is hard – Moving terabytes of information off-premise –once- can be challenging enough, moving it as changes occur is even more challenging.
2) Auditing – to turn bad data good, means lots of changes. Keeping track of these changes and offering roll-back capabilities and full auditing is critical. How can these be easily managed when they are off-premise?
3) Customization– every company is unique how they approach data, even data like address information which would seem commonplace. Most on-premise data quality engine solutions have some type of customizable rules approach whereas many 3rd party solutions are using generic approaches.
4) Profiling – the forgotten aspect of data cleansing is first understanding and seeing the stain. Off-premise data cleansing solutions assume that the data needs cleansing, but the element of profiling needs to be applied on-premise within the enterprise wide data-centric applications. That’s not necessarily in a single source or a single data warehouse.
5)Trust – It is part psychology and part technology. Companies are likely to outsource certain aspects of their data. For example, a bank might outsource check scans, but only to validate what’s already typed into the system at the bank ATM. Companies will chose to keep most of their core data on-premise, so they’re still going to need an on-premise data quality solution to manage it.

If these data Laundromats sound utopian, it is because they are. I believe we may see some type of outsourced data quality, especially when they need to access outside information, for example DUNS, UNSPC, but not for the critical core business assets of the bulk of their data, I would first run them through an on-premise cleansing cycle.

According to Chris Kanaracus from Computerworld, Gartner has lowered its worldwide enterprise software spending forecast from $229.2 billion, instead of the $231.2 billion that it predicted in September. This was impacted by "a combination of economic, technical and regional forces." aka the global economic meltdown. What's interesting to me in the article is that there are some recession proof elements in today's IT spend:

But software aimed at optimizing how organizations are run, such as BPM (business process management) and MDM (master data management) will fare better, Gartner said.

This is definitely true. In our own research, we've identified that Data Integration and Management helps companies reduce development costs by 30%, improve the speed of handling data by 50%, and improve business process execution times by 70%. These efficiency gains are critical in today's challenging global economic climate.

Learn more about this vibrant market in a new Oracle whitepaper, "The State of the Data Integration Market," compiled from leading analyst reports, articles, and a global survey to over 350 top companies.

Recently Timo Elliot, wrote four top reasons for "What Might Go Wrong in Business Intelligence in 2009". In his reason #2 he cites:

Corporate cutbacks, "thou shalt not buy anything" policies, and new levels of sign-off will encourage some people to attempt to do analysis without extra software investment: hand-coded data extraction in SQL, data manipulation using Excel macros, etc. Over time, the work involved in developing and maintaining these solutions will cost much more than purchased packages.

I couldn't agree more with why it might fail. BI can't succeed without a dedicated data integration fabric that can automate and manage data movement, data synchronization, and data quality. The costs for investing in Data Integration solutions can immediately provide unexpected returns. For example: by implementing ETL/ELT, costs can be saved in custom code and the SQL develepment associated with enterprise-class business intelligence projects.

I'd argue that another reason to think about why BI solutions might fail, is today that they aren't perceived as "actionable" and perceived more as a looking glass. That looking glass is a luxury if it can't immediately provide recourse for action. This goes back to my Data Integration point. If the data is automated already, then acting on it becomes swift, efficient and can lead to greater efficiencies, risk reduction and countless other business benefits.

Nevertheless, there are many smart people out there that can utilize these best practices effectively - or perhaps invent some new ones. BI (thanks to the help of data integration) can definitely succeed in today's gloom and doom markets. I'm definitely more optimistic and hence bullish on BI for 2009.

For more details on why Business Intelligence is on the rise, take a look at what we've identified in our "State of the Data Integration Market" report which outlines the rise of Data Integration and Management and how it's supporting more actionable and real-time BI inititiaves.

All I wanted for the holidays: Oracle Data Integrator 10.1.3.5

What's in the box?

Well what's notable is that ODI now gives its users an edge in Business Intelligence. Specifically, ODI can be used as the data integration platform of choice for OBI Analytic Applications using EBS 11i10 as a source. More on this detail can be found on this datasheet.

If you've been reading some earlier blogs on the importance of BI, you'll see why this is so compelling.

Got Global Quality Issues? Oracle Data Quality and Data Profiling now includes: Support for EBCDIC Code Pages, additional country support (Russia, Norway, Greece, Poland, Czech Republic, Finland, Luxembourg, etc). In addition DQ/DP includes an enhanced Schema Editor, New Expression builder operators (String Joining, Contains, etc.).

Trying to cut down the cost of integrating disparate applications? Try these new knowledge modules:

. Oracle eBusiness Suite Knowledge Modules (v11i10 and higher)
. Siebel CRM Knowledge Modules (v7.7 and higher)
. PeopleSoft Knowledge Modules (v8.x and higher)
. Oracle OLAP Knowledge Modules (v10g)
. Teradata Optimizations (new KMs leveraging v12 features)
. Row-by-row processing Knowledge Modules (for de-bugging)

Check it out now. Download it and be the first to take advantage of these new capabilities. Or check out the OTN page for the latest and greatest documentation and more.

So rather than my own ramblings of the best and worst in 2008, I thought I would point you to other folks who have also reviewed the landscape. Consider this a review of other year in reviews!

Number 5: Fernando Labastida’s Integration Blog. This isn't so much a year in review so much is about the year of Data Integration and Management. CIOs are all the buzz with it!

On to the top IT priorities for 2008. According to CIO Insight magazine, the number 1 and number 3 top priorities in terms of technology that will improve a company’s business strategy are business intelligence and data and application integration, at 44% and 29% respectively. Since business intelligence requires data integration tools to build the data warehouses that BI tools sit on top of, then two of the top five priorities regarding technology for business improvement for 2008 will require application integration tools.

Number 4: Dan Power's In one of the top 10 posts, Dan Power points to one of the keys to successful Master Data Management which I think is very true.

"Use a holistic approach – people, process, technology and information."

Number 3: Predicts 2009: Technology Changes Will Shape the Future of Data Management and Integration. This isn't a top 10 blog, but I thought it was relevant as sometimes the best way to look back is to look forward - or is it the other way around?

Number 2:Top 10 Disruptive Technologies by Billy Cripe. This is more focused on Enterprise 2.0 and content managment, but actually I thought it was a good prediction. It also has a really good photo!

Number 1: BI The Year in Review. Stephen Swoyer writes that the Data Integration market enjoyed healthy adoption:

Despite the tumult in the global financial markets, 2008 was relatively calm for BI professionals. In contrast to 2007, when the three biggest BI pure-play vendors (Hyperion, Business Objects, and Cognos) were acquired by larger, non-BI vendors, the year past was a sleepy one. There were the requisite acquisitions, to be sure, but nothing comparable to the domino-toppling wave of consolidation that swept the industry in 2007.

I think overall we saw a lot of interesting developments. I'm definitely looking forward to an interesting 2009.