Data Integration and Management

So rather than my own ramblings of the best and worst in 2008, I thought I would point you to other folks who have also reviewed the landscape. Consider this a review of other year in reviews!

Number 5: Fernando Labastida’s Integration Blog. This isn't so much a year in review so much is about the year of Data Integration and Management. CIOs are all the buzz with it!

On to the top IT priorities for 2008. According to CIO Insight magazine, the number 1 and number 3 top priorities in terms of technology that will improve a company’s business strategy are business intelligence and data and application integration, at 44% and 29% respectively. Since business intelligence requires data integration tools to build the data warehouses that BI tools sit on top of, then two of the top five priorities regarding technology for business improvement for 2008 will require application integration tools.

Number 4: Dan Power's In one of the top 10 posts, Dan Power points to one of the keys to successful Master Data Management which I think is very true.

"Use a holistic approach – people, process, technology and information."

Number 3: Predicts 2009: Technology Changes Will Shape the Future of Data Management and Integration. This isn't a top 10 blog, but I thought it was relevant as sometimes the best way to look back is to look forward - or is it the other way around?

Number 2:Top 10 Disruptive Technologies by Billy Cripe. This is more focused on Enterprise 2.0 and content managment, but actually I thought it was a good prediction. It also has a really good photo!

Number 1: BI The Year in Review. Stephen Swoyer writes that the Data Integration market enjoyed healthy adoption:

Despite the tumult in the global financial markets, 2008 was relatively calm for BI professionals. In contrast to 2007, when the three biggest BI pure-play vendors (Hyperion, Business Objects, and Cognos) were acquired by larger, non-BI vendors, the year past was a sleepy one. There were the requisite acquisitions, to be sure, but nothing comparable to the domino-toppling wave of consolidation that swept the industry in 2007.

I think overall we saw a lot of interesting developments. I'm definitely looking forward to an interesting 2009.

All I wanted for the holidays: Oracle Data Integrator 10.1.3.5

What's in the box?

Well what's notable is that ODI now gives its users an edge in Business Intelligence. Specifically, ODI can be used as the data integration platform of choice for OBI Analytic Applications using EBS 11i10 as a source. More on this detail can be found on this datasheet.

If you've been reading some earlier blogs on the importance of BI, you'll see why this is so compelling.

Got Global Quality Issues? Oracle Data Quality and Data Profiling now includes: Support for EBCDIC Code Pages, additional country support (Russia, Norway, Greece, Poland, Czech Republic, Finland, Luxembourg, etc). In addition DQ/DP includes an enhanced Schema Editor, New Expression builder operators (String Joining, Contains, etc.).

Trying to cut down the cost of integrating disparate applications? Try these new knowledge modules:

. Oracle eBusiness Suite Knowledge Modules (v11i10 and higher)
. Siebel CRM Knowledge Modules (v7.7 and higher)
. PeopleSoft Knowledge Modules (v8.x and higher)
. Oracle OLAP Knowledge Modules (v10g)
. Teradata Optimizations (new KMs leveraging v12 features)
. Row-by-row processing Knowledge Modules (for de-bugging)

Check it out now. Download it and be the first to take advantage of these new capabilities. Or check out the OTN page for the latest and greatest documentation and more.

Recently Timo Elliot, wrote four top reasons for "What Might Go Wrong in Business Intelligence in 2009". In his reason #2 he cites:

Corporate cutbacks, "thou shalt not buy anything" policies, and new levels of sign-off will encourage some people to attempt to do analysis without extra software investment: hand-coded data extraction in SQL, data manipulation using Excel macros, etc. Over time, the work involved in developing and maintaining these solutions will cost much more than purchased packages.

I couldn't agree more with why it might fail. BI can't succeed without a dedicated data integration fabric that can automate and manage data movement, data synchronization, and data quality. The costs for investing in Data Integration solutions can immediately provide unexpected returns. For example: by implementing ETL/ELT, costs can be saved in custom code and the SQL develepment associated with enterprise-class business intelligence projects.

I'd argue that another reason to think about why BI solutions might fail, is today that they aren't perceived as "actionable" and perceived more as a looking glass. That looking glass is a luxury if it can't immediately provide recourse for action. This goes back to my Data Integration point. If the data is automated already, then acting on it becomes swift, efficient and can lead to greater efficiencies, risk reduction and countless other business benefits.

Nevertheless, there are many smart people out there that can utilize these best practices effectively - or perhaps invent some new ones. BI (thanks to the help of data integration) can definitely succeed in today's gloom and doom markets. I'm definitely more optimistic and hence bullish on BI for 2009.

For more details on why Business Intelligence is on the rise, take a look at what we've identified in our "State of the Data Integration Market" report which outlines the rise of Data Integration and Management and how it's supporting more actionable and real-time BI inititiaves.

According to Chris Kanaracus from Computerworld, Gartner has lowered its worldwide enterprise software spending forecast from $229.2 billion, instead of the $231.2 billion that it predicted in September. This was impacted by "a combination of economic, technical and regional forces." aka the global economic meltdown. What's interesting to me in the article is that there are some recession proof elements in today's IT spend:

But software aimed at optimizing how organizations are run, such as BPM (business process management) and MDM (master data management) will fare better, Gartner said.

This is definitely true. In our own research, we've identified that Data Integration and Management helps companies reduce development costs by 30%, improve the speed of handling data by 50%, and improve business process execution times by 70%. These efficiency gains are critical in today's challenging global economic climate.

Learn more about this vibrant market in a new Oracle whitepaper, "The State of the Data Integration Market," compiled from leading analyst reports, articles, and a global survey to over 350 top companies.

There’s been some interesting discussion around what’s next for data quality and the fascinating challenges of cleaning data for data warehouses and business intelligence applications. I am always intrigued by blogs that discuss the challenges of data management and applying cleansing principles for complex data-centric applications across the enterprise. However, I’m dismayed by discussions that quickly jump to the conclusion of out-sourcing data quality as a software-as-a -service model. For example David Rosenberg writes on his blog:

Look for the emergence of third party B2B integration and commerce management service providers that support data entry and validation for all trading partners. Integrated suites of direct system-to-system integration and Web portal services will be supplemented with combined e-mail and smart-form technologies solving the data quality problem associated with paper-based exchanges with small and occasional trading partners.

While it sounds good on paper, I think this is more marketing spin than a realistic use case. I think we can meet the goal of achieving clean, trusted authoritative data without going off-premise. When we ask companies to deliver their most critical asset into 3rd party hands, it’s going to lead to more challenges that aren’t easily solved. Terabytes of data aren’t easily moved like sacks of dirty laundry. here are 5 reasons why the business model of outsourced data quality is ahead of its time:

1) Moving Data is hard – Moving terabytes of information off-premise –once- can be challenging enough, moving it as changes occur is even more challenging.
2) Auditing – to turn bad data good, means lots of changes. Keeping track of these changes and offering roll-back capabilities and full auditing is critical. How can these be easily managed when they are off-premise?
3) Customization– every company is unique how they approach data, even data like address information which would seem commonplace. Most on-premise data quality engine solutions have some type of customizable rules approach whereas many 3rd party solutions are using generic approaches.
4) Profiling – the forgotten aspect of data cleansing is first understanding and seeing the stain. Off-premise data cleansing solutions assume that the data needs cleansing, but the element of profiling needs to be applied on-premise within the enterprise wide data-centric applications. That’s not necessarily in a single source or a single data warehouse.
5)Trust – It is part psychology and part technology. Companies are likely to outsource certain aspects of their data. For example, a bank might outsource check scans, but only to validate what’s already typed into the system at the bank ATM. Companies will chose to keep most of their core data on-premise, so they’re still going to need an on-premise data quality solution to manage it.

If these data Laundromats sound utopian, it is because they are. I believe we may see some type of outsourced data quality, especially when they need to access outside information, for example DUNS, UNSPC, but not for the critical core business assets of the bulk of their data, I would first run them through an on-premise cleansing cycle.

Recently, Rick Sherman describes the pitfalls of misaligned data-centric architectures in
his article series:, the Accidental Architecture and Recovering from the Accidental Architecture.

A word of caution: do not get too wrapped up in the architecture. Some companies will get so fixated on the final architecture that they take months or years trying to develop it. The architecture is not the result of your BI/DW project, but rather a means to an end. Do not spend time on a monstrous, complicated architecture that solves world hunger; design something that you can start developing toward and that you can evolve over time.

I couldn’t agree more with what Rick is saying. I also have my own top 5 list of how to immediately capitalize on the return from Data Integration and Management investments while at the same time reducing development costs helping consolidate IT.

Here’s an excerpt from the State of the Data Integration Market White Paper where we polled over 350 global top companies to get to the bottom of what was bugging them: We identified these top 5 lessons learned:

- Avoid fragmented solutions
- Solve IT/Business alignment challenges first
- Consider data governance early in the process
- Implement data services for improved agility
- Start small, show incremental value, and repeat.

This last point, I’ll expand on because it’s one that I think is most important and circles back to what Rick is discussing as well. One of the recent lessons learned from SOA implementations is to start projects on a smaller scale—despite the urge to cross enterprise boundaries for immediate agility benefits. The same lesson applies to larger data warehouse, MDM, and BI projects that expand in scope across the company. The most successful data integration projects are ones that solve a manageable problem that exists across the organization, while still providing incremental value to the business. For example, using data services enables the incremental reuse of information by the processes and applications that need them for a particular project, while leaving existing infrastructure in place.

So try not to get wrapped up in the complexities of data. I welcome you to use some of the data integration tools available to insure against those nasty architecture accidents!

In respose to a comment on our Data Integration blog, we're seeing that Data Integration as well as MDM tends to be an overloaded term. Not surprising! Let's dive into this in a little more detail and see some of the trends.

Today there is disagreement about the role that MDM plays in managing data-centric applications and the future role that it has in redefining data integration platforms. What does is it mean to master your data? Why is it essential to consider as part of a data integration platform?

Let's look at the existing definition of MDM which is focused on managing data per a specific domain. For example, customer data integration (CDI) hubs focused on the customer data domain, while product information management (PIM) products focuses on the product data domain. Domain-specific MDM addresses a single view of product, customer, supplier, site, or financial data depending on the need. There are multiple modes for these MDM domain models depending on the industry or the requirement.

In addition a separate thread of MDM development has emerged which is focused on downstream analytical MDM requirements and tends to be more data-domain agnostic. With domain-agnostic MDM, there are functional capabilities that relate directly to components found in data integration platforms. These include data movement, data synchronization, data quality, data federation, and especially data management, which take into consideration metadata management. This approach masters data for any domain - often seen as a single view of the truth. In this definition, MDM includes platform capabilities for creating (what many analysts have named) a 'single view of the business', see a good blog on the subject on TDWI. These MDM platform approaches require comprehensive data integration capabilities to ensure that all parts of the enterprise cooperate and work toward common goals.

A single view of the truth for all enterprise data might initially be perceived as a luxury, but it is an important obstacle that needs to be surmounted. Today’s companies continue to struggle in their MDM initiatives because most vendors have yet to deliver unified, comprehensive MDM solutions that combine both domain-specific and domain-agnostic aspects. In fact, platforms require significant customization and professional services to ensure a successful MDM implementation.

As a result, enterprise architects and data stewards should exercise caution before undertaking an MDM strategy without first implementing core data integration solutions that integrate their data-centric applications. Despite the chaos and uncertainty in the diverging MDM definitions, data integration can be seen as the cornerstone for successful data-centric architectures and provide authoritative master data for a single view of business.

You can read more survey and analyst information we collected on MDM and the importance of Data Integration platforms in our State of the Data Integration White Paper. The paper is available for download here: www.oracle.com/goto/ODI

Ok we've seen these blogs all over the place now. We know we're in a recession. We know we want to do more with less in our IT budgets. Tell us something we don't already know. But this is taking it just way too far. Now I'm starting see something a bit more outragious. How IT technologies can prevent meltdowns from starting in the first place. Is it possible? Is it even relevant? I'll let you decide:

Todd Goldman (No relation to Goldman-Sachs) writes in his blog:

One of the reasons for the meltdown is that the banks don't have the ability to price these mortgage backed securities because they lack traceability back to the source. They can't tell you what mortgages map to a particular financial product and as a result, no one can say if a certain set of securities is better or worse than another set. Now this particular problem is all about data management. The ability to accurately show the lineage that links the underlying asset to a complex financial instrument based on that asset is not a skill that most banks have. As a result, this only inflames the lack of confidence that has been injected into the market due to the housing bubble bursting, people not being able to pay their variable rate mortgages or refinance those mortgages at a lower rate. So while all of those non-data effects cause concern, lack of good data management to price out the true cost of those effects only pours gasoline on an already raging fire.

My two cents... or make that 750 qua-trillion cents... Definitely definitely consider Data Management for your enterprise architecture, your SOA, your Business Intelligence applications, your data warehouse, your composite applications even... but... Hmmm... I think I need a physics PHD to understand how credit default swaps work.

If you think that Data Management can help for helping with the mortgage crisis, contact the Treasury Department today. But for now, I'm going to use it for preventing Enterprise IT meltdowns on main street.

The growth of the Data Integration segment has exceeded market expectations as companies recognize the fundamental importance of unified enterprise data. The market is now expected to exceed US $3 billion by 2012. Because the business drivers for data integration are so compelling, it remains an investment area even in times of economic stress.

In an effort to help customers cut through the noise of this vibrant market, we completed a thorough analysis of the Data Integration market, drawing on leading analyst reports, articles, and a market survey to over 350 top companies. Thanks for submitting your feedback to us if you did! This white paper explores the rise of Data Integration, the current trajectory, and the top market trends to watch.

Also included in this paper:

- Data Integration market size, growth and maturity
- Top Business and IT drivers of Data Integration
- How Data integration matured from its extract, transform, and load (ETL) roots
- Influence Data warehousing, SOA, BI and MDM
- Emerging trends in Data Services, Actionable Business Intelligence
- Impact of of Data Management and Data Governance
- Best practices of data integration solutions at work

Also we included examples of next generation data warehousing, data federation, and real-time business intelligence for you! Data Integration has helped organizations save costs through IT consolidation and development efficiencies, improve customer intimacy with single views of customer data, and increase agility by better re-using and acting on real-time information.

Check it out today!

How do you get more trust in your data? As Tina Turner would possibly say: "What's Gov got to do with it?"

It has everything to do with it. But technology alone does not deliver trusted data. Information managers need to define what data means to their organizations through data governance. Data governance is analogous—though not identical—to SOA governance, a discipline that has saved many SOA implementations from certain failure.

I really think Mike's blog on the subject says it best when he states...on http://blogs.oracle.com/governance : “Governance, regardless of what 'specialty' you are talking about, is a series of activities associated with influencing actions and behavior of an environment. SOA governance, data, governance, process governance, application governance, etc are all related to, but not dependent on, one another. It's not that one encompasses the other, but rather the activities associated with each should work in conjunction with the rest of the governance discipline.”

Data governance helps define not only data quality rules but also the processes for how the rules are maintained, approved, and iterated. As companies scale and grow, these established processes are critical to managing the lifecycle of enterprise data-centric architectures.

Data governance must include multiple data quality and data management capabilities, as well as allow for the human element in implementing a governed data-centric environment.
For example, a company might define certain data as off-limits to a set of roles that is integrated across multiple data hubs. This type of governance can be implemented by combining identity management and data access services or through an entitlement policy that is executed at runtime. In other cases, data quality might require a complex set of business logic to be specified as a business rule, or business processes might automate a workflow of data exception management. In each example, governance processes are key to successful enterprise implementations...

...and key to adding more visibility and control (perhaps not love) but almost as priceless.