Data Integration and Management

How do you get more trust in your data? As Tina Turner would possibly say: "What's Gov got to do with it?"

It has everything to do with it. But technology alone does not deliver trusted data. Information managers need to define what data means to their organizations through data governance. Data governance is analogous—though not identical—to SOA governance, a discipline that has saved many SOA implementations from certain failure.

I really think Mike's blog on the subject says it best when he states...on http://blogs.oracle.com/governance : “Governance, regardless of what 'specialty' you are talking about, is a series of activities associated with influencing actions and behavior of an environment. SOA governance, data, governance, process governance, application governance, etc are all related to, but not dependent on, one another. It's not that one encompasses the other, but rather the activities associated with each should work in conjunction with the rest of the governance discipline.”

Data governance helps define not only data quality rules but also the processes for how the rules are maintained, approved, and iterated. As companies scale and grow, these established processes are critical to managing the lifecycle of enterprise data-centric architectures.

Data governance must include multiple data quality and data management capabilities, as well as allow for the human element in implementing a governed data-centric environment.
For example, a company might define certain data as off-limits to a set of roles that is integrated across multiple data hubs. This type of governance can be implemented by combining identity management and data access services or through an entitlement policy that is executed at runtime. In other cases, data quality might require a complex set of business logic to be specified as a business rule, or business processes might automate a workflow of data exception management. In each example, governance processes are key to successful enterprise implementations...

...and key to adding more visibility and control (perhaps not love) but almost as priceless.

The growth of the Data Integration segment has exceeded market expectations as companies recognize the fundamental importance of unified enterprise data. The market is now expected to exceed US $3 billion by 2012. Because the business drivers for data integration are so compelling, it remains an investment area even in times of economic stress.

In an effort to help customers cut through the noise of this vibrant market, we completed a thorough analysis of the Data Integration market, drawing on leading analyst reports, articles, and a market survey to over 350 top companies. Thanks for submitting your feedback to us if you did! This white paper explores the rise of Data Integration, the current trajectory, and the top market trends to watch.

Also included in this paper:

- Data Integration market size, growth and maturity
- Top Business and IT drivers of Data Integration
- How Data integration matured from its extract, transform, and load (ETL) roots
- Influence Data warehousing, SOA, BI and MDM
- Emerging trends in Data Services, Actionable Business Intelligence
- Impact of of Data Management and Data Governance
- Best practices of data integration solutions at work

Also we included examples of next generation data warehousing, data federation, and real-time business intelligence for you! Data Integration has helped organizations save costs through IT consolidation and development efficiencies, improve customer intimacy with single views of customer data, and increase agility by better re-using and acting on real-time information.

Check it out today!

Ok we've seen these blogs all over the place now. We know we're in a recession. We know we want to do more with less in our IT budgets. Tell us something we don't already know. But this is taking it just way too far. Now I'm starting see something a bit more outragious. How IT technologies can prevent meltdowns from starting in the first place. Is it possible? Is it even relevant? I'll let you decide:

Todd Goldman (No relation to Goldman-Sachs) writes in his blog:

One of the reasons for the meltdown is that the banks don't have the ability to price these mortgage backed securities because they lack traceability back to the source. They can't tell you what mortgages map to a particular financial product and as a result, no one can say if a certain set of securities is better or worse than another set. Now this particular problem is all about data management. The ability to accurately show the lineage that links the underlying asset to a complex financial instrument based on that asset is not a skill that most banks have. As a result, this only inflames the lack of confidence that has been injected into the market due to the housing bubble bursting, people not being able to pay their variable rate mortgages or refinance those mortgages at a lower rate. So while all of those non-data effects cause concern, lack of good data management to price out the true cost of those effects only pours gasoline on an already raging fire.

My two cents... or make that 750 qua-trillion cents... Definitely definitely consider Data Management for your enterprise architecture, your SOA, your Business Intelligence applications, your data warehouse, your composite applications even... but... Hmmm... I think I need a physics PHD to understand how credit default swaps work.

If you think that Data Management can help for helping with the mortgage crisis, contact the Treasury Department today. But for now, I'm going to use it for preventing Enterprise IT meltdowns on main street.

In respose to a comment on our Data Integration blog, we're seeing that Data Integration as well as MDM tends to be an overloaded term. Not surprising! Let's dive into this in a little more detail and see some of the trends.

Today there is disagreement about the role that MDM plays in managing data-centric applications and the future role that it has in redefining data integration platforms. What does is it mean to master your data? Why is it essential to consider as part of a data integration platform?

Let's look at the existing definition of MDM which is focused on managing data per a specific domain. For example, customer data integration (CDI) hubs focused on the customer data domain, while product information management (PIM) products focuses on the product data domain. Domain-specific MDM addresses a single view of product, customer, supplier, site, or financial data depending on the need. There are multiple modes for these MDM domain models depending on the industry or the requirement.

In addition a separate thread of MDM development has emerged which is focused on downstream analytical MDM requirements and tends to be more data-domain agnostic. With domain-agnostic MDM, there are functional capabilities that relate directly to components found in data integration platforms. These include data movement, data synchronization, data quality, data federation, and especially data management, which take into consideration metadata management. This approach masters data for any domain - often seen as a single view of the truth. In this definition, MDM includes platform capabilities for creating (what many analysts have named) a 'single view of the business', see a good blog on the subject on TDWI. These MDM platform approaches require comprehensive data integration capabilities to ensure that all parts of the enterprise cooperate and work toward common goals.

A single view of the truth for all enterprise data might initially be perceived as a luxury, but it is an important obstacle that needs to be surmounted. Today’s companies continue to struggle in their MDM initiatives because most vendors have yet to deliver unified, comprehensive MDM solutions that combine both domain-specific and domain-agnostic aspects. In fact, platforms require significant customization and professional services to ensure a successful MDM implementation.

As a result, enterprise architects and data stewards should exercise caution before undertaking an MDM strategy without first implementing core data integration solutions that integrate their data-centric applications. Despite the chaos and uncertainty in the diverging MDM definitions, data integration can be seen as the cornerstone for successful data-centric architectures and provide authoritative master data for a single view of business.

You can read more survey and analyst information we collected on MDM and the importance of Data Integration platforms in our State of the Data Integration White Paper. The paper is available for download here: www.oracle.com/goto/ODI

Recently, Rick Sherman describes the pitfalls of misaligned data-centric architectures in
his article series:, the Accidental Architecture and Recovering from the Accidental Architecture.

A word of caution: do not get too wrapped up in the architecture. Some companies will get so fixated on the final architecture that they take months or years trying to develop it. The architecture is not the result of your BI/DW project, but rather a means to an end. Do not spend time on a monstrous, complicated architecture that solves world hunger; design something that you can start developing toward and that you can evolve over time.

I couldn’t agree more with what Rick is saying. I also have my own top 5 list of how to immediately capitalize on the return from Data Integration and Management investments while at the same time reducing development costs helping consolidate IT.

Here’s an excerpt from the State of the Data Integration Market White Paper where we polled over 350 global top companies to get to the bottom of what was bugging them: We identified these top 5 lessons learned:

- Avoid fragmented solutions
- Solve IT/Business alignment challenges first
- Consider data governance early in the process
- Implement data services for improved agility
- Start small, show incremental value, and repeat.

This last point, I’ll expand on because it’s one that I think is most important and circles back to what Rick is discussing as well. One of the recent lessons learned from SOA implementations is to start projects on a smaller scale—despite the urge to cross enterprise boundaries for immediate agility benefits. The same lesson applies to larger data warehouse, MDM, and BI projects that expand in scope across the company. The most successful data integration projects are ones that solve a manageable problem that exists across the organization, while still providing incremental value to the business. For example, using data services enables the incremental reuse of information by the processes and applications that need them for a particular project, while leaving existing infrastructure in place.

So try not to get wrapped up in the complexities of data. I welcome you to use some of the data integration tools available to insure against those nasty architecture accidents!