Saturday Aug 31, 2013

How Important Is Metadata To An Information Management Strategy?

It's been a while since my first post and a lot has happened. I have now formed an Information Management Architecture team within our European Data Integration Solutions team to engage more strategically with our customers and partners around helping them to develop their information management strategy.

I'll be discussing aspects of information management strategy in this blog along the lines of the first blog posting. Let's start with a subject that always starts an interesting debate - metadata.

What is Metadata?

Metadata is 'data about data' and can be divided into four basic classifications:

  • Business Metadata - the business meaning of data. It includes business definitions of the objects and metrics, hierarchies, business rules, and aggregation rules.

  • Operational Metadata - Operational metadata stores information about who accessed what and when.

  • Technical Metadata - Technical Metadata describes the data structures and formats such as table types, data types, indexes, and partitioning method.

  • Process Metadata - Process Metadata describes the data input process.

Typical Use Cases

  • Introspection - discovering and harvesting of information into a metadata repository

  • Impact Analysis - analyse dependencies across the architecture. Requires end to end view of interdependencies

  • Data Lineage - description of the origins of a piece of data and the proves by which it arrived in the database. The 'provenance' or 'pedigree.'

Why Is Metadata Important To An Organization?

Accurate and comprehensive metadata enables the evaluation of an architecture's efficiency and eases the process of changing or evolving the architecture. For example, when you want to change a particular component then what is the impact on other components in the architecture?

Metadata can have a key part to play in regulatory compliance initiatives or certification processes to clearly demonstrate what components exist in the organization's architecture, their capabilities and interdependencies.

Information and data management initiatives such as common reference data across an organization as part of an Master Data Management strategy. A clean, accurate view of corporate reference data requires clean, accurate metadata.

Enabling cost savings and efficiency through better insight and more efficient troubleshooting and impact analysis and helping to avoid data duplication.

What Approaches Do Customers Take?

Shared Metadata Repository

A common repository used by different tools to share metadata across the toolsets. This is seen as the 'nirvana' of data integration and has many advantages but has some practical limitations that need to be overcome:

  • There are many different roles involved in an integration strategy from business users, Data Stewards, ETL Developers to report builders. The tool needs to cater for all these types of users or each individual tool needs to provide a specific interface into the metadata repository.

  • Common naming standards and definitions need to be defined, so all users understand they are addressing the same underlying data.

Individual product innovation should not be stifled by having to adhere to a common metadata repository such as having to wait for a new release of the repository before releasing a new version of the tool or not being able to release new functionality because the common repository won't support it.

Metadata Bridge

A common view of metadata across different tools holding metadata about the individual metadata repositories. This is a more loosely coupled approach than a common repository and allows flexibility in introducing new tools and technologies and a more agile approach to data integration as well a single point of reference for regulatory reporting, for example.

There is an overhead in administration and making sure there is an up to date view of metadata information in a timely manner.

Specific Metadata Repository Integration Between Tools

Where tools have their own individual metadata repositories for describing functionality and capability within their tool set, specific points of integration can be developed or the metadata repository extended where it adds value to the business.

For example, sharing metadata between an ETL tools and a reporting tool would achieve a level of data lineage from report to data sources if the reporting tool could interrogate the ETL tool's metadata or import metadata from the ETL tool. A specific development is needed between the tools in this case.

What is Oracle's Approach?

The Oracle technology stack is driven by metadata and all the types of metadata mentioned above from metadata within the business applications to technical metadata within the database and Fusion Middleware products. A 'one size fits all' approach is not viable across the whole stack but there are some standards being used and some specific points of integration. Let's take a look at some of these:

Oracle Metadata Services (MDS)

MDS is a metadata standard across the Fusion Middleware stack and can be used as a central metadata store for multiple Fusion Middleware components such as SOA Suite, Oracle Application Development Framework (ADF) and Oracle WebCenter. It is described in detail in the following article Storing SCA Metadata in the Oracle Metadata Services Repository.

Oracle Enterprise Repository

A 'master' repository for runtime services, SOA assets and their dependencies as part of a SOA Governance strategy. It provides automated artifact harvesting and a central point for analysing relationships between components for impact analysis as well as management analytics. See Oracle Enterprise Repository for more information.

Oracle Data Integration Solutions and Metadata

Oracle Data Integrator (ODI) is a metadata driven tool and has its own metadata repository, which is extensible via FlexFields and has a Smart Export/Import utility to easily exchange data with other tools. A specific integration between ODI and Oracle Business Intelligence (OBIEE) has been done to share metadata for data lineage and impact analysis. this is described in the white paper Managing Metadata With Oracle Data Integrator.

Oracle Enterprise Data Quality (EDQ) also has its own repository but can share metadata with ODI via Export/Import functions and EDQ functionality can be directly included within ODI process flows.

ODI and Application Adapter for Hadoop

ODI builds Hadoop metadata through it Knowledge Modules and the Application Adapter for Hadoop. Map Reduce jobs can be created, orchestrated and coordinated through an easy to use interface and information loaded through to an Oracle relational database using ODI and the Oracle Loader for Hadoop. This enables shared metadata between Hadoop and Oracle and a bridge between the two types of technology as part of an overal information management strategy. For more information see Oracle Data Integrator Application Adapter for Hadoop.

Metadata Bridging Technology

Third party tools are available to act as a bridge or central point for multiple metadata repositories. An example is the Meta Integration Model Bridge (MIMB) from Meta Integration Technology Inc., which is the foundation technology for several vendor's metadata solutions such as IBM and Informatica.


Here's a few key points to finish with: 

Take a Pragmatic Approach - Specific metadata solutions for specific use cases based on defined business benefits

Integration at a Product level Where Possible - Look for specific points of integration at a solution level to reduce development effort and overall product maintenance

Metadata Sharing Promotes Operational Efficiency - Share metadata between data structures as well as with SOA and Middleware components will help with troubleshooting and impact analysis

Better Insight and Cost Savings - Insight into relationships between data structures to analyse the cost of change and make it easier to know how the data structures have been derived

Bridge the divide between structured and unstructured information - As part of a Big Data strategy to gain a better understanding of your overall information architecture through a standardised metadata approach.

I hope this has given an insight into how important metadata can be to an information management strategy and how Oracle take a pragmatic approach to metadata management. Watch this space to see how Oracle will continue to develop this strategy.

Special thanks to Stephen Bennett of Oracle's Global Enterprise Architecture team for some of the definitions around metadata from his soon to be published Information Management Reference Architecture white paper.


Thursday May 02, 2013

What is Oracle's Unique Value Proposition?

This is my first Blog entry on Oracle Blogs, so where do I start? How do you start a Blog? What do you deem important? What would people be interested in? I've been at Oracle for over sixteen years and worked across a range of products and industries and seen a lot of changes and I'd like to share some views and experiences. Let's start at the beginning.

The first question I get asked sometimes by customers and a question I ask myself frequently is 'What is Oracle's Unique Value Proposition?' How do we differentiate ourselves from the competition? Why would any customer make an investment in Oracle? The corporate message is about 'Simplifying IT' by providing a complete stack of software, vertically integrated, engineered to work together with a choice of deployment models - private cloud, public cloud or both.

 I believe like Aristotle that "The whole is greater than the sum of its parts." The value of Oracle is when you can leverage the sum of its parts into something bigger and better. This is the main theme that will run through this blog. Let's look at some of the specific areas where our products are integrated to work together to provide value to our customers.

Data Integration Architectures

In my domain area of Data Integration, there are many different parts of our solution sets that touch other domain areas and form part of a greater solution set. This can be summarised below in a platform view of our solutions:

I would describe this as a product architecture view and this is by no means the full capability but illustrates some of the main components of our platform for managing information. It can be used to show where we have developed specific integration points between products and how they work together. The following describes some of the specific integration points between the solution areas on the diagram above.

Oracle Data Integration

  • Integration between Oracle GoldenGate (OGG) and Oracle Data Integrator (ODI) for “real-time” data feeds into target operational data stores or enterprise data warehouses
  • Integration between ODI and Enterprise Data Quality (EDQ) to provide data quality processing as part of an ETL flow
  • Integration between EDQ and Master Data Management (MDM) to provide data quality services as part of operational MDM processes 
  • Sharing of metadata between key components such as ODI and the Oracle Business Intelligence tools (OBIEE) for data lineage and impact analysis

Oracle Master Data Management 

  • Integration between EDQ and Master Data Management (MDM) to provide data quality services as part of operational MDM processes
  • Integration between MDM and ODI for bulk loading operations across MDM infrastructure
  • Integration between MDM and OBIEE for insight into business information

 Oracle Enterprise Performance Management 

  • Integration between EDQ and Master Data Management (MDM) to provide data quality services as part of operational MDM processes
  • Integration between MDM and ODI for bulk loading operations across MDM infrastructure
  • Integration between MDM and OBIEE for insight into business information

Oracle Business Intelligence 

  • Integration between MDM and OBIEE for insight into business information
  • Sharing of metadata between key components such as ODI and the Oracle Business Intelligence tools (OBIEE) for data lineage and impact analysis
  • Integration between MDM and OBIEE for insight into business information
  • Integration of ODI as part of BI Applications – pre-built BI as part of Oracle Applications

Oracle Data Warehousing and Advanced Analytics

  • All tool repositories across all product sets are stored and run in the Oracle Database (RDBMS)
  • Integration between RDBMS Data Warehousing platform and Oracle Data Integrator (ODI) for ETL/|ELT code generation optimised for the Oracle database
  • ODI can run its optimised ELT code directly on the target Oracle Database
  • Oracle GoldenGate Capture and Delivery processes optimised for the Oracle Database
  • Integrated management framework – Oracle Enterprise Manager – to manage all components of the platform
  • Integrated security framework to secure all components of the platform


We'll look at some specific use cases and drill into some of the product architectures in future posts. Well, I've got to start somewhere!

Bye for now.

Ian Thomas - Director of Technology & Architecture, EMEA Data Integration Solutions


Collateral and content related to how Oracle Data Integration Solutions are architected and how they fit into other solution areas.


« July 2016