Friday Apr 25, 2014

Big Data Challenges & Considerations

By: Murad Fatehali

While 'Big Data' is dominating a lot of media and executive attention (it's a Top-5 Initiative according to IDC Retail Insights 2014 Predictions), the underlying considerations & challenges of Big Data are unfortunately getting trivialized.  As corporations and people continue to make the Internet a more fundamental way of life (from social and transactional perspectives), the scope of the underlying data that gets created, stored, retrieved, and analyzed will commensurately and exponentially grow as well. Although many factors and assumptions go into Big Data discussions, outlined below are five pivotal dimensions:

1. Size: This is the most obvious and most talked about factor.  Everybody seems to understand that given the rise of the Internet, coupled with the plethora of apps in the hands of consumers and users, growth in the scale and volume of data is truly astonishing (some say, quantity is doubling every 2 years). However, what gets neglected are the challenges related to storage and analysis of vast volumes of data, captured in a variety of formats, for example: text-based (tweets, SMSs), graphically illustrated (pictures and drawings), and audio-visually represented (podcasts, movies, etc.).  For anyone interested in making sense of Big Data, how efficiently the data gets stored and retrieved will be key factors - no wonder we are seeing an increase in the number and capacity of purpose-built storage systems and lightening-fast data-crunching machines.

2. Sources: Usually Big Data discussions refer to ‘external’ sources like the Internet (Facebook, Twitter) and media (digital or otherwise), but often overlooked are the ‘internal’ data sources that can be scraped for Insight - these would be the transactional and operational systems supporting multiple marketing, sales, fulfillment, and service systems, in addition to the company’s own BI/reporting systems.  Ask any CMO and they will explain to you the widening gaps in data due to lack of integration and data governance.  Many companies struggle when answering how many customers they have, not to mention the difficulties in identifying those most profitable or the most loyal. Since the data opportunities inside the company have not been fully explored, diving into the external sources of Big Data without adequate forethought, can potentially not only add costs, but also complexity and confusion. In our engagement with customers, while ERP systems & POS data remain good inputs into a company’s Business Intelligence and reporting efforts, more and more we see executives asking for data from other sources to be able to develop ‘personalized’ offers and solutions, for example:

a. Machine usage data about performance and diagnostics using sensors (vehicle-to-smartphone connectivity devices for smarter driving from:,,, etc.)

b. Medical data from patients/public records and research studies (fitbit and wearable technologies, research findings, etc.)

c. Geographic and telemetric data from devices, maps, GPS signals, user tags and flags (Google maps, in-vehicle trackers, etc.)

d. Smartphone check-ins (four-square, etc.)

e. Social networks that capture trends (Twitter, etc.)

f. Content sites (Wikipedia, etc.)

3. Speed: The flow of data is something that can no longer go unnoticed - it is instant and constant.  According to Google CEO Eric Schmidt, “There were 5 exabytes of information created between the dawn of civilization through 2003, but that much information is now created every 2 days, and the pace is increasing.”  It is clear that the rate of incoming data can quickly alter any trends captured off historical data and render meaningless any campaigns that don’t take into account real-time response tracking.   For example, without compute-intensive servers tied-into the large-scale infrastructure hosting Facebook posts, Twitter feeds, and Google searches it would be impossible to capitalize on fads, keep pace with up-to-the-minute news, understand critical events, and predict emerging trends [case in point: who knew that tracking the number of flu queries on Google can help predict localized outbreaks or that analyzing global weather patterns can help predict harvest outputs or crop failures]. The implications are huge for those who can robustly and quickly find data relationships: commonality, lineage, and correlation are going to be big differentiators.

4. Standards: Unlike organizational transactions designed to meet compliance standards for financial reporting, Big Data does not need to conform to any such rules or convention. The vast majority (some say as much as 90%) of data coming from uploads-social network chatter-comments-likes or tweets is neither structured, nor precise, not to mention anything about data reliability (accuracy). The sheer variety of posts representing wide-ranging interests/agendas across multiple sites/sources being repeated/re-circulated requires significant analytical prowess, speed, and talent to make intelligible.

5. Strategy: Big Data has the potential to help a company truly cross channels (understand personal profiles and preferences) and deliver world-class experiences to their customers; it can also help inform a company’s strategy by making sense of wide-spread data coming from public and private networks, internal and external systems, and individual and institutional sources.  How Big Data plays into a company’s success will depend on the priorities of its leadership - and their willingness to make data-driven decisions a reality, and turn Big Data into more than just a buzzword. Linked-In, for example, is analyzing vast quantities of data to generate billions of personalized recommendations every week – no small feat when looking through the silo and antiquated lens of yesterday’s IT landscape. According to Baseline, a 10% increase in data accessibility translates into an additional $65.7M in net income for a typical Fortune 1000 company.

Big Data challenges and considerations outlined above are further impacted by availability of skilled talent - people who understand data in all its forms and can help govern and master it.  Additionally, ethical and legal challenges further compound the problem since perspectives differ on what is personal, private, and sensitive information.  To be sure, none of these Big Data challenges are expected to get resolved anytime soon – all the more reason, therefore, for executives to be cognizant of the five Big Data considerations (size, sources, speed, standards, and strategy) as they chart new ground in their Big Data journey to unlock its value.

For additional details on the Insight program, please visit:

Murad Fatehali is a Senior Director with the Insight & Customer Strategy Team at Oracle Corporation; he focuses on helping Oracle customers solve Data Strategy & Integration challenges.

Friday Mar 21, 2014

Master Data Management: How to Avoid Big Mistakes in Big Data

Big Data Quality MDM

Master Data Management: How to Avoid Big Mistakes in Big Data

The paradigm-changing potential benefits of big data can't be overstated—but big changes can deliver big risks as well. For example, exploding data volumes naturally create a corresponding increase in data correlations, but as leading experts warn, correlations should not be mistaken for causes.

To avoid drawing the wrong conclusions from big data, organizations first need a way to assemble reliable master data to analyze. Then they need a way to put those conclusions and that data to work operationally, in the systems that govern and facilitate their day-to-day operations.

Master data management (MDM) helps deliver insightful information in context to aid decision-making. It can be used to filter big data, isolating and identifying key entities and shrinking the dataset to a manageable size for parsing, tagging, and associating with operational system records. And it provides the key intersecting point that enables organizations to map big data results to operational systems that are built on relational databases and structured information.

Adopting master data management capabilities helps organizations create consolidated, consistent, and authoritative master data across the enterprise, enabling the distribution of master information to all operational and analytical applications, including those that contain customer, product, supplier, site, and financial information.

Oracle Master Data Management drives results by delivering the ability to cleanse, govern, and manage the quality and lifecycle of master data.

To learn more about the importance of MDM as an underlying technology that facilitates big data initiatives, read an in-depth Oracle C-Central article, "Masters of the Data: CIOs Tune into the Importance of Data Quality, Data Governance, and Master Data Management."

And don't miss the new Oracle MDM resource center. Visit today to download white papers, read customer stories, view videos, and learn more about the full range of features for ensuring data quality and mastering data in the key domains of customer, product, supplier, site and financial data.

Friday Mar 07, 2014

Master Data Management and Big Data: Perfect Together!

By Gino Fortunato

Master Data Management and Big Data: Perfect together!

The 'hot' button around gathering customer insight is Big Data.  And justifiably so.  Using Big Data is a great way to harness previously unusable data to look for patterns in the data crumbs that customers leave behind.  By rapidly processing this data in real time, Big Data allows customer insight that was previously impossible. 

Much of this insight is statistical.  Customers have similar patterns.  They abandon shopping carts when something is out of stock or when they see the final price.  At least compared to other points of the buying process.  It's just human nature.  By using statistical and other techniques, driving insight about what the customer is doing, or might be doing next, can drive a lot of value. 

But wouldn't it be great to use that Big Data insight along with what you already know about that customer?  That's where MDM comes in.  MDM is the spot to operationalize what you already know about the customer.  By using what you already know, plus the insight you have gleaned from Big Data, you can make informed decisions about how to react to the customer's next click.  And do it in real time.   To properly use the insight, it is necessary to properly idenfity the customer.  Again, an area that master data management can help.  With it's built in identity resolution capabilities, MDM can help in two ways.  One is to add to what is being derived based on the Big Data source.  The other is to prevent mistakes when the statistical analysis is wrong.  For example, the customer surfing the gaming site may be grouped into a category that has a number of traits.  One of those traits might be an expected age range.  But if the organization knew the birthdate of the person was outside that age range, they can propose different cross sell/ upsell possiblities and perhaps lead to the discovery of a new subcategory to further open the market.

To learn more about the importance of MDM as an underlying technology that facilitates big data initiatives, read an in-depth Oracle C-Central article, "Masters of the Data: CIOs Tune into the Importance of Data Quality, Data Governance, and Master Data Management."

Tuesday Aug 28, 2012

Master Data Management – A Foundation for Big Data Analysis

While Master Data Management has crossed the proverbial chasm and is on its way to becoming mainstream, businesses are being hammered by a new megatrend called Big Data. Big Data is characterized by massive volumes, its high frequency, the variety of less structured data sources such as email, sensors, smart meters, social networks, and Weblogs, and the need to analyze vast amounts of data to determine value to improve upon management decisions.

Businesses that have embraced MDM to get a single, enriched and unified view of Master data by resolving semantic discrepancies and augmenting the explicit master data information from within the enterprise with implicit data from outside the enterprise like social profiles will have a leg up in embracing Big Data solutions. This is especially true for large and medium-sized businesses in industries like Retail, Communications, Financial Services, etc that would find it very challenging to get comprehensive analytical coverage and derive long-term success without resolving the limitations of the heterogeneous topology that leads to disparate, fragmented and incomplete master data.

For analytical success from Big Data or in other words ROI from Big Data Investments, businesses need to acquire, organize and analyze the deluge of data to make better decisions. There will need to be a coexistence of structured and unstructured data and to maintain a tight link between the two to extract maximum insights. MDM is the catalyst that helps maintain that tight linkage by providing an understanding about the identity, characteristics of Persons, Companies, Products, Suppliers, etc. associated with the Big Data and thereby help accelerate ROI.

In my next post I will discuss about patterns for co-existing Big Data Solutions and MDM.

Feel free to provide comments and thoughts on above as well as Integration or Architectural patterns.  For more information on Oracle Master Data Management click here.


Get the latest on all things related to Oracle Master Data Management. Join Oracle's MDM Community today.

Follow us on twitter Catch Us on YouTube


« August 2016