Four Steps to a Successful Big Data Project

By adopting a systematic approach, companies can move from the promise of big data to valuable analytics.

by Christopher Sowa

May 2015

Too often, investments in big data yield little value. Companies tend to get stuck in constant proof-of-concept efforts and investments that never capture the full potential of their big data vision.


Big data projects often fall into two proof-of-concept traps.

The IT science project:

This is often led by architects who decide to create a data pool, or even a data lake, from data that no one has had any need to use previously. These architects hope that business users and value will magically appear from newly accessible data, and that they can get funding for an even grander big data project. Not surprisingly, what often results is a big data investment that yields low user adoption, a negligible return on investment, and little hope of continued funding.

Only by targeting strategic data domains of meaningful value to the business will big data have a big impact.”

Thinking too small for big data:

This data trap is created by business users. Rather than viewing the potential value of a big data lake, business users more often target just a few variables for correlation or for multivariate analysis as they do with traditional manual data analysis. Once this problem is solved, few additional queries of value can be run, so little value comes from the big data investment. For example, executives at one large energy company seemed consumed by using big data to reduce rogue procurement where only a couple of million of dollars of benefits were possible. Instead, they could have figured out how to use big data to improve their energy extraction and compliance processes, which were core to their business. In another example, executives at a large bank focused on using big data to reduce employee attrition, instead of applying big data to improve talent management and compliance with human resources policies, where the impact of tens of millions of dollars was possible.

To move from the promise of big data to smart analytics that yield results, companies need to follow a systematic approach focused on deriving outcome-driven analytics from both structured and unstructured data. To be successful in making the right big data investments, companies should follow four steps:

1. Determine the most important business objectives and the largest or most important business problems to solve to achieve these objectives. For example, a financial services company should be thinking not only about how to lower the cost of stress testing, but also about how to increase the return on risk from an asset class, or the risk-adjusted return from a group of asset classes (in addition to stress testing reporting).

2. Establish a cross-functional business domain team. This team should discuss potential broad use cases that can solve the company’s most urgent needs (outlined in step one). It should include IT architects, IT analytics support/reporting experts, analytical modelers, and executive decision-makers.

3. Compile a list of structured and unstructured data sources that could yield valuable answers to targeted business use cases. This should include easily available data inside the company, as well as data that can be obtained externally, e.g. from social media or partners. Since the data repository should be useful over a long time, it is important that data for discovery include related data elements (rather than only the specific data elements that are central to answering known use cases). This related data will create option value beyond known use cases and ensure that big data can yield large value in this critical business domain for an extended period of time. For example, if executives at an insurance company are interested in doing analysis to reduce claims cycle time, they should also capture data related to claims customer interactions as part of a larger strategic claims management data domain. This information could be used in the future to allow analysis of, say, fraud and claims severity, in addition to claims cycle time.

Thought Leadercsowa headshot small

Christopher Sowa is Vice President, Customer and Insight Strategy at Oracle.

4. Create big data architecture to support the priority business domains and the option value of future analysis going forward. This architecture should be aligned with the business needs for access and data latency (e.g. real-time data versus near real time) to prioritized strategic data domains. It should also have the required scale, security, resilience, and governance needed to work with this data. It is also important to integrate big data architecture with existing data architecture. This will help speed data integration, allow insights modeled with big data to be translated into effective management dashboards, and allow data to trigger process actions.

Only by targeting strategic data domains of meaningful value to the business will big data have a big impact. Also, by capturing data that is related to this domain (and beyond the scope of known use cases), executives can create a source of sustainable value for decision-making from their big data investment.

Photography by Shutterstock