The core business reason to build a Data Factory as it is presented here
is to implement a cost savings strategy by placing long-running batch
jobs on a cheaper system. The project is often funded by not spending
money on the more expensive system – for example by switching Mainframe
MIPS off - and instead leveraging that cost savings to fund the Data
Factory. The first figure shows a simplified implementation of the Data Factory.
As the image below shows, the data factory must be scalable, flexible
and (more) cost effective for processing the data. The typical system
used to build a data factory is Apache Hadoop or in the case of Oracle’s
Big Data Appliance – Cloudera’s Distribution including Apache Hadoop
Hadoop (and therefore Big Data Appliance and CDH) offers an extremely
scalable environment to process large data volumes (or a large number of
small data sets) and jobs. Most typical is the offload of large batch
updates, matching and de-duplication jobs etc. Hadoop also offers a very
flexible model, where data is interpreted on read, rather than on
write. This idea enables a data factory to quickly accommodate all types
of data, which can then be processed in programs written in Hive, Pig
As shown in above the data factory is an integration platform, much like
an ETL tool. Data sets land in the data factory, batch jobs process
data and this processed data moves into the upstream systems. These
upstream systems include RDBMS’s which are then used for various
information needs. In the case of a Data Warehouse, this is very close
to pattern 2 described below, with the difference that in the data
factory data is often transient and removed after the processing is
This transient nature of data is not a required feature, but it is often
implemented to keep the Hadoop cluster relatively small. The aim is
generally to just transform data in a more cost effective manner.
In the case of an upstream system in NoSQL systems, data is often
prepared in a specific key-value format to be served up to end
applications like a website. NoSQL databases work really well for that
purpose, but the batch processing is better left to Hadoop cluster.
It is very common for data to flow in the reverse order or for data from
RDBMS or NoSQL databases to flow into the data factory. In most cases
this is reference data, like customer master data. In order to process
new customer data, this master data is required in the Data Factory.
Because of its low risk profile – the logic of these batch processes is
well known and understood – and funding from savings in other systems,
the Data Factory is typically an IT department’s first attempt at a big
data project. The down side of a Data Factory project is that business
users see very little benefits in that they do not get new insights out
of big data.
The common way to drive new insights out of big data is pattern two.
Expanding the data warehouse with a data reservoir enables an
organization to expand the raw data captured in a system that is able to
add agility to the organization. The graphical pattern is shown in
A Data Reservoir – like the Data Factory from Pattern 1 – is based on
Hadoop and Oracle Big Data Appliance, but rather then have transient
data and just process data and then hand the data off, a Data Reservoir
aims to store data at a lower than previously stored grain for a period
much longer than previous periods.
The Data Reservoir is initially used to capture data, aggregate new
metrics and augment (not replace) the data warehouse with new and
expansive KPIs or context information. A very typical addition is the
sentiment of a customer towards a product or brand which is added to a
customer table in the data warehouse.
The addition of new KPIs or new context information is a continuous
process. That is, new analytics on raw and correlated data should find
their way into the upstream Data Warehouse on a very, very regular
As the Data Reservoir grows and starts to become known to exist because
of the new KPIs or context, users should start to look at the Data
Reservoir as an environment to “experiment” and “play” with data. With
some rudimentary programming skills power users can start to combine
various data elements in the Data Reservoir, using for example Hive.
This enables the users to verify a hypotheses without the need to build a
new data mart. Hadoop and the Data Reservoir now becomes an
economically viable sandbox for power users driving innovation, agility
and possibly revenue from hitherto unused data.
Agility for power users and expert programmers is one thing, but eventually the goal is to enable business users to discover new and exciting things in the data. Pattern 3 combines the data reservoir with a special information discovery system to provide a Graphical User Interface specifically for data discovery. This GUI emulates in many
ways how an end user today searches for information on the internet.
To empower a set of business users to truly discover information, they first and foremost require a Discovery tool. A project should therefore always start with that asset.
Once the Discovery tool (like Oracle Endeca) is in place, it pays to start to leverage the Data Reservoir to feed the Discovery tool. As is shown above, the Data Reservoir is continuously fed with new data. The Discovery tool is a business user’s tool to create ad-hoc data marts in the discovery tool. Having the Data Reservoir simplifies the acquisition
by end users because they only need to look in one place for data.
In essence, the Data Reservoir now is used to drive two different systems; the Data Warehouse and the Information Discovery environment and in practice users will very quickly gravitate to the appropriate system. But no matter which system they use, they now have the ability to drive value from data into the organization.
So far, most of what was discussed was analytics and batch based. But a lot of organizations want to come to some real time interaction model with their end customers (or in the world of the Internet of Things – with other machines and sensors).
Hadoop is very good at providing the Data Factory and the Data Reservoir, at providing a sandbox, at providing massive storage and processing capabilities, but it is less good at doing things in real time. Therefore, to build a closed loop recommendation system – which should react in real time – Hadoop is only one of the components .
Typically the bottom half of the last figure is akin to pattern 2 and is used to catch all data, analyze the correlations between recorded events (detected fraud for example) and generate a set of predictive models describing something like “if a, b and c during a transaction – mark as suspect and hand off to an agent”. This model would for example block a credit card transaction.
To make such a system work it is important to use the right technology at both levels. Real time technologies like Oracle NoSQL Database, Oracle Real Time Decisions and Oracle Event Processing work on the data stream in flight. Oracle Big Data Appliance, Oracle Exadata/Database and Oracle Advanced Analytics provide the infrastructure to create, refine and expose the models.
Today’s big data technologies offer a wide variety of capabilities. Leveraging these capabilities with the existing environment and skills already in place according to the four patterns described does enable an organization to benefit from big data today. It is a matter of identifying the applicable pattern for your organization and then to start on the implementation.
The technology is ready. Are you?