Authored by: Madhu Raviendran nair
Everyday Data Integration
It is not every day that the average citizen wakes up thinking “Let me integrate some data today,” but that is precisely what they end up doing in many ways.
From waking up to catching a favorite morning program on cable TV, to grabbing that cup of coffee from a major coffee retailer around the block, to binge-watching a favorite series on one of the leading subscription entertainment services, we come in daily contact with the business end of data integration.
Of course, there is a nice human interface that contextualizes and enhances our experiences delivered through our mobile phones and web browsers on-top of the data that we interact with.
Insightful businesses and business leaders want to provide the best services and they also look to delight and enhance customer experiences by providing personalized content and offerings. For this, deep insights and a complete understanding of customers is required.
Data provides the raw information for these insights. Data also serves as the key ingredient, which when combined with other related data, bubbles up key recommendations that go into many every day and strategic business decisions that allow companies to differentiate themselves by providing customized and excellent capabilities.
Many times, businesses tend to overlook the amount of planning and foundational work that goes into constructing the right data integration foundation for many transformation practices.
In this series of blogs, I hope to bring out the role of data integration through various stories where Oracle Data Integration plays a critical role.
LinkedIn is a great example of how Oracle Data Integration works behind the scenes to keep user experiences seamless.
From ensuring that user profile data is accessible across the globe to making sure that user updates flow through instantaneously at the click of a refresh button, Oracle Data Integration underpins a well thought out architecture for one of the largest networking platforms.
Here is a quick overview of how LinkedIn uses Oracle Data Integration, specifically Oracle GoldenGate for Big Data, to pass on the benefits of real-time updates and synchronization to its user community.
A Deeper Dive On How LinkedIn Achieved Real Time Glory
The primary goal was to ensure that data is efficiently moved across a number of online data stores (Oracle and MySQL) to downstream data stores, eliminating bulky data dumps, multiple hops, and standardizing the data formats.
Oracle GoldenGate, the real-time replication engine that captures data changes and streams them to any required destination instantaneously, is at the core of this implementation. Oracle GoldenGate comes with off-the-shelf integration with many sources and targets making it easier to implement and integrate with existing systems that were being used. This allows greater compatibility with Oracle and MySQL databases also eliminating the multiple hops that the data used to go through to reach its final data store.
Another critical criterion to select Oracle GoldenGate was the low impact on the source data base when capturing data to be streamed. Oracle GoldenGate for Big Data has a very light footprint and has minimal impact on business-critical source systems, an important consideration when fiddling around with carefully tuned applications and databases.
Big Data Considerations – Kafka\HDFS
Big Data platforms provide a cost-effective alternative to specialized hardware for both storage and computing for many scenarios. While data warehouses and high-speed analytics still benefit from optimized hardware, big data has gained tremendous viability for deep data storage used for machine learning and artificial intelligence requirements.
Oracle Data Integration has always recognized this need for heterogeneity, a factor that played into the decision to use Oracle GoldenGate for Big Data for this project.
Oracle GoldenGate for Big Data has handlers (a handler being a prebuilt bit of technology to integrate with specific systems) for many big data standards and technologies, among others, Kafka and Hadoop Distributed File System (HDFS). Both came in useful in pilot stages when determining which Big Data technology to adopt. Take a look at this datasheet for a more exhaustive capability list for Oracle GoldenGate for Big Data.
But in short, this is how it can look:
The Oracle Data Integration Platform
LinkedIn is one of many customers who is innovating in their core businesses with cutting edge technology from Oracle.
Oracle Data Integration, in turn, is pushing the boundaries of data integration to enable customers solve data challenges and turn them into opportunities for excellence. Oracle Data Integration’s products and services are evolving into a comprehensive and unified cloud service that brings together all the capabilities required for a data integration solution. The new platform, Oracle Data Integration Platform Cloud (DIPC), combines rich capabilities, a wide breadth of features, persona-based user experience, simple pricing, and packaging to make our customers’ data integration journey easier.
Learn more about Oracle Data Integration Platform Cloud here.
Learn more about Oracle Data Integration here.
Learn more about Oracle GoldenGate for Big Data here.