Customer's use case and architecture for the custom stage capabilities

November 8, 2023 | 7 minute read
Nisharahmed Soneji
Distinguished Data Management Specialist
Text Size 100%:

As part of my 3-part blog series, I am writing about the GoldenGate Stream Analytics custom stage functionality, how it has been used in one of the customer's organization scenarios with examples, and what the architecture was.

In this part-2 blog, I mention the scenario where "Custom-stage" functionality is beneficial. Hence, I will explain the use case and architecture for better understanding.

OSA Pipeline with custom-stage and reference data

As per the above pipeline, the organization has a Siebel CRM source that has been generating the "Customer" information data stream(Customer table in our example), and it has been captured using GoldenGate for Oracle technologies in real time. 
However, the other transaction information is stored in the Parent Database ("Contact Tables" in our example), and it requires to be referenced. At the same time, the data stream arrives in the GoldenGate Stream Analytics. The reference data must then be merged(joined) with "Customer" stream data. 
The merged data must be converted into a specific format, the Avro Kafka Messages in Nested structure. Because a customer can have multiple contact information that is required to be stored along with Customer records. 

The organization wants a solution where "Customer" stream data is correlated with "Contacts" Table data in real time. Then, the entire transactions are replicated into the Kafka System in a specific format, Nested Avro Message. The organization also wants to ensure that the performance shouldn't be impacted by referencing the contact tables in a database. The transaction flow is very high, and hence, the real-time referencing of data and processing converting into a specific format all have to happen quickly. The latency should be minimal as target systems require the whole transaction within milliseconds to be consumed by other Third-party applications.
It is just one example(Customer and Contact details merging), and the organization has set up many such pipelines inside GoldenGate Stream Analytics for various use cases possible with GoldenGate Stream Analytics.

The Oracle GoldenGate technologies, like Real-time replication, help organizations to capture the data in real-time from Oracle Databases, and it becomes the stream of data to ingest into GoldenGate Stream Analytics for further processing like enrichment of data, merging or joining the data. Finally, it helps Organizations to convert the data in the format that they want. All this processing happens on a voluminous data stream within a milliseconds window.

In the next blog, I shall mention how to write the custom stage Java program and integrate it with GoldenGate Stream Analytics technology.

Nisharahmed Soneji

Distinguished Data Management Specialist

Nishar is the "Distinguished Data Management Specialist", leading the sales opportunity for Data Integration products, OCI GoldenGate & GoldenGate On-premises in the EMEA region. The GoldenGate is the best database replication technology available in the market. He has been advocating for the customers on new-age technology and best practices and helping them achieve strategic goals using Oracle solutions and Products. He has been evangelizing GoldenGate Products, Solutions, and new-age architects(Data Mesh, Data Fabric) in Conferences, enabling and coaching the Tech hub engineers on the Products' functionality & insights. The family of GoldenGate products includes OCI GoldenGate, GoldenGate for Oracle, GoldenGate for Non-Oracle, GoldenGate for Big Data, GoldenGate Stream Analytics, GoldenGate Foundation Suite(Veridata, GoldenGate OEM PlugIn), and OCI Data Integration.

Before joining the sales team, he was a Senior Principal Product Manager at Oracle. He led the OCI GoldenGate Cloud Service, GoldenGate Foundation Suite Products, and GoldenGate core products globally.

Nishar has been a key leadership team member who created new cloud services from inception to growing Oracle Data Integration business services. He has also spearheaded multiple Oracle internal innovations, resulting in top-line and bottom-line growth in Database Replication.

Nishar has over 22+ years of experience in technology, specializing in enterprise data replication and integration products, with a demonstrated track record of creating and scaling world-class businesses. For the last 10+ years, He has been focused on driving revenue through sales and product management, making the business strategy, executing the product execution, and leading the cross-functional teams and products.

Before joining the Sales and Product Management role, he was the software development architect; he led various complex technology projects for several companies, including Endress+Hauser, Cummins Inc, and GoldenGate Inc.

Previous Post

Oracle GoldenGate for Big Data Articles get a new look!

Anuradha Chepuri | 3 min read

Next Post

How do you convert Data Stream into Avro Kafka Messages in Nested format using GoldenGate Stream Analytics?

Nisharahmed Soneji | 15 min read