As part of my 3-part blog series, I am writing about the GoldenGate Stream Analytics custom stage functionality, how it has been used in one of the customer’s organization scenarios with examples, and what the architecture was.
In this part-2 blog, I mention the scenario where “Custom-stage” functionality is beneficial. Hence, I will explain the use case and architecture for better understanding.
As per the above pipeline, the organization has a Siebel CRM source that has been generating the “Customer” information data stream(Customer table in our example), and it has been captured using GoldenGate for Oracle technologies in real time.
However, the other transaction information is stored in the Parent Database (“Contact Tables” in our example), and it requires to be referenced. At the same time, the data stream arrives in the GoldenGate Stream Analytics. The reference data must then be merged(joined) with “Customer” stream data.
The merged data must be converted into a specific format, the Avro Kafka Messages in Nested structure. Because a customer can have multiple contact information that is required to be stored along with Customer records.
The organization wants a solution where “Customer” stream data is correlated with “Contacts” Table data in real time. Then, the entire transactions are replicated into the Kafka System in a specific format, Nested Avro Message. The organization also wants to ensure that the performance shouldn’t be impacted by referencing the contact tables in a database. The transaction flow is very high, and hence, the real-time referencing of data and processing converting into a specific format all have to happen quickly. The latency should be minimal as target systems require the whole transaction within milliseconds to be consumed by other Third-party applications.
It is just one example(Customer and Contact details merging), and the organization has set up many such pipelines inside GoldenGate Stream Analytics for various use cases possible with GoldenGate Stream Analytics.
The Oracle GoldenGate technologies, like Real-time replication, help organizations to capture the data in real-time from Oracle Databases, and it becomes the stream of data to ingest into GoldenGate Stream Analytics for further processing like enrichment of data, merging or joining the data. Finally, it helps Organizations to convert the data in the format that they want. All this processing happens on a voluminous data stream within a milliseconds window.
In the next blog, I shall mention how to write the custom stage Java program and integrate it with GoldenGate Stream Analytics technology.