Friday Mar 08, 2013

Implementing Coherence Cache-Aside pattern using Coherence Goldengate Adapter

A common requirement in Enterprise SOA Initiatives is to create a Business Look-up data cache which contains business look-up data. This data is used across the organization and controls the behavior of various organization processes, business calculations and is also used for populating drop-down lists in the Business Application User Interfaces.
This look-up data also requires to be maintained because as the Business changes, there is change to this look-up data. Hence a business friendly user interface needs to be designed for this look-up data.

Oracle Coherence can be used to store this data in memory for efficient and fast look-ups. The data can be persisted in an Oracle Database.  It is recommended to have the entire data set being stored in the cache so that there are no cache misses.  This kind of a solution typically ends up using a Cache-Aside Pattern. In this pattern the cache data is entirely managed by the application developer.  A representation of this pattern is given here.

One of the key challenges in this architecture is identifying what data has changed in the persistent store and updating the cache with the changed data. As represented above, this requires complex custom programming for each data element. This is a very brittle integration and will become a maintenance nightmare as the data structure changes over time.
This is where the Coherence Goldengate Adapter (yet to be released as of the date of writing this blog) will play a big role. Instead of having some kind of custom programming, triggers etc on the persistent data store, Goldengate can extract the changed data from the persistent data store database logs and update the coherence cache directly. This greatly simplifies the architecture and makes the implementation of the Cache-Aside Pattern very easy.  The new pattern is represented here.

Benefits of this architecture –
1.    Simplify development of Cache-Aside Pattern for Coherence implementations
2.    Maintenance of the Cache-Data Store integration becomes simpler and easier
3.    Update the cache without firing any query on the persistent data store
4.    Capture the changes in real-time and update the cache automatically

References and further reading-



Wednesday Jan 30, 2013

Capture Application Events using Oracle Goldengate

In my past I have worked on SOA, BPM and Event Processing systems. I was lucky enough to have an opportunity to be part of teams building large SOA applications, Application Integrations and Application Enhancements.

One of the KEY problems that we faced when building Event processing or SOA systems is the problem of capturing the application events WITHOUT -
1.    Re-engineering/modifying the existing application
2.    Adding Database Triggers
3.    Build/utilize application provided hooks

All of the above methods are either intrusive or have a processing overhead on the Source Application processing. Thus most customers have to evaluate which of the above three methods is suitable for their environment and decide which method to select. As these methods require changes to the application architecture (code changes) or have performance implications some customers have second thoughts on going ahead with the project though they can see the value that they can derive from SOA, BPM and Event Processing systems. The customers who went ahead with these projects also had to factor additional cost in terms of services to re-engineer the application or hardware to handle the additional load.

Oracle Goldengate provides non-intrusive log-based data capture and extraction mechanism. A very little talked -about feature of Oracle Goldengate is its Java/JMS integration. Using Oracle Goldengate it is possible to capture transactions happening in the Application and write the transaction/event to a JMS queue.

With the knowledge of the existing application architecture and Data Model, it is possible to configure the Oracle Goldengate extracts to read transactions on from the specific tables impacted during an application event.

The diagram depicts how Goldengate can be used to capture Application Events non-intrusively.

Goldengate with JMS

As can be seen from the above representation

•    GG Processes run outside the Database
•    GG processes read the data from the Log files without hitting the source database
•    GG Pump process pushes the data in an online fashion to the Data Integration Server with minimal lag
•    GG Replicat processes read the data from the Trail files and post it to JMS
•    From JMS it can be fed into multiple different destinations as listed

This paves the way for non-invasive event capture from existing applications. Now it is possible to capture Application events without re-engineering the application or adding any significant load on the application.

This enables the extension of Existing applications for-

1.    Capturing events from the existing application and process those using Event processing engines like Oracle CEP.
2.    Captur
ing events from the existing application and trigger BPEL or BPM workflows using products like Oracle BPEL or BPM
3.    Captur
ing events from the existing application and build real-time dashboards using tools like Oracle BAM
4.    Extending the application functionality by capturing the events and performing Java actions (Integrating with Java Apps)
5.    Captured events can also be put into a Coherence Grid for any kind of in-memory grid processing of the events

Thus Oracle Goldengate can be used to capture application events for an SOA WITHOUT

1.    Re-engineering/modifying the existing application

2.    Adding Database Triggers

3.    Build/utilize application provided hooks

Goldengate enables non-invasive capture of the application events and deliver them to an SOA.

References and further reading-

1. A detailed step-by-step guide to capture events and writing to JMS Queues is available on the Oracle-By-Example site here.

2. Data Sheet for the Goldengate Application Adapters (which includes JMS support) is available here


This Blog discusses new ways of using Oracle Technology to achieve Integration, Harmonization and Orchestration for New and Existing Applications


« July 2016