Oracle GoldenGate for Distributed Applications and Analytics 23ai is now generally available

July 17, 2024 | 4 minute read
Deniz Sendil
Senior Principal Product Manager
Text Size 100%:

Oracle GoldenGate announces the general availability of GoldenGate for Distributed Applications and Analytics (GG for DAA 23ai) for on-premises and OCI GoldenGate. GG for DAA 23ai introduces new, advanced capabilities that enable heterogeneous, hybrid, and multi-cloud data synchronization for operational, analytical, and distributed applications. This blog highlights the following major new features:

  • New source and target technologies
  • Stage & Merge replication improvements
  • Snowflake replication improvements
  • MongoDB bi-directional replication
  • New graphical user experience
  • Nested Avro formatting

New source and target technologies

Databricks delivery: GG for DAA Databricks Handler stages change data records in micro-batches at a temporary staging location (ADLS for Azure, S3 for AWS and GCS for GCP) and then merges into the Databricks target tables using a merge SQL statement. Databricks handler can create the target tables if they don't exist. Databricks handler supports real-time data replication to both managed and external delta tables. Users can also sync initial load and real-time data replication using Databricks handler that allows no downtime data migrations to Databricks. Databricks delivery is currently available only in on-premises GG for DAA 23ai.

Google Pub/Sub delivery: GG for DAA Google Pub/Sub Handler streams real-time data into Google Pub/Sub topics with very low latencies in various formats. With pluggable formatters, users can enrich the produced messages with source meta details which helps down stream application message consumptions. Google PubSub delivery is currently available only in on-premises GG for DAA 23ai.

Stage & Merge replication improvements

Stage and Merge is the most preferred replication design for cloud data warehouse replications. It's a micro-batch process that can use various staging locations. In GG for DAA 23ai, the most powerful stage and merge improvement is optimized operation aggregation. GG for DAA 23ai moves the aggregation process to local systems. New design will result in replications that performs better with less resources. This will also help customers with operating larger batch windows that would help managing target cloud warehouse ingestion costs, especially for Snowflake.

GG for DAA 23ai introduces SQL Delete-Insert statements as an option for writing to target tables. Prior to GG for DAA 23ai, only SQL Merge was supported. SQL Merge is recommended and will perform better; but now users will have the option to SQL Delete-Insert statements for writing to target tables.

GG for DAA 23ai also improves logging with localization. This will allow users to localize log messages. In the logfiles, users will be able to see from which service that log is generated. This will improve trouble shooting and help users with identifying the issues more easily. For example, in the below sample Databricks log file, you can see different services in logged in the log file.

GG for DAA 23ai

Snowflake Replication Improvements

GG for DAA 23ai introduces two major improvements for Snowflake replication:

  1. Snowflake Streaming Replication: Snowflake Streaming Handler replicates data into Snowflake using the Snowpipe Streaming API. Snowflake Streaming Handler supports insert-only workloads and helps with lowering latencies and Snowflake ingestion costs. Snowflake Streaming Handler provides low latency loading of rows directly into the target table and also eliminates the need for a staging area.

    Differences between Stage and Merge, and Streaming Handlers:

    GoldenGate

    2. Snowflake Target Table Instantiation: Both Snowflake Stage & Merge and Streaming replication now supports target table creation. During real-time replication or initial data load, GG for DAA will create the target Snowflake tables if they don't exist. Snowflake target table creation is set to true by default; but users can turn this property off optionally.

MongoDB Bi-Directional Replication

GG for DAA 23ai MongoDB bi-directional replication enables customers with application high-availability and live MongoDB migrations. In MongoDB bi-directional replication, changes that are made to one source collection are replicated to target collection, and changes that are made to the second collection are replicated back to the first collection. Integrated loop detection filters duplicate messages and avoids loops.

MongoDB Bidirectional Replication

New graphical user experience

To enable a more productive user experience, GoldenGate 23ai simplifies setup and administration with new user experiences for workflows, configuration, management, and troubleshooting. GG for DAA 23ai updates graphical user experiences for building replication, enabling more automated configurations and visibility to operational data.

GG for DAA

 

Nested Avro formatting

With the new external schema-based Avro formatting, users can now create nested Avro files. The GoldenGate trail structure is a flat structure of key value pairs which model the flat column name and column value structure of an RDBMS; but Avro can be a hierarchical structure where records can be nested inside of fields and records. The Avro Metadata provider has been modified to traverse the Avro schema and to flatten the Avro structure to return the target metadata to the GoldenGate mapping.

In Summary

GG for DAA 23ai supports new technologies, provides a simplified usability backed with improved GUI, improves performance, security, governance, and developer APIs.

GoldenGate for Distributed Applications and Analytics 23ai Data Sheet

GoldenGate for Distributed Applications and Analytics 23ai Release Notes

GoldenGate for Distributed Applications and Analytics 23ai Product Document

GoldenGate for Distributed Applications and Analytics 23ai Quick Starts

 

 

Deniz Sendil

Senior Principal Product Manager

Deniz Sendil is a Senior Principle Product Manager in Oracle GoldenGate product group with a focus on GoldenGate for Big Data. Deniz has over 17 years of experience in data management, analytics and data integration domains. 


Previous Post

OCI GoldenGate Data Transforms extracting data from Oracle Fusion ERP

Elói Lopes | 7 min read

Next Post


Migrating Oracle Data Integrator to Data Transforms

Jayant Mahto | 3 min read