Oracle GoldenGate announces the general availability of GoldenGate for Distributed Applications and Analytics (GG for DAA 23ai) for on-premises and OCI GoldenGate. GG for DAA 23ai introduces new, advanced capabilities that enable heterogeneous, hybrid, and multi-cloud data synchronization for operational, analytical, and distributed applications. This blog highlights the following major new features:
Databricks delivery: GG for DAA Databricks Handler stages change data records in micro-batches at a temporary staging location (ADLS for Azure, S3 for AWS and GCS for GCP) and then merges into the Databricks target tables using a merge SQL statement. Databricks handler can create the target tables if they don't exist. Databricks handler supports real-time data replication to both managed and external delta tables. Users can also sync initial load and real-time data replication using Databricks handler that allows no downtime data migrations to Databricks. Databricks delivery is currently available only in on-premises GG for DAA 23ai.
Google Pub/Sub delivery: GG for DAA Google Pub/Sub Handler streams real-time data into Google Pub/Sub topics with very low latencies in various formats. With pluggable formatters, users can enrich the produced messages with source meta details which helps down stream application message consumptions. Google PubSub delivery is currently available only in on-premises GG for DAA 23ai.
Stage and Merge is the most preferred replication design for cloud data warehouse replications. It's a micro-batch process that can use various staging locations. In GG for DAA 23ai, the most powerful stage and merge improvement is optimized operation aggregation. GG for DAA 23ai moves the aggregation process to local systems. New design will result in replications that performs better with less resources. This will also help customers with operating larger batch windows that would help managing target cloud warehouse ingestion costs, especially for Snowflake.
GG for DAA 23ai introduces SQL Delete-Insert statements as an option for writing to target tables. Prior to GG for DAA 23ai, only SQL Merge was supported. SQL Merge is recommended and will perform better; but now users will have the option to SQL Delete-Insert statements for writing to target tables.
GG for DAA 23ai also improves logging with localization. This will allow users to localize log messages. In the logfiles, users will be able to see from which service that log is generated. This will improve trouble shooting and help users with identifying the issues more easily. For example, in the below sample Databricks log file, you can see different services in logged in the log file.
GG for DAA 23ai introduces two major improvements for Snowflake replication:
Differences between Stage and Merge, and Streaming Handlers:
2. Snowflake Target Table Instantiation: Both Snowflake Stage & Merge and Streaming replication now supports target table creation. During real-time replication or initial data load, GG for DAA will create the target Snowflake tables if they don't exist. Snowflake target table creation is set to true by default; but users can turn this property off optionally.
GG for DAA 23ai MongoDB bi-directional replication enables customers with application high-availability and live MongoDB migrations. In MongoDB bi-directional replication, changes that are made to one source collection are replicated to target collection, and changes that are made to the second collection are replicated back to the first collection. Integrated loop detection filters duplicate messages and avoids loops.
To enable a more productive user experience, GoldenGate 23ai simplifies setup and administration with new user experiences for workflows, configuration, management, and troubleshooting. GG for DAA 23ai updates graphical user experiences for building replication, enabling more automated configurations and visibility to operational data.
With the new external schema-based Avro formatting, users can now create nested Avro files. The GoldenGate trail structure is a flat structure of key value pairs which model the flat column name and column value structure of an RDBMS; but Avro can be a hierarchical structure where records can be nested inside of fields and records. The Avro Metadata provider has been modified to traverse the Avro schema and to flatten the Avro structure to return the target metadata to the GoldenGate mapping.
GG for DAA 23ai supports new technologies, provides a simplified usability backed with improved GUI, improves performance, security, governance, and developer APIs.
GoldenGate for Distributed Applications and Analytics 23ai Data Sheet
GoldenGate for Distributed Applications and Analytics 23ai Release Notes
GoldenGate for Distributed Applications and Analytics 23ai Product Document
GoldenGate for Distributed Applications and Analytics 23ai Quick Starts
Deniz Sendil is a Senior Principle Product Manager in Oracle GoldenGate product group with a focus on GoldenGate for Big Data. Deniz has over 17 years of experience in data management, analytics and data integration domains.
Previous Post