Geo replicating OCI streams

Oracle Cloud Infrastructure (OCI) Streaming service is a fully managed streaming service that has compatibility with Kafka APIs. OCI Streaming is the cornerstone of many large deployments for enterprise workloads processing trillions of messages and supporting mission-critical applications. Providing resiliency and business continuity is a top priority for all enterprises using OCI Streaming.

Streaming is highly available within a region, but large enterprise applications need resiliency against regional outages. In this article, we highlight recovery support from cluster downtime and how we can build and deploy a multiregion OCI Streaming implementation.

This blog showcases a solution that meets the following use cases:

  • Regional replication of Oracle streams for disaster recovery
  • Replicating from on-premises to Oracle streams or the reverse

Solution brief

The solution uses the MirrorMaker 2 (MM2) tool, part of the open source Apache Kafka ecosystem. MM2 2 uses the Kafka Connect ecosystem to start three connectors: MirrorSourceConnector, MirrorCheckpointConnector, and MirrorHeartbeatConnector.

Within OCI, the Streaming service Kafka Connect harness supports the open source Kafka Connect ecosystem. The Connect harness provides the compacted streams for config, offset, and status topics.

The following diagram is a reference setup for MM2 to replicate streams from one region to another:

A graphic depicting the architecture for the MirrorMaker setup.

The two virtual cloud networks (VCNs) in different regions connect through the dynamic routing gateway (DRG) and can be in the same or different tenancies. The stream pool in the source region has private endpoints in the private subnet of the VCN. The same applies to the stream pool in the target region but for the VCN in the target region.

Create the OCI Streaming Connect harness configuration to make compacted topics for status, offset, and config, which by the Kafka Connect framework uses.

Then, set up a Mirror Maker cluster in a different subnet with virtual machines (VMs) running MM2, which comes bundled with Apache Kafka. You can connect to these MM2 VMs from the Bastion service in OCI.

Try it for yourself

Use this GitHub repository to clone and run the setup using the Terraform CLI. This sample is a Quick Start that demonstrates the MM2 setup where you can see things in action.

This setup has the following limitations:

  • OCI Streaming currently doesn’t support compacted topics. So, the creation of offsets, heartbeat, and checkpointing topics by MirrorMaker throws an exception. But when auto-create topics are enabled in the Stream Pool, at the time of publishing data to these topics, these topics are created but not compacted.
  • sync.group.offsets.enabled=true is not supported in OCI Streaming.
  • sync.topic.acls.enabled=true is not supported in OCI Streaming.

For customers migrating from a source cluster to a target cluster, you can use the class org.apache.Kafka.connect.mirror.RemoteClusterUtils, which maps the offsets from the source cluster stream to the target cluster stream.

Summary

You can achieve geo replication of Oracle streams by using open source tools like MirrorMaker 2. Try out the solution yourself. Sign up for Oracle Cloud Free Tier or sign in to your account.