OCI Streaming is a managed event streaming service for high-volume event data. it is highly available within an OCI region: In multi-availability-domain regions, stream data is replicated across availability domains; in single-availability-domain regions, data is replicated across fault domains. Reference: OCI Streaming Service.
However some workloads need a recovery target in another region. Common business use cases include:
- Regional resiliency and business continuity.
- Regulatory, audit, or operational recovery requirements.
- Migration or cut over from one region or stream pool to another.
- Cross-region validation, analytics, or shadow processing.
This post describes a validated pattern for replicating selected OCI Streaming streams across regions by using Apache Kafka MirrorMaker 2, hereafter MM2. The pattern was validated with records produced by Kafka-compatible clients and by the OCI Streaming HTTPS/REST API.
Architecture overview
The pattern is an active-standby replication design. The source region remains the primary write path during normal operation. MM2 runs in the target region, reads selected source streams through the source stream pool Kafka-compatible endpoint, and writes replicated records to the target stream pool. Records can originate from Kafka-compatible clients or from the OCI Streaming HTTPS/REST API.
At a high level, the architecture follows the pattern shown in Figure 1.
Figure 1: OCI Streaming cross-region replication architecture with Apache Kafka MirrorMaker 2.
This pattern is a good fit when the requirement is asynchronous replication of selected streams. Replication latency is not fixed; it depends on workload, stream capacity, MM2 sizing, and cross-region network behavior. The validation for this post used Kafka-compatible clients and also tested records produced through the OCI Streaming HTTPS/REST API. MM2 reads from the source stream and writes to the target stream through OCI Streaming Kafka-compatible endpoints. For REST-originated records, the HTTPS/REST API is only the producer ingress path; after the records are stored in the source stream, MM2 replicates them through the Kafka-compatible endpoints.
Topic selection and naming
MM2 is configured by topic pattern. It is not a tenancy-level replication switch. MM2 connects to the source and target stream pool Kafka-compatible endpoints, then replicates selected streams or topics that match the configured pattern.
With the default MM2 replication policy, target topics are prefixed with the source cluster alias. For example, a source topic named:
orders
is replicated to a target topic named:
source_cluster.orders
This naming pattern helps identify the source cluster and avoid target-topic collisions. Applications that read from the target region must use the target topic name created by MM2 unless the MM2 naming policy is customized and validated.
Implementation considerations
MM2 consumes capacity on both sides: source-region read capacity and target-region write capacity. OCI Streaming throughput is partition-based, so size the design by expected MB per second, payload size, partition count, MM2 worker capacity, and peak traffic.
The current OCI service limits list 1 MB per second write per partition and 2 MB per second read per partition. Customers should still check Limits, Quotas and Usage in the OCI Console for tenancy-specific values. Reference: OCI service limits
If MM2 cannot read from the source and write to the target fast enough, replication lag grows and affects the recovery point objective.
Plan partition count before starting replication. More partitions can allow more MM2 parallelism, assuming enough MM2 tasks and worker capacity. Create and validate the required partition count intentionally instead of relying on MM2 or Kafka admin operations to resize topics after creation.
MM2 can resume replication when the required MM2 state and source records are still available. Recovery depends on source retention, valid authentication, target availability, network connectivity, and Kafka Connect internal storage topics. For production, run MM2 under systemd or an approved orchestration platform.
Monitor MM2 process availability, task errors, authentication failures, source and target throughput, replication delay, throttling, target topic creation, target consumer validation, and VM CPU, memory, disk, and network utilization.
Recommendations
First, size MM2 against OCI Streaming limits. For each replicated stream, account for source producer writes, existing source consumers, MM2 source reads, MM2 target writes, and any target-region consumers. Size streams by expected MB per second, leave headroom on both source and target, monitor throughput and throttling, and set source retention long enough for MM2 to recover after outages or lag.
Second, plan consumer offset handling separately. Exact source-to-target consumer offset continuation is not automatic in this OCI Streaming MM2 pattern. Applications should define whether target-region consumers start from the earliest available record, latest record, timestamp, or an application-managed checkpoint. Use idempotency, deduplication, and replay handling where duplicate processing is possible.
If a customer requires offset inspection, offset mapping, offset migration, or stricter offset continuity, they should open a ticket or service request with OCI Streaming. Offsets are not currently exposed in this MM2 pattern in a way that customers can directly use for exact cross-region consumer offset continuity.
Validated paths
The validation used a source stream pool in us-phoenix-1 and a target stream pool in us-ashburn-1. MM2 ran on a compute VM in the target region and connected to both OCI Streaming Kafka-compatible endpoints by using SASL_SSL authentication with OCI auth tokens.
Two producer paths were validated:
- Kafka-compatible producer path: Records were produced through the source Kafka-compatible endpoint and consumed from the replicated target topic.
- OCI Streaming HTTPS/REST producer path: Three records were published through the OCI Streaming HTTPS/REST data-plane API. The API response returned failures: 0 and offsets 0, 1, and 2. MM2 created the target topic, and the three replicated records were consumed from the target Kafka-compatible endpoint.
Conclusion
Apache Kafka MirrorMaker 2 can be used with OCI Streaming Kafka-compatible endpoints to replicate selected streams from one OCI region to another. This pattern can support regional standby, migration preparation, validation, analytics, and shadow processing use cases.
Before using this pattern for production recovery, validate throughput, partition count, replication delay, source retention, target capacity, MM2 operations, and consumer offset handling with realistic traffic. See the companion tutorial, Configure cross-region replication for OCI Streaming with MirrorMaker 2, for the validated setup flow.
To learn more, review the OCI Streaming documentation, Streaming resource limits, and your tenancy-specific Limits, Quotas and Usage values in the OCI Console.