This companion post walks through the validated configuration flow for replicating selected OCI Streaming streams across regions with Apache Kafka MirrorMaker 2, hereafter MM2. Use the first post for the architecture, recommendations, and design considerations. Use this post as the operator-oriented setup sequence.
The examples use placeholders instead of environment-specific values.
The exact steps depend on your networking model, IAM model, region choices, and operational standards. The following runbook shows the validated flow in a clearer operator sequence.
The configuration has four main parts:
| OCI Streaming resources | Source stream pool, target stream pool, source streams/topics, target capacity | Provides the Kafka-compatible endpoints and stream storage |
| Kafka Connect state | Config, offset, and status storage topics | Lets MM2 store connector runtime state |
| MM2 runtime | Compute VM or worker environment, Java, Kafka binaries, network access | Runs the replication process |
| Client and MM2 configuration | Source auth, target auth, topic selection, cluster aliases, storage topics | Tells MM2 what to replicate and where to write it |
Before you start, decide these values:
| Source region | us-phoenix-1 | Region where producers write during normal operation |
| Target region | us-ashburn-1 | Region that receives replicated records |
| Source cluster alias | source_cluster | Used by MM2 and included in default target topic names |
| Target cluster alias | target_cluster | Used by MM2 to identify the target Kafka endpoint |
| Topic selection pattern | mm2-continuous-.* | Start narrow, then expand after validation |
| Runtime placement | Target-region VM | Keeps the replication runtime close to the target stream pool |
| Authentication method | SASL_SSL with OCI auth token | Do not expose auth tokens in shared docs or screenshots |
Step 1: Confirm the replication goal
Start by defining why replication is needed. Common goals include regional recovery readiness, migration support, target-region analytics, or validation of a standby processing path.
Output of this step:
- A named replication use case.
- A list of streams/topics to replicate.
- A target recovery or validation objective.
- A known topic naming expectation for target-region consumers.
Step 2: Create a compartment and required access
Create or choose the compartment that will contain the resources. Confirm that the operators or automation principals have permission to manage or read the required OCI Streaming, Kafka Connect configuration, networking, compute, logging, and monitoring resources.
Validate:
- The compartment exists in the expected tenancy.
- The required IAM policies are in place.
- Operators can create or view stream pools, Kafka Connect configuration, compute, networking, logs, and metrics.
Step 3: Create the source stream pool
Create the source OCI Streaming stream pool in the source region. This is the stream pool that producers use during normal operation.
In the OCI Console:
- Switch to the source region.
- Go to Analytics & AI > Streaming > Stream Pools.
- Create a stream pool in the selected compartment.
- Choose the endpoint type for the environment.
- Record the Kafka-compatible endpoint.
Record:
- Source region.
- Source stream pool OCID.
- Source Kafka-compatible bootstrap endpoint.
- Source stream names or Kafka topic names.
- Source stream retention settings.
- Source stream partition counts.
Validate:
- The source stream pool is active.
- The source Kafka-compatible endpoint is visible.
- The streams/topics selected for replication exist or are planned.
Step 4: Create the target stream pool
Create the target OCI Streaming stream pool in the target region. This is where MM2 writes replicated records.
Record:
- Target region.
- Target stream pool OCID.
- Target Kafka-compatible bootstrap endpoint.
- Target retention settings.
- Expected target topic naming pattern.
Validate:
- The target stream pool is active.
- The target Kafka-compatible endpoint is visible.
- Target capacity and retention are appropriate for replicated data.
Important: With the default MM2 replication policy, target topics are prefixed with the source cluster alias. For example, a source topic named orders is replicated as source_cluster.orders when the source alias is source_cluster.
Step 5: Create the Kafka Connect configuration
Create a Kafka Connect configuration in the target region. MM2 uses Kafka Connect internally, and the Kafka Connect configuration provides the internal storage topics for configuration, offsets, and status.
Record the generated values for:
config.storage.topic
offset.storage.topic
status.storage.topic
These topics are part of MM2 runtime state. Protect and monitor them as part of the replication system.
Validate:
- The Kafka Connect configuration exists in the expected target region.
- The config, offset, and status storage topic names are available.
- The storage topic values are copied into the private deployment record.
Step 6: Prepare the MM2 runtime
Create the compute VM or approved worker environment that will run MM2. In the validated design, the MM2 VM ran in the target region.
Install Java, Apache Kafka, and basic connectivity tools.
Example setup:
sudo dnf install -y java-11-openjdk java-11-openjdk-devel wget tar gzip nmap-ncat java -version wget https://archive.apache.org/dist/kafka/2.5.1/kafka_2.12-2.5.1.tgz tar -xzf kafka_2.12-2.5.1.tgz ln -s kafka_2.12-2.5.1 kafka ./kafka/bin/kafka-topics.sh --version
Validate that the MM2 host can reach both endpoints.
Example connectivity check:
nc -vz <source-streaming-endpoint> 9092 nc -vz <target-streaming-endpoint> 9092
Validate:
- Java is installed.
- Kafka command-line tools are installed.
- The MM2 host can connect to the source endpoint on port 9092.
- The MM2 host can connect to the target endpoint on port 9092.
Step 7: Create source and target Kafka client configs
Create Kafka client configuration files for the source and target stream pools by using SASL_SSL and the required OCI Streaming Kafka-compatible authentication format.
SOURCE_POOL="<source-stream-pool-ocid>" TARGET_POOL="<target-stream-pool-ocid>" SOURCE_BOOTSTRAP="<source-streaming-endpoint>:9092" TARGET_BOOTSTRAP="<target-streaming-endpoint>:9092" SOURCE_USER="<oci-kafka-username-for-source-stream-pool>" TARGET_USER="<oci-kafka-username-for-target-stream-pool>"
Create a source client config:
security.protocol=SASL_SSL sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<source-user>" password="<auth-token>";
Create a target client config:
security.protocol=SASL_SSL sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<target-user>" password="<auth-token>";
Validate source authentication:
./kafka/bin/kafka-topics.sh \ --bootstrap-server "$SOURCE_BOOTSTRAP" \ --command-config source-client.config \ --list
Validate target authentication:
./kafka/bin/kafka-topics.sh \ --bootstrap-server "$TARGET_BOOTSTRAP" \ --command-config target-client.config \ --list
Expected result:
- The source topic list command completes.
- The target topic list command completes.
- Authentication failures, timeout errors, and endpoint errors are resolved before MM2 is started.
Step 8: Create the MM2 configuration
Create mm2.config with the source cluster alias, target cluster alias, source and target endpoints, authentication settings, topic selection pattern, and Kafka Connect storage topics.
At minimum, confirm that the configuration includes:
- Source cluster alias.
- Target cluster alias.
- Source bootstrap endpoint.
- Target bootstrap endpoint.
- Source and target authentication settings.
- Kafka Connect storage topics.
- Topic selection pattern.
- Replication policy settings.
- Task count and connector settings appropriate for the workload.
Representative structure:
clusters=source_cluster,target_cluster source_cluster.bootstrap.servers=<source-streaming-endpoint>:9092 target_cluster.bootstrap.servers=<target-streaming-endpoint>:9092 source_cluster.security.protocol=SASL_SSL source_cluster.sasl.mechanism=PLAIN source_cluster.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<source-user>" password="<auth-token>"; target_cluster.security.protocol=SASL_SSL target_cluster.sasl.mechanism=PLAIN target_cluster.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<target-user>" password="<auth-token>"; source_cluster->target_cluster.enabled=true source_cluster->target_cluster.topics=<topic-regex> config.storage.topic=<connect-config-storage-topic> offset.storage.topic=<connect-offset-storage-topic> status.storage.topic=<connect-status-storage-topic> tasks.max=1
Validate:
- The source and target cluster aliases are consistent throughout the file.
- The topic pattern matches only the intended source topics.
- The Kafka Connect storage topic names are copied from the target-region Kafka Connect configuration.
- The file does not contain stale endpoints, stale stream pool OCIDs, or expired credentials.
Step 9: Start MM2
Start MM2 and confirm the process is running.
Example:
./kafka/bin/connect-mirror-maker.sh mm2.config --clusters target_cluster
Validate:
- The MM2 process is running.
- Logs show the MirrorSourceConnector or WorkerSourceTask starting.
- Logs do not show persistent authentication, timeout, or endpoint failures.
Example process check:
ps -ef | grep -E '[m]m2.config|[c]onnect-mirror-maker|[C]onnectDistributed'
Example log check:
grep -E 'ERROR|Exception|FAILED|SaslAuthenticationException|TimeoutException|MirrorSourceConnector|WorkerSourceTask' mm2.log | tail -n 100
Step 10: Validate continuous replication
Produce records to a source topic after MM2 starts. Confirm that MM2 creates the expected target topic and that records can be consumed from the target region.
A useful smoke test is:
- Start MM2.
- Produce messages 1-20 to the source topic.
- Consume from the target topic and confirm messages 1-20.
- Produce messages 21-40 while MM2 continues running.
- Consume from the target topic again and confirm messages 1-40.
This confirms that replication is continuous, not only a one-time copy.
Validate:
- The target topic is created with the expected MM2 naming policy.
- Records produced after MM2 starts are visible in the target region.
- Records continue replicating while MM2 remains running.
- The application team understands which target topic name consumers must use during failover or validation.
Conclusion:
This setup shows how to configure MM2 to replicate selected OCI Streaming streams from one region to another. The key steps are to create the source and target stream pools, prepare the Kafka Connect state, configure the MM2 runtime, and validate that records produced in the source region can be consumed from the target region.
Before using this for production, test with realistic traffic, monitor replication lag, and document how applications will handle failover, replay, deduplication, and consumer offsets.