Apache Kafka is built for scale. Topics can be split into partitions, partitions can be replicated across brokers, and consumer groups can process those partitions in parallel. But one of the most common scaling questions we hear from teams using or evaluating OCI Streaming with Apache Kafka is:
“When should I increase my topic partition count, and what happens when I do?”
The short answer: increase partitions when you need more parallelism, not simply because you need more infrastructure capacity.
OCI Streaming with Apache Kafka is a fully managed service that lets you create and run Kafka clusters in an OCI tenancy with Apache Kafka functionality. Brokers store Kafka data and process client requests. Topics organize event streams. Partitions split those topics for data distribution and parallel processing. You can use standard Apache Kafka APIs and CLI tools to create and update topics, partitions, records, producers, and consumers. OCI Streaming with Apache Kafka documentation
This post explains how to think about partition resizing in OCI Streaming with Apache Kafka, when to use it, what impact it has, and how to perform the change safely.
Clarification: In this article, “partition resizing” means changing the total number of partitions configured for a topic. It does not mean resizing the physical storage size, disk allocation, or capacity of an individual Kafka partition. OCI Streaming with Apache Kafka does not expose a customer operation to resize the physical size of an individual partition; customers manage topic partition count, where supported, not physical partition sizing.
Kafka partitioning: a quick refresher
A Kafka topic is a logical stream of records. Each topic is split into one or more partitions. Each partition is an ordered log, and each record in a partition has an offset.
A simplified model looks like this:
Partitions are important because they are the unit of parallelism in Kafka. Producers can write to multiple partitions, brokers can host partition leaders and replicas, and consumers in the same consumer group can process partitions in parallel. Confluent partition guidance
Topic: orders
Partition 0: offset 0, offset 1, offset 2 ...
Partition 1: offset 0, offset 1, offset 2 ...
Partition 2: offset 0, offset 1, offset 2 ...
Partitions are important because they are the unit of parallelism in Kafka. Producers can write to multiple partitions, brokers can host partition leaders and replicas, and consumers in the same consumer group can process partitions in parallel. Confluent partition guidance
Replicas are copies of partitions placed across brokers for availability. For each partition, one replica is the leader and the others are followers. Producers write to the leader, followers replicate from the leader, and if a broker fails, Kafka can elect a new leader from an in-sync replica. Kafka automatically detects broker shutdowns or failures and elects new leaders for affected partitions. Apache Kafka operations documentation
A topic with:
12 partitions
replication factor 3
has:
12 partition logs
36 total partition replicas
distributed across brokers.
Partitions versus brokers: what are you really scaling?
One of the most important decisions for Kafka operators is whether to scale partitions or brokers.
Use this mental model:
Partitions scale concurrency.
Brokers scale capacity.
Increase partitions when the bottleneck is parallelism. For example, if a topic has 8 partitions and you want to run 20 consumer instances in the same consumer group, only up to 8 consumers can actively consume from that topic at the same time. That is because a partition can be assigned to only one consumer within a consumer group.
Scale out the cluster or add broker capacity when the bottleneck is infrastructure capacity. For example, if brokers are saturated on CPU, disk, network, or storage, adding more partitions may add overhead without solving the underlying capacity problem.
| Situation | Best first action |
| Consumer group cannot keep up and consumers are limited by partition count | Increase partitions |
| More consumers than partitions, leaving consumers idle | Increase partitions |
| Producer throughput needs more parallelism across partition leaders | Consider increasing partitions |
| Broker CPU, disk, or network is saturated | Scale cluster or broker capacity |
| Storage pressure is high | Scale capacity; partitions alone do not add storage |
| A single key is hot | Review key strategy; more partitions may not help |
| Strict per-key ordering is required | Be very careful with partition increases |
Partition resizing should be treated as an application scaling change, not just a cluster administration change.
When should you increase partitions?
Consider increasing partitions when you need more parallel processing across producers or consumers.
Common examples include:
- Consumer lag is growing.
- Consumers are healthy but cannot process fast enough.
- There are more consumer instances than partitions.
- The workload needs more parallel write/read lanes.
- The team expects future growth in consumer count or throughput.
However, partition count should not be increased casually. More partitions also mean more metadata, more log segments, more file handles, more replication work, more leader election work, and potentially more expensive consumer group rebalances.
Choose a partition count that supports current and expected near-term scale, but avoid creating excessive partitions “just in case.”
What happens when you increase partitions?
Increasing a topic’s partition count is an online Kafka operation, but it is not completely invisible to applications.
This section focuses on increasing partition count, the most common scaling scenario. The operation adds more partitions to the topic; it does not expand the physical size of existing partitions.
The most important impacts are:
| Area | Impact |
| Existing records | Not moved, deleted, or rewritten |
| Existing offsets | Preserved |
| New partitions | Created empty |
| New records | May be written to the expanded partition set |
| Consumer groups | May rebalance |
| Keyed records | May map to different partitions after the change |
| Ordering | Preserved within a partition, but key-based ordering assumptions can be affected |
Kafka does not support reducing the number of partitions for an existing topic. To use fewer partitions, create a new topic with the desired partition count and migrate producers and consumers to that new topic. Apache Kafka topic operations
The most important warning: keyed messages and ordering
For many workloads, producers send records with keys such as customerId, orderId, accountId, or deviceId. Kafka uses the record key to determine which partition receives the record. When the partition count changes, the same key may map to a different partition for future records.
For example:
Before resize:
customer-123 -> partition 4
After resize:
customer-123 -> partition 17
Existing records for customer-123 remain in partition 4. New records for customer-123 may start going to partition 17.
Kafka still preserves ordering within each partition, but if the application assumes that all records for a key always land on the same partition forever, increasing partitions can break that assumption across the resize boundary.
Important: Do not increase partitions on a keyed topic with strict per-key ordering requirements unless the application owner has reviewed and accepted the impact. For critical ordered workloads, consider creating a new topic with the desired partition count and migrating traffic in a controlled way.
Consumer group impact: expect a rebalance
When a topic’s partition count changes, consumer groups subscribed to that topic may rebalance. A rebalance is Kafka’s process of redistributing partition ownership across consumers in the group. Kafka’s consumer group rebalance protocol determines how partitions are assigned within a group and how assignments change when consumers join, leave, or topic metadata changes. Confluent consumer group design
For example, before the resize:
Topic: orders
Partitions: 4
Consumers: C1, C2
C1 -> P0, P1
C2 -> P2, P3
After increasing to 6 partitions:
Topic: orders
Partitions: 6
C1 -> P0, P1, P4
C2 -> P2, P3, P5
The new partitions start empty, but they still need to be assigned to consumers.
During a rebalance, you may observe:
- A short processing pause
- Temporary consumer lag increase
- New partition assignments
- Potential duplicate processing if offsets are not committed safely
- Application-level latency spikes
For production workloads, consumers should be designed to handle rebalances safely. That means committing offsets carefully, stopping work on revoked partitions, and making downstream processing idempotent where possible.
Best practices before resizing partitions
Before increasing a partition count in OCI Streaming with Apache Kafka, review the workload.
Ask these questions:
- What problem are we solving: throughput, lag, parallelism, or capacity?
- What is the current partition count?
- What is the target partition count?
- What is the replication factor?
- How many brokers are in the cluster?
- How many consumer groups read this topic?
- How many consumer instances are in each group?
- Are records keyed?
- Does the application require strict per-key ordering?
- Are Kafka Streams, Flink, Spark, Kafka Connect, or other stateful consumers involved?
- Is broker CPU, disk, memory, or network already under pressure?
For stateful stream processing applications, be extra cautious. Partition changes can affect task assignment, state distribution, local stores, and rebalance behavior.
Steps to resize partitions in OCI Streaming with Apache Kafka
OCI Streaming with Apache Kafka is compatible with standard Apache Kafka operations. After the cluster is created and authentication is configured, you can set up a Kafka client and run Kafka commands to create, update, produce, and consume data. Getting started with OCI Streaming with Apache Kafka
The following runbook uses standard Kafka CLI tools with an OCI Streaming with Apache Kafka bootstrap endpoint.
Step 1: Gather connection details
Collect the following from the OCI Console or your cluster configuration:
OCI Streaming with Apache Kafka bootstrap endpoint
Topic name
Target total partition count
Authentication mechanism
TLS or SASL configuration, if enabled
Client properties file
Most secured environments use a Kafka client properties file with –command-config.--bootstrap-server <oci-streaming-with-apache-kafka-bootstrap-endpoint>
--command-config <client.properties>
Step 2: Describe the current topic
Run:
kafka-topics.sh \
--bootstrap-server <oci-streaming-with-apache-kafka-bootstrap-endpoint> \
--command-config <client.properties> \
--describe \
--topic <topic-name>
Review:Current partition count
Replication factor
Partition leaders
Replica placement
ISR health
Offline partitions, if any
Example output may show:Topic: orders PartitionCount: 12 ReplicationFactor: 3
Step 3: Check consumer group lag before the change
For each important consumer group:
kafka-consumer-groups.sh \
--bootstrap-server <oci-streaming-with-apache-kafka-bootstrap-endpoint> \
--command-config <client.properties> \
--describe \
--group <consumer-group-id>
Capture the baseline:Current lag
Current assignments
Active members
Whether the group is stable
Whether lag is already growing
If the consumer group is already unstable or continuously rebalancing, stabilize it before changing partition count.
Step 4: Increase the partition count
Run:
kafka-topics.sh \
--bootstrap-server <oci-streaming-with-apache-kafka-bootstrap-endpoint> \
--command-config <client.properties> \
--alter \
--topic <topic-name> \
--partitions <new-total-partition-count>
Important: –partitions is the new total partition count, not the number of partitions to add.
For example, if the topic currently has 12 partitions and the target is 24 total partitions:kafka-topics.sh \
--bootstrap-server <oci-streaming-with-apache-kafka-bootstrap-endpoint> \
--command-config <client.properties> \
--alter \
--topic orders \
--partitions 24
This changes the topic to 24 total partitions. It does not add 24 more.
Step 5: Verify the new topic layout
Run:
kafka-topics.sh \
--bootstrap-server <oci-streaming-with-apache-kafka-bootstrap-endpoint> \
--command-config <client.properties> \
--describe \
--topic <topic-name>
Confirm:PartitionCount equals the expected target
All partitions have leaders
Replicas are in sync
No offline partitions
No under-replicated partitions
Step 6: Monitor consumer groups after the resize
Run:
kafka-consumer-groups.sh \
--bootstrap-server <oci-streaming-with-apache-kafka-bootstrap-endpoint> \
--command-config <client.properties> \
--describe \
--group <consumer-group-id>
Watch for:Consumer rebalances
Temporary lag increase
Consumers receiving new partition assignments
Unexpected idle consumers
Application processing errors
Duplicate processing symptoms
End-to-end latency changes
Step 7: Verify producer distribution
After the resize, confirm that new records are using the expanded partition set.
If new partitions remain empty, possible explanations include:The producer is explicitly writing to fixed partitions.
The producer uses a custom partitioner.
Message keys are highly skewed.
Traffic volume is too low to observe yet.
Only a small set of keys is active.
Adding partitions does not automatically fix a hot-key problem. If most traffic uses the same key, that traffic can still concentrate on one partition.
Step 8: Continue monitoring cluster and application health
After resizing, monitor both Kafka and application-level signals:
Consumer lag
Produce request latency
Fetch request latency
Broker CPU
Broker network throughput
Broker disk usage
Under-replicated partitions
Offline partitions
ISR shrink and expand events
Application error rate
Duplicate processing rate
End-to-end processing latency
OCI Monitoring and OCI Logging can also be used with OCI Streaming with Apache Kafka for cluster metrics and cluster-level logs. OCI Streaming with Apache Kafka overview
What about broker and replica rebalancing?
Partition resizing and broker rebalancing are different operations. Increasing partitions creates new partitions. Existing records are not redistributed across the new partitions. Similarly, adding brokers to a Kafka cluster does not necessarily move existing partition replicas to the new brokers automatically. Replica redistribution is typically a separate administrative or automated balancing operation.
Caution: partition reassignment can cause significant data movement. If reassignment is performed as part of a broader balancing activity, review the current partition assignment and the proposed movement plan before applying it. Kafka’s generic reassignment tooling can generate broad or randomized movement plans, which may cause a cluster-wide shuffle of data if applied without review. For high-workload clusters, throttle reassignment traffic and proceed gradually, moving only a small number of partitions at a time where possible to avoid a data movement storm.
The distinction is:
Partition resize: Creates more partitions for a topic.
Consumer rebalance: Redistributes partition ownership across consumers.
Replica rebalance: Moves partition replicas across brokers.
Leader rebalance: Changes which broker hosts the leader for existing partition replicas.
If the problem is consumer or producer parallelism, partition resizing may help. If the problem is broker capacity or uneven broker load, evaluate cluster scaling or broker/replica balancing options instead.
Key takeaway
Increasing a topic’s partition count in OCI Streaming with Apache Kafka is an online Kafka-compatible operation used primarily to increase producer and consumer parallelism. Existing records are not moved, deleted, or repartitioned, and committed offsets for existing partitions remain valid. New partitions are created empty and can receive future records.
The main impacts are possible consumer group rebalances, temporary lag or processing pauses, and possible key-to-partition remapping for future records on keyed topics.
Validate ordering requirements, perform the change during a controlled window, and monitor consumer lag, producer distribution, and broker health afterward.
Final checklist
Before resizing partitions
- Confirm the issue is parallelism, not broker capacity.
- Confirm the operation is changing topic partition count, not resizing the physical size of an individual partition.
- Confirm the current and target partition counts.
- Validate keyed-message and ordering requirements.
- Review consumer group count and consumer instance count.
- Check consumer lag and group stability.
- Check broker CPU, network, disk, and ISR health.
- If replica reassignment or balancing is planned, review the assignment, throttle movement, and proceed carefully on high-workload clusters.
- Plan the change during a controlled window for critical workloads.
During the resize
- Use
kafka-topics.sh --alter --partitions <new-total>. - Remember that the value is the new total partition count.
- Do not expect existing records to move.
- Expect subscribed consumer groups to rebalance.
After the resize
- Verify the new partition count.
- Confirm all partitions have leaders and healthy replicas.
- Monitor consumer lag and application latency.
- Confirm producers are writing to the expanded partition set.
- Watch for hot partitions or skewed keys.
Partition resizing is a powerful Kafka scaling tool. Used carefully, it can help OCI Streaming with Apache Kafka users increase application parallelism and improve throughput. The key is to resize for the right reason, understand the ordering and rebalance impacts, and monitor the workload closely after the change.