Partition count is one of the first decisions you make when creating a Kafka topic, and one of the hardest to change later. The conventional advice is to partition generously because you can always add partitions but you cannot remove them. That advice is not wrong, but it is incomplete. Over-partitioned topics have real costs that accumulate quietly.

What partitions actually do

Partitions are the unit of parallelism in Kafka. More partitions means more consumers can read from a topic simultaneously. They are also the unit of replication: each partition has a leader and a configurable number of followers. Every partition has a cost in memory, file handles, and rebalance time. A topic with 500 partitions and three replicas means 1,500 partition replicas distributed across your brokers. That is manageable at small scale and expensive at large scale.

The rebalance cost of over-partitioning

When a consumer joins or leaves a consumer group, Kafka triggers a rebalance. During a rebalance, all consumers in the group stop processing. The duration of the rebalance is roughly proportional to the number of partitions being reassigned. A consumer group reading from a 500-partition topic will have a noticeably longer rebalance pause than one reading from a 20-partition topic. If your consumers restart frequently, this adds up.

How to estimate the right partition count

A reasonable starting formula: divide your target throughput in messages per second by the throughput a single partition can sustain on your hardware. For most workloads on modern hardware, a single partition can sustain 10-50 MB/s of write throughput. Measure your actual message size and target write rate, do the division, and add a 30% buffer. Start there. You can always add partitions later if you need more parallelism.

Adding partitions to an existing topic

Adding partitions to an existing topic is possible but has consequences. New messages are distributed across all partitions, including the new ones. Old messages remain in the original partitions. If your consumers rely on key-based partitioning for ordering guarantees, adding partitions breaks those guarantees for new messages. Test this in a staging environment before doing it in production, and make sure your downstream consumers can handle the change.

When to consider repartitioning

Repartitioning, meaning creating a new topic with a different partition count and migrating data to it, is the right answer when the partition count is so wrong that adding partitions will not fix it. We have done this twice for clients. It requires a parallel-run period and a careful cutover sequence, similar to a batch-to-streaming migration. It is not a quick fix, but it is sometimes the correct one.

Partition tuning is one of the findings that appears most often in our pipeline audits. If you are unsure whether your current partition counts are appropriate for your workload, the audit will give you a specific answer. The story behind the team explains how we approach this kind of work.