Things that go bump in the night: Kafka operational issues and how to survive them

A quick framing before we start: most Kafka problems aren't Kafka bugs. They're generally misconfigurations that made sense at the time, missing observability that hid a real signal, or a reasonable decision made without full context. Kafka is usually behaving exactly how you told it to. The config was wrong, misunderstood, or out of date, because the volume changed and nobody went back to change the config.

The first story starts with some consumer group basics. A consumer group is a set of consumers sharing the work of consuming a topic, and each partition is assigned to exactly one member at a time. The group coordinator is a special broker that manages membership and assignments, and when membership changes, a rebalance occurs and partitions get reassigned. Rebalances matter because consumption can pause during one, depending on the protocol you're using, and that can be a stop-the-world event where every partition of the topic stops consuming. The more members in a group, the more work the coordinator does, and frequent or slow rebalances mean lag builds up on your topic.

The setup for this story: a service was consuming from hundreds of topics, each with around 10 partitions, and the service had scaled to hundreds of instances. Every instance started a new consumer for each of those topics, so every instance had hundreds of threads with a unique consumer per topic, and every instance shared the same consumer group across all of them. The result was thousands of idle consumer members doing mostly nothing. The math: 400 instances, 100 consumers per instance, so already 40,000 members. But with 100 topics and 10 partitions each, at any point in time you can only have 1,000 assignments, which means 39,000 idle members sitting there doing nothing.

Rebalances started taking longer and consumer lag started building up. The team's initial diagnosis was that lag was building because there weren't enough application instances, so they deployed a hotfix and doubled the service to 800 instances. That's 80,000 members, but the same math still applies: still only 1,000 active assignments, so now 79,000 idle members instead of 39,000. Nothing improved, rebalances took even longer, and consumer lag climbed higher. One broker was now sitting at 100% CPU, so the next decision was to scale Kafka and add more brokers. Unfortunately, adding brokers moves partitions onto the new broker to share cluster load, which drives more partition reassignment, more rebalance activity, and more coordinator load, on a coordinator that was already at 100%. Twice in a row, scaling the thing that isn't broken didn't fix the thing that is, and made it worse.

The missing signal was that there were no metrics on consumer group membership size, no alerts on coordinator request rates, and nothing in the dashboards pointing at consumer group size being wrong. It took a vendor support case for the vendor to say there were way too many consumer groups, which had overwhelmed the group coordinator, which was the broker sitting at 100%. It had too much to do and couldn't reassign partitions in time, which led to lag, which led to the series of wrong decisions to try to solve it.

The immediate fix was to scale the deployment back down from 800 instances to 100, which relieved load on the group coordinator. The longer-term fix was separating consumer groups per logical consumer boundary, so instead of one group covering every topic, groups were split along the boundaries where it actually made sense for topics to share one. The lesson is to know the limits of your consumer group topology before you scale: if you're scaling because of lag, check that scaling is actually the correct move, and that you're scaling the right thing. Share groups for Kafka can help with fan-out, but you'll still want to limit membership count, since the same coordinator load problem applies whether you have 80,000 consumers as individual group members or as part of a share group.

The second story is the config that came back to haunt us. Some broker replication basics: each partition has a leader and one or more replicas, and replicas fetch from the leader to stay in sync. Two settings matter here: the number of replica fetch threads controls how many threads exist to pull data in, and the number of I/O threads controls disk I/O thread count. These settings have real implications for disk and network.

The cluster wasn't having problems; the goal was just to make rebuilds and rebalancing faster, so that if a broker was lost, the rebuild would be quicker. This was on MSK. The provisioned storage throughput was scaled up per the vendor's recommendation, and the number of replica fetch threads and I/O threads were increased to match. The cluster upgrade failed mid-process and automatically rolled back, which MSK is meant to do. The provisioned storage throughput reverted, but the replica fetch and I/O thread counts that had been increased did not revert, and stayed at the new, higher level.

It ran fine for six months. Load increased occasionally, some topics got deleted to relieve pressure, and it masked the underlying disk I/O pressure that was building. Then one day a single disk failed on one broker, and Kafka did exactly what it was meant to: it started rebuilding the broker and bringing those partitions back online. The problem was that the disk couldn't handle the IOPS from the aggressive replica fetching during recovery, because the increased threads and I/O throughput were pulling in more data than the disk could handle. Producer errors spiked and the cluster looked unstable. One team started getting producer timeout exceptions, then two, then ten, and that's when it became clear the problem was systemic. All the timeouts traced back to one broker, and buried in the vendor dashboards was an IOPS saturation metric on exactly that broker, the one being rebuilt.

It took someone looking specifically at that broker's config to notice it didn't match the vendor's best practices, and going back six months revealed what had happened: the update had been partial, the disk change hadn't gone through, but nobody checked that the thread count increases had also rolled back. A rollback isn't done until every change is verified and reverted, and config drift is silent until it isn't. The fix was to temporarily stop that broker from taking leadership, using the preferred leadership API, since you only get produce requests when you're a leader. Once the broker rebuilt, leadership was restored, the drifted config was rolled back, and the change was retried properly.

The third story is the messages that were never read. A team scaled a service by increasing the partition count on their topics: a deliberate, planned change to get more throughput once they hit the limits of the existing partition count. Producers refreshed quickly and started writing to the new partitions. Consumers refreshed too, but took a little longer to rebalance and start reading from the new partitions. In that gap, messages landed on partitions the consumers didn't know existed yet.

The problem was auto.offset.reset set to latest. By the time consumers discovered the new partitions, they started reading from the end of each one, so any messages produced before they were assigned were silently skipped. Across around a hundred partitions, hundreds of messages per partition were quietly dropped. There were no errors and no alerts, because the system behaved exactly as configured. It wasn't noticed for days, until each of those missed messages, which should have moved a workflow into a new state, left workflows stuck, and business process rules started flagging SLA breaches. The alert fired on a business logic error, not on Kafka, and the initial complaint was that Kafka had lost messages, which it hadn't: inspecting the topic showed the messages were still there, just never read.

The fix was painful, since this was a real-time system that couldn't stop processing. A parallel consumer was spun up specifically to replay the missed offsets, consuming the historical messages without interfering with real-time processing. Tens of thousands of messages were replayed and the stuck workflows recovered. The lesson is that partition count increases aren't zero cost for consumers, and you need to know where your consumers will start reading from before you scale. Using auto.offset.reset set to earliest avoids this particular trap, but it comes with its own risk: a new consumer starting from earliest on a topic with terabytes of data can take months to catch up, causing noisy-neighbour load and laggy consumption. If a consumer doesn't need the full topic history, don't use earliest.

The last of the main stories is the 20-minute timeout. A team wanted to increase max.poll.interval.ms to 20 minutes, because they wanted to pre-batch thousands of messages into one before sending to Kafka, on the reasoning that this would be more efficient than sending many small messages. Because it then took longer for the consumer to process a batch, they wanted to raise the poll interval to match. That was strongly discouraged in favour of Kafka's native batching, but the exception was granted, and the interval was set to 20 minutes.

What max.poll.interval.ms actually controls is the maximum time between calls to poll. If a consumer doesn't poll within that window, the coordinator assumes it's dead, removes it from the group, and triggers a rebalance, reassigning its partitions to other consumers. Setting this to 20 minutes means a dead or stuck consumer won't be detected for 20 minutes either. It ran fine for 12 months, with processing time staying under the 20-minute window, until one day a batch didn't process: a large batch, effectively a poison pill, exceeded the window. The consumer failed to poll, the coordinator declared it dead, a rebalance triggered, a new consumer picked up the partition, tried the same batch, exceeded the timeout again, and repeated. This happened at around 2am.

The resolution that night was to vertically scale the service, but that was luck: vertical scaling won't always save you, especially if you're already on the largest instance your organisation allows, or if the real bottleneck is a third-party rate limit rather than compute. Eventually the service was rearchitected to remove the pre-batching entirely and send a single message at a time, letting Kafka handle batching at the producer and consumer level. The premature optimisation they were trying to avoid building turned out to be the one they built anyway, twelve months later, at much greater cost. The lesson: max.poll.interval.ms is a failure detection timeout, not a processing budget. Pre-batching in the application fights against Kafka's own efficient, adaptive batching, and one slow or broken message can take out an entire partition. Incremental cooperative rebalancing reduces the blast radius, but a poison pill will still get a partition stuck.

A few quick wins and patterns to close with. Don't set auto.offset.reset to earliest in production just in case: it's easy to forget about, and if a consumer group ID ever changes, even for an innocent reason like renaming a service, it will reprocess everything from the beginning and cause huge lag. Use latest in production and make offset resets a deliberate, manual operation.

Keep max.poll.interval.ms small. It's a failure detection timeout, not a processing budget, and a large value slows both rebalances and failure detection. Five minutes is a reasonable default; if processing a message takes longer than that, Kafka is probably the wrong tool for that particular workload, and something like SQS, which caps out at 15 minutes and isn't configurable beyond that, might be more appropriate, or the message might need to move somewhere else entirely.

Watch out for the dead letter queue carousel: when an ETL reads from and writes back to the same DLQ or retry topic, a failure can immediately re-enter the loop and grow the backlog indefinitely. Add a retry count header so that once a message exceeds a limit, it errors out instead of looping. The same pattern shows up with MirrorMaker, where replicating a topic from region A to region B, and accidentally back again, sets up a loop where message throughput scales rapidly. Any process that writes back to a topic it reads from needs a way to break that loop.

Log retention set too high on high-throughput topics is a slow-burn problem. Seven days is a common default, but gigabytes per hour for seven days adds up to a lot of disk, and disk fills slowly over weeks until it's suddenly urgent. Running out of disk is about the worst state Kafka can be in, and very hard to recover from. Monitor disk usage per topic, and set retention bytes as well as retention time on high-throughput topics, so a topic's on-disk size is capped even if that means deleting data earlier than the time-based retention would.

Max message bytes set too high is a trap that tends to spring later. A topic might start out low-throughput, so a larger message size limit seems harmless, but six months later the topic becomes popular and the large messages cap its throughput, with no real fix at that point. Kafka handles millions of small messages a second extremely well, and handles large messages poorly. Where possible, keep messages small, and if a payload is genuinely large, store it externally, such as in S3, and put a reference token on the topic instead.

Linger.ms, which defaults to 5 milliseconds since Kafka 4, is worth using for most workloads. A lower value increases CPU usage and network connections, and reduces throughput in exchange for lower latency. Reserve linger.ms of zero for the topics that genuinely need low latency, and leave the rest at the default.

A bonus problem: not using the official Apache Kafka or Confluent client libraries. Differences in protocol interpretation between third-party client implementations cause quiet, compounding problems. If you're not on the JVM, use a client library that wraps librdkafka rather than one that reimplements the Kafka protocol itself. One case involved a message header that should have written four bytes but wrote three, which looked harmless until an official client started rebalancing whenever it encountered those messages, and the effect built up across a cluster over days.

One more: idempotent versus transactional producers. Only use transactional producers if you're producing across multiple topics or consuming across multiple topics as part of the same transaction, and pair them with a read-committed consumer. There's a common misconception that transactional producers eliminate duplicates in Kafka; they don't. Within a single batch you won't get duplicates, but that doesn't remove duplicates altogether, so a client that assumes otherwise, and isn't itself idempotent, will still end up with duplicate message processing. Transactional producers also add overhead for the brokers, so save them for when you're actually using the Kafka transactions API rather than turning them on everywhere by default.

On signals that actually matter: most teams already track consumer lag, broker CPU and memory, network throughput, and under-replicated partitions. These are useful for telling you something is wrong, but not why. Something worth internalising is that managed services also need monitoring: Kafka health is a shared responsibility. Confluent, AWS, and Aiven do a great job of hosting Kafka, but they aren't responsible for how your client code behaves, and client code has a major effect on cluster health. Data inspection tooling is essential for handling lost-message complaints, since the large majority of the time, the message was on Kafka all along. And make changes judiciously: one change at a time, monitored before the next, rather than rolling a config change and a disk increase into the same window.

Treat broker config changes like code where you can: version-controlled and applied through CI. After any rollback, diff the running config against your baseline, and periodically audit the running config against your documented best practices, since some environments keep a break-glass option for emergency config changes that never quite makes it back into GitOps.

The pattern behind these incidents is consistent: a reasonable decision made without full context, a signal that was missing or misread, and an action that made things worse before the cause was understood. The fix always looks obvious in hindsight, which is the nature of hindsight. You can't prevent every incident, but you can get faster at recognising the shape of them. Don't panic, think before you act, and don't scale Kafka as a reflex during an incident; scaling Kafka afterwards, once you understand the cause, is a different matter. And don't wait until you're in a crisis to open a support case: some of these incidents were only cracked open by a vendor pointing out something the team couldn't see from the inside, so use support as a debugging tool rather than a last resort.