Kafka: The Complete Guide

Guides

Chad Harris·August 12, 2026·12 min read

Apache Kafka is a distributed event streaming platform that stores records in ordered, partitioned, replayable logs called topics. Producers append records to a topic, consumers read them at their own pace, and the log persists independently of whether anyone has read it yet. That single design decision, keeping the log rather than deleting on delivery, is what separates Kafka from a traditional message queue.

This hub is the entry point to everything we have written about running Kafka. It is organised in four parts: the fundamentals, day-to-day operations, governance and security, and the tooling landscape. Each section links out to the detailed guides.

I am a Solutions Architect at Factor House, and before that I ran Kafka in production at Block, Square and Cash App. Most of what follows comes from incidents I have either worked or watched closely.

Fundamentals topics · partitions · offsets · consumer groups Producer Consumer Operations brokers · consumer lag · rebalancing · KRaft Broker lag Governance & security mTLS · ACLs · quotas · audit mTLS ACLs Tools & alternatives Kpow · Kafka Streams · ksqlDB · Connect Kpow processor ksqlDB

Kafka fundamentals

A Kafka topic is an append-only log split into partitions. Each partition is an ordered sequence, and ordering is guaranteed within a partition, never across a whole topic. A partition is assigned to exactly one consumer in a group at a time, which is the constraint that governs how far a consumer group can usefully scale.

Records are retained by time or by size, not by whether a consumer has read them. That is why a Kafka consumer can be replayed from an earlier offset, and why retention is an operational decision rather than a cleanup detail.

Start with Kafka architecture for how the pieces fit together, then Kafka topics and partitions for the design decisions that are hard to reverse later.

F1 Kafka in production, on the public record /kafka/

12M/sec Messages per second at Uber, August 2021

200,000 Partitions in Uber's deployment

1.3T/day Messages per day at PayPal

2T/day Events per day at Netflix, at the top of its range

Figures as published by each company. They use different units and different reporting periods, so they show range rather than a like-for-like ranking.

Kafka operations

Five metric families cover a cluster’s operational state. Active controller count must be exactly 1: a value of 0 means no broker is coordinating the cluster, and a value of 2 means the cluster has split and two brokers each believe they are in charge. Under-replicated partitions should sit at 0, and a sustained non-zero value points to broker overload, a disk bottleneck or a network problem. Offline partitions must be 0. Consumer lag is the offset distance between the newest record on a partition and the last one a consumer group processed, measured per partition rather than per topic.

Those metrics tell you something is wrong. They do not tell you why, and that gap is where an incident’s minutes actually go.

Here is what that looks like in practice. A team running on AWS MSK had 400 service instances in a single consumer group, against 100 topics of 10 partitions each. Because each partition is assigned to exactly one member, only 1,000 assignments were possible, which left 39,000 idle members in the group, all still heartbeating to the group coordinator. Consumer lag climbed, so the team scaled up. At 800 instances there were 80,000 members, 79,000 of them idle, and still only 1,000 doing work. Rebalances took longer, lag got worse, and the group coordinator sat at 100% CPU. Adding brokers made it worse again, because that increased partition reassignment work on a coordinator already at its limit.

The fix was to scale back down to 100 instances.

The reason it took so long to find is the point of this section: the root cause was a lack of metrics on consumer group membership size and coordinator request rates. Every dashboard was green on the metrics they had. The number that would have explained it was not being collected.

For the metric-by-metric detail see Kafka monitoring and how to monitor Kafka consumer lag. For the alert thresholds we recommend, see Kafka cluster monitoring.

F2 Scaling the thing that is not broken /kafka/

What the team did What was actually happening

Starting state 400 service instances, 100 consumers each, against 100 topics of 10 partitions. 40,000 group members competing for 1,000 possible assignments. 39,000 sat idle.

Lag started building Doubled to 800 instances, on the reading that more consumers clear a backlog faster. 80,000 members, still 1,000 assignments. The extra 40,000 added heartbeats, not throughput.

Rebalances slowed Added brokers, to give the cluster more capacity. More partition reassignment work for a group coordinator already pinned at 100% CPU.

The fix Cut back to 100 instances. The ceiling was never instances. Partition count sets it, and nothing above it does work.

Partition count fixes the assignment ceiling, so every instance above it becomes coordination load rather than capacity.

Governance and security

Governance is the part teams instrument after they need it rather than before. Access control, audit trails and data masking are straightforward to add to a cluster that is already running, and painful to retrofit under a regulator’s deadline.

The practical questions are narrow: who can read this topic, who changed that retention setting and when, and does anything in this payload need masking before it reaches a screen. Those are answerable with configuration, but only if someone decided to collect the evidence in advance.

See Kafka security architecture for the underlying model.

F3 On knowing which dashboard to build /kafka/

It is hard to know what dashboards you should build until you have found the next problem.
Chad Harris, Solutions Architect at Factor House

From "Things that go bump in the night: Kafka operational issues and how to survive them".

Tools and alternatives

Kafka exposes its metrics over JMX, so every monitoring tool is a way of collecting, storing and displaying the same underlying data. Prometheus with the JMX Exporter, paired with Grafana, is the common self-hosted stack. Burrow, from LinkedIn, evaluates consumer lag as a trend over time rather than a single threshold, which avoids alerting on a brief spike. Datadog and Confluent Control Center bundle collection, storage and dashboards into a commercial product.

A Kafka console is a different tool answering a different question. A metrics stack tells you a number moved. A console shows you the topic, partition or consumer group the number belongs to. Teams that own only one of the two pay for it during an incident, and at scale the difference between a UI that surfaces under-replicated partitions clearly and one that forces an operator back to kafka-topics.sh is the difference between a five-minute fix and a thirty-minute incident.

For the landscape see Kafka UI: the ultimate guide, the best Kafka monitoring tools and the best free Kafka UI tools.

F4 A metrics stack and a console answer different questions /kafka/

A metrics stack tells you A console tells you

Controller Active controller count is 0, or it is 2. Which broker holds it, and which one also believes it does.

Replication Under-replicated partitions is above 0. Which topic and which partition, and which broker is behind.

Consumer lag Lag is climbing on a consumer group. Which partition it sits on, and which member owns that partition.

During an incident That a number moved. What the number belongs to, which is where the minutes actually go.

Both are worth having. Owning only one is what turns a five-minute fix into a thirty-minute incident.

Everything Kafka

Everything the Kafka cluster covers

Apache Kafka is an open-source distributed event streaming platform that stores records in ordered, partitioned, replayable logs. This is what it is, how the pieces fit, and when it is the right choice.

Kafka fundamentals

Kafka operations

Governance and security

Tools and alternatives

Everything the Kafka cluster covers

Related reading

Keep reading

Kafka consumers in production

Kafka in Docker