An offset is a record’s position in a partition, and a consumer group’s committed offset is its bookmark: the position it will resume from after a restart. The two most common operational tasks around consumer groups are exactly what this page covers, offset resets and lag investigation. Group behaviour itself, rebalances and the coordinator, lives on the consumer page.
Immediate operational needs
The working tool is kafka-consumer-groups.sh. Describing a group shows, per partition, the CURRENT-OFFSET the group has committed and the LOG-END-OFFSET of the partition, and the difference between them is the lag. Resets use the same tool with a strategy flag: --to-earliest to replay everything retained, --to-latest to skip to now, --shift-by with a positive or negative count to nudge, or a timestamp to land at a moment. A reset is a data decision as much as a command: backwards means reprocessing records, forwards means never processing them.
Lag must be read per partition, not per group. Monitoring lag per consumer group and per partition is one of the most operationally important signals in a Kafka-backed system, because a group total averages one stuck partition against its healthy siblings and hides it. The CLI answers all of this, and it answers it across several tools at once: debugging a stuck consumer typically means switching between kafka-consumer-groups.sh, kafka-topics.sh and whatever dashboards exist, and practitioners describe the CLI surface as an ocean of options.
My standing rule, and the offset-reset default from my quick-win list: set auto.offset.reset to latest in production, and make every reset a deliberate, manual operation. The reason is what the earliest default does on the day nobody expects it: change a consumer group id, deploy a service under a new name, or lose committed offsets past retention, and an earliest-configured group silently reprocesses everything from the beginning of every partition. With latest, the same mistake costs you a gap you can see and choose to replay. With earliest, it costs you a duplicate storm you discover from side effects.
The corollary is that resets deserve ceremony. During an incident the reset command is the fastest-looking fix on the screen, and it is also the one whose damage is decided by a flag. Decide forward or backward against what the downstream consumers can tolerate, write it down, then run it. Never let the reset be the reflex.
Technical contexts
Offsets are per partition, full stop. Partition 0, 1 and 2 of the same topic each have their own offset sequence starting at zero, and a consumer group’s position is the set of per-partition offsets, not one number. That is why every offset command and every lag reading is qualified by a partition id, and it is why a group coordinator exists: a designated broker manages the group’s membership and assigns each partition to exactly one member at a time.
Committed offsets are stored in Kafka itself, in the internal topic __consumer_offsets, keyed by group, topic and partition. Older legacy systems committed offsets to ZooKeeper, and that path is history: modern clients commit to the internal topic, which is replicated and compacted like any other Kafka data. The pattern generalises across the ecosystem, with Kafka Connect keeping its configuration, offsets and status in dedicated Kafka topics of its own.
The context that catches people during scaling: partition count increases are not free for consumers, and you need to know where your consumers will start reading from before you scale. New partitions have no committed offsets, so each consumer’s auto.offset.reset policy, earliest or latest, silently decides whether the group replays nothing or everything on the partitions that just appeared. The topic vs partition page covers the other half of that decision.
The detail that ties this section to the last one: a fresh partition starts at offset 0 with no committed position for any group, so the moment you add partitions, every consumer’s reset policy silently fires on the new partitions. The reset default you chose above and the partition increase you are planning in a capacity review are the same decision wearing two hats.
Migrations expose offsets as physical facts rather than logical positions. JPMorgan Chase found consumer group offset synchronisation to be a real operational complexity in active-passive replication with MirrorMaker 2.0, because the primary and replica clusters have different partition assignments and log segment layouts: offset 4,502,117 on the primary is not offset 4,502,117 on the replica. Any DR plan that says “consumers resume from their offsets” without offset translation is a plan to replay or skip data at the worst possible moment. Verify translated positions before cutover, with the same seriousness you verify the data itself.
FAQ
What is the purpose of an offset in Kafka?
An offset gives every record a stable position in its partition, and gives every consumer group an independent bookmark. Because the broker never tracks delivery, the committed offset IS the consumption state: it decides where a restarted consumer resumes, what lag means, and what replay means. One log can serve many readers at different positions precisely because each reader carries only a number.
How do I manage offsets in Kafka?
Day to day, offset management is three practices. Commit deliberately: disable auto-commit on any consumer doing real work and commit after the result is durable. Monitor per partition: lag is CURRENT-OFFSET against LOG-END-OFFSET, read per partition so a stuck partition cannot hide. Reset deliberately: kafka-consumer-groups.sh --reset-offsets with an explicit strategy, chosen against what downstream consumers can tolerate, never as an incident reflex. The sections above cover each in detail.