Spring Boot integrates with Apache Kafka through Spring for Apache Kafka, the spring-kafka library. It wraps the Kafka Java client in the programming model Spring developers already use: KafkaTemplate for producing, the @KafkaListener annotation for consuming, and listener containers that manage threads, polling and offset commits behind the annotation.
The integration is production infrastructure, not a convenience layer. Large consumer-facing platforms run their Kafka producer and consumer services on Spring Boot, including Adidas, whose application-layer event streaming services are Spring Boot applications, and Goldman Sachs, whose services use the Spring Kafka client library.
A production-grade Spring Boot Kafka service is defined by four decisions, and they map to the sections of this page: how the dependency and configuration are set up, how failures are handled without stopping the consumer, how the connection to a secured cluster is authenticated, and how the listener is tuned for throughput. The hub holds the wider cluster context.
Quick setup and configuration
The integration starts with one dependency: spring-kafka from Maven or Gradle, which Spring Boot auto-configures from properties under spring.kafka in application.yml. Bootstrap servers, serializers, consumer group id and every producer or consumer property can be set there without writing configuration code.
Serialization is the first real decision. String and JSON serializers cover early development, and Avro with a schema registry is the production norm because it versions the message contract instead of trusting both sides to agree. Deserialization is where setups break silently: a message the consumer cannot deserialize fails before the listener method is ever called. Spring Kafka catches deserialization errors through its error-handling deserializer, while Kafka Streams requires a custom handler for the same failure.
Modern client versions have safer defaults than older tutorials assume. From Kafka clients 3.2.0 the idempotent producer is enabled without extra configuration in a producing application, so exactly-once writes to a partition no longer require hand-set flags.
Here is a Spring-specific trap I have had to explain more than once, because it looks like a monitoring bug and is actually a configuration choice. If you use @KafkaListener with topicPartitions to pin specific partitions, Spring calls .assign() on the consumer instead of .subscribe(). The consumer works, but it never joins a consumer group, so it shows up in tooling under simple consumers, with no group membership, no rebalancing and no group-level lag tracking. Teams then go hunting for their “missing” consumer group. Nothing is missing. The annotation opted them out of group management, and the docs do not make that loud. Use topicPartitions only when you genuinely want manual partition assignment, and know that group semantics leave with it.
On serialization, my advice is boring and firm: pay the Avro and schema registry cost on day one. The Netflix numbers make the efficiency case on their own, roughly 3 to 5 times smaller than JSON on the wire, but the real reason is the contract. Every JSON-topic team eventually has the week where a renamed field takes down a consumer fleet.
Production reliability
The default consumer behaviour is to commit offsets automatically, which means a message can be marked consumed before your code has finished processing it. Production listeners switch to manual acknowledgment so the offset commits only after the work is done, and a crash mid-processing replays the message instead of losing it.
A failing message must not stop the partition. The standard Spring Kafka pattern is DefaultErrorHandler with DeadLetterPublishingRecoverer: the listener retries a failed record a bounded number of times, then publishes it to a dead letter topic and moves on. Spring Kafka preserves the original message key on the dead-lettered record by default, so the failed message can be traced back to its partition and replayed in order.
Blocking retries hold up every message behind the failed one, which is why non-blocking retry topics exist. A dedicated retry topic per consumer isolates replay workloads, scales independently of the main topic, and prevents one consumer’s failures from causing duplicate processing in another. The dead letter topic is the end of that chain: the give-up path for records that need manual intervention.
On the producing side, idempotence is the reliability floor. An idempotent producer requires acks=all and retries greater than zero, and Kafka enforces both automatically when idempotence is enabled, which removes the duplicate-on-retry failure mode. The full producer picture is on the producer page.
The dead letter topic is where I see the most designs that work in the demo and fail in the estate. Two traps specifically.
The first trap is the replay path. Putting dead-lettered messages back on the main topic works at low scale or with a single consumer. With multiple consumers on that topic, every one of them replays the message, including the ones that processed it fine the first time. That is why I push the dedicated retry stream per topic and consumer pair, with the DLQ strictly as the give-up, needs-a-human path: failures stay attributed to the consumer that failed, and replay reaches only the consumer that needs it.
The second trap is the carousel. An ETL that reads from and writes back to the same DLQ or retry topic can loop a poison message indefinitely, quietly, at full throughput. Add a retry-count header and enforce a ceiling; it is one header and it breaks the loop.
Neither trap is hypothetical. Uber’s insurance engineering documented the tiered version of the retry-topic pattern, payments.retry-1, payments.retry-2, then a final DLQ, and Robinhood found the opposite edge: a bare DLQ topic stores failures fine but gives you no way to query them by failure reason, fix them, or selectively replay a subset. The topic is the easy half. The operations on it are the design.
Security and connectivity
A secured Kafka cluster rejects the plaintext connection every getting-started guide assumes. Connecting a Spring Boot service to one means setting three things in configuration: the security protocol, the SASL mechanism, and the JAAS credentials. In properties terms that is spring.kafka.security.protocol, spring.kafka.properties.sasl.mechanism and the sasl.jaas.config string.
For new deployments without existing Kerberos infrastructure, SCRAM-SHA-512 over TLS is the most practical starting point: username and password authentication with the credentials never crossing the wire in the clear. Managed cloud Kafka services commonly run exactly this combination, SASL_SSL with SCRAM-SHA-512, with credentials held in the cloud provider’s secret vault rather than in application configuration.
Enterprise clusters go further. Production deployments at large companies enforce authentication and encryption on every client connection with SSL and mutual TLS, and brokers expose multiple listeners with different security protocols so internal, external and controller traffic are isolated from each other. A Spring Boot service connects to the listener intended for application clients, not the inter-broker one.
Two additions from the field. First, where your credentials live matters as much as which mechanism you chose, and Spring’s property binding has real edges here: a documented Spring Boot 3.4.4 case reports unbound-property failures when cluster configuration is injected via environment variables, which blocked pulling secrets from AWS Secrets Manager. Test your exact injection path in staging with the real secret store, not with an application.yml full of literals that will never ship.
The second addition is the direction of travel: SASL/OAUTHBEARER is projected to be the standard mechanism for cloud-native Kafka deployments. SCRAM gets a new service connected today, and if your organisation is standardising identity on OIDC, wiring the callback handler now beats migrating credentials later. The cautionary shape of getting this wrong at scale is old and well documented: before Pinterest built a client abstraction, applications carried direct Kafka client dependencies with hardcoded broker hostnames and SSL passwords, and every maintenance window was an outage risk. Centralise the connection config, whatever mechanism you pick.
Performance and scaling
Listener concurrency is bounded by partition count. Setting the container’s concurrency gives you that many consumer threads, and any thread beyond the number of partitions sits idle, because a partition is consumed by exactly one member of a group at a time. Historically that ceiling forced teams to over-partition topics; the topic vs partition page covers why that decision is made at creation.
The biggest tuning win needs no architecture change: producer compression combined with sensible batching. Compressed batches move more records per network round trip and store more records per disk byte. On the consumer side, batch listening processes a whole poll of records in one listener call, which suits high-volume, low-cost-per-record work, while single-record processing keeps failure isolation simple.
Two limits deserve respect rather than tuning. max.poll.interval.ms is a failure-detection timeout, not a processing budget: raising it to accommodate slow processing means one slow message can take out an entire partition before the group notices. And Kafka excels with small messages at enormous volume while performing poorly with large ones, so the fix for slow processing is usually smaller records or more partitions, not bigger timeouts. Well-tuned modern clusters sustain 50 to 100+ MB/s per partition.
The mistake I see most in Spring services specifically is pre-batching in the application: collecting records into an application-level buffer before processing, on the theory that batches are efficient. Kafka already batches, efficiently and adaptively, at the producer, in the protocol, on disk. An application-level batch on top of it fights that design, and it welds your processing time to your batch size, which is exactly the coupling that turns one slow record into a partition-wide stall. Process records as they come, size the poll with max.poll.records, and let the infrastructure do the batching it was built for.
Kafka’s sweet spot is millions of small messages a second, and it is genuinely bad at large ones. When throughput is the goal, the ceiling is real and high: DoorDash peaks around 8 million messages per minute. If your per-message processing cannot keep up inside that shape, the answer is smaller messages or more partitions, and the partition decision belongs at topic creation.
FAQ
What is the use of Apache Kafka in Spring Boot?
Spring Boot uses Kafka as its messaging backbone through the spring-kafka library: KafkaTemplate publishes events from your services, and @KafkaListener methods consume them, with Spring Boot auto-configuring both from application properties. The practical uses are the event-driven patterns this page covers: publishing domain events between microservices, consuming streams for processing, and wiring retry topics and dead letter topics around the listeners so one bad message never stops a partition.