Apache Spark is a general-purpose engine for large-scale data processing. Its Structured Streaming API reads from sources such as Apache Kafka in micro-batches and writes to sinks such as files, Kafka topics and Apache Iceberg tables, with a checkpoint that records the query’s progress.
This hub is the entry point to what we have written about running Spark alongside Kafka. Factor House does not make a Spark product. Kpow manages Kafka and Flex manages Apache Flink, so the pages here cover the Kafka side of Spark pipelines and compare Spark with the tools Factor House does build for. The only Spark material Factor House publishes itself is in its open source Factor House Local environment, which includes Spark labs. The hub is new and will grow as we publish more.
Everything the Spark cluster covers
Why Spark does not commit offsets to Kafka, so a consumer group view shows no lag for it, how to read lag from Spark's own progress metrics, and how to restart a query without losing or repeating data.
Learn moreWhy PySpark fails with Failed to find data source: kafka, the Kafka connector package it is missing, and how to add it at launch with --packages.
Learn moreFive ways to monitor a Spark job reading Kafka, scored on query rates, offsets behind latest, history, alerting and portability. Factor House has no Spark product and is not scored.
Learn moreHow the two engines differ on execution model, state, Kafka offsets and delivery guarantees, from the Apache Spark and Flink documentation, and when each fits.
Learn moreKafka stores and moves streams of events and Spark processes data at scale. They are not alternatives, and most pipelines use both.
Learn moreHow to write a Kafka topic into an Apache Iceberg table with Spark Structured Streaming, which settings matter for commit rate and partitioned tables, and what maintenance a streaming table needs.
Learn more