Journey to an open lakehouse: real-world lessons from streaming Kafka into Apache Iceberg

Karel Sague spent the last year migrating a production data platform from Snowflake to Apache Iceberg. In this session, he shares what he learned throughout that process, including what worked well and what he would approach differently.

Karel covers why the traditional split between operational and analytical data stores tends to create vendor lock-in and duplicated governance overhead, and how an open table format can address this. He walks through streaming Kafka data into Iceberg using Kafka Connect, including a standard CloudEvents-based schema he built to handle ingestion consistently across tables, GitOps-based self-serve provisioning, and the metrics and SLOs used to keep pipelines observable in production.

He then discusses migration in more detail. Karel explains why an all-or-nothing approach to migration can create unnecessary risk, how to prioritize which workloads move first, and the schema and partitioning challenges that arise when Kafka’s ingestion-time partitioning doesn’t align with analytics queries filtered by business key. He also covers consumer-aligned tables, materialized views, and upcoming developments in Iceberg, including v3 features, secondary indexes, and pluggable file formats for AI workloads.

This session will be useful if you are running Kafka and considering Iceberg, or if you are already migrating and want to compare notes on the practical challenges involved.