Migrating to open source Kafka: cutting TCO without the operational burden

Chad Harris (Factor House) and Justin George (NetApp Instaclustr) opened by mapping the three ways teams consume Kafka today: the proprietary Confluent platform, true Apache-licensed open source Kafka, and a managed open source offering such as Instaclustr, where a provider takes on the operational overhead of patching, securing, and staffing for a cluster while the underlying engine stays vanilla Kafka. Confluent has since been acquired by IBM, and part of its platform now includes Kafka-protocol-compatible technology that sits outside the open source project itself, which is one of the reasons teams end up evaluating a move.

The session set out seven dimensions to weigh before deciding whether a migration pays off: cost and predictability, support and risk, innovation pace, flexibility and optionality, timing, developer experience, and vendor or cloud lock-in. On cost, a recurring problem is that the true bill is rarely visible on one line item — Amazon MSK costs can be split across networking, cross-availability-zone transfer, and ENI registration charges that never show up under "MSK" itself, making it hard to attribute spend fully to the Kafka footprint. Support often follows the same pattern, billed separately from infrastructure in a way that obscures what a self-managed setup actually costs to run.

Innovation was framed as one of the strongest cases for staying on open source: proprietary vendors build real capability, but keep it behind their own platform, while the pace of Kafka Improvement Proposals reaching the open source project keeps increasing year over year. Running Kafka in your own infrastructure also preserves flexibility and data sovereignty, since nothing needs to leave your cloud or VPC, and keeps procurement leverage open for enterprises that want cloud providers bidding for their business rather than assuming it. Timing matters too — forced broker-type migrations, such as MSK's move to Express brokers or Confluent's newer cluster types, offer no in-place upgrade path, so a team already paying for a forced migration should use that moment to evaluate whether to move providers entirely rather than just cluster types.

On developer experience and lock-in specifically, schema registry migration was called out as a frequent hidden cost: different registries support different lengths of schema identity, so moving between them, for example from MSK's Glue schema registry to Karapace or Confluent's registry, requires translating every schema ID rather than a straight import. Authentication choices carry a similar tradeoff — TLS, mTLS, or SASL/SCRAM travel cleanly between providers without touching client applications, but AWS IAM-native auth, while a strong fit inside AWS, locks every connecting client to that ecosystem and has to be re-implemented client by client on a move. Kafka Connect connectors carry the same risk: roughly half of Confluent's connector catalog is open source and portable, but a sink and its complementary source connector are often licensed differently, so a team can find the piece it depends on isn't the piece it can take with it.

Cluster-to-cluster replication was covered as the core mechanism for any migration. Cluster-linking style tools built by individual vendors, including Confluent's cluster linking, WarpStream's Orbit, and AutoMQ's equivalent, make it easy to bring data into a platform by extending Kafka's native replication protocol and preserving offsets, but none of that tooling has been contributed back to open source, so it only ever runs one way. KIP-1279 was highlighted as the proposal to watch, since it aims to bring that same offset-preserving replication into open source Kafka itself and make cross-vendor migration symmetric. Until then, MirrorMaker 2 remains the open, vendor-neutral option, translating consumer offsets between clusters even though the offset values themselves change, and Kroxylicious, a Kafka-aware proxy that sits in front of a cluster and can redirect producers and consumers during a cutover, was named as the technology quietly powering Confluent's own managed proxy offering.

The talk's central worked example priced the same workload, same throughput, retention, and multi-availability-zone architecture, across Confluent Cloud, Amazon MSK, and an Instaclustr-managed cluster with Kpow, and paired that hard-dollar cost with an availability-based cost of risk. Using a representative $25,000-per-hour cost of business interruption, MSK's three-nines SLA implies roughly $219,000 of acceptable annual exposure, Confluent's four nines brings that down to around $22,000, and a five-nines managed service brings it to roughly $2,190 — before even pricing in each provider's SLA exclusions, which on MSK explicitly rule out coverage for broker overload from legitimate traffic growth and for failures caused by bugs in the Kafka or ZooKeeper engine itself. Factoring that cost of risk into the total cost of ownership changed the ranking of the three options materially compared to looking at infrastructure and labor cost alone.

Audience questions closed the session. Zero-downtime migration was described as achievable when moving from Kafka-native sources, though harder to execute without an experienced partner. On tooling for teams running several Kafka providers at once, Kpow was positioned as giving a consistent experience across all of them during a migration, with open source alternatives like AKHQ covering the basics but lacking the RBAC, governance, and data inspection depth enterprise teams need. On the IBM acquisition of Confluent, the read was cautious optimism rather than alarm, with several vendors now investing in easier migration tooling as a signal that customers are demanding real portability. And on whether a migration is worth the cost at all, the guidance was a straightforward payback calculation: divide the one-time migration cost by the expected annual saving, and treat anything paying back within a year or two as worth pursuing.