MirrorMaker 2 on Connect: migrating off connect-mirror-maker.sh

Key takeaways

• Dedicated mode is not deprecated. Move for the control plane rather than the replication, and only if you already run Kafka Connect.
• Replication progress does not migrate. A new source connector starts at offset zero with auto.offset.reset=earliest, so seed every partition before you start it.
• sync.group.offsets.enabled defaults to false. MM2 writes checkpoint records but never commits translated offsets, so set the flag or translate manually.
• Translation is unavailable for groups that were already behind when MM2 started, and it only lands on groups that are idle on the target. Let every group reach zero lag before you flip.
• tasks.max defaults to 1, so omitting it hands every partition to a single task.
• Do not run old and new over the same topics. They share internal topics and heartbeats, so split by topic set or freeze and cut over.

Introduction

Most teams that end up migrating MirrorMaker 2 onto Kafka Connect started in dedicated mode, because that is where the documentation sends them. Apache’s MirrorMaker 2 documentation only covers the dedicated cluster. Its opening line is that the following sections “describe how to configure and run a dedicated MirrorMaker cluster,” and running MM2 inside an existing Connect cluster is deferred to a KIP.

Dedicated mode is also the easier start: one file, one script, no Connect cluster to stand up or size.

The limitations show up later: there is no management API, so changing anything means editing the file and restarting the process, and on a multi-node cluster that restart could quietly fail to apply the change at all. Scope is next, then the argument for and against moving. Everything after that is what breaks on the way.

Scope

You run connect-mirror-maker.sh with an mm2.properties. You want those three connectors deployed on a Kafka Connect cluster you operate, so you get the REST API, per-connector lifecycle control, and the monitoring and tooling your Connect cluster already has.

Out of scope: MirrorMaker 1, removed in Kafka 4.0. It has none of the machinery below, so migrating from it is building a new deployment rather than converting one. Also out of scope are operator- or service-managed platforms like Strimzi’s KafkaMirrorMaker2 and MSK Connect.

Those hand the Connect worker to something else, so the commands below do not apply verbatim. The seeding step does survive on Strimzi: from 0.44 you can list, alter and reset offsets for KafkaMirrorMaker2 connectors with the strimzi.io/connector-offsets and strimzi.io/mirrormaker-connector annotations plus a ConfigMap (Strimzi). MSK Connect has no equivalent mechanism, so the freeze-window path is the one to use there.

Throughout, the example is a one-way flow from us-west to us-east, with a consumer group called orders-api.

You need Kafka 3.6 or later on the Connect side. The step this article depends on is altering connector offsets: MirrorSourceConnector.alterOffsets() and the PATCH /connectors/{name}/offsets route both arrived in 3.6. Reading offsets is older than that. Writing them is what carries the version bound. Below 3.6 the seeding strategy is unavailable, and the freeze-window path is the one to use.

Should you do this at all?

Dedicated mode is not deprecated. It still ships in Kafka 4.3, and the Kafka team explicitly rejected abandoning its distributed mode.

So the reason to move is not that dedicated mode is broken, but that it gives you no control plane. There is no management API: the config file is the control plane, so changing anything means editing mm2.properties and restarting the process. KIP-710 added an internal REST server, controlled by dedicated.mode.enable.internal.rest, but it registers only the internal endpoints (InternalMirrorResource: writing task configurations and fencing zombies) and none of the public management API.

There is no /connectors, no /connectors/{name}/config, and no offsets endpoint. Whether that was deliberate is murky. The maintainer who fixed the fallout concluded the public API had been “intentionally left out of the implementation” to avoid a security divergence.

Internal endpoints are secured by default. The connector-config endpoint is not. Two consequences followed:

• KAFKA-15372: on a multi-node cluster, a rolling restart could silently drop configuration changes. Leadership moves between nodes as each one restarts, so a node can miss the leadership window and fail to apply its config. The workaround was to fully stop and start the whole deployment, because the internal REST does not expose the connector-config endpoint.
• KAFKA-10582: a new source topic got its remote topic created but never started replicating until MM2 was restarted. Users reported it on releases from 2.5 through 3.3, and the fix only takes effect if you also set dedicated.mode.enable.internal.rest=true.

In Connect-managed mode that whole class of problem disappears, because PUT /connectors/{name}/config is an ordinary REST request and Connect forwards it to the leader for you.

Four things follow, permanently:

• Config changes stop bouncing the process. PUT /connectors/{name}/config restarts the connector and its tasks, so replication still pauses briefly. What you gain is that the worker stays up and Connect rebalances the tasks itself.
• Per-connector and per-task control, including pausing the checkpoint connector without stopping replication and restarting a single failed task.
• Offset management through GET, PATCH and DELETE /connectors/{name}/offsets, which is the only supported way to skip a poison record, rewind a topic, or recover a connector that has started re-replicating.
• A desired-state surface for runbooks, CI checks and GitOps reconcilers.

That said, this is operational benefit, and it only pays off if you operate at a rate where it matters. If your replication config changes a few times a year, you have never needed to intervene in a connector’s offsets, and you don’t already run Connect, the migration buys you very little and costs you per-flow isolation. The strongest reason to move is usually not capability.

It is consolidation: MM2 as a tenant of the Connect topology you already run, monitor, secure and upgrade, rather than a second process with its own fleet and no API. Some platforms also leave you no choice.

Move if one of the four capabilities above is something you have wanted and could not have. Also move if you already run Connect and want MM2 on the same control plane, or if your platform leaves you no choice.

Don’t move if your replication config is stable, you have never needed to intervene in a connector’s offsets, and standing up or joining a Connect cluster would be new work. Dedicated mode is supported and it works, and this migration is a trade rather than an upgrade. Here is what you would give up:

• Per-flow isolation. Each flow gets its own logical Connect cluster: its own config, offset and status topics, and its own worker group. On a shared Connect cluster every MM2 connector shares one worker fleet and one connect-offsets topic, so a hot flow can starve the others.
• One file for the whole topology. Fan-out, aggregation and active/active fall out of mm2.properties. On Connect, topology becomes a deployment-placement decision.
• Worker state on the target, and heartbeat wiring that is automatic.

KIP-710’s authors rejected documenting distributed dedicated mode as unsupported, on the grounds that it “provides essential improvements over the vanilla Connect mode”. That reasoning still holds, which is why this article covers how to move rather than whether to.

Why this migration is dangerous

connect-mirror-maker.sh is a Connect driver, not a wrapper. It creates one internal Connect cluster per replication flow (a worker plus a herder each), and that cluster’s elected leader writes the three connector configs itself. The worker’s config store, offset store, and status store live on the target cluster, named after the source: mm2-configs.<source>.internal, mm2-offsets.<source>.internal, mm2-status.<source>.internal, with group.id = <source>-mm2 (MirrorMakerConfig).

So the aliases are the deployment’s identity, and the driver’s state is what makes replication resume where it left off. In Connect-managed mode the connectors become ordinary configs in your Connect cluster, their progress moves to your connect-offsets topic, and their identity becomes the connector name you chose.

The consequence runs through the rest of this article: configuration can be retyped, but state cannot be moved. Almost every serious failure here comes from something that lived in the driver’s state and was left behind.

The eight catches that change what you do

Catch 1: Replication progress does not migrate, and the naive move duplicates everything

What breaks. You deploy the connectors, point them at the same source and target, and the target fills with a second copy of every record from the beginning of time.

Why. The source task stores its position in the Connect worker’s offset store under {cluster, topic, partition} and seeks to storedOffset + 1. In dedicated mode that store is mm2-offsets.<source>.internal on the target. On your Connect cluster it is connect-offsets, and it is empty for these connectors. MM2 also sets auto.offset.reset=earliest unless you override it (it is a putIfAbsent, so consumer.auto.offset.reset wins), and with no stored offset that means “start at zero”.

Do this. Seed every partition before replication starts, using MirrorSourceConnector.alterOffsets(). Connect refuses unless the connector is stopped. modifyConnectorOffsetsChecks requires target state STOPPED and zero tasks. Otherwise Connect returns a 400 reading “Connectors must be in the STOPPED state before their offsets can be modified.” Setting enabled=false does not satisfy it. That is MM2’s own switch, and it leaves Connect’s target state at RUNNING.

CONN=http://connect:8083/connectors/<connector>

curl -sS -X PUT $CONN/stop          # required before offsets can be altered

curl -sS -X PATCH $CONN/offsets -H 'Content-Type: application/json' -d '{
  "offsets": [
    {"partition": {"cluster": "us-west", "topic": "orders", "partition": 0},
     "offset":    {"offset": 14829371}}
  ]}'

curl -sS -X PUT $CONN/resume        # begin replicating from the seeded position

The keys are cluster (the source alias), topic and partition. The offset is the last source offset already replicated. Get the values by freezing producers, confirming the old deployment is at zero lag, and seeding HWM - 1 per partition, or by decoding the old mm2-offsets.<source>.internal topic.

Then prove it: after resume, confirm target high-water marks do not move until new records are produced. A connector that reports no lag seconds after starting on a large topic has more likely failed to start than caught up.

Catch 2: The data path is the Connect worker’s producer, not MM2’s

What breaks. You deploy onto a Connect cluster whose worker is attached to the source cluster, and replicated records are produced back into the source. Or you tune target.producer.compression.type and nothing happens. Or, on a shared cluster still using the default JsonConverter, replicated payloads come out re-encoded.

Why. MirrorSourceTask.poll() returns SourceRecords and has no producer of its own. The Connect framework’s producer writes them, and that producer is built from the worker’s bootstrap.servers plus the worker’s producer.* settings, with the connector’s producer.override.* layered on top. MM2’s own producer.* and target.producer.* keys configure only its internal clients: MirrorSourceConfig uses them for the offset-syncs producer and nothing else. This never bit anyone in dedicated mode, because the driver copies the target alias’s client config into the worker config and its worker is the target cluster. On a Connect cluster you supply, neither is automatic.

Do this. Three things need to be right:

1. Point the worker at the target. The worker’s bootstrap.servers is where replicated records are produced. target.cluster.bootstrap.servers is not the data destination: it is what MM2’s admin clients use to create topics and sync configs.
2. Tune the data path in the right place. acks, compression.type, batch.size, linger.ms and max.request.size belong in the worker’s producer.*, or in the connector’s producer.override.* . The override policy has to permit it, and the distributed default All does. MM2’s producer.* will not reach them. The transactional producer used for exactly-once is the framework’s too, which is why exactly.once.source.support is a worker setting.
3. Set the converters to ByteArrayConverter. The dedicated driver sets key.converter, value.converter and header.converter to ByteArrayConverter for its worker. A shared Connect cluster usually defaults to JsonConverter, which re-encodes MM2’s byte payloads. Set them on the connector so you do not disturb anything else on the worker.

Catch 3: Offset translation is off by default in the sense that matters

What breaks. Your runbook says consumers will resume near where they left off. After cutover they start wherever auto.offset.reset puts them.

Why. MirrorCheckpointConnector always writes checkpoint records to <source>.checkpoints.internal on the target. It only commits translated offsets into the target’s __consumer_offsets when sync.group.offsets.enabled=true, and that defaults to false. Checkpoints on their own are a data source that nothing acts on, and Kafka clients never read that topic.

Do this. Choose deliberately. Automatic: set sync.group.offsets.enabled=true on the checkpoint connector. Manual: translate with RemoteClusterUtils.translateOffsets() and apply the result yourself. There is no Apache CLI for the translation, so at scale this means a small tool plus kafka-consumer-groups.sh --reset-offsets --from-file.

Catch 4: Translation only lands on idle groups, and is unavailable for lagging ones

What breaks. Some groups get sensible offsets, some get offsets behind their real progress, and some get nothing at all, silently.

Why. Three mechanisms. (a) Checkpoint records are written for every group that passes the topic filter, whatever its state. What the EMPTY check gates is the commit. refreshIdleConsumerGroupOffset() only records target offsets for groups in state EMPTY, and alterConsumerGroupOffsets fails with UnknownMemberIdException if the group is active on the target. (b) OffsetSyncStore refuses to translate when the newest sync is ahead of the group’s committed offset. The javadoc says translation is unavailable “if replication started after the position of the consumer group,” so a group already behind when MM2 started gets nothing. (c) Elsewhere it is approximate by design: when the group is ahead of the newest usable sync, MM2 translates to at most one record past that sync, preferring re-delivery over data loss. offset.lag.max (default 100) bounds sync frequency, and the config doc gives the arithmetic: “Partition Count x offset.lag.max = Approximate duplicated record count.”

Do this. Design consumers for at-least-once across the cut. Let groups reach zero lag before flipping. Use a fresh group ID for every rehearsal: resetting offsets does not clear MM2’s checkpoint state, so stale checkpoints keep winning.

Catch 5: tasks.max defaults to 1, and the ten-partition rule you may have read is obsolete

What breaks. Two separate problems get conflated here. Set tasks.max too low and one task carries every partition, so replication lags well behind what the cluster can absorb. On Kafka 3.4.0 and earlier there was a second, worse failure: assigning more than ten partitions to a task could stall offset translation for those partitions permanently.

Why. tasks.max defaults to 1. Omitting it hands every partition to a single task, which is the usual cause of “MM2 is slower than it should be”. The ten-partition problem came from a bug rather than a design limit. OffsetSyncWriter caps in-flight offset-sync records with MAX_OUTSTANDING_OFFSET_SYNCS = 10, a producer backpressure buffer that has nothing to do with partition counts. In KAFKA-12558 the partition state was mutated before the send was confirmed. Once that buffer saturated, the sync was never retried. The reporter described the result as “additional offset syncs are unlikely to arrive for a long time, if ever.” It is fixed in 3.4.1 and 3.5.0. KAFKA-12558 lists 3.3.3 as a fix version, but no 3.3.3 release exists: 3.3 ended at 3.3.2. The constant remains because it is a buffer rather than a cap.

Do this. Set tasks.max from your partition count and throughput target, and confirm it under load rather than trusting a formula. On a version older than the fix, keep it to ten partitions per task.

Catch 6: You cannot cleanly run old and new side by side over the same topics

What breaks. A plan to run both deployments side by side and compare them produces duplicated heartbeat records, two writers on one checkpoint topic, and, if both are dedicated-mode, one silently winning the config for the whole cluster pair.

Why. With matching aliases the deployments share mm2-offset-syncs.<peer>.internal, heartbeats, and <source>.checkpoints.internal. Apache’s docs warn that MM2 processes sharing a target cluster share configuration: “either the topic foo or the topic bar is replicated, but not both.” And you can’t partially disable checkpointing: emit.checkpoints.enabled=false is rejected by validation, so the whole connector must be disabled.

Do this. Prefer a freeze-window cutover (see the playbook). If you must run in parallel, split by topic set, prove the sets are disjoint by listing them, and set heartbeats.replication.enabled=false on exactly one side.

Catch 7: Aliases and separator are baked in, and alias drift disables cycle detection

What breaks. Renaming an alias after go-live renames every replicated topic and orphans the internal topics. Worse: if the two directions ever disagree on the alias strings, cycle detection stops working. Each side re-replicates the other’s prefixed topics, so names grow by a prefix per hop. At Kafka’s 249-character topic-name limit (Topic.MAX_NAME_LENGTH), topic creation fails and every connector task dies (Instaclustr).

Why. The alias appears in the internal topic names, the worker group.id, the default topic prefix, metric tags, and the cluster key in every stored offset. Cycle detection works by parsing the alias back out of a topic name, so it depends on every configuration agreeing on those strings.

Do this. Treat aliases and separator as immutable, and keep them short because they eat the 249-character budget. Add an automated check that every connector for a pair agrees on both aliases. Disagreement is a correctness problem rather than a cosmetic one. If you choose IdentityReplicationPolicy instead (the usual choice when consumers must keep unprefixed names), accept that cycle detection is impossible by construction, and heartbeat topics are renamed even then.

Catch 8: offset-syncs.topic.location defaults to source, and the topic is named for the peer

What breaks. The connector fails on startup needing write access to a cluster your policy treats as read-only. Or you look for the offset-syncs topic and can’t find it.

Why. The default puts mm2-offset-syncs.<targetAlias>.internal on the source cluster: note the target alias in the name. Setting location=target gives mm2-offset-syncs.<sourceAlias>.internal on the target instead. Moving it later means a new topic and losing all offset-sync history.

Do this. Choose explicitly, document it, and grant the ACLs it implies on both clusters. This is a one-way door.

Also check

These are real but do not change the migration plan, so they get a line each.

• Internal topic RFs default to 3: and a config key on the wrong connector is a silent no-op, because ConfigDef.validateAll() ignores unknown keys and PUT /config still returns 200.
• MM2 recreates a remote topic you delete and resumes from its stored offset, so a bare delete leaves a topic with only recent data and a healthy-looking lag metric.
• Topic config sync will delete target-side tuning. use.defaults.from=target issues DELETE ops for anything at default on the source, every 600s. It never copies min.insync.replicas, and it does revert your retention.ms.
• ACL sync is narrow: topic LITERAL only, WRITE dropped, ALL downgraded to READ. Its “no authorizer” diagnostic points at the source and logs at INFO.
• source.cluster.alias is required, and target.cluster.alias silently defaults to "target": a wrong value there corrupts metric tags and checkpoint attribution without erroring.
• A “one-way” deployment still runs a reverse herder, because heartbeats need it. Inventory by listing topics and *-mm2 consumer groups, not by reading the file.
• MM2 ships no replication-lag metric, so derive one from heartbeat lag (the replicated <source>.heartbeats topic on the target) and from connector metrics. The two fail differently, so keep both alerts. The legacy names are scheduled for removal in Kafka 5.0.
• Metric names can change across this migration. The legacy kafka.connect.mirror namespace does not, and the claim that the MBean prefix differs traces to a name in KIP-382 that never shipped. If you opted into KIP-1280’s newer names via metric.names.formats in 4.3, the connector label changes from MirrorSourceConnector to whatever you named the connector.
• replication.factor defaults to 2 for newly created remote topics, set by MirrorSourceConfig rather than inherited from the source or the target broker default. Under-replicated DR topics do not show up in MM2’s metrics.
• Worker-level config is invisible to translation: config.providers, plugin version, internal-topic settings, and exactly.once.source.support (a worker setting, not a connector key). EOS can silently degrade to at-least-once.

Playbook

1. Inventory what’s actually running. Not the file: it may be years out of date, and unknown keys were silently discarded. List both clusters’ internal topics, *-mm2 consumer groups, every consumer group with its lag (lagging groups get no clean translation, catch 4), and partition counts for the tasks.max calculation.

2. Decide the target. A Connect cluster dedicated to MM2 is simpler to reason about. A shared one is cheaper, but it couples MM2’s throughput to its neighbours and hands you its worker config. Either way, record where replication progress will live, because that’s catch 1.

3. Translate config, with one owner per key. The rows that actually bite:

mm2.properties	Connector JSON	Goes on
us-west.bootstrap.servers / us-east.bootstrap.servers	source.cluster.bootstrap.servers / target.cluster.bootstrap.servers	all three
us-west.consumer.*	consumer.* (or source.consumer.*)	all three
us-east.producer.*	worker producer.*, or producer.override.* per connector (catch 2)	Connect worker / connector
us-east.key.converter / value.converter / header.converter	ByteArrayConverter on the connector	source, checkpoint, heartbeat
us-west->us-east.topics and .topics.exclude	topics, topics.exclude	source and checkpoint, identical
us-west->us-east.groups and .groups.exclude	groups, groups.exclude	checkpoint only
replication.policy.class, replication.policy.separator, offset-syncs.topic.location	same keys	all three, identical
replication.factor / checkpoints.topic.replication.factor / offset-syncs.topic.replication.factor	same keys	source / checkpoint / source
tasks.max	tasks.max	source and checkpoint (heartbeat is always one task)
exactly.once.source.support	worker config	Connect worker, not a connector

Two rules apply. Every key needs an owner: validate each connector against the plugin’s own validator and diff “keys I set” against “keys the connector knows,” because a misplaced key is silent. And for the data path, producer.override.* is the right place (or worker producer.*), because the framework producer is what writes replicated records. MM2’s own producer.* keys never reach it (catch 2).

4. Pre-flight the Connect cluster. Five things to confirm before you deploy:

• The worker version matches the MM2 classes, and there is one plugin location.
• The internal topics exist and are compacted, with the replication factor you want. connect-offsets now holds your replication progress.
• config.providers is configured.
• ACLs are granted, including CreateTopics and Write on the source if you kept offset-syncs.topic.location=source.
• tasks.max is set from your partition count and throughput target, not left at the default of 1.

5. Seed, then prove nothing is re-read. Do not create the connector running. It consumes from earliest immediately, and whatever it writes before you stop it is duplicated by your seed. Create it stopped, with "initial_state": "STOPPED" on 3.7 or later, or enabled=false on 3.6. Then: freeze producers, confirm the old deployment is at zero lag, seed HWM - 1 per partition (catch 1) skipping empty partitions, and resume. Record target high-water marks, wait one full refresh.topics.interval.seconds (600s), and re-check: they must be unchanged. Wait the full interval, because the source connector only re-reads the topic list that often and a shorter wait proves nothing.

6. Validate translation in a shadow group. Take a group that is not yet in production (orders-api in this example), and consume, commit and stop it on the source. Wait emit.checkpoints.interval.seconds + sync.group.offsets.interval.seconds (60 + 60) plus margin. Compare its target offsets against source. If zero or absent, work through: flag off, group still active on target, connector not running or unauthorized, offset precedes the earliest sync, topics filter mismatch.

7. Cut over in this order.

1. Stop source producers.
2. Wait for source consumers to reach zero lag and commit.
3. Wait for MM2 to translate those offsets. This is the synchronization point.
4. Update target ACLs and start target producers.
5. Start target consumers.
6. Stop the old deployment.

The order matters. Starting producers early drops records. Starting consumers early means MM2 cannot write into an active group, and you get extra re-delivery.

8. Decommission and roll back. Keep the old internal topics (mm2-offsets.<source>.internal, mm2-offset-syncs.<peer>.internal, <source>.checkpoints.internal) for the full confidence window. Rollback is not a clean restart: the old process resumes from its own offsets and re-replicates everything the new one did. Plan for that duplicate window, or hand-write the old deployment’s offsets into its internal topics, since dedicated mode has no offsets API.

Checklist

• GET /connectors/<connector>/offsets returns an offset for every topic-partition in the filter.
• Target high-water marks are unchanged after a full refresh.topics.interval.seconds with producers still frozen.
• tasks.max is set explicitly and confirmed under load, not left at the default of 1.
• replication.factor set explicitly, and --describe on a new remote topic shows the intended RF.
• sync.group.offsets.enabled=true, or a documented manual translation that has been run end to end.
• A shadow group translated correctly, compared with kafka-consumer-groups.sh --describe on both clusters.
• Every cutover group had zero lag on the source before the flip, or is documented as accepting re-delivery.
• Internal topics exist on the expected clusters with the intended RF and cleanup.policy=compact.
• replication.policy.*, both aliases, and offset-syncs.topic.location are identical across all connectors for the pair, checked programmatically.
• topics and topics.exclude are identical between source and checkpoint connectors.
• A config diff of each connector shows no keys its ConfigDef doesn’t own.
• Heartbeat lag is under threshold, and a lag alert has been fired and resolved artificially.
• Topic configs re-read on the target after one sync interval, and target-local configs are either excluded or accepted as transient.
• The topic-recreation runbook exists and has been walked through by whoever is on call.
• Old internal topics retained with a written deletion date, and the rollback runbook states the duplicate window.

Closing

Migrate for the control plane rather than for the replication. Keep the aliases stable, seed the offsets before you start the new connectors, and let every consumer group reach zero lag before you flip. Done in that order, the cutover is uneventful.

Further reading

Kafka’s behaviour is cited inline to the connect/mirror classes on GitHub. The references actually worth your time, and why:

• Ryanne Dolan, KIP-382: MirrorMaker 2.0: the original design, and the reason a driver exists at all. Its MBean name (kafka.mirror.connect) never shipped, so do not cite it for JMX.
• Daniel Urban, KIP-710: one nested Connect node per replication flow. Implemented in 3.5.0, but only the internal REST surface, which is why dedicated mode still has no management API. The most useful KIP for understanding what you are moving between.
• Omnia Ibrahim, KIP-690 and KIP-716: internal-topic naming and offset-syncs location. Read these if you are confused about why a topic name contains the other cluster’s alias.
• Apache Kafka geo-replication docs: the operational source of truth for dedicated mode: config-sharing conflicts, metrics, exactly-once requirements.
• Greg Harris in KAFKA-19607: the cutover sequence and the “reach zero lag or tolerate re-delivery” rule, from the person who maintains this code. KAFKA-12558 (Alan Ning) is where the offset-sync buffering bug came from.
• Instaclustr’s field notes: the only write-up I found that documents alias drift silently breaking cycle detection, plus independent confirmation of the replication.factor=2 default.

Related reading

• Kafka Connect
• Kafka consumer offsets