// Module included in the following assemblies:
//
// assembly-config-mirrormaker2.adoc

[id='con-mirrormaker-{context}']
= MirrorMaker 2.0 data replication

[role="_abstract"]
Data replication across clusters supports scenarios that require:

* Recovery of data in the event of a system failure
* Aggregation of data for analysis
* Restriction of data access to a specific cluster
* Provision of data at a specific location to improve latency

include::../overview/con-overview-mirrormaker2.adoc[leveloffset=+1]

== Topic configuration synchronization

MirrorMaker 2.0 supports topic configuration synchronization between source and target clusters. 
You specify source topics in the MirrorMaker 2.0 configuration.
MirrorMaker 2.0 monitors the source topics.
MirrorMaker 2.0 detects and propagates changes to the source topics to the remote topics.
Changes might include automatically creating missing topics and partitions.

NOTE: In most cases you write to local topics and read from remote topics. Though write operations are not prevented on remote topics, they should be avoided. 

== Offset tracking
MirrorMaker 2.0 tracks offsets for consumer groups using internal topics.

`offset-syncs` topic:: The `offset-syncs` topic maps the source and target offsets for replicated topic partitions from record metadata.
`checkpoints` topic:: The `checkpoints` topic maps the last committed offset in the source and target cluster for replicated topic partitions in each consumer group.

As they used internally by MirrorMaker 2.0, you do not interact directly with these topics. 

`MirrorCheckpointConnector` emits _checkpoints_ for offset tracking.
Offsets for the `checkpoints` topic are tracked at predetermined intervals through configuration.
Both topics enable replication to be fully restored from the correct offset position on failover.

The location of the `offset-syncs` topic is the `source` cluster by default.
You can use the `offset-syncs.topic.location` connector configuration to change this to the `target` cluster.
You need read/write access to the cluster that contains the topic.
Using the target cluster as the location of the `offset-syncs` topic allows you to use MirrorMaker 2.0 even if you have only read access to the source cluster.

== Synchronizing consumer group offsets

The `__consumer_offsets` topic stores information on committed offsets for each consumer group.
Offset synchronization periodically transfers the consumer offsets for the consumer groups of a source cluster into the consumer offsets topic of a target cluster.

Offset synchronization is particularly useful in an _active/passive_ configuration.
If the active cluster goes down, consumer applications can switch to the passive (standby) cluster and pick up from the last transferred offset position.

To use topic offset synchronization, enable the synchronization by adding `sync.group.offsets.enabled` to the checkpoint connector configuration, and setting the property to `true`.
Synchronization is disabled by default.

When using the `IdentityReplicationPolicy` in the source connector, it also has to be configured in the checkpoint connector configuration.
This ensures that the mirrored consumer offsets will be applied for the correct topics.

Consumer offsets are only synchronized for consumer groups that are not active in the target cluster.
If the consumer groups are in the target cluster, the synchronization cannot be performed and an `UNKNOWN_MEMBER_ID` error is returned. 

If enabled, the synchronization of offsets from the source cluster is made periodically.
You can change the frequency by adding `sync.group.offsets.interval.seconds` and `emit.checkpoints.interval.seconds` to the checkpoint connector configuration.
The properties specify the frequency in seconds that the consumer group offsets are synchronized, and the frequency of checkpoints emitted for offset tracking.
The default for both properties is 60 seconds.
You can also change the frequency of checks for new consumer groups using the `refresh.groups.interval.seconds` property, which is performed every 10 minutes by default.

Because the synchronization is time-based, any switchover by consumers to a passive cluster will likely result in some duplication of messages.

NOTE: If you have an application written in Java, you can use the `RemoteClusterUtils.java` utility to synchronize offsets through the application. The utility fetches remote offsets for a consumer group from the `checkpoints` topic. 

== Connectivity checks

`MirrorHeartbeatConnector` emits _heartbeats_ to check connectivity between clusters.

An internal `heartbeat` topic is replicated from the source cluster.
Target clusters use the `heartbeat` topic to check the following:

* The connector managing connectivity between clusters is running
* The source cluster is available

