Setting up Apache Kafka Replication and Disaster Recovery using Kafka MirrorMaker

Author: Mohinder Kumar, Director, CloudTechner

---

Problem Statement

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is an AWS streaming data service that manages Apache Kafka infrastructure and operations, making it easy for developers and DevOps managers to run Apache Kafka applications and Kafka Connect connectors on AWS, without the need to become experts in operating Apache Kafka.

Now a days modernised applications try to use Apache Kafka due to distributed system, scalability and real time processing capabilities. Although Amazon MSK is fully managed Streaming for Apache Kafka but the real challenge comes when we need to setup Disaster Recovery for it for business continuity. Unlike other AWS Managed services like RDS, AWS doesn’t support OOTB replication/failover feature for Kafka topics and messages to another region.

This blog describes setting up Disaster Recovery for AWS Managed Streaming for Apache Kafka(Amazon MSK) using too called Kafka Mirror Maker

Before we deep dive, lets revisit some basic concepts of Apache Kafka and Kafka Mirror Maker tool

Apache Kafka is a distributed data store optimised for ingesting and processing streaming data in real-time. Streaming data is data that is continuously generated by thousands of data sources, which typically send the data records in simultaneously. A streaming platform needs to handle this constant influx of data, and process the data sequentially and incrementally.

Kafka provides three main functions to its users:

• Publish and subscribe to streams of records
• Effectively store streams of records in the order in which records were generated
• Process streams of records in real time

Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data.

Brokers and Clusters

Kafka runs on clusters where each cluster consists of multiple servers, generally called brokers (and sometimes called nodes). That’s what makes Kafka a distributed system: data in the Kafka cluster is distributed amongst multiple brokers. And multiple copies (replicas) of the same data exist in a Kafka cluster. This mechanism makes Kafka more stable, fault-tolerant, and reliable; if an error or failure occurs with one broker, another broker steps in to perform the functions of the malfunctioning component, and the information is not lost.

Topics

A Kafka topic is an immutable log of events (sequences). Producers publish events to Kafka topics; consumers subscribe to topics to access their desired data. Each topic can serve data to many consumers. Continuing with our example, the registration component of the website publishes “new user” events (via Kafka) into the “registration” topic. Subscribers such as analytics apps, newsfeed apps, monitoring apps, databases, and so on in turn consume events from the “registration” topic and use it with other data as the foundation for delivering their own products or services.

Partitions

A partition is the smallest storage unit in Kafka. Partitions serve to split data across brokers to accelerate performance. Each Kafka topic is divided into partitions and each partition can be placed on a separate broker.

Kafka Mirror Maker

Kafka MirrorMaker is a stand-alone tool for copying data between two Apache Kafka clusters. Mirror Maker automatically replicates existing topics and messages and also detects new topics and partitions, while also ensuring the topic configurations are synced between clusters. Data will be read from topics in the origin cluster and written to a topic with the same name in the destination cluster. You can run many such mirroring processes to increase throughput and for fault-tolerance.

Mirror Maker comes with a default replication policy that can be customised by providing a custom replication policy class. Remote topics in Mirror Maker are replicated topics on the target cluster. These topics reference the source cluster using a naming convention as shown in the following figure.

Mirror maker supports multi directional flows to replicate data from source to target

• fan-out — fan-out is a replication strategy where you have a single source cluster and multiple target clusters

• aggregation– aggregation is a replication strategy where you have multiple source clusters and a single target cluster

• active/active, where you have a bidirectional flow between a source cluster and a target cluster

• active/passive, where you have a unidirectional flow from a source cluster to a target cluster

Setting up Kafka Mirror Maker on AWS EKS Cluster

Pre-requisites

Following prerequisites should be met before setting up Mirror Maker between two clusters

• AWS MSK Kafka Clusters is already setup in Primary and DR region
• Connectivity is established between Primary and DR region using VPC Peering
• All the necessary routes have been established to allow traffic between Primary and DR Region
• Allow EKS Cluster CIDR range of Primary region in AWS MSK Kafka Cluster Security group of both Primary and DR region so that Mirror Maker can talk to AWS MSK Kafka Cluster in both regions
• Docker Installed on local machine/or any AWS EC2
• Helm Installed on the local machine/or AWS EC2
• Kafka CLI utilities installed on the local machine/or AWS EC2

Setup Instructions

We have created Helm Chart templates for deploying Kakfa Mirror Maker on Kubernetes Cluster. These helm charts are fully templatized and requires to update few values in values yaml along with mirror maker properties having kafka cluster details.

1. Clone GIT Repository
2. https://github.com/cloudtechner/ct-kafka-mm-k8s-setup . This repository contains the dockerfile to create the base image for Kafka Mirror Maker and Helm charts templates for deploying mirror-maker into Kubernetes cluster
3. Build docker image with Dockerfile and push to Private Image registry
   docker build -t <IMAGE_NAME:TAG> .
4. Update following details in mm2.properties file
   – Source (primary cluster) and target (secondary) kafka clusters names
   – Source and target kafka clusters bootstrap servers details
5. Modify values.yaml and update image name and image tag for kafka mirror maker with your own docker registry details
6. Connect to Kubernetes Cluster and install helm charts on Kubernetes with updated values.yaml file from the current directory
   helm install mirror-maker .
7. Once helm charts are installed successfully, check the status of pods. You should see mirror-maker and zookeeper pods running
   kubectl get pods

Testing

Now that the MirrorMaker setup is done, we can replicate changes done on primary Kafka cluster to secondary cluster.

Get the brokers list deployed in Kafka Cluster. In case of self managed Kafka Cluster get IP and port of the Kafka Nodes
aws kafka get-bootstrap-brokers --cluster-arn "kafka-cluster-arn"

• Create a topic in Primary Kafka Cluster using one of the broker IP
  kafka-topics --bootstrap-server <primary-cluster-broker-host:9092> --create --topic sampletopic

• List Topics in Primary Kafka Cluster and validate if topic is successfully created or not
  kafka-topics --bootstrap-server <primary-cluster-broker-host:9092> --list

• List Topics in Secondary Kafka Cluster and validate if same topic is successfully replicated
  kafka-topics --bootstrap-server <secondary-cluster-broker-host:9092> --list

Now lets produce some messages and validate messages replication

• Add messages in Primary Kafka Cluster Topic
  kafka-console-producer --bootstrap-server <primary-cluster-broker-host:9092> --topic <topic_name>

• Enter Messages and validate in secondary cluster if messages are getting successfully replicated
  kafka-console-consumer --bootstrap-server <secondary-cluster-borker-host:9092> --topic <topic-created-in-primary-cluster> --from-beginning

The above example shows the unidirectional Kafka replication (active-passive). For bi-directional or any other replication topology, we just need to modify mm2. properties as per requirement.

This shows how to setup Disaster recovery environment for AWS MSK (Amazon Managed Service for Kafka) using MirrorMaker 2.0 with the help of helm charts on a Kubernetes Cluster. It does not need a running Connect cluster, and leverages a high-level driver which generates a set of Connect workers based on the mm2.properties configuration file.