Multi-Cluster Apache Kafka with Cluster Linking ft. Nikhil Bhatia

Note: This episode was recorded when Cluster Linking was in preview mode. It’s now generally available as part of the Confluent Q3 ‘21 release on August 17, 2021. 

Infrastructure needs to react in real time to support globally distributed events, such as cloud migration, IoT, edge data collection, and disaster recovery. To provide a seamless yet cloud-native, cross-cluster topic replication experience, Nikhil Bhatia (Principal Engineer I, Product Infrastructure, Confluent) and the team engineered a solution called Cluster Linking. Available on Confluent Cloud, Cluster Linking is an API that enables Apache Kafka® to work across multi-datacenters, making it possible to design globally available distributed systems. 

As industries adopt multi-cloud usage and depart from on-premises and single cluster operations, we need to rethink how clusters operate across regions in the cloud. Cluster Linking as an inter-cluster replication layer into Confluent Server, allowing you to connect clusters together and replicate topics asynchronously without the need for Connect. 

Cluster Linking requires zero external components when moving messages from one cluster to another. It replicates data into its destination by partition and byte for byte, preserving offsets from the source cluster. Different from Confluent Replicator and MirrorMaker2, Cluster Linking simplifies failover in high availability and disaster recovery scenarios, improving overall efficiency by avoiding recompression. As a great cost-effective alternative to Multi-Region Cluster, Cluster Linking reduces traffic between data centers and enables inter-cluster replication without the need to deploy and manage a separate Connect cluster. 

With low recovery point objective (RPO) and recovery time objective (RTO), Cluster Linking enables scenarios such as: 

• Migration to cloud: Remove the complexity layer of self-run datacenters with fully managed cloud services. 
• Global reads: Enable users to connect to Kafka from around the globe and consume data locally. Empowering better performance and improving cost effectiveness. 
• Disaster recovery: Prepare your system for fault tolerance, from datacenter, regional, or cloud-level disasters, ensuring zero data loss and high availability. 

Find out more about Cluster Linking architecture and set your data in motion with global Kafka.

EPISODE LINKS

• Announcing the Confluent Q3 '21 Release
• Introducing Cluster Linking in Confluent Platform 6.0
• What is Cluster Linking? 
• Resurrecting In-Sync Replicas with Automatic Observer Promotion ft. Anna McDonald
• Watch the video version of this podcast
• Join the Confluent Community
• Learn Kafka at Confluent Developer
• Live demo: Event-Driven Microservices with Confluent
• Use PODCAST100 to get $100 of Confluent Cloud usage (details)