Kafka KRaft (Kafka without Zookeeper)

↪ In 2020, the Apache Kafka project started to work to remove the Zookeeper dependency from Kafka.

↪ Kafka 4.0 completely removes support for Zookeeper.

↪ KRaft stands for Kafka Raft metadata mode. It replaces Zookeeper as the metadata manager in Apache Kafka.

↪ Zookeeper shows scaling issues when Kafka cluster have > 100000 partitions.

↪ By removing Zookeeper, Apache Kafka can:

• Scale to millions of partitions, and becomes easier to maintain and set-up
• Improve stability, makes it easier to monitor, support and administer
• Single security model for the whole system
• Single process to start with Kafka
• Faster controller shutdown and recovery time

Quorum Controller

↪  In KRaft mode (Kafka without ZooKeeper), Quorum Controller is a group of Kafka nodes that manage the metadata and control plane of the Kafka cluster.

↪ One controller node is elected as the Leader (active controller).

↪ Other nodes are followers (they replicate the metadata log).

↪ This quorum ensures:

• Leader election
• Metadata consistency
• High availability of cluster control