A historic dependency
Since its inception Apache Kafka has used ZooKeeper to manage cluster metadata: the list of active brokers, partition assignment to nodes, controller election and topic configuration. ZooKeeper is a robust and battle-tested distributed coordination service, but its presence introduces operational complexity. A Kafka deployment requires two separate clusters — Kafka and ZooKeeper — each with its own monitoring, scaling, upgrade and backup needs.
The dual infrastructure has a concrete cost: more components to manage, more failure modes to anticipate, more expertise required from operations teams. For small deployments the complexity is disproportionate; for large ones, metadata scalability in ZooKeeper itself becomes a constraint.
KRaft: the internal metadata quorum
Kafka 2.8, released in April 2021, introduces KRaft (Kafka Raft metadata mode) in early access. KRaft replaces ZooKeeper with an internal quorum of controllers based on the Raft protocol, implemented directly within the Kafka brokers. Cluster metadata are managed as a replicated log inside the same system, without external dependencies.
The KRaft quorum is formed by a subset of brokers designated as controllers. One of them is the active leader, responsible for metadata management; the others are followers that replicate the log and stand ready to take over in case of failure. Leader election occurs through the Raft protocol, which guarantees consensus even in the presence of network partitions.
Architectural advantages
Eliminating ZooKeeper brings benefits at multiple levels. Deployment becomes simpler: a single process type to configure, upgrade and monitor. Metadata scale better because the Raft log is optimised for Kafka’s specific use case, without ZooKeeper’s generic constraints. Controller recovery time improves: the new leader already has a local copy of the metadata and can become active quickly.
The maximum number of partitions per cluster, historically limited by ZooKeeper’s capacity to manage watches across thousands of nodes, can grow significantly with KRaft. For organisations managing tens of thousands of topics with hundreds of thousands of partitions, this is a real constraint that KRaft relaxes.
A gradual transition
KRaft in Kafka 2.8 is declared early access: suitable for evaluation and testing, not yet recommended for production. The roadmap envisions progressive stabilisation in subsequent versions and the definitive removal of the ZooKeeper dependency in a future release. Migration from existing clusters will be supported by dedicated tooling.
KRaft’s architectural rewrite represents the deepest structural change in Kafka’s history: removing an external dependency present since day one to replace it with an internal mechanism designed for the platform’s specific needs.
Link: kafka.apache.org