kafka network_error ai_generated partial

org.apache.kafka.common.errors.ZooKeeperConnectionLossException: Connection loss during KRaft controller election

ID: kafka/zookeeper-connection-loss-during-kraft-election

Also available as: JSON · Markdown · 中文
80%Fix Rate
87%Confidence
1Evidence
2024-08-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Kafka 3.5.0 active — — —
Kafka 3.6.0 active — — —
Kafka 3.7.0 active — — —

Root Cause

During KRaft migration or mixed-mode operation, a ZooKeeper connection loss disrupts the controller election process, leaving the cluster without a leader.

generic

中文

在KRaft迁移或混合模式操作期间，ZooKeeper连接丢失中断了控制器选举过程，导致集群没有领导者。

Official Documentation

https://kafka.apache.org/documentation/#kraft_controller

Workarounds

  1. 85% success Restart the KRaft controller process with 'bin/kafka-server-stop.sh && bin/kafka-server-start.sh config/kraft/server.properties' to force a new election. 
    Restart the KRaft controller process with 'bin/kafka-server-stop.sh && bin/kafka-server-start.sh config/kraft/server.properties' to force a new election.
  2. 90% success Ensure ZooKeeper quorum is stable and network latency between ZooKeeper and Kafka brokers is below 10ms. Use 'echo stat | nc zookeeper-host 2181' to check ZooKeeper state. 
    Ensure ZooKeeper quorum is stable and network latency between ZooKeeper and Kafka brokers is below 10ms. Use 'echo stat | nc zookeeper-host 2181' to check ZooKeeper state.

中文步骤

  1. Restart the KRaft controller process with 'bin/kafka-server-stop.sh && bin/kafka-server-start.sh config/kraft/server.properties' to force a new election.
  2. Ensure ZooKeeper quorum is stable and network latency between ZooKeeper and Kafka brokers is below 10ms. Use 'echo stat | nc zookeeper-host 2181' to check ZooKeeper state.

Dead Ends

Common approaches that don't work:

  1. Restarting only ZooKeeper servers 90% fail

    Restarting ZooKeeper alone does not re-trigger the KRaft election; the controller process must be restarted.

  2. Raising zookeeper.session.timeout.ms 80% fail

    Increasing ZooKeeper session timeout delays the failure but does not prevent connection loss due to network issues.