kafka config_error ai_generated true

org.apache.kafka.common.errors.BrokerIdConflictException: Broker ID 1 is already registered by another broker with address 192.168.1.10:9092

ID: kafka/broker-id-conflict

Also available as: JSON · Markdown · 中文
95%Fix Rate
90%Confidence
1Evidence
2023-11-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
kafka_2.13-3.4.0 active — — —
kafka_2.13-3.5.1 active — — —
kafka_2.13-3.6.0 active — — —

Root Cause

Two Kafka brokers in the same cluster have the same `broker.id` value, causing a conflict in ZooKeeper or KRaft metadata.

generic

中文

同一 Kafka 集群中的两个代理具有相同的 `broker.id` 值，导致 ZooKeeper 或 KRaft 元数据中的冲突。

Official Documentation

https://kafka.apache.org/documentation/#brokerconfigs_broker.id

Workarounds

  1. 95% success Change the `broker.id` in one broker's `server.properties` to a unique value (e.g., from 1 to 2) and restart that broker. Verify no other broker uses ID 2. 
    Change the `broker.id` in one broker's `server.properties` to a unique value (e.g., from 1 to 2) and restart that broker. Verify no other broker uses ID 2.
  2. 90% success If using KRaft, check `meta.properties` in the log directory and ensure `broker.id` is unique across all nodes; regenerate the cluster ID if necessary. 
    If using KRaft, check `meta.properties` in the log directory and ensure `broker.id` is unique across all nodes; regenerate the cluster ID if necessary.
  3. 93% success Use a script to list all registered brokers: `zookeeper-shell.sh localhost:2181 get /brokers/ids/1` and compare the `endpoints` field to identify the conflicting broker, then update its ID. 
    Use a script to list all registered brokers: `zookeeper-shell.sh localhost:2181 get /brokers/ids/1` and compare the `endpoints` field to identify the conflicting broker, then update its ID.

中文步骤

  1. Change the `broker.id` in one broker's `server.properties` to a unique value (e.g., from 1 to 2) and restart that broker. Verify no other broker uses ID 2.
  2. If using KRaft, check `meta.properties` in the log directory and ensure `broker.id` is unique across all nodes; regenerate the cluster ID if necessary.
  3. Use a script to list all registered brokers: `zookeeper-shell.sh localhost:2181 get /brokers/ids/1` and compare the `endpoints` field to identify the conflicting broker, then update its ID.

Dead Ends

Common approaches that don't work:

  1. 98% fail

    The conflict persists because the same IDs are still assigned; ZooKeeper still sees both with the same ID.

  2. 90% fail

    This removes the conflicting registration but the broker re-registers with the same ID on restart, causing the error again.

  3. 85% fail

    This only affects automatic ID assignment, not manual conflicts; it does not resolve existing duplicate IDs.