kafka protocol_error ai_generated true

org.apache.kafka.common.errors.ProducerFencedException: Producer epoch fenced due to partition reassignment

ID: kafka/transactional-producer-epoch-fenced-during-partition-reassignment

Also available as: JSON · Markdown · 中文
81%Fix Rate
88%Confidence
1Evidence
2024-06-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Kafka 3.3.0 active — — —
Kafka 3.5.0 active — — —
Kafka 3.7.0 active — — —

Root Cause

A transactional producer's epoch was invalidated because a partition it was writing to was reassigned to a different broker, causing the transaction coordinator to fence the producer.

generic

中文

事务性生产者的纪元被无效化，因为它正在写入的分区被重新分配到不同的代理，导致事务协调器隔离了该生产者。

Official Documentation

https://kafka.apache.org/documentation/#producerconfigs_transactional.id

Workarounds

  1. 90% success Initialize a new transactional producer with a new transactional.id after catching the exception. Example in Java: try { producer.initTransactions(); } catch (ProducerFencedException e) { producer.close(); producer = createNewProducerWithNewTransactionalId(); producer.initTransactions(); } 
    Initialize a new transactional producer with a new transactional.id after catching the exception. Example in Java: try { producer.initTransactions(); } catch (ProducerFencedException e) { producer.close(); producer = createNewProducerWithNewTransactionalId(); producer.initTransactions(); }
  2. 85% success Ensure partition reassignment is completed before starting transactional writes. Use 'bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --verify --reassignment-json-file reassign.json' to confirm completion. 
    Ensure partition reassignment is completed before starting transactional writes. Use 'bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --verify --reassignment-json-file reassign.json' to confirm completion.

中文步骤

  1. Initialize a new transactional producer with a new transactional.id after catching the exception. Example in Java: try { producer.initTransactions(); } catch (ProducerFencedException e) { producer.close(); producer = createNewProducerWithNewTransactionalId(); producer.initTransactions(); }
  2. Ensure partition reassignment is completed before starting transactional writes. Use 'bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --verify --reassignment-json-file reassign.json' to confirm completion.

Dead Ends

Common approaches that don't work:

  1. Setting transaction.timeout.ms to 300000 (5 minutes) 90% fail

    Increasing transaction.timeout.ms does not prevent epoch fencing; it only delays transaction expiration.

  2. Restarting the producer application without changing transactional.id 95% fail

    Restarting the producer without resetting the transactional.id will still use the fenced epoch.