kafka runtime_error ai_generated true

org.apache.kafka.common.errors.PartitionNotAssignedException: Partition my-topic-0 is not assigned to this consumer instance

ID: kafka/partition-not-assigned

Also available as: JSON · Markdown · 中文
85%Fix Rate
86%Confidence
1Evidence
2023-09-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
kafka 2.8.0 active — — —
kafka 3.0.0 active — — —
kafka 3.3.0 active — — —

Root Cause

The consumer attempted to commit offsets or fetch data for a partition that was not assigned to it, often due to a stale partition assignment after a rebalance.

generic

中文

消费者尝试提交偏移或获取未分配给它的分区的数据，通常是由于再平衡后分区分配过期。

Official Documentation

https://kafka.apache.org/documentation/#consumerconfigs_session.timeout.ms

Workarounds

  1. 90% success Use a ConsumerRebalanceListener to update partition assignments after a rebalance, and ensure commitSync() is only called for assigned partitions. 
    Use a ConsumerRebalanceListener to update partition assignments after a rebalance, and ensure commitSync() is only called for assigned partitions.
  2. 80% success Increase 'session.timeout.ms' and 'heartbeat.interval.ms' to reduce the likelihood of spurious rebalances that cause stale assignments. 
    Increase 'session.timeout.ms' and 'heartbeat.interval.ms' to reduce the likelihood of spurious rebalances that cause stale assignments.

中文步骤

  1. Use a ConsumerRebalanceListener to update partition assignments after a rebalance, and ensure commitSync() is only called for assigned partitions.
  2. Increase 'session.timeout.ms' and 'heartbeat.interval.ms' to reduce the likelihood of spurious rebalances that cause stale assignments.

Dead Ends

Common approaches that don't work:

  1. 60% fail

    poll() may not trigger a rebalance if the group is stable; it can also cause a busy loop that degrades performance.

  2. 75% fail

    This bypasses the group coordinator and can lead to offset conflicts or duplicate consumption if other consumers exist.