kafka consumer_error ai_generated true

CommitFailedException: Commit cannot be completed since the group has already rebalanced

ID: kafka/consumer-rebalance-revoked

Also available as: JSON · Markdown
78%Fix Rate
85%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
3 active — — —

Root Cause

Consumer group rebalanced while processing messages. Processing took longer than session.timeout.ms or max.poll.interval.ms.

generic

Workarounds

  1. 88% success Increase max.poll.interval.ms to match expected processing time 
    max.poll.interval.ms=600000  # 10 minutes; adjust to your processing time

    Sources: https://kafka.apache.org/documentation/

  2. 85% success Reduce max.poll.records to process fewer messages per poll 
    max.poll.records=100  # process smaller batches to stay within poll interval
  3. 80% success Move heavy processing to async worker pool 
    Poll quickly, dispatch to thread pool, commit after completion

Dead Ends

Common approaches that don't work:

  1. Set session.timeout.ms to very large value 72% fail

    Delays detection of genuinely failed consumers; dead consumers block partition processing

  2. Disable auto-commit and never commit offsets 85% fail

    Without commits, all messages are reprocessed on restart, causing duplicates