kafka runtime_error ai_generated true

org.apache.kafka.common.errors.InterruptException: Consumer fetch thread interrupted during rebalance

ID: kafka/consumer-fetch-thread-interrupted-during-rebalance

Also available as: JSON · Markdown · 中文
78%Fix Rate
83%Confidence
1Evidence
2024-05-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Kafka 3.4.0 active — — —
Kafka 3.6.1 active — — —
Kafka 3.8.0 active — — —

Root Cause

A consumer's background fetch thread was interrupted while the group was in a rebalance state, often due to thread pool shutdown or timeout in poll() loop.

generic

中文

消费者后台拉取线程在组处于重平衡状态时被中断，通常是由于线程池关闭或poll()循环超时。

Official Documentation

https://kafka.apache.org/documentation/#consumerconfigs_max.poll.interval.ms

Workarounds

  1. 92% success Use a dedicated thread pool for consumer operations and ensure graceful shutdown with consumer.wakeup() before interrupting threads. Example: ExecutorService executor = Executors.newSingleThreadExecutor(); executor.submit(() -> { try { while (true) { consumer.poll(Duration.ofMillis(100)); } } catch (WakeupException e) { consumer.close(); } }); 
    Use a dedicated thread pool for consumer operations and ensure graceful shutdown with consumer.wakeup() before interrupting threads. Example: ExecutorService executor = Executors.newSingleThreadExecutor(); executor.submit(() -> { try { while (true) { consumer.poll(Duration.ofMillis(100)); } } catch (WakeupException e) { consumer.close(); } });
  2. 75% success Set session.timeout.ms and max.poll.interval.ms to values that give the consumer enough time to complete processing before rebalance interrupts. 
    Set session.timeout.ms and max.poll.interval.ms to values that give the consumer enough time to complete processing before rebalance interrupts.

中文步骤

  1. Use a dedicated thread pool for consumer operations and ensure graceful shutdown with consumer.wakeup() before interrupting threads. Example: ExecutorService executor = Executors.newSingleThreadExecutor(); executor.submit(() -> { try { while (true) { consumer.poll(Duration.ofMillis(100)); } } catch (WakeupException e) { consumer.close(); } });
  2. Set session.timeout.ms and max.poll.interval.ms to values that give the consumer enough time to complete processing before rebalance interrupts.

Dead Ends

Common approaches that don't work:

  1. Raising max.poll.interval.ms to 10 minutes 85% fail

    Increasing max.poll.interval.ms gives more time but doesn't address the thread interruption source.

  2. Restarting the consumer application 95% fail

    Restarting the consumer without fixing the thread pool management just re-triggers the same interruption.