ERR redis data_error ai_generated true

Error: Stream entries added to consumer group PEL without acknowledgment

ID: redis/stream-entries-added-to-consumer-group-pel

Also available as: JSON · Markdown · 中文
85%Fix Rate
85%Confidence
1Evidence
2024-03-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
redis 6.2 active — — —
redis 7.0 active — — —
redis 7.2 active — — —

Root Cause

Consumer group pending entries list (PEL) grows unbounded as consumers fail to acknowledge processed messages, potentially causing memory exhaustion and performance degradation.

generic

中文

消费者组待处理条目列表(PEL)因消费者未能确认已处理的消息而无限制增长，可能导致内存耗尽和性能下降。

Official Documentation

https://redis.io/docs/latest/commands/xack/

Workarounds

  1. 90% success Periodically acknowledge processed entries using XACK: `XACK mystream mygroup 1526569498055-0` for each message ID. Automate with a background job that calls XACK after successful processing. 
    Periodically acknowledge processed entries using XACK: `XACK mystream mygroup 1526569498055-0` for each message ID. Automate with a background job that calls XACK after successful processing.
  2. 85% success Set a consumer group PEL size limit with `XGROUP SETID mystream mygroup $` and use XTRIM to cap stream length: `XTRIM mystream MAXLEN ~ 10000`. 
    Set a consumer group PEL size limit with `XGROUP SETID mystream mygroup $` and use XTRIM to cap stream length: `XTRIM mystream MAXLEN ~ 10000`.
  3. 80% success Use XAUTOCLAIM to reassign idle PEL entries to other consumers: `XAUTOCLAIM mystream mygroup consumer2 3600000 0-0` after setting a reasonable idle timeout. 
    Use XAUTOCLAIM to reassign idle PEL entries to other consumers: `XAUTOCLAIM mystream mygroup consumer2 3600000 0-0` after setting a reasonable idle timeout.

中文步骤

  1. Periodically acknowledge processed entries using XACK: `XACK mystream mygroup 1526569498055-0` for each message ID. Automate with a background job that calls XACK after successful processing.
  2. Set a consumer group PEL size limit with `XGROUP SETID mystream mygroup $` and use XTRIM to cap stream length: `XTRIM mystream MAXLEN ~ 10000`.
  3. Use XAUTOCLAIM to reassign idle PEL entries to other consumers: `XAUTOCLAIM mystream mygroup consumer2 3600000 0-0` after setting a reasonable idle timeout.

Dead Ends

Common approaches that don't work:

  1. Increasing consumer group size to handle more messages 85% fail

    Adding more consumers does not clear the PEL; each consumer still accumulates unacknowledged entries, worsening the issue.

  2. Deleting and recreating the stream 90% fail

    Deletes all data and disrupts active consumers; PEL rebuilds quickly if root cause (lack of acknowledgment) persists.

  3. Setting XREADGROUP COUNT to a very high value 80% fail

    High COUNT does not clear PEL; it only returns more entries, and unacknowledged entries remain, continuing to grow the PEL.