ERR redis resource_error ai_generated partial

ERR PEL too large for consumer group 'mygroup' on stream 'mystream'

ID: redis/stream-consumer-group-pending-entries-overload

Also available as: JSON · Markdown · 中文
75%Fix Rate
85%Confidence
1Evidence
2024-03-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
7.0 active — — —
7.2 active — — —
8.0-m3 active — — —

Root Cause

The Pending Entries List (PEL) for a consumer group has grown excessively large, exceeding Redis internal limits or causing performance degradation.

generic

中文

消费者组的待处理条目列表 (PEL) 增长过大，超出了 Redis 内部限制或导致性能下降。

Official Documentation

https://redis.io/docs/latest/commands/xreadgroup/

Workarounds

  1. 85% success Use XACK to acknowledge processed messages and shrink the PEL. Example: XACK mystream mygroup 1234567890-0 1234567891-0 
    Use XACK to acknowledge processed messages and shrink the PEL. Example: XACK mystream mygroup 1234567890-0 1234567891-0
  2. 80% success Use XAUTOCLAIM to reclaim idle pending messages and acknowledge them programmatically. Example: XAUTOCLAIM mystream mygroup other-consumer 3600000 0-0 COUNT 100 
    Use XAUTOCLAIM to reclaim idle pending messages and acknowledge them programmatically. Example: XAUTOCLAIM mystream mygroup other-consumer 3600000 0-0 COUNT 100
  3. 90% success Monitor and limit the number of pending entries per consumer by setting a maximum processing time and acknowledging messages promptly in application code. 
    Monitor and limit the number of pending entries per consumer by setting a maximum processing time and acknowledging messages promptly in application code.

中文步骤

  1. Use XACK to acknowledge processed messages and shrink the PEL. Example: XACK mystream mygroup 1234567890-0 1234567891-0
  2. Use XAUTOCLAIM to reclaim idle pending messages and acknowledge them programmatically. Example: XAUTOCLAIM mystream mygroup other-consumer 3600000 0-0 COUNT 100
  3. Monitor and limit the number of pending entries per consumer by setting a maximum processing time and acknowledging messages promptly in application code.

Dead Ends

Common approaches that don't work:

  1. Delete and recreate the consumer group without acknowledging pending messages. 60% fail

    This loses all pending messages and disrupts processing state; messages may be lost permanently.

  2. Increase the stream's max-length to a very high value. 80% fail

    The PEL size is not directly tied to stream length; it depends on unacknowledged messages. Increasing max-length does not help.

  3. Restart the Redis server to clear the PEL. 90% fail

    The PEL is persisted and will be restored on restart; this only causes downtime without solving the underlying issue.