ERR redis resource_error ai_generated true

ERR PEL too large for consumer group 'mygroup' on stream 'mystream': 50000 entries, consider trimming

ID: redis/stream-pending-entry-list-too-large

Also available as: JSON · Markdown · 中文
79%Fix Rate
86%Confidence
1Evidence
2024-08-22First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
7.2.0 active — — —
7.4.0 active — — —
8.0.0 active — — —

Root Cause

The pending entries list (PEL) for a consumer group has grown excessively large, consuming memory and slowing down XREADGROUP and XACK operations.

generic

中文

消费者组的待处理条目列表（PEL）增长过大，消耗内存并拖慢 XREADGROUP 和 XACK 操作。

Official Documentation

https://redis.io/docs/data-types/streams/

Workarounds

  1. 85% success Use XAUTOCLAIM to reclaim and acknowledge old pending entries: 'XAUTOCLAIM mystream mygroup myconsumer 3600000 0-0 COUNT 1000' then XACK each claimed entry. Repeat until PEL is reduced. 
    Use XAUTOCLAIM to reclaim and acknowledge old pending entries: 'XAUTOCLAIM mystream mygroup myconsumer 3600000 0-0 COUNT 1000' then XACK each claimed entry. Repeat until PEL is reduced.
  2. 80% success Implement a periodic cleanup job in the consumer that checks PEL size via XINFO GROUPS and automatically trims old entries using XDEL after confirming they are no longer needed 
    Implement a periodic cleanup job in the consumer that checks PEL size via XINFO GROUPS and automatically trims old entries using XDEL after confirming they are no longer needed

中文步骤

  1. Use XAUTOCLAIM to reclaim and acknowledge old pending entries: 'XAUTOCLAIM mystream mygroup myconsumer 3600000 0-0 COUNT 1000' then XACK each claimed entry. Repeat until PEL is reduced.
  2. Implement a periodic cleanup job in the consumer that checks PEL size via XINFO GROUPS and automatically trims old entries using XDEL after confirming they are no longer needed

Dead Ends

Common approaches that don't work:

  1. Increasing maxmemory to accommodate the large PEL 80% fail

    This only delays the problem; the PEL will continue to grow until memory is exhausted again, and it does not improve performance.

  2. Deleting the consumer group and recreating it 90% fail

    This loses all pending entry state, potentially causing data loss if entries were not processed.

  3. Setting a very high XREADGROUP COUNT to read more entries 70% fail

    This increases load on the server and does not reduce the PEL size; it may actually grow it faster.