ERR redis runtime_error ai_generated partial

Error: Timeout waiting for replica acknowledgment during failover

ID: redis/redis-timeout-while-waiting-for-replica-ack

Also available as: JSON · Markdown · 中文
78%Fix Rate
82%Confidence
1Evidence
2024-04-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
redis 6.2 active — — —
redis 7.0 active — — —
redis 7.2 active — — —

Root Cause

During a Redis Sentinel or cluster failover, the master waits for replicas to acknowledge data synchronization, but network latency or replica lag causes the timeout to expire.

generic

中文

在Redis哨兵或集群故障转移期间，主节点等待副本确认数据同步，但网络延迟或副本滞后导致超时过期。

Official Documentation

https://redis.io/docs/latest/operate/oss_and_stack/management/replication/

Workarounds

  1. 80% success Check replica lag with `INFO replication` and reduce it by ensuring replicas have sufficient resources (CPU, network). If lag is high, temporarily increase `repl-timeout` in redis.conf: `repl-timeout 120`. 
    Check replica lag with `INFO replication` and reduce it by ensuring replicas have sufficient resources (CPU, network). If lag is high, temporarily increase `repl-timeout` in redis.conf: `repl-timeout 120`.
  2. 75% success Use `CLUSTER FAILOVER FORCE` on a replica that is synced within acceptable lag to trigger a forced failover without waiting for all replicas. 
    Use `CLUSTER FAILOVER FORCE` on a replica that is synced within acceptable lag to trigger a forced failover without waiting for all replicas.
  3. 70% success Optimize network between master and replicas: use dedicated links or lower latency paths. Monitor with `redis-cli --latency` to identify bottlenecks. 
    Optimize network between master and replicas: use dedicated links or lower latency paths. Monitor with `redis-cli --latency` to identify bottlenecks.

中文步骤

  1. Check replica lag with `INFO replication` and reduce it by ensuring replicas have sufficient resources (CPU, network). If lag is high, temporarily increase `repl-timeout` in redis.conf: `repl-timeout 120`.
  2. Use `CLUSTER FAILOVER FORCE` on a replica that is synced within acceptable lag to trigger a forced failover without waiting for all replicas.
  3. Optimize network between master and replicas: use dedicated links or lower latency paths. Monitor with `redis-cli --latency` to identify bottlenecks.

Dead Ends

Common approaches that don't work:

  1. Increasing the timeout in redis.conf to 300 seconds 65% fail

    Higher timeout may mask underlying issues like replica lag or network problems; failover can still fail if replicas are too far behind.

  2. Manually promoting a replica to master without waiting 90% fail

    Forcing promotion can cause data loss if the replica is not fully synchronized; the new master may miss recent writes.

  3. Disabling replication checks altogether 95% fail

    Turning off replication checks compromises data consistency and can lead to split-brain scenarios.