SENTINEL redis network_error ai_generated partial

SENTINEL: arbiter node unreachable, failover delayed for 30000ms

ID: redis/sentinel-arbiter-down-failover-delay

Also available as: JSON · Markdown · 中文
77%Fix Rate
84%Confidence
1Evidence
2024-01-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
6.2.0 active — — —
7.0.0 active — — —
7.2.0 active — — —

Root Cause

Sentinel's arbiter node is not reachable, preventing the quorum from being reached for failover, causing a delay.

generic

中文

Sentinel 的仲裁节点不可达，导致无法达到故障转移所需的法定人数，从而造成延迟。

Official Documentation

https://redis.io/docs/management/sentinel/

Workarounds

  1. 80% success Check network connectivity from sentinel to arbiter: 'telnet <arbiter-ip> 26379' and ensure firewall rules allow traffic. If the arbiter is down, restart it and verify it rejoins the quorum: 'redis-cli -p 26379 SENTINEL CKQUORUM <mymaster>' 
    Check network connectivity from sentinel to arbiter: 'telnet <arbiter-ip> 26379' and ensure firewall rules allow traffic. If the arbiter is down, restart it and verify it rejoins the quorum: 'redis-cli -p 26379 SENTINEL CKQUORUM <mymaster>'
  2. 70% success Temporarily reconfigure sentinel to use a different arbiter or increase the down-after-milliseconds to allow more time for arbiter recovery before failover is triggered 
    Temporarily reconfigure sentinel to use a different arbiter or increase the down-after-milliseconds to allow more time for arbiter recovery before failover is triggered

中文步骤

  1. Check network connectivity from sentinel to arbiter: 'telnet <arbiter-ip> 26379' and ensure firewall rules allow traffic. If the arbiter is down, restart it and verify it rejoins the quorum: 'redis-cli -p 26379 SENTINEL CKQUORUM <mymaster>'
  2. Temporarily reconfigure sentinel to use a different arbiter or increase the down-after-milliseconds to allow more time for arbiter recovery before failover is triggered

Dead Ends

Common approaches that don't work:

  1. Increasing sentinel quorum to 2 to bypass arbiter requirement 70% fail

    This reduces fault tolerance; if the arbiter is down but other nodes are up, failover may still be blocked if quorum is not met.

  2. Restarting the arbiter node without checking network connectivity 85% fail

    If the network between Sentinel and arbiter is partitioned, restarting alone won't fix the connectivity issue.

  3. Removing the arbiter from the sentinel configuration 80% fail

    This changes the quorum requirement and can lead to split-brain scenarios if the arbiter was intentionally placed there.