SENTINEL redis runtime_error ai_generated partial

SENTINEL: failover aborted - no reachable quorum after master change

ID: redis/redis-sentinel-down-after-master-failover

Also available as: JSON · Markdown · 中文
85%Fix Rate
83%Confidence
1Evidence
2024-06-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
redis 6.2 active — — —
redis 7.0 active — — —
redis 7.2 active — — —
redis 7.4 active — — —

Root Cause

After a master failover, the new master's configuration epochs are inconsistent, causing sentinel instances to lose quorum and abort the failover process.

generic

中文

在主节点故障转移后，新主节点的配置纪元不一致，导致哨兵实例失去仲裁并中止故障转移过程。

Official Documentation

https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/

Workarounds

  1. 85% success Manually reset the sentinel state for the affected master: `SENTINEL RESET mymaster` on each sentinel. Then monitor with `SENTINEL MASTER mymaster` to verify quorum. 
    Manually reset the sentinel state for the affected master: `SENTINEL RESET mymaster` on each sentinel. Then monitor with `SENTINEL MASTER mymaster` to verify quorum.
  2. 80% success Force a new failover by running `SENTINEL FAILOVER mymaster` on the leader sentinel after ensuring all sentinels agree on the new master's configuration. 
    Force a new failover by running `SENTINEL FAILOVER mymaster` on the leader sentinel after ensuring all sentinels agree on the new master's configuration.
  3. 90% success Check and synchronize sentinel configuration files across all nodes, ensuring `sentinel monitor` and `sentinel auth-pass` are identical, then restart sentinels one by one. 
    Check and synchronize sentinel configuration files across all nodes, ensuring `sentinel monitor` and `sentinel auth-pass` are identical, then restart sentinels one by one.

中文步骤

  1. Manually reset the sentinel state for the affected master: `SENTINEL RESET mymaster` on each sentinel. Then monitor with `SENTINEL MASTER mymaster` to verify quorum.
  2. Force a new failover by running `SENTINEL FAILOVER mymaster` on the leader sentinel after ensuring all sentinels agree on the new master's configuration.
  3. Check and synchronize sentinel configuration files across all nodes, ensuring `sentinel monitor` and `sentinel auth-pass` are identical, then restart sentinels one by one.

Dead Ends

Common approaches that don't work:

  1. Restarting all sentinel instances simultaneously 75% fail

    Simultaneous restart can cause all sentinels to lose state, delaying failover detection and potentially causing split-brain scenarios.

  2. Manually setting the old master back as leader 95% fail

    The old master may be unreachable or have stale data; forcing it back can cause data loss and inconsistency with replicas.

  3. Increasing sentinel quorum to 5 80% fail

    Higher quorum makes it harder to reach consensus, especially in degraded networks, and does not fix the epoch mismatch.