redis network_error ai_generated partial

SENTINEL: arbiter node unreachable, failover delayed for master 'mymaster'

ID: redis/sentinel-arbiter-unreachable-failover-delayed

Also available as: JSON · Markdown · 中文
82%Fix Rate
86%Confidence
1Evidence
2024-11-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Redis 6.2.0 active — — —
Redis 7.0.0 active — — —
Redis 7.2.0 active — — —

Root Cause

Sentinel cannot connect to the arbiter node (a Redis instance configured as quorum participant) within the failover timeout, preventing the quorum from being reached and delaying failover.

generic

中文

Sentinel 无法在故障转移超时内连接到仲裁节点（配置为法定人数参与的 Redis 实例），阻止达成法定人数并延迟故障转移。

Official Documentation

https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/

Workarounds

  1. 85% success Check the arbiter's network connectivity using ping and redis-cli -h <arbiter-ip> -p <port> PING, then fix firewall rules or DNS resolution if needed. 
    Check the arbiter's network connectivity using ping and redis-cli -h <arbiter-ip> -p <port> PING, then fix firewall rules or DNS resolution if needed.
  2. 75% success Temporarily adjust the quorum to exclude the arbiter by reducing sentinel monitor mymaster <ip> <port> <quorum> to a lower value (e.g., from 3 to 2) until the arbiter is restored. 
    Temporarily adjust the quorum to exclude the arbiter by reducing sentinel monitor mymaster <ip> <port> <quorum> to a lower value (e.g., from 3 to 2) until the arbiter is restored.
  3. 80% success Ensure the arbiter Redis instance is running with a minimal configuration (e.g., port 26379) and is reachable from all sentinel nodes; verify sentinel.conf for correct arbiter IP. 
    Ensure the arbiter Redis instance is running with a minimal configuration (e.g., port 26379) and is reachable from all sentinel nodes; verify sentinel.conf for correct arbiter IP.

中文步骤

  1. 使用 ping 和 redis-cli -h <arbiter-ip> -p <port> PING 检查仲裁节点的网络连接，然后根据需要修复防火墙规则或 DNS 解析。
  2. 通过将 sentinel monitor mymaster <ip> <port> <quorum> 降低到较小值（例如从 3 降到 2），临时调整法定人数以排除仲裁节点，直到仲裁节点恢复。
  3. 确保仲裁 Redis 实例以最小配置运行（例如端口 26379），并且所有 Sentinel 节点都可访问；验证 sentinel.conf 中的仲裁节点 IP 是否正确。

Dead Ends

Common approaches that don't work:

  1. 60% fail

    Increasing sentinel failover timeout (sentinel failover-timeout mymaster 60000) may allow more time but doesn't fix the underlying network or arbiter issue.

  2. 80% fail

    Removing the arbiter from the configuration reduces quorum resilience and may cause split-brain scenarios.

  3. 70% fail

    Restarting the arbiter without checking its network connectivity or configuration will result in the same unreachable state.