ERM redis runtime_error ai_generated partial

ERR Cluster node timeout exceeded, possible split-brain detected

ID: redis/cluster-node-timeout-split-brain

Also available as: JSON · Markdown · 中文
75%Fix Rate
82%Confidence
1Evidence
2023-08-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
6.2 active — — —
7.0 active — — —
7.2 active — — —

Root Cause

A cluster node failed to respond within cluster-node-timeout, leading to a potential split-brain scenario where multiple nodes assume master roles.

generic

中文

集群节点在 cluster-node-timeout 内未响应，导致可能的脑裂场景，多个节点假定为主节点角色。

Official Documentation

https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/

Workarounds

  1. 80% success Run 'CLUSTER FORGET <node-id>' on all healthy masters to remove the unresponsive node, then rejoin it after network recovery. 
    Run 'CLUSTER FORGET <node-id>' on all healthy masters to remove the unresponsive node, then rejoin it after network recovery.
  2. 88% success Configure sentinel monitors with quorum = majority of nodes to prevent split-brain. Example: 'sentinel monitor mymaster 127.0.0.1 6379 2'. 
    Configure sentinel monitors with quorum = majority of nodes to prevent split-brain. Example: 'sentinel monitor mymaster 127.0.0.1 6379 2'.

中文步骤

  1. 在所有健康主节点上运行 'CLUSTER FORGET <node-id>' 移除无响应节点，网络恢复后重新加入。
  2. 配置 sentinel monitor 使用多数节点作为法定人数，防止脑裂。示例：'sentinel monitor mymaster 127.0.0.1 6379 2'。

Dead Ends

Common approaches that don't work:

  1. 70% fail

    Delays detection but does not prevent network partitions; can worsen split-brain.

  2. 85% fail

    May cause data loss if RDB/AOF not synced; does not resolve underlying network issues.

  3. 95% fail

    Split-brain can cause inconsistent data; manual intervention required.