ERR redis network_error ai_generated true

ERR Cluster node timeout exceeded: node <node_id> unreachable for <seconds> seconds

ID: redis/cluster-node-timeout-exceeded

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2023-06-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
5.0.0 active — — —
6.0.0 active — — —
7.0.0 active — — —

Root Cause

A cluster node has not responded to gossip pings within the cluster-node-timeout period, causing the cluster to mark it as possibly failed and triggering a failover attempt.

generic

中文

集群节点在cluster-node-timeout周期内未响应gossip ping，导致集群将其标记为可能故障并触发故障转移尝试。

Official Documentation

https://redis.io/docs/management/scaling/

Workarounds

  1. 90% success Check network connectivity between nodes: `redis-cli -h <node_ip> -p <port> PING` from each node. If unreachable, verify firewall rules (e.g., iptables) and ensure cluster bus port (base port + 10000) is open. 
    Check network connectivity between nodes: `redis-cli -h <node_ip> -p <port> PING` from each node. If unreachable, verify firewall rules (e.g., iptables) and ensure cluster bus port (base port + 10000) is open.
  2. 75% success If the node is overloaded, reduce client traffic or scale up resources. Use `CLUSTER SET-NODE-TIMEOUT <milliseconds>` temporarily to increase tolerance, then investigate root cause. 
    If the node is overloaded, reduce client traffic or scale up resources. Use `CLUSTER SET-NODE-TIMEOUT <milliseconds>` temporarily to increase tolerance, then investigate root cause.
  3. 85% success If the node is permanently down, use `CLUSTER FORGET <node_id>` on all other nodes to remove it, then re-add a replacement node. 
    If the node is permanently down, use `CLUSTER FORGET <node_id>` on all other nodes to remove it, then re-add a replacement node.

中文步骤

  1. 检查节点间网络连通性：从每个节点运行 `redis-cli -h <node_ip> -p <port> PING`。如果不可达，验证防火墙规则（例如iptables）并确保集群总线端口（基础端口+10000）已开放。
  2. 如果节点过载，减少客户端流量或扩展资源。临时使用 `CLUSTER SET-NODE-TIMEOUT <毫秒>` 增加容限，然后调查根本原因。
  3. 如果节点永久宕机，在所有其他节点上使用 `CLUSTER FORGET <node_id>` 移除它，然后添加替换节点。

Dead Ends

Common approaches that don't work:

  1. 85% fail

    Simply restarting the unreachable node without checking network connectivity or firewall rules often leads to the same timeout recurring.

  2. 80% fail

    Increasing cluster-node-timeout to a very high value delays failure detection and can mask underlying network issues, leading to split-brain scenarios.