redis config_error ai_generated true

ERR Node timeout mismatch: expected 15000 but got 30000 from node 127.0.0.1:7001

ID: redis/cluster-node-timeout-mismatch

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2023-11-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Redis 7.0.0 active — — —
Redis 7.2.4 active — — —
Redis 6.2.14 active — — —

Root Cause

Redis cluster nodes have inconsistent cluster-node-timeout values, preventing them from forming a coherent cluster.

generic

中文

Redis 集群节点的 cluster-node-timeout 值不一致，阻止它们形成一致的集群。

Official Documentation

https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/

Workarounds

  1. 90% success Set consistent cluster-node-timeout on all nodes by editing redis.conf: cluster-node-timeout 15000. Then restart each node gracefully: redis-cli -p 7001 SHUTDOWN NOSAVE; redis-server /path/to/redis.conf. Verify with CLUSTER INFO. 
    Set consistent cluster-node-timeout on all nodes by editing redis.conf: cluster-node-timeout 15000. Then restart each node gracefully: redis-cli -p 7001 SHUTDOWN NOSAVE; redis-server /path/to/redis.conf. Verify with CLUSTER INFO.
  2. 85% success Use CONFIG SET cluster-node-timeout 15000 on each node, then CONFIG REWRITE to persist. Example: redis-cli -p 7001 CONFIG SET cluster-node-timeout 15000 && redis-cli -p 7001 CONFIG REWRITE. 
    Use CONFIG SET cluster-node-timeout 15000 on each node, then CONFIG REWRITE to persist. Example: redis-cli -p 7001 CONFIG SET cluster-node-timeout 15000 && redis-cli -p 7001 CONFIG REWRITE.
  3. 75% success If nodes are already in a failed state, use CLUSTER FORGET to remove misconfigured nodes, fix their config, and rejoin them with CLUSTER MEET. 
    If nodes are already in a failed state, use CLUSTER FORGET to remove misconfigured nodes, fix their config, and rejoin them with CLUSTER MEET.

中文步骤

  1. Set consistent cluster-node-timeout on all nodes by editing redis.conf: cluster-node-timeout 15000. Then restart each node gracefully: redis-cli -p 7001 SHUTDOWN NOSAVE; redis-server /path/to/redis.conf. Verify with CLUSTER INFO.
  2. Use CONFIG SET cluster-node-timeout 15000 on each node, then CONFIG REWRITE to persist. Example: redis-cli -p 7001 CONFIG SET cluster-node-timeout 15000 && redis-cli -p 7001 CONFIG REWRITE.
  3. If nodes are already in a failed state, use CLUSTER FORGET to remove misconfigured nodes, fix their config, and rejoin them with CLUSTER MEET.

Dead Ends

Common approaches that don't work:

  1. 95% fail

    Restarting doesn't change configuration values. The mismatch is in the config files, not a transient state.

  2. 80% fail

    This command sets the config epoch for failover, not the cluster-node-timeout. It doesn't address the timeout mismatch.

  3. 70% fail

    CONFIG SET changes the runtime value but may not persist after restart. Also, the cluster may reject the change if nodes are already in a mixed state.