2 pytorch network_error ai_generated partial

RuntimeError: [torch.distributed] AllReduce timed out in rank 0 (timeout: 600)

ID: pytorch/nccl-timeout-during-allreduce

Also available as: JSON · Markdown · 中文
80%Fix Rate
82%Confidence
1Evidence
2024-01-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
pytorch>=2.1.0 active — — —
nccl>=2.18.0 active — — —

Root Cause

NCCL allreduce operation timed out due to network congestion, a slow or crashed worker, or mismatched NCCL versions across nodes.

generic

中文

NCCL allreduce 操作由于网络拥塞、工作节点缓慢或崩溃、或节点间 NCCL 版本不匹配而超时。

Official Documentation

https://pytorch.org/docs/stable/distributed.html#troubleshooting

Workarounds

  1. 85% success Check network connectivity between all nodes using `nccl-tests` or `ping`. Ensure all nodes can communicate on the required ports (e.g., 29500). Then set `NCCL_SOCKET_IFNAME=eth0` to specify the correct network interface. 
    Check network connectivity between all nodes using `nccl-tests` or `ping`. Ensure all nodes can communicate on the required ports (e.g., 29500). Then set `NCCL_SOCKET_IFNAME=eth0` to specify the correct network interface.
  2. 75% success Reduce the batch size or model size to lower communication volume, or use gradient accumulation to decrease allreduce frequency. 
    Reduce the batch size or model size to lower communication volume, or use gradient accumulation to decrease allreduce frequency.
  3. 80% success Switch to Gloo backend for allreduce if NCCL is problematic: `torch.distributed.init_process_group(backend='gloo', ...)`. Note: Gloo may be slower but more stable on certain networks. 
    Switch to Gloo backend for allreduce if NCCL is problematic: `torch.distributed.init_process_group(backend='gloo', ...)`. Note: Gloo may be slower but more stable on certain networks.

中文步骤

  1. 使用 `nccl-tests` 或 `ping` 检查所有节点之间的网络连接。确保所有节点可以在所需端口（例如 29500）上通信。然后设置 `NCCL_SOCKET_IFNAME=eth0` 以指定正确的网络接口。
  2. 减小批量大小或模型大小以降低通信量，或使用梯度累积减少 allreduce 频率。
  3. 如果 NCCL 有问题，切换到 Gloo 后端进行 allreduce：`torch.distributed.init_process_group(backend='gloo', ...)`。注意：Gloo 可能较慢但在某些网络上更稳定。

Dead Ends

Common approaches that don't work:

  1. 70% fail

    If the underlying issue (e.g., network congestion or slow node) persists, the operation will still fail after the longer timeout.

  2. 90% fail

    The network issue is likely persistent; a restart only resets the state temporarily.

  3. 95% fail

    While logging helps diagnosis, it does not resolve the timeout itself.