NCCL_TIMEOUT_600 pytorch network_error ai_generated partial

RuntimeError: [Rank 0] NCCL communicator was aborted on rank 2. Original reason for failure was: AllReduce timed out after 600000 ms

ID: pytorch/nccl-timeout-allreduce

Also available as: JSON · Markdown · 中文
75%Fix Rate
85%Confidence
1Evidence
2023-06-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
torch>=1.10 active — — —
cuda>=11.3 active — — —
nccl>=2.9 active — — —

Root Cause

NCCL collective operation (AllReduce) timed out due to network congestion, GPU hang, or uneven workload distribution across distributed ranks.

generic

中文

由于网络拥塞、GPU挂起或分布式等级间工作负载不均衡，NCCL集合操作（AllReduce）超时。

Official Documentation

https://pytorch.org/docs/stable/distributed.html#common-environment-variables

Workarounds

  1. 80% success Set NCCL_IB_TIMEOUT and NCCL_SOCKET_IFNAME to optimize InfiniBand and Ethernet interfaces. Example: export NCCL_IB_TIMEOUT=22; export NCCL_SOCKET_IFNAME=eth0; torchrun --nproc_per_node=8 train.py 
    Set NCCL_IB_TIMEOUT and NCCL_SOCKET_IFNAME to optimize InfiniBand and Ethernet interfaces. Example: export NCCL_IB_TIMEOUT=22; export NCCL_SOCKET_IFNAME=eth0; torchrun --nproc_per_node=8 train.py
  2. 70% success Use gradient accumulation to reduce communication frequency: model.zero_grad(); for i in range(accum_steps): loss = loss_fn(); loss.backward(); optimizer.step() 
    Use gradient accumulation to reduce communication frequency: model.zero_grad(); for i in range(accum_steps): loss = loss_fn(); loss.backward(); optimizer.step()
  3. 60% success Switch to Gloo backend for debugging: dist.init_process_group(backend='gloo'). Note: slower but more stable on Ethernet. 
    Switch to Gloo backend for debugging: dist.init_process_group(backend='gloo'). Note: slower but more stable on Ethernet.

中文步骤

  1. 设置NCCL_IB_TIMEOUT和NCCL_SOCKET_IFNAME以优化InfiniBand和以太网接口。示例：export NCCL_IB_TIMEOUT=22; export NCCL_SOCKET_IFNAME=eth0; torchrun --nproc_per_node=8 train.py
  2. 使用梯度累积减少通信频率：model.zero_grad(); for i in range(accum_steps): loss = loss_fn(); loss.backward(); optimizer.step()
  3. 切换到Gloo后端进行调试：dist.init_process_group(backend='gloo')。注意：速度较慢但在以太网上更稳定。

Dead Ends

Common approaches that don't work:

  1. export NCCL_TIMEOUT=1200 70% fail

    Increasing NCCL timeout without addressing root cause (e.g., network or workload imbalance) just delays failure.

  2. torchrun --nproc_per_node=2 50% fail

    Reducing world size may mask the issue if the real problem is network topology or GPU memory fragmentation.