cuda distributed_communication ai_generated true

RuntimeError: NCCL communicator was aborted on rank 0. Original reason for aborting was: Timeout at NCCL sequence 42

ID: cuda/nccl-timeout

Also available as: JSON · Markdown
74%Fix Rate
72%Confidence
29Evidence
2023-07-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —
2 active — — —

Root Cause

NCCL timeout errors in multi-GPU PyTorch 2.1 setups are caused by network misconfiguration, asymmetric GPU workloads, or NCCL selecting the wrong network interface. Diagnosing with NCCL_DEBUG=INFO and fixing the network configuration resolves most cases. The root cause is almost never NCCL itself but rather the networking layer it operates over.

generic

Workarounds

  1. 70% success Set NCCL_DEBUG=INFO to diagnose the actual failure point 
    export NCCL_DEBUG=INFO && export NCCL_DEBUG_SUBSYS=ALL && torchrun --nproc_per_node=NUM_GPUS your_script.py 2>&1 | tee nccl_debug.log

    Sources: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html

  2. 82% success Fix network configuration by specifying the correct interface with NCCL_SOCKET_IFNAME 
    export NCCL_SOCKET_IFNAME=eth0 (replace eth0 with the correct high-bandwidth interface; find it with 'ip link show' or 'ibstat' for InfiniBand)

    Sources: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html https://github.com/pytorch/pytorch/issues/105390

  3. 76% success Ensure all ranks can communicate by fixing firewall and routing 
    Verify bidirectional connectivity: on each node, run 'nc -zv <other_node_ip> <port>' for the NCCL port range (default 29500+). Disable firewall for inter-node traffic: 'sudo ufw allow from <subnet>' or configure iptables accordingly.

    Sources: https://pytorch.org/docs/stable/distributed.html

  4. 73% success Use NCCL_IB_DISABLE=1 on systems without InfiniBand 
    export NCCL_IB_DISABLE=1 && export NCCL_SOCKET_IFNAME=eth0

    Sources: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html

Dead Ends

Common approaches that don't work:

  1. Increasing the NCCL timeout value alone (NCCL_TIMEOUT or dist.init_process_group timeout parameter) 85% fail

    Increasing the timeout masks the underlying issue without resolving it. If NCCL is timing out because it selected the wrong network interface (e.g., a loopback or a slow management NIC), a longer timeout just delays the failure. The root cause is a communication path problem, not an insufficient timeout.

  2. Restarting the training script without fixing the network configuration 92% fail

    NCCL timeout errors are deterministic when caused by network misconfiguration. Restarting does not change the network interface selection, routing tables, or firewall rules. The same failure will recur at the same point in training (or earlier if NCCL state is not cleanly torn down).

  3. Setting NCCL_P2P_DISABLE=1 as a blanket fix 68% fail

    Disabling peer-to-peer communication forces NCCL to fall back to shared-memory or network-based transfers, which are significantly slower. On NVLink-equipped systems (like multi-GPU A100 nodes), this destroys the performance advantage of NVLink. It may also not fix the timeout if the underlying issue is interface selection.

Error Chain

Leads to:
Preceded by: