pytorch distributed_error ai_generated true

RuntimeError: NCCL communicator was aborted on rank 2. Original reason for failure was: watchdog callback timed out

ID: pytorch/nccl-timeout

Also available as: JSON · Markdown
75%Fix Rate
85%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —

Root Cause

NCCL collective operation timed out in distributed training. One or more ranks fell behind, a GPU hung, or inter-node networking failed.

generic

Workarounds

  1. 88% success Check that all ranks reach the collective at the same point 
    Add torch.distributed.barrier() before the collective; print rank and step to find the straggler

    Sources: https://pytorch.org/tutorials/

  2. 85% success Set NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=ALL for detailed diagnostics 
    export NCCL_DEBUG=INFO; look for 'connection refused' or 'timeout' between specific ranks
  3. 82% success Verify network interfaces and firewall between nodes 
    export NCCL_SOCKET_IFNAME=eth0; ensure the correct NIC is used; check firewall allows NCCL ports

Dead Ends

Common approaches that don't work:

  1. Increase NCCL_TIMEOUT to a very large value 72% fail

    Hides the real issue; training hangs silently for minutes before the error surfaces again

  2. Set NCCL_P2P_DISABLE=1 globally 68% fail

    Disabling P2P forces NCCL to use shared memory paths, drastically slowing multi-GPU communication