ERROR cuda runtime_error ai_generated partial

RuntimeError: NCCL error: Timeout (NCCL_ERROR_TIMEOUT)

ID: cuda/nccl-timeout-distributed

Also available as: JSON · Markdown
65%Fix Rate
82%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
12 active — — —

Root Cause

NCCL collective operation timed out during distributed training. One or more GPUs fell behind or lost network connectivity.

generic

Workarounds

  1. 88% success Set NCCL_DEBUG=INFO to identify which rank and operation hangs 
    NCCL_DEBUG=INFO torchrun --nproc_per_node=8 train.py
  2. 82% success Check NVLink/InfiniBand connectivity between GPUs 
    nvidia-smi topo -m  # verify NVLink topology
  3. 85% success Ensure all ranks reach the collective at the same point (no conditional branching) 
    All processes must call the same collectives in the same order; check for if/else around all_reduce

Dead Ends

Common approaches that don't work:

  1. Increase NCCL_TIMEOUT to very large values 80% fail

    Masks the real issue (GPU hang, network failure); training will hang silently instead of crashing

  2. Disable NCCL and use Gloo backend 88% fail

    Gloo is CPU-based for GPU tensors; 10-100x slower for large models

Error Chain

Preceded by: