NCCL_ERROR_TIMEOUT pytorch network_error ai_generated partial

RuntimeError: [torch.distributed] Barrier timeout after 600000 ms. NCCL communicator was aborted on rank 2. Original reason for failure was: watchdog callback timed out.

ID: pytorch/nccl-timeout-barrier

Also available as: JSON · Markdown · 中文
75%Fix Rate
86%Confidence
1Evidence
2024-01-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
torch>=1.11 active — — —
torch>=2.0 active — — —
NCCL 2.12+ active — — —
CUDA 11.x+ active — — —
NVIDIA A100/H100 active — — —

Root Cause

A distributed barrier timed out because one or more ranks (e.g., rank 2) stalled due to NCCL watchdog timeout, often caused by hardware failures, network congestion, or GPU compute hung.

generic

中文

分布式屏障超时，因为一个或多个 rank（例如 rank 2）因 NCCL 看门狗超时而停滞，通常由硬件故障、网络拥塞或 GPU 计算挂起引起。

Official Documentation

https://pytorch.org/docs/stable/distributed.html#troubleshooting

Workarounds

  1. 80% success Set NCCL timeout in environment: export NCCL_TIMEOUT=600 (seconds); and ensure all GPUs are healthy with nvidia-smi -pm 1 
    Set NCCL timeout in environment: export NCCL_TIMEOUT=600 (seconds); and ensure all GPUs are healthy with nvidia-smi -pm 1
  2. 70% success Reduce network load by using NCCL_IB_DISABLE=1 and NCCL_SOCKET_IFNAME=eth0 to force TCP instead of InfiniBand 
    Reduce network load by using NCCL_IB_DISABLE=1 and NCCL_SOCKET_IFNAME=eth0 to force TCP instead of InfiniBand
  3. 85% success Run with torch.distributed.run --nproc_per_node=N with NCCL_DEBUG=INFO to identify the hanging rank and check its GPU memory or process status 
    Run with torch.distributed.run --nproc_per_node=N with NCCL_DEBUG=INFO to identify the hanging rank and check its GPU memory or process status

中文步骤

  1. Set NCCL timeout in environment: export NCCL_TIMEOUT=600 (seconds); and ensure all GPUs are healthy with nvidia-smi -pm 1
  2. Reduce network load by using NCCL_IB_DISABLE=1 and NCCL_SOCKET_IFNAME=eth0 to force TCP instead of InfiniBand
  3. Run with torch.distributed.run --nproc_per_node=N with NCCL_DEBUG=INFO to identify the hanging rank and check its GPU memory or process status

Dead Ends

Common approaches that don't work:

  1. Increasing barrier timeout to 1200 seconds via torch.distributed.init_process_group(timeout=timedelta(seconds=1200)) 70% fail

    A hung rank will eventually hit the watchdog timeout regardless of barrier timeout; the root cause is not duration but a hang.

  2. Restarting only the failed rank 2 without resetting NCCL communicator 90% fail

    NCCL communicator state is global; restarting one rank leaves the group in an inconsistent state, causing further errors.