pytorch network_error ai_generated partial

RuntimeError: [torch.distributed] Broadcast failed on rank 3: ncclInternalError: Internal error - unexpected event

ID: pytorch/distributed-broadcast-error

Also available as: JSON · Markdown · 中文
75%Fix Rate
88%Confidence
1Evidence
2024-02-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
torch>=1.13 active — — —
torch<=2.5.1 active — — —
nccl>=2.14 active — — —
nccl<=2.22 active — — —

Root Cause

NCCL internal error during a broadcast collective operation, often caused by GPU hardware faults, driver issues, or network timeouts in multi-node setups.

generic

中文

广播集合操作期间的 NCCL 内部错误，通常由 GPU 硬件故障、驱动程序问题或多节点设置中的网络超时引起。

Official Documentation

https://pytorch.org/docs/stable/distributed.html#torch.distributed.broadcast

Workarounds

  1. 80% success Set environment variable NCCL_IB_DISABLE=1 to disable InfiniBand and fall back to TCP, which is more stable on some clusters. For example: export NCCL_IB_DISABLE=1; torchrun --nproc_per_node=8 train.py 
    Set environment variable NCCL_IB_DISABLE=1 to disable InfiniBand and fall back to TCP, which is more stable on some clusters. For example: export NCCL_IB_DISABLE=1; torchrun --nproc_per_node=8 train.py
  2. 85% success Update NCCL and CUDA drivers to compatible versions. Check NCCL compatibility matrix and use torch's bundled NCCL if possible: pip install torch --index-url https://download.pytorch.org/whl/cu118 
    Update NCCL and CUDA drivers to compatible versions. Check NCCL compatibility matrix and use torch's bundled NCCL if possible: pip install torch --index-url https://download.pytorch.org/whl/cu118

中文步骤

  1. Set environment variable NCCL_IB_DISABLE=1 to disable InfiniBand and fall back to TCP, which is more stable on some clusters. For example: export NCCL_IB_DISABLE=1; torchrun --nproc_per_node=8 train.py
  2. Update NCCL and CUDA drivers to compatible versions. Check NCCL compatibility matrix and use torch's bundled NCCL if possible: pip install torch --index-url https://download.pytorch.org/whl/cu118

Dead Ends

Common approaches that don't work:

  1. Rebooting all nodes without changing NCCL configuration 70% fail

    If the root cause is a driver mismatch or NCCL version incompatibility, a simple reboot will not fix the issue.

  2. Increasing the number of GPUs per node to improve bandwidth 80% fail

    More GPUs can exacerbate NCCL internal errors due to increased contention and resource pressure.