pytorch network_error ai_generated partial

运行时错误：[torch.distributed] 广播在 rank 3 上失败：ncclInternalError：内部错误 - 意外事件

RuntimeError: [torch.distributed] Broadcast failed on rank 3: ncclInternalError: Internal error - unexpected event

ID: pytorch/distributed-broadcast-error

其他格式: JSON · Markdown 中文 · English
75%修复率
88%置信度
1证据数
2024-02-10首次发现

版本兼容性

版本状态引入弃用备注
torch>=1.13 active — — —
torch<=2.5.1 active — — —
nccl>=2.14 active — — —
nccl<=2.22 active — — —

根因分析

广播集合操作期间的 NCCL 内部错误，通常由 GPU 硬件故障、驱动程序问题或多节点设置中的网络超时引起。

English

NCCL internal error during a broadcast collective operation, often caused by GPU hardware faults, driver issues, or network timeouts in multi-node setups.

generic

官方文档

https://pytorch.org/docs/stable/distributed.html#torch.distributed.broadcast

解决方案

  1. Set environment variable NCCL_IB_DISABLE=1 to disable InfiniBand and fall back to TCP, which is more stable on some clusters. For example: export NCCL_IB_DISABLE=1; torchrun --nproc_per_node=8 train.py
  2. Update NCCL and CUDA drivers to compatible versions. Check NCCL compatibility matrix and use torch's bundled NCCL if possible: pip install torch --index-url https://download.pytorch.org/whl/cu118

无效尝试

常见但无效的做法:

  1. Rebooting all nodes without changing NCCL configuration 70% 失败

    If the root cause is a driver mismatch or NCCL version incompatibility, a simple reboot will not fix the issue.

  2. Increasing the number of GPUs per node to improve bandwidth 80% 失败

    More GPUs can exacerbate NCCL internal errors due to increased contention and resource pressure.