NCCL_TIMEOUT_600 pytorch network_error ai_generated partial

运行时错误：[等级0] NCCL通信器在等级2上被中止。原始失败原因：AllReduce在600000毫秒后超时

RuntimeError: [Rank 0] NCCL communicator was aborted on rank 2. Original reason for failure was: AllReduce timed out after 600000 ms

ID: pytorch/nccl-timeout-allreduce

其他格式: JSON · Markdown 中文 · English
75%修复率
85%置信度
1证据数
2023-06-15首次发现

版本兼容性

版本状态引入弃用备注
torch>=1.10 active — — —
cuda>=11.3 active — — —
nccl>=2.9 active — — —

根因分析

由于网络拥塞、GPU挂起或分布式等级间工作负载不均衡，NCCL集合操作（AllReduce）超时。

English

NCCL collective operation (AllReduce) timed out due to network congestion, GPU hang, or uneven workload distribution across distributed ranks.

generic

官方文档

https://pytorch.org/docs/stable/distributed.html#common-environment-variables

解决方案

  1. 设置NCCL_IB_TIMEOUT和NCCL_SOCKET_IFNAME以优化InfiniBand和以太网接口。示例：export NCCL_IB_TIMEOUT=22; export NCCL_SOCKET_IFNAME=eth0; torchrun --nproc_per_node=8 train.py
  2. 使用梯度累积减少通信频率：model.zero_grad(); for i in range(accum_steps): loss = loss_fn(); loss.backward(); optimizer.step()
  3. 切换到Gloo后端进行调试：dist.init_process_group(backend='gloo')。注意：速度较慢但在以太网上更稳定。

无效尝试

常见但无效的做法:

  1. export NCCL_TIMEOUT=1200 70% 失败

    Increasing NCCL timeout without addressing root cause (e.g., network or workload imbalance) just delays failure.

  2. torchrun --nproc_per_node=2 50% 失败

    Reducing world size may mask the issue if the real problem is network topology or GPU memory fragmentation.