# 运行时错误：[等级0] NCCL通信器在等级2上被中止。原始失败原因：AllReduce在600000毫秒后超时

- **ID:** `pytorch/nccl-timeout-allreduce`
- **领域:** pytorch
- **类别:** network_error
- **错误码:** `NCCL_TIMEOUT_600`
- **验证级别:** ai_generated
- **修复率:** 75%

## 根因

由于网络拥塞、GPU挂起或分布式等级间工作负载不均衡，NCCL集合操作（AllReduce）超时。

## 版本兼容性

| 版本 | 状态 | 引入 | 弃用 |
|------|------|------|------|
| torch>=1.10 | active | — | — |
| cuda>=11.3 | active | — | — |
| nccl>=2.9 | active | — | — |

## 解决方案

1. ```
   设置NCCL_IB_TIMEOUT和NCCL_SOCKET_IFNAME以优化InfiniBand和以太网接口。示例：export NCCL_IB_TIMEOUT=22; export NCCL_SOCKET_IFNAME=eth0; torchrun --nproc_per_node=8 train.py
   ```
2. ```
   使用梯度累积减少通信频率：model.zero_grad(); for i in range(accum_steps): loss = loss_fn(); loss.backward(); optimizer.step()
   ```
3. ```
   切换到Gloo后端进行调试：dist.init_process_group(backend='gloo')。注意：速度较慢但在以太网上更稳定。
   ```

## 无效尝试

- **export NCCL_TIMEOUT=1200** — Increasing NCCL timeout without addressing root cause (e.g., network or workload imbalance) just delays failure. (70% 失败率)
- **torchrun --nproc_per_node=2** — Reducing world size may mask the issue if the real problem is network topology or GPU memory fragmentation. (50% 失败率)