# RuntimeError: [Rank 0] NCCL communicator was aborted on rank 2. Original reason for failure was: AllReduce timed out after 600000 ms

- **ID:** `pytorch/nccl-timeout-allreduce`
- **Domain:** pytorch
- **Category:** network_error
- **Error Code:** `NCCL_TIMEOUT_600`
- **Verification:** ai_generated
- **Fix Rate:** 75%

## Root Cause

NCCL collective operation (AllReduce) timed out due to network congestion, GPU hang, or uneven workload distribution across distributed ranks.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| torch>=1.10 | active | — | — |
| cuda>=11.3 | active | — | — |
| nccl>=2.9 | active | — | — |

## Workarounds

1. **Set NCCL_IB_TIMEOUT and NCCL_SOCKET_IFNAME to optimize InfiniBand and Ethernet interfaces. Example: export NCCL_IB_TIMEOUT=22; export NCCL_SOCKET_IFNAME=eth0; torchrun --nproc_per_node=8 train.py** (80% success)
   ```
   Set NCCL_IB_TIMEOUT and NCCL_SOCKET_IFNAME to optimize InfiniBand and Ethernet interfaces. Example: export NCCL_IB_TIMEOUT=22; export NCCL_SOCKET_IFNAME=eth0; torchrun --nproc_per_node=8 train.py
   ```
2. **Use gradient accumulation to reduce communication frequency: model.zero_grad(); for i in range(accum_steps): loss = loss_fn(); loss.backward(); optimizer.step()** (70% success)
   ```
   Use gradient accumulation to reduce communication frequency: model.zero_grad(); for i in range(accum_steps): loss = loss_fn(); loss.backward(); optimizer.step()
   ```
3. **Switch to Gloo backend for debugging: dist.init_process_group(backend='gloo'). Note: slower but more stable on Ethernet.** (60% success)
   ```
   Switch to Gloo backend for debugging: dist.init_process_group(backend='gloo'). Note: slower but more stable on Ethernet.
   ```

## Dead Ends

- **export NCCL_TIMEOUT=1200** — Increasing NCCL timeout without addressing root cause (e.g., network or workload imbalance) just delays failure. (70% fail)
- **torchrun --nproc_per_node=2** — Reducing world size may mask the issue if the real problem is network topology or GPU memory fragmentation. (50% fail)