# RuntimeError: [torch.distributed] AllReduce timed out in rank 0 (timeout: 600)

- **ID:** `pytorch/nccl-timeout-during-allreduce`
- **Domain:** pytorch
- **Category:** network_error
- **Error Code:** `2`
- **Verification:** ai_generated
- **Fix Rate:** 80%

## Root Cause

NCCL allreduce operation timed out due to network congestion, a slow or crashed worker, or mismatched NCCL versions across nodes.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| pytorch>=2.1.0 | active | — | — |
| nccl>=2.18.0 | active | — | — |

## Workarounds

1. **Check network connectivity between all nodes using `nccl-tests` or `ping`. Ensure all nodes can communicate on the required ports (e.g., 29500). Then set `NCCL_SOCKET_IFNAME=eth0` to specify the correct network interface.** (85% success)
   ```
   Check network connectivity between all nodes using `nccl-tests` or `ping`. Ensure all nodes can communicate on the required ports (e.g., 29500). Then set `NCCL_SOCKET_IFNAME=eth0` to specify the correct network interface.
   ```
2. **Reduce the batch size or model size to lower communication volume, or use gradient accumulation to decrease allreduce frequency.** (75% success)
   ```
   Reduce the batch size or model size to lower communication volume, or use gradient accumulation to decrease allreduce frequency.
   ```
3. **Switch to Gloo backend for allreduce if NCCL is problematic: `torch.distributed.init_process_group(backend='gloo', ...)`. Note: Gloo may be slower but more stable on certain networks.** (80% success)
   ```
   Switch to Gloo backend for allreduce if NCCL is problematic: `torch.distributed.init_process_group(backend='gloo', ...)`. Note: Gloo may be slower but more stable on certain networks.
   ```

## Dead Ends

- **** — If the underlying issue (e.g., network congestion or slow node) persists, the operation will still fail after the longer timeout. (70% fail)
- **** — The network issue is likely persistent; a restart only resets the state temporarily. (90% fail)
- **** — While logging helps diagnosis, it does not resolve the timeout itself. (95% fail)