# RuntimeError: [torch.distributed] Barrier timeout after 600000 ms. NCCL communicator was aborted on rank 2. Original reason for failure was: watchdog callback timed out.

- **ID:** `pytorch/nccl-timeout-barrier`
- **Domain:** pytorch
- **Category:** network_error
- **Error Code:** `NCCL_ERROR_TIMEOUT`
- **Verification:** ai_generated
- **Fix Rate:** 75%

## Root Cause

A distributed barrier timed out because one or more ranks (e.g., rank 2) stalled due to NCCL watchdog timeout, often caused by hardware failures, network congestion, or GPU compute hung.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| torch>=1.11 | active | — | — |
| torch>=2.0 | active | — | — |
| NCCL 2.12+ | active | — | — |
| CUDA 11.x+ | active | — | — |
| NVIDIA A100/H100 | active | — | — |

## Workarounds

1. **Set NCCL timeout in environment: export NCCL_TIMEOUT=600 (seconds); and ensure all GPUs are healthy with nvidia-smi -pm 1** (80% success)
   ```
   Set NCCL timeout in environment: export NCCL_TIMEOUT=600 (seconds); and ensure all GPUs are healthy with nvidia-smi -pm 1
   ```
2. **Reduce network load by using NCCL_IB_DISABLE=1 and NCCL_SOCKET_IFNAME=eth0 to force TCP instead of InfiniBand** (70% success)
   ```
   Reduce network load by using NCCL_IB_DISABLE=1 and NCCL_SOCKET_IFNAME=eth0 to force TCP instead of InfiniBand
   ```
3. **Run with torch.distributed.run --nproc_per_node=N with NCCL_DEBUG=INFO to identify the hanging rank and check its GPU memory or process status** (85% success)
   ```
   Run with torch.distributed.run --nproc_per_node=N with NCCL_DEBUG=INFO to identify the hanging rank and check its GPU memory or process status
   ```

## Dead Ends

- **Increasing barrier timeout to 1200 seconds via torch.distributed.init_process_group(timeout=timedelta(seconds=1200))** — A hung rank will eventually hit the watchdog timeout regardless of barrier timeout; the root cause is not duration but a hang. (70% fail)
- **Restarting only the failed rank 2 without resetting NCCL communicator** — NCCL communicator state is global; restarting one rank leaves the group in an inconsistent state, causing further errors. (90% fail)