# RuntimeError: NCCL communicator was aborted on rank 2. Original reason for failure was: watchdog callback timed out.

- **ID:** `pytorch/nccl-communicator-aborted-watchdog-timeout`
- **Domain:** pytorch
- **Category:** network_error
- **Verification:** ai_generated
- **Fix Rate:** 78%

## Root Cause

A NCCL watchdog callback timed out, indicating that a collective operation (e.g., allreduce) hung for too long, often due to network congestion, GPU compute imbalance, or a single slow node.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| torch>=1.13 | active | — | — |
| NCCL 2.16 | active | — | — |
| CUDA 11.8 | active | — | — |
| Slurm 23.02 | active | — | — |

## Workarounds

1. **Set environment variables to improve NCCL stability:

export NCCL_IB_TIMEOUT=22
export NCCL_SOCKET_IFNAME=eth0
export NCCL_DEBUG=INFO

Then rerun the job to identify the slow rank.** (85% success)
   ```
   Set environment variables to improve NCCL stability:

export NCCL_IB_TIMEOUT=22
export NCCL_SOCKET_IFNAME=eth0
export NCCL_DEBUG=INFO

Then rerun the job to identify the slow rank.
   ```
2. **Add gradient accumulation to reduce communication frequency:

accumulation_steps = 4
for i, (inputs, labels) in enumerate(dataloader):
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss = loss / accumulation_steps
    loss.backward()
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()** (80% success)
   ```
   Add gradient accumulation to reduce communication frequency:

accumulation_steps = 4
for i, (inputs, labels) in enumerate(dataloader):
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss = loss / accumulation_steps
    loss.backward()
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()
   ```
3. **Use torch.nn.parallel.DistributedDataParallel with find_unused_parameters=True and gradient_as_bucket_view=True to reduce communication overhead.** (75% success)
   ```
   Use torch.nn.parallel.DistributedDataParallel with find_unused_parameters=True and gradient_as_bucket_view=True to reduce communication overhead.
   ```

## Dead Ends

- **Increasing NCCL_TIMEOUT environment variable to a very large value** — This only delays the timeout but does not fix the root cause (e.g., network congestion or slow node); the job will eventually hang or fail later. (80% fail)
- **Restarting the job with the same number of GPUs** — If the underlying issue (e.g., network topology or GPU imbalance) is not addressed, the same failure will recur. (90% fail)
- **Disabling NCCL watchdog with NCCL_DEBUG=WARN** — Disabling the watchdog hides the error but does not prevent the hang; the job will stall indefinitely without feedback. (95% fail)