pytorch network_error ai_generated true

RuntimeError: NCCL communicator was aborted on rank 2. Original reason for failure was: watchdog callback timed out.

ID: pytorch/nccl-communicator-aborted-watchdog-timeout

Also available as: JSON · Markdown · 中文

78%Fix Rate

86%Confidence

1Evidence

2023-07-01First Seen

Version Compatibility

Version	Status	Introduced	Deprecated	Notes
torch>=1.13	active	—	—	—
NCCL 2.16	active	—	—	—
CUDA 11.8	active	—	—	—
Slurm 23.02	active	—	—	—

Root Cause

A NCCL watchdog callback timed out, indicating that a collective operation (e.g., allreduce) hung for too long, often due to network congestion, GPU compute imbalance, or a single slow node.

generic

中文

NCCL 看门狗回调超时，表明一个集合操作（例如 allreduce）挂起时间过长，通常是由于网络拥塞、GPU 计算不平衡或单个慢节点导致。

Official Documentation

https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html

Workarounds

85% success Set environment variables to improve NCCL stability: export NCCL_IB_TIMEOUT=22 export NCCL_SOCKET_IFNAME=eth0 export NCCL_DEBUG=INFO Then rerun the job to identify the slow rank.
```
Set environment variables to improve NCCL stability:

export NCCL_IB_TIMEOUT=22
export NCCL_SOCKET_IFNAME=eth0
export NCCL_DEBUG=INFO

Then rerun the job to identify the slow rank.
```
80% success Add gradient accumulation to reduce communication frequency: accumulation_steps = 4 for i, (inputs, labels) in enumerate(dataloader): outputs = model(inputs) loss = criterion(outputs, labels) loss = loss / accumulation_steps loss.backward() if (i + 1) % accumulation_steps == 0: optimizer.step() optimizer.zero_grad()
```
Add gradient accumulation to reduce communication frequency:

accumulation_steps = 4
for i, (inputs, labels) in enumerate(dataloader):
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss = loss / accumulation_steps
    loss.backward()
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()
```
75% success Use torch.nn.parallel.DistributedDataParallel with find_unused_parameters=True and gradient_as_bucket_view=True to reduce communication overhead.
```
Use torch.nn.parallel.DistributedDataParallel with find_unused_parameters=True and gradient_as_bucket_view=True to reduce communication overhead.
```

中文步骤

Set environment variables to improve NCCL stability:

export NCCL_IB_TIMEOUT=22
export NCCL_SOCKET_IFNAME=eth0
export NCCL_DEBUG=INFO

Then rerun the job to identify the slow rank.

Add gradient accumulation to reduce communication frequency:

accumulation_steps = 4
for i, (inputs, labels) in enumerate(dataloader):
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss = loss / accumulation_steps
    loss.backward()
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

Use torch.nn.parallel.DistributedDataParallel with find_unused_parameters=True and gradient_as_bucket_view=True to reduce communication overhead.

Dead Ends

Common approaches that don't work:

Increasing NCCL_TIMEOUT environment variable to a very large value 80% fail
This only delays the timeout but does not fix the root cause (e.g., network congestion or slow node); the job will eventually hang or fail later.
Restarting the job with the same number of GPUs 90% fail
If the underlying issue (e.g., network topology or GPU imbalance) is not addressed, the same failure will recur.
Disabling NCCL watchdog with NCCL_DEBUG=WARN 95% fail
Disabling the watchdog hides the error but does not prevent the hang; the job will stall indefinitely without feedback.