pytorch network_error ai_generated true

RuntimeError: NCCL 通信器在 rank 2 上被中止。原始失败原因：看门狗回调超时。

RuntimeError: NCCL communicator was aborted on rank 2. Original reason for failure was: watchdog callback timed out.

ID: pytorch/nccl-communicator-aborted-watchdog-timeout

其他格式: JSON · Markdown 中文 · English
78%修复率
86%置信度
1证据数
2023-07-01首次发现

版本兼容性

版本状态引入弃用备注
torch>=1.13 active — — —
NCCL 2.16 active — — —
CUDA 11.8 active — — —
Slurm 23.02 active — — —

根因分析

NCCL 看门狗回调超时，表明一个集合操作（例如 allreduce）挂起时间过长，通常是由于网络拥塞、GPU 计算不平衡或单个慢节点导致。

English

A NCCL watchdog callback timed out, indicating that a collective operation (e.g., allreduce) hung for too long, often due to network congestion, GPU compute imbalance, or a single slow node.

generic

官方文档

https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html

解决方案

  1. Set environment variables to improve NCCL stability:

export NCCL_IB_TIMEOUT=22
export NCCL_SOCKET_IFNAME=eth0
export NCCL_DEBUG=INFO

Then rerun the job to identify the slow rank.
  2. Add gradient accumulation to reduce communication frequency:

accumulation_steps = 4
for i, (inputs, labels) in enumerate(dataloader):
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss = loss / accumulation_steps
    loss.backward()
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()
  3. Use torch.nn.parallel.DistributedDataParallel with find_unused_parameters=True and gradient_as_bucket_view=True to reduce communication overhead.

无效尝试

常见但无效的做法:

  1. Increasing NCCL_TIMEOUT environment variable to a very large value 80% 失败

    This only delays the timeout but does not fix the root cause (e.g., network congestion or slow node); the job will eventually hang or fail later.

  2. Restarting the job with the same number of GPUs 90% 失败

    If the underlying issue (e.g., network topology or GPU imbalance) is not addressed, the same failure will recur.

  3. Disabling NCCL watchdog with NCCL_DEBUG=WARN 95% 失败

    Disabling the watchdog hides the error but does not prevent the hang; the job will stall indefinitely without feedback.