cudaErrorPeerAccessUnsupported cuda runtime_error ai_generated partial

运行时错误：CUDA 错误：这两个设备之间不支持对等访问 (cudaErrorPeerAccessUnsupported)

RuntimeError: CUDA error: peer access is not supported between these two devices (cudaErrorPeerAccessUnsupported)

ID: cuda/peer-access-unsupported-by-hardware

其他格式: JSON · Markdown 中文 · English
70%修复率
84%置信度
1证据数
2023-06-10首次发现

版本兼容性

版本状态引入弃用备注
CUDA 11.0 active — — —
CUDA 12.0 active — — —
CUDA 12.3 active — — —

根因分析

两个 GPU 不支持直接对等 (P2P) 内存访问，通常是由于硬件拓扑（例如不同的 PCIe 交换机）或驱动程序中禁用了 P2P。

English

The two GPUs do not support direct peer-to-peer (P2P) memory access, typically due to hardware topology (e.g., different PCIe switches) or disabled P2P in the driver.

generic

官方文档

https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__PEER.html

解决方案

  1. Disable peer-to-peer access in your code by setting the environment variable NCCL_P2P_DISABLE=1 before launching the script. For PyTorch DistributedDataParallel, use: os.environ['NCCL_P2P_DISABLE'] = '1'. This forces NCCL to use shared memory or network-based communication instead.
  2. If using multiple GPUs, assign each GPU to a separate process (e.g., via torch.multiprocessing) to avoid P2P requirements. For example, use torch.cuda.set_device(rank) and communicate via torch.distributed with NCCL_SHM_DISABLE=1.

无效尝试

常见但无效的做法:

  1. 90% 失败

    Enabling P2P via software flags cannot override hardware limitations; it will still fail.

  2. 80% 失败

    Rebooting the system does not change GPU topology; if P2P is unsupported by hardware, it remains unsupported.