cuda runtime_error ai_generated partial

RuntimeError: CUDA error: peer access is not supported between these two devices

ID: cuda/multi-gpu-peer-access-error

Also available as: JSON · Markdown
65%Fix Rate
80%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
12 active — — —

Root Cause

GPUs cannot directly access each other's memory. No NVLink between the devices or PCIe topology prevents P2P.

generic

Workarounds

  1. 85% success Disable P2P and let NCCL fall back to shared host memory 
    export NCCL_P2P_DISABLE=1  # uses host-staged copies instead
  2. 88% success Check GPU topology to verify which GPUs have P2P support 
    nvidia-smi topo -m  # NV# = NVLink, PHB = no direct P2P
  3. 80% success Place communicating tensors on GPUs with direct NVLink connections

Dead Ends

Common approaches that don't work:

  1. Force enable peer access with environment variables 90% fail

    P2P is a hardware capability; if the bus topology doesn't support it, no software config can enable it

  2. Move all tensors to a single GPU to avoid P2P 82% fail

    Defeats the purpose of multi-GPU; model may not fit on one GPU

Error Chain

Preceded by: