GPA tensorflow system_error ai_generated partial

InternalError：当前CUDA驱动程序或设备拓扑不支持从GPU:0到GPU:1的直连访问

InternalError: Peer access from GPU:0 to GPU:1 is not supported by the current CUDA driver or device topology

ID: tensorflow/gpu-peer-access-error

其他格式: JSON · Markdown 中文 · English
80%修复率
84%置信度
1证据数
2024-02-14首次发现

版本兼容性

版本状态引入弃用备注
tensorflow>=2.13.0 active — — —
cuda>=11.8 active — — —
nvidia-driver>=525 active — — —

根因分析

由于硬件限制、驱动程序版本或PCIe拓扑约束，系统中的GPU不支持点对点内存访问（例如通过NVLink），但TensorFlow的多GPU分发策略尝试启用它。

English

The GPUs in the system do not support peer-to-peer memory access (e.g., via NVLink) due to hardware limitations, driver version, or PCIe topology constraints, but TensorFlow's multi-GPU distribution strategy attempted to enable it.

generic

官方文档

https://www.tensorflow.org/guide/gpu#multi-gpu_setup

解决方案

  1. Disable peer access in TensorFlow by setting the environment variable `TF_GPU_ALLOCATOR=cuda_malloc_async` or using `tf.config.experimental.set_memory_growth` per GPU. Alternatively, use `tf.distribute.MirroredStrategy` with `cross_device_ops=tf.distribute.HierarchicalCopyAllReduce()` which does not require peer access.
  2. Check GPU topology with `nvidia-smi topo -m` and if peer access is unsupported, place GPUs on the same PCIe switch if possible, or use a distribution strategy that avoids peer access (e.g., `tf.distribute.experimental.MultiWorkerMirroredStrategy` with RPC).

无效尝试

常见但无效的做法:

  1. Upgrading to the latest CUDA toolkit without checking driver compatibility. 65% 失败

    Peer access support depends on both hardware (e.g., NVLink) and driver version; a newer CUDA toolkit may not help if the driver is outdated or hardware lacks NVLink.

  2. Setting CUDA_VISIBLE_DEVICES to a single GPU to avoid multi-GPU errors. 50% 失败

    This bypasses the error but reduces the effective GPU count to 1, defeating the purpose of multi-GPU training; the error is not fixed, just avoided.