# InternalError: Peer access from GPU:0 to GPU:1 is not supported by the current CUDA driver or device topology

- **ID:** `tensorflow/gpu-peer-access-error`
- **Domain:** tensorflow
- **Category:** system_error
- **Error Code:** `GPA`
- **Verification:** ai_generated
- **Fix Rate:** 80%

## Root Cause

The GPUs in the system do not support peer-to-peer memory access (e.g., via NVLink) due to hardware limitations, driver version, or PCIe topology constraints, but TensorFlow's multi-GPU distribution strategy attempted to enable it.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| tensorflow>=2.13.0 | active | — | — |
| cuda>=11.8 | active | — | — |
| nvidia-driver>=525 | active | — | — |

## Workarounds

1. **Disable peer access in TensorFlow by setting the environment variable `TF_GPU_ALLOCATOR=cuda_malloc_async` or using `tf.config.experimental.set_memory_growth` per GPU. Alternatively, use `tf.distribute.MirroredStrategy` with `cross_device_ops=tf.distribute.HierarchicalCopyAllReduce()` which does not require peer access.** (80% success)
   ```
   Disable peer access in TensorFlow by setting the environment variable `TF_GPU_ALLOCATOR=cuda_malloc_async` or using `tf.config.experimental.set_memory_growth` per GPU. Alternatively, use `tf.distribute.MirroredStrategy` with `cross_device_ops=tf.distribute.HierarchicalCopyAllReduce()` which does not require peer access.
   ```
2. **Check GPU topology with `nvidia-smi topo -m` and if peer access is unsupported, place GPUs on the same PCIe switch if possible, or use a distribution strategy that avoids peer access (e.g., `tf.distribute.experimental.MultiWorkerMirroredStrategy` with RPC).** (75% success)
   ```
   Check GPU topology with `nvidia-smi topo -m` and if peer access is unsupported, place GPUs on the same PCIe switch if possible, or use a distribution strategy that avoids peer access (e.g., `tf.distribute.experimental.MultiWorkerMirroredStrategy` with RPC).
   ```

## Dead Ends

- **Upgrading to the latest CUDA toolkit without checking driver compatibility.** — Peer access support depends on both hardware (e.g., NVLink) and driver version; a newer CUDA toolkit may not help if the driver is outdated or hardware lacks NVLink. (65% fail)
- **Setting CUDA_VISIBLE_DEVICES to a single GPU to avoid multi-GPU errors.** — This bypasses the error but reduces the effective GPU count to 1, defeating the purpose of multi-GPU training; the error is not fixed, just avoided. (50% fail)