CUDA error: an NVLink error was detected on the link (cudaErrorNvlinkUncorrectable)
ID: cuda/nvlink-error
Version Compatibility
| Version | Status | Introduced | Deprecated | Notes |
|---|---|---|---|---|
| 12 | active | — | — | — |
Root Cause
An uncorrectable error was detected on an NVLink connection between GPUs. NVLink provides high-bandwidth GPU-to-GPU communication. Uncorrectable errors indicate physical link degradation, overheating, or hardware failure. This error causes the affected GPUs to be unable to communicate via NVLink, falling back to PCIe or failing entirely. Common in multi-GPU servers (DGX, HGX) under heavy inter-GPU communication workloads.
genericWorkarounds
-
75% success Check GPU temperatures and cooling, reseat GPUs if possible, and run NVLink diagnostics
Check temperatures: nvidia-smi -q -d TEMPERATURE. Run NVLink diagnostics: nvidia-smi nvlink --status -i 0. Check error counts: nvidia-smi nvlink -e -i 0. If errors are intermittent and temperature-related, improve server cooling. If the server allows it, reseat the GPU to fix a loose NVLink connector. Check dmesg for Xid 74 errors for details.
-
90% success Replace the affected GPU or NVLink bridge if errors persist
Persistent NVLink uncorrectable errors indicate hardware failure. Run nvidia-smi -q to identify which GPU and which NVLink link is affected. In DGX/HGX systems, the NVLink bridge may be replaceable separately. Contact NVIDIA or the server vendor for RMA. As a temporary workaround, use CUDA_VISIBLE_DEVICES to exclude the affected GPU from workloads.
Dead Ends
Common approaches that don't work:
-
Disabling NVLink in software and using PCIe for all GPU-to-GPU communication
80% fail
There is no software toggle to disable NVLink. CUDA automatically uses NVLink when available. Setting CUDA_VISIBLE_DEVICES to isolate GPUs avoids NVLink but also prevents multi-GPU workloads. The underlying hardware issue persists and may indicate impending GPU failure.
-
Increasing GPU power limits to compensate for NVLink errors
90% fail
NVLink errors are caused by link signal integrity issues, not insufficient power. Increasing power limits does not improve link quality and may increase temperatures, potentially worsening the problem.