CUDA error: system not ready (cudaErrorSystemNotReady)
ID: cuda/cuda-system-not-ready
Version Compatibility
| Version | Status | Introduced | Deprecated | Notes |
|---|---|---|---|---|
| 12 | active | — | — | — |
Root Cause
The CUDA subsystem is not ready to accept API calls. This typically occurs when the NVIDIA kernel module is loaded but the GPU has not completed initialization, during system resume from suspend/hibernate, when the GPU is being reset after an error (e.g., Xid error), or in containers where the GPU device files are not yet available.
genericWorkarounds
-
88% success Wait and retry the CUDA initialization with exponential backoff
The system-not-ready state is usually transient. Implement a retry loop: attempt cudaGetDeviceCount or cudaSetDevice, if it returns cudaErrorSystemNotReady, sleep for 1-2 seconds and retry up to 30 seconds. This is common after system resume or GPU reset. In containers, wait for nvidia-container-runtime to expose the device.
-
80% success Check and reload the NVIDIA kernel modules if the GPU is stuck
Run: nvidia-smi to check GPU state. If nvidia-smi also fails, the driver may need reloading: sudo rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia && sudo modprobe nvidia. Check dmesg for Xid errors indicating GPU hardware issues. If the GPU experienced an Xid 79 (GPU fallen off the bus), a full system reboot may be required.
Dead Ends
Common approaches that don't work:
-
Reinstalling the CUDA toolkit to fix the system not ready state
90% fail
The error is a transient hardware/driver state issue, not a software installation problem. Reinstalling CUDA does not change the driver state or GPU hardware readiness. The GPU needs time to initialize or recover from a reset.
-
Calling cudaDeviceReset() to force the system into a ready state
85% fail
cudaDeviceReset destroys the current CUDA context but does not affect the underlying driver or hardware state. If the system is not ready at the driver level, resetting the CUDA context does not help and may return the same error.