# CUDA error: driver is in a state that is invalid for the requested operation (cudaErrorDriverNotReady)

- **ID:** `cuda/cuda-error-driver-unloading`
- **Domain:** cuda
- **Category:** runtime_error
- **Error Code:** `cudaErrorDriverNotReady (804)`
- **Verification:** ai_generated
- **Fix Rate:** 78%

## Root Cause

The CUDA driver is in the process of being unloaded or has been partially unloaded due to a race condition in multi-threaded application shutdown, often when cudaDeviceReset() is called while other threads still hold CUDA contexts.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| CUDA 11.8 | active | — | — |
| CUDA 12.0 | active | — | — |
| CUDA 12.1 | active | — | — |
| CUDA 12.2 | active | — | — |
| PyTorch 2.1.0 | active | — | — |
| PyTorch 2.2.0 | active | — | — |

## Workarounds

1. **Ensure all CUDA contexts are destroyed before calling cudaDeviceReset() by using a thread-safe reference counter. For example, in Python with PyTorch: `import torch; torch.cuda.synchronize(); torch.cuda.empty_cache(); torch.cuda.reset_peak_memory_stats(); del model; torch.cuda.reset_max_memory_cached()`** (85% success)
   ```
   Ensure all CUDA contexts are destroyed before calling cudaDeviceReset() by using a thread-safe reference counter. For example, in Python with PyTorch: `import torch; torch.cuda.synchronize(); torch.cuda.empty_cache(); torch.cuda.reset_peak_memory_stats(); del model; torch.cuda.reset_max_memory_cached()`
   ```
2. **Avoid calling cudaDeviceReset() in multi-threaded environments; instead, rely on the driver to clean up contexts at process exit. In C++, remove explicit `cudaDeviceReset()` calls from destructors or atexit handlers.** (90% success)
   ```
   Avoid calling cudaDeviceReset() in multi-threaded environments; instead, rely on the driver to clean up contexts at process exit. In C++, remove explicit `cudaDeviceReset()` calls from destructors or atexit handlers.
   ```
3. **Use a try-catch around the reset call and ignore the error if it occurs during shutdown: `try { cudaDeviceReset(); } catch (const std::exception&) { /* ignore during shutdown */ }`** (75% success)
   ```
   Use a try-catch around the reset call and ignore the error if it occurs during shutdown: `try { cudaDeviceReset(); } catch (const std::exception&) { /* ignore during shutdown */ }`
   ```

## Dead Ends

- **** — The error occurs during shutdown, so restarting only delays the issue; the race condition persists on subsequent shutdowns. (70% fail)
- **** — Synchronization does not guarantee that all threads have released their contexts; the driver may still be in an invalid state if other threads are mid-operation. (60% fail)