cuda init_error ai_generated partial

RuntimeError: CUDA error: initialization error

ID: cuda/cuda-initialization-error

Also available as: JSON · Markdown
78%Fix Rate
85%Confidence
50Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
12 active — — —

Root Cause

CUDA driver failed to initialize. Driver crash, GPU reset, or process forking issue.

generic

Workarounds

  1. 85% success Restart the GPU: sudo nvidia-smi --gpu-reset 
    sudo nvidia-smi --gpu-reset

    Sources: https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__ERROR.html

  2. 90% success If using multiprocessing, set start method to 'spawn' not 'fork' 
    import multiprocessing
multiprocessing.set_start_method('spawn')

    Sources: https://pytorch.org/docs/stable/notes/multiprocessing.html

  3. 88% success Check nvidia-smi for GPU health and driver status 
    Check nvidia-smi for GPU health and driver status

    Sources: https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__ERROR.html

Dead Ends

Common approaches that don't work:

  1. Reinstall CUDA toolkit 70% fail

    Usually a driver issue, not toolkit issue

Error Chain

Frequently confused with: