cuda device ai_generated true

RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable

ID: cuda/all-cuda-capable-devices-busy

Also available as: JSON · Markdown
88%Fix Rate
90%Confidence
50Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
12 active — — —

Root Cause

All GPUs are in use by other processes or the CUDA driver failed to initialize.

generic

Workarounds

  1. 93% success Check GPU usage: nvidia-smi — identify which processes are using the GPUs 
    Look for processes with high memory usage that can be stopped safely
  2. 90% success Set CUDA_VISIBLE_DEVICES to use a specific free GPU 
    export CUDA_VISIBLE_DEVICES=1 — restricts to GPU 1 only

    Sources: https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars

Dead Ends

Common approaches that don't work:

  1. Killing all GPU processes with nvidia-smi --gpu-reset 75% fail

    May corrupt running training jobs; data loss

  2. Restarting the CUDA driver 70% fail

    Requires root; kills all GPU processes

Error Chain

Frequently confused with: