cuda configuration_error ai_generated true

CUDA error: invalid device ordinal (cudaErrorInvalidDevice)

ID: cuda/invalid-device-ordinal

Also available as: JSON · Markdown
92%Fix Rate
95%Confidence
50Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
12 active — — —

Root Cause

Code requested a GPU device number that does not exist. For example, requesting device 2 when only 2 GPUs are present (devices 0 and 1). Common when CUDA_VISIBLE_DEVICES hides some GPUs or when moving code between different GPU configurations.

generic

Workarounds

  1. 95% success Check available devices and use a valid ordinal: nvidia-smi -L 
    Run nvidia-smi -L to list GPUs. Adjust code to use device ordinal in range [0, num_gpus-1].
  2. 90% success Set CUDA_VISIBLE_DEVICES correctly and use remapped ordinals starting from 0 
    export CUDA_VISIBLE_DEVICES=0,1 then use torch.cuda.set_device(0) for the first visible GPU
  3. 88% success Use torch.cuda.device_count() to dynamically select devices 
    device = torch.device(f'cuda:{min(desired_id, torch.cuda.device_count()-1)}')

Dead Ends

Common approaches that don't work:

  1. Set CUDA_VISIBLE_DEVICES to include the requested device number 60% fail

    CUDA_VISIBLE_DEVICES remaps device ordinals; device N in the visible set becomes device 0,1,2... in order. Setting it to '2' makes that GPU device 0, not device 2.

  2. Install more GPU drivers hoping the device appears 95% fail

    The device ordinal is about physical hardware; drivers cannot create GPUs that do not exist

Error Chain

Leads to:
Preceded by: