huggingface device_error ai_generated true

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

ID: huggingface/pipeline-device-mismatch

Also available as: JSON · Markdown
88%Fix Rate
90%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —
4 active — — —

Root Cause

Model and input tensors are on different devices. Pipeline loaded on CPU while inputs on GPU, or mixed device_map.

generic

Workarounds

  1. 92% success Use device_map='auto' in from_pretrained to handle placement automatically 
    model = AutoModelForCausalLM.from_pretrained('model', device_map='auto')

    Sources: https://huggingface.co/docs/transformers/

  2. 90% success Set device explicitly in pipeline constructor 
    pipe = pipeline('text-generation', model='model', device=0)  # or device='cuda:0'
  3. 85% success Ensure input tensors match model device before forward pass 
    inputs = tokenizer(text, return_tensors='pt').to(model.device)  # use model.device for consistency

Dead Ends

Common approaches that don't work:

  1. Move individual tensors with .to() after each operation 75% fail

    Manually moving tensors is error-prone and slow. Use device_map='auto' or ensure consistent device assignment.

  2. Set CUDA_VISIBLE_DEVICES to empty to force CPU 70% fail

    Forcing CPU is a workaround, not a fix. The model will run orders of magnitude slower.