huggingface memory_error ai_generated true

OutOfMemoryError: CUDA out of memory (Transformers)

ID: huggingface/cuda-oom-transformers

Also available as: JSON · Markdown
80%Fix Rate
85%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
4 active — — —

Root Cause

Model too large for GPU VRAM during training or inference.

generic

Workarounds

  1. 90% success Use model.half() or load_in_8bit/4bit quantization 
    model=AutoModel.from_pretrained(name,load_in_8bit=True)
  2. 85% success Enable gradient checkpointing to reduce memory

Dead Ends

Common approaches that don't work:

  1. Increase CUDA memory fraction to 1.0 85% fail

    No headroom for PyTorch cache, fragmentation crash

  2. Switch to CPU only 70% fail

    Extremely slow for large models