huggingface memory_error ai_generated true

torch.cuda.OutOfMemoryError: CUDA out of memory during model.generate()

ID: huggingface/generate-oom

Also available as: JSON · Markdown
80%Fix Rate
88%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

OOM during text generation. KV cache grows linearly with sequence length, consuming all VRAM.

generic

Workarounds

  1. 90% success Load model in lower precision 
    model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.float16, device_map='auto')

    Sources: https://huggingface.co/docs/transformers/

  2. 88% success Use quantization (4-bit or 8-bit) 
    model = AutoModelForCausalLM.from_pretrained(name, load_in_4bit=True, device_map='auto')
  3. 85% success Set reasonable max_new_tokens and use streaming 
    model.generate(inputs, max_new_tokens=256, do_sample=True)

Dead Ends

Common approaches that don't work:

  1. Set max_new_tokens very high for better generation quality 78% fail

    Longer sequences need quadratically more KV cache memory. Set reasonable limits.

  2. Call torch.cuda.empty_cache() between generate calls 65% fail

    Does not free the KV cache allocated during generate; only helps if there is fragmented cached memory