huggingface memory_error ai_generated true

OutOfMemoryError during attention computation

ID: huggingface/attention-sink-oom

Also available as: JSON · Markdown
78%Fix Rate
82%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
4 active — — —

Root Cause

Attention mechanism OOM on long sequences.

generic

Workarounds

  1. 90% success Use Flash Attention 2: model.from_pretrained(attn_implementation="flash_attention_2")
  2. 85% success Reduce max_length or use sliding window attention

Dead Ends

Common approaches that don't work:

  1. Pad all sequences to max length 78% fail

    Wastes memory on short sequences

  2. Remove attention mask 88% fail

    Wrong attention computation