cuda runtime_error ai_generated true

RuntimeError: FlashAttention only supports Ampere GPUs or newer (sm >= 80)

ID: cuda/flash-attention-unsupported

Also available as: JSON · Markdown
85%Fix Rate
88%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
12 active — — —

Root Cause

FlashAttention requires GPU compute capability >= 8.0 (Ampere: A100, RTX 3090+). Older GPUs are not supported.

generic

Workarounds

  1. 90% success Use torch.nn.functional.scaled_dot_product_attention with math backend as fallback 
    with torch.backends.cuda.sdp_kernel(enable_flash=False, enable_math=True): output = F.scaled_dot_product_attention(q, k, v)
  2. 88% success Set attn_implementation='eager' when loading HuggingFace models 
    model = AutoModelForCausalLM.from_pretrained('...', attn_implementation='eager')
  3. 82% success Use xformers memory_efficient_attention as alternative 
    pip install xformers && set use_memory_efficient_attention=True

Dead Ends

Common approaches that don't work:

  1. Install an older version of flash-attn 92% fail

    No version of flash-attn supports pre-Ampere GPUs; the hardware limitation is fundamental

  2. Set CUDA_VISIBLE_DEVICES to a different GPU hoping it works 85% fail

    If no Ampere+ GPU is available on the machine, no device will work