CUDA OOM llm resource_error ai_generated partial

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has a total capacity of 79.15 GiB of which 0 bytes is free.

ID: llm/vllm-cuda-oom-during-prefill

Also available as: JSON · Markdown · 中文
80%Fix Rate
86%Confidence
1Evidence
2024-04-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
vllm==0.4.3 active — — —
torch==2.3.0 active — — —
cuda==12.1 active — — —

Root Cause

vLLM's prefill phase (processing the prompt) allocates KV cache memory proportional to prompt length; a very long prompt or high batch size exhausts GPU memory despite available capacity being reported as zero due to fragmentation.

generic

中文

vLLM 的预填充阶段（处理提示）分配与提示长度成比例的 KV 缓存内存；非常长的提示或高批次大小耗尽 GPU 内存，尽管由于碎片化报告可用容量为零。

Official Documentation

https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html

Workarounds

  1. 85% success Enable automatic prefix caching (`--enable-prefix-caching`) in vLLM to reuse KV cache for common prompt prefixes, reducing memory for repeated prompts. 
    Enable automatic prefix caching (`--enable-prefix-caching`) in vLLM to reuse KV cache for common prompt prefixes, reducing memory for repeated prompts.
  2. 90% success Set `max_model_len` to a lower value than the model's maximum (e.g., `--max-model-len 4096`) to limit KV cache allocation per request. 
    Set `max_model_len` to a lower value than the model's maximum (e.g., `--max-model-len 4096`) to limit KV cache allocation per request.
  3. 80% success Use `--enforce-eager` to disable CUDA graphs, which reduces memory fragmentation at the cost of throughput. 
    Use `--enforce-eager` to disable CUDA graphs, which reduces memory fragmentation at the cost of throughput.

中文步骤

  1. Enable automatic prefix caching (`--enable-prefix-caching`) in vLLM to reuse KV cache for common prompt prefixes, reducing memory for repeated prompts.
  2. Set `max_model_len` to a lower value than the model's maximum (e.g., `--max-model-len 4096`) to limit KV cache allocation per request.
  3. Use `--enforce-eager` to disable CUDA graphs, which reduces memory fragmentation at the cost of throughput.

Dead Ends

Common approaches that don't work:

  1. 75% fail

    This limits throughput severely and may cause request queuing; the OOM may still occur if a single long prompt exceeds the reduced budget.

  2. 70% fail

    vLLM already manages memory; increasing utilization can cause it to allocate more KV cache, worsening fragmentation.

  3. 80% fail

    This changes the model entirely and may not be feasible for the application; the OOM can still happen with long prompts on smaller models.