CUDA OOM llm resource_error ai_generated partial

torch.cuda.OutOfMemoryError：CUDA 内存不足。尝试分配 2.00 GiB。GPU 0 总容量为 79.15 GiB，其中 0 字节空闲。

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has a total capacity of 79.15 GiB of which 0 bytes is free.

ID: llm/vllm-cuda-oom-during-prefill

其他格式: JSON · Markdown 中文 · English
80%修复率
86%置信度
1证据数
2024-04-05首次发现

版本兼容性

版本状态引入弃用备注
vllm==0.4.3 active — — —
torch==2.3.0 active — — —
cuda==12.1 active — — —

根因分析

vLLM 的预填充阶段（处理提示）分配与提示长度成比例的 KV 缓存内存；非常长的提示或高批次大小耗尽 GPU 内存，尽管由于碎片化报告可用容量为零。

English

vLLM's prefill phase (processing the prompt) allocates KV cache memory proportional to prompt length; a very long prompt or high batch size exhausts GPU memory despite available capacity being reported as zero due to fragmentation.

generic

官方文档

https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html

解决方案

  1. Enable automatic prefix caching (`--enable-prefix-caching`) in vLLM to reuse KV cache for common prompt prefixes, reducing memory for repeated prompts.
  2. Set `max_model_len` to a lower value than the model's maximum (e.g., `--max-model-len 4096`) to limit KV cache allocation per request.
  3. Use `--enforce-eager` to disable CUDA graphs, which reduces memory fragmentation at the cost of throughput.

无效尝试

常见但无效的做法:

  1. 75% 失败

    This limits throughput severely and may cause request queuing; the OOM may still occur if a single long prompt exceeds the reduced budget.

  2. 70% 失败

    vLLM already manages memory; increasing utilization can cause it to allocate more KV cache, worsening fragmentation.

  3. 80% 失败

    This changes the model entirely and may not be feasible for the application; the OOM can still happen with long prompts on smaller models.