llm resource_error ai_generated true

RuntimeError: OutOfMemoryError: Unable to allocate memory for KV cache block. Requested block size: 16, free blocks: 0

ID: llm/vllm-block-manager-out-of-memory

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2024-01-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
vLLM 0.3.0 active — — —
vLLM 0.4.0 active — — —
vLLM 0.4.2 active — — —
CUDA 12.1 active — — —
CUDA 12.2 active — — —

Root Cause

vLLM's PagedAttention block manager exhausts all available GPU memory for KV cache blocks due to insufficient gpu_memory_utilization setting or concurrent requests exceeding memory budget.

generic

中文

vLLM的PagedAttention块管理器因gpu_memory_utilization设置不足或并发请求超出内存预算，耗尽了KV缓存块的所有可用GPU内存。

Official Documentation

https://vllm.readthedocs.io/en/latest/models/engine_args.html

Workarounds

  1. 90% success Reduce gpu_memory_utilization to 0.85 or lower and set --max-num-seqs appropriately: python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-7b-hf --gpu-memory-utilization 0.80 --max-num-seqs 64 
    Reduce gpu_memory_utilization to 0.85 or lower and set --max-num-seqs appropriately:

python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-7b-hf --gpu-memory-utilization 0.80 --max-num-seqs 64
  2. 85% success Enable swap space via --swap-space to offload KV cache to CPU when GPU memory is tight: python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-7b-hf --swap-space 16 
    Enable swap space via --swap-space to offload KV cache to CPU when GPU memory is tight:

python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-7b-hf --swap-space 16

中文步骤

  1. 将gpu_memory_utilization降低到0.85或更低，并适当设置--max-num-seqs：

python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-7b-hf --gpu-memory-utilization 0.80 --max-num-seqs 64
  2. 通过--swap-space启用交换空间，在GPU内存紧张时将KV缓存卸载到CPU：

python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-7b-hf --swap-space 16

Dead Ends

Common approaches that don't work:

  1. 60% fail

    Larger batch tokens consume more KV cache memory without freeing space; the error persists or worsens.

  2. 40% fail

    Truncates all inputs, degrading model quality for long-context tasks; may still OOM if concurrent requests are high.