# torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has a total capacity of 79.15 GiB of which 0 bytes is free.

- **ID:** `llm/vllm-cuda-oom-during-prefill`
- **Domain:** llm
- **Category:** resource_error
- **Error Code:** `CUDA OOM`
- **Verification:** ai_generated
- **Fix Rate:** 80%

## Root Cause

vLLM's prefill phase (processing the prompt) allocates KV cache memory proportional to prompt length; a very long prompt or high batch size exhausts GPU memory despite available capacity being reported as zero due to fragmentation.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| vllm==0.4.3 | active | — | — |
| torch==2.3.0 | active | — | — |
| cuda==12.1 | active | — | — |

## Workarounds

1. **Enable automatic prefix caching (`--enable-prefix-caching`) in vLLM to reuse KV cache for common prompt prefixes, reducing memory for repeated prompts.** (85% success)
   ```
   Enable automatic prefix caching (`--enable-prefix-caching`) in vLLM to reuse KV cache for common prompt prefixes, reducing memory for repeated prompts.
   ```
2. **Set `max_model_len` to a lower value than the model's maximum (e.g., `--max-model-len 4096`) to limit KV cache allocation per request.** (90% success)
   ```
   Set `max_model_len` to a lower value than the model's maximum (e.g., `--max-model-len 4096`) to limit KV cache allocation per request.
   ```
3. **Use `--enforce-eager` to disable CUDA graphs, which reduces memory fragmentation at the cost of throughput.** (80% success)
   ```
   Use `--enforce-eager` to disable CUDA graphs, which reduces memory fragmentation at the cost of throughput.
   ```

## Dead Ends

- **** — This limits throughput severely and may cause request queuing; the OOM may still occur if a single long prompt exceeds the reduced budget. (75% fail)
- **** — vLLM already manages memory; increasing utilization can cause it to allocate more KV cache, worsening fragmentation. (70% fail)
- **** — This changes the model entirely and may not be feasible for the application; the OOM can still happen with long prompts on smaller models. (80% fail)