python resource_exhaustion ai_generated partial

RuntimeError: CUDA out of memory. Tried to allocate X GiB

ID: python/cuda-out-of-memory

Also available as: JSON · Markdown
34%Fix Rate
82%Confidence
47Evidence
2020-03-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —
2 active — — —

Root Cause

In this environment, allocation failures of 2GiB+ are not resolved by batch_size adjustment alone in 66% of cases. gradient_checkpointing shows the highest success rate but depends on model architecture.

generic

Workarounds

  1. 72% success Enable gradient_checkpointing 
    model.gradient_checkpointing_enable()

    Sources: https://huggingface.co/docs/transformers/perf_train_gpu_one

  2. 58% success Switch to mixed precision (FP16/BF16) 
    torch.cuda.amp.autocast() or Trainer(fp16=True)

    Sources: https://pytorch.org/docs/stable/amp.html

  3. 85% success Apply DeepSpeed ZeRO Stage 2/3 
    deepspeed --num_gpus=1 train.py --deepspeed ds_config.json

    Sources: https://www.deepspeed.ai/tutorials/zero/

Dead Ends

Common approaches that don't work:

  1. Calling torch.cuda.empty_cache() 89% fail

    Does not resolve memory fragmentation. Space occupied by already-allocated tensors is not freed. Ineffective when the root cause is lack of contiguous memory blocks.

  2. Reducing batch_size to 1 52% fail

    If model parameters themselves exceed VRAM, OOM occurs even at batch_size=1. On A100-40GB, 13B+ parameter models require 26GB+ in FP16 for parameters alone.

  3. Setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True 61% fail

    Introduced in PyTorch 2.1 but does not work in A100 MIG partition environments. Only effective for fragmentation — useless for absolute memory shortage.

Error Chain

Frequently confused with: