cuda oom_error ai_generated partial

torch.cuda.OutOfMemoryError: CUDA out of memory

ID: cuda/torch-cuda-oom-new

Also available as: JSON · Markdown
85%Fix Rate
90%Confidence
50Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —

Root Cause

New PyTorch 2.x OOM error type. GPU memory exhausted during training or inference.

generic

Workarounds

  1. 95% success Reduce batch size — most direct fix for OOM 
    # Halve the batch size:
batch_size = 16  # was 32

# Or use gradient accumulation to maintain effective batch size:
accumulation_steps = 2
for i, batch in enumerate(dataloader):
    loss = model(batch) / accumulation_steps
    loss.backward()
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

    Sources: https://pytorch.org/docs/stable/notes/cuda.html

  2. 88% success Use gradient checkpointing to trade compute for memory 
    model.gradient_checkpointing_enable()

    Sources: https://pytorch.org/docs/stable/checkpoint.html

  3. 90% success Use mixed precision training (fp16/bf16) to halve memory 
    scaler = torch.amp.GradScaler(); with torch.autocast('cuda'):

    Sources: https://pytorch.org/docs/stable/amp.html

  4. 82% success Use DeepSpeed ZeRO or FSDP for multi-GPU memory distribution 
    Use DeepSpeed ZeRO or FSDP for multi-GPU memory distribution

    Sources: https://pytorch.org/docs/stable/fsdp.html

Dead Ends

Common approaches that don't work:

  1. Add torch.cuda.empty_cache() in training loop 75% fail

    Only frees cached memory, not memory held by tensors — negligible effect

  2. Set PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb 65% fail

    Only helps with fragmentation, not actual OOM

Error Chain

Frequently confused with: