pytorch memory_error ai_generated true

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB

ID: pytorch/cuda-out-of-memory

Also available as: JSON · Markdown
85%Fix Rate
92%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —

Root Cause

GPU memory exhausted during training or inference. Batch size too large, model too big, or memory leak from accumulated gradients.

generic

Workarounds

  1. 92% success Reduce batch size or use gradient accumulation 
    Halve batch_size; use optimizer.step() every N mini-batches to simulate larger batch
  2. 88% success Use mixed precision training (AMP) 
    with torch.amp.autocast('cuda'): ... # FP16 uses ~half the memory
  3. 85% success Detach loss and move metrics to CPU 
    loss_val = loss.item()  # not loss_val = loss; prevents computation graph accumulation
  4. 80% success Use gradient checkpointing for large models 
    model.gradient_checkpointing_enable()  # trades compute for memory

Dead Ends

Common approaches that don't work:

  1. Set PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb to a small value 70% fail

    Reduces fragmentation but does not free memory; often makes OOM happen sooner with a different error

  2. Call torch.cuda.empty_cache() in the training loop 72% fail

    empty_cache releases unused cached memory back to CUDA but does not free tensors still referenced; has no effect if all memory is in use

  3. Upgrade to a GPU with more VRAM as first response 60% fail

    Often the code has a memory leak (not detaching losses, storing all predictions). Fix the code before throwing hardware at it.