EXECUTION cuda runtime_error ai_generated true

RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

ID: cuda/cudnn-execution-plan-failed

Also available as: JSON · Markdown
80%Fix Rate
80%Confidence
60Evidence
2021-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
12 active — — —

Root Cause

cuDNN fails to execute a convolution or attention kernel. Often caused by cuDNN benchmark mode selecting an algorithm that doesn't work for the specific input shape, or by a cuDNN/CUDA version incompatibility.

generic

Workarounds

  1. 85% success Disable cuDNN benchmark mode to use deterministic algorithm selection 
    import torch
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
# This prevents cuDNN from auto-tuning algorithms, avoiding the problematic one

    Sources: https://pytorch.org/docs/stable/backends.html#torch.backends.cudnn.benchmark

  2. 80% success Ensure cuDNN version matches the CUDA toolkit and driver version 
    # Check versions:
python -c "import torch; print(torch.backends.cudnn.version())"
nvidia-smi  # Check driver version
nvcc --version  # Check CUDA toolkit
# Install matching PyTorch:
pip install torch --index-url https://download.pytorch.org/whl/cu121

    Sources: https://pytorch.org/get-started/locally/

  3. 75% success Use torch.backends.cuda.preferred_linalg_library('default') and disable TF32 for debugging 
    import torch
torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cudnn.allow_tf32 = False
# If error disappears, TF32 precision is the issue. A100-specific.

    Sources: https://pytorch.org/docs/stable/notes/cuda.html#tf32-on-ampere

Dead Ends

Common approaches that don't work:

  1. Reinstalling PyTorch without checking cuDNN version compatibility 65% fail

    PyTorch bundles its own cuDNN, but if CUDA driver version is too old for the bundled cuDNN, or if LD_LIBRARY_PATH points to a system cuDNN that conflicts, reinstalling PyTorch alone doesn't resolve the mismatch.

  2. Reducing batch size as a general fix 60% fail

    CUDNN_STATUS_EXECUTION_FAILED is not an OOM error. Reducing batch size changes input dimensions, which may cause cuDNN to select a different algorithm that happens to work, but this masks the real issue and breaks reproducibility.

Error Chain

Preceded by:
Frequently confused with: