EXECUTION pytorch cuda_error ai_generated true

RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED when calling cudnnRNNBackwardData

ID: pytorch/cudnn-rnn-backward-error

Also available as: JSON · Markdown
78%Fix Rate
85%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —

Root Cause

cuDNN RNN backward pass fails. Often caused by non-contiguous hidden states, wrong CUDA/cuDNN version, or variable-length sequences without packing.

generic

Workarounds

  1. 90% success Make hidden states contiguous before passing to RNN 
    hidden = hidden.contiguous()  # or h_0, c_0 = h_0.contiguous(), c_0.contiguous()

    Sources: https://pytorch.org/tutorials/

  2. 88% success Use pack_padded_sequence for variable-length inputs 
    packed = nn.utils.rnn.pack_padded_sequence(input, lengths, batch_first=True, enforce_sorted=False)
  3. 80% success Set torch.backends.cudnn.deterministic=True as a diagnostic step 
    If deterministic mode works, the issue is a cuDNN non-deterministic kernel bug; update CUDA/cuDNN

Dead Ends

Common approaches that don't work:

  1. Disable cuDNN entirely with torch.backends.cudnn.enabled=False 72% fail

    Disabling cuDNN massively slows down RNN operations (10-50x slower); fix the root cause instead

  2. Downgrade to a much older PyTorch version 65% fail

    Older versions have different cuDNN bugs; the real fix is proper input handling