运行时错误：调用 cudnnSetRNNDescriptor_v8 时返回 CUDNN_STATUS_BAD_PARAM，参数为 hiddenSize=4096, numLayers=4, dropout=0.0
RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM when calling cudnnSetRNNDescriptor_v8 with hiddenSize=4096, numLayers=4, and dropout=0.0
ID: cuda/cudnn-rnn-hidden-size-mismatch
版本兼容性
| 版本 | 状态 | 引入 | 弃用 | 备注 |
|---|---|---|---|---|
| cuDNN 8.9 | active | — | — | — |
| cuDNN 9.0 | active | — | — | — |
| PyTorch 2.1 | active | — | — | — |
| PyTorch 2.2 | active | — | — | — |
根因分析
cuDNN RNN 描述符初始化失败，因为隐藏层大小不是所选 RNN 模式和数据类型对齐要求（通常为 32 或 64）的倍数。
English
cuDNN RNN descriptor initialization fails because the hidden size is not a multiple of the alignment requirement (typically 32 or 64) for the chosen RNN mode and data type.
官方文档
https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnSetRNNDescriptor_v8解决方案
-
将隐藏层大小填充到最接近的 64 的倍数。例如：hidden_size = ((hidden_size + 63) // 64) * 64。然后相应调整模型的隐藏维度。
-
在创建 RNN 模块之前设置 torch.backends.cudnn.enabled = False，切换到非 cuDNN 的 RNN 实现。这会使用 PyTorch 的原生 RNN，没有对齐约束。
-
使用不同的 RNN 模式，如 GRU 或 LSTM，并保持相同的隐藏层大小；有时不同模式的对齐要求不同。
无效尝试
常见但无效的做法:
-
95% 失败
The error is caused by hidden size alignment, not layer count. Reducing layers may change the model architecture but doesn't fix the alignment issue.
-
98% 失败
Dropout doesn't affect descriptor alignment requirements; it only controls regularization. Changing it has no effect on the BAD_PARAM error.
-
70% 失败
Older cuDNN versions may have different alignment constraints but often are more restrictive. This can introduce other compatibility issues with PyTorch or CUDA.