pytorch runtime_error ai_generated true

RuntimeError: torch.compile: Recompilation limit reached (100 recompilations) for function 'forward'. Consider using dynamic=True or reducing tensor shape variability.

ID: pytorch/compile-recompilation-limit

Also available as: JSON · Markdown · 中文
90%Fix Rate
84%Confidence
1Evidence
2024-04-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
torch>=2.0 active — — —
torch<=2.5.1 active — — —

Root Cause

torch.compile's guard mechanism triggers recompilation when tensor shapes change, and after 100 recompilations, it stops to prevent infinite loops or performance degradation.

generic

中文

torch.compile 的守卫机制在张量形状变化时触发重新编译，经过 100 次重新编译后停止，以防止无限循环或性能下降。

Official Documentation

https://pytorch.org/docs/stable/compile.html

Workarounds

  1. 90% success Use dynamic=True in torch.compile to handle dynamic shapes: torch.compile(model, dynamic=True). This allows the compiler to handle shape variations without recompiling. 
    Use dynamic=True in torch.compile to handle dynamic shapes: torch.compile(model, dynamic=True). This allows the compiler to handle shape variations without recompiling.
  2. 85% success Pad or resize tensors to a fixed set of shapes during training to reduce shape variability. For example, use torch.nn.utils.rnn.pad_sequence to pad sequences to a fixed length. 
    Pad or resize tensors to a fixed set of shapes during training to reduce shape variability. For example, use torch.nn.utils.rnn.pad_sequence to pad sequences to a fixed length.

中文步骤

  1. Use dynamic=True in torch.compile to handle dynamic shapes: torch.compile(model, dynamic=True). This allows the compiler to handle shape variations without recompiling.
  2. Pad or resize tensors to a fixed set of shapes during training to reduce shape variability. For example, use torch.nn.utils.rnn.pad_sequence to pad sequences to a fixed length.

Dead Ends

Common approaches that don't work:

  1. Disabling torch.compile entirely to avoid recompilation 60% fail

    This removes the performance benefits of compilation, which may be critical for large models.

  2. Setting torch._dynamo.config.recompile_limit = 0 to disable the limit 90% fail

    This can lead to infinite recompilation loops, causing severe performance degradation or crashes.