huggingface runtime_error ai_generated partial

RuntimeError: torch.compile is not supported for PEFT models with dynamic adapters

ID: huggingface/torch-compile-incompatible-with-peft

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2024-03-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
transformers>=4.38.0 active — — —
peft>=0.9.0 active — — —
torch>=2.2.0 active — — —

Root Cause

torch.compile fails on PEFT models when adapters are dynamically loaded or switched because the computation graph changes during runtime.

generic

中文

当适配器被动态加载或切换时，torch.compile在PEFT模型上失败，因为计算图在运行时发生变化。

Official Documentation

https://huggingface.co/docs/peft/main/en/developer_guides/torch_compile

Workarounds

  1. 85% success Use `torch.compile` only after loading all adapters and disable dynamic switching: `model = torch.compile(model, dynamic=False)` 
    Use `torch.compile` only after loading all adapters and disable dynamic switching: `model = torch.compile(model, dynamic=False)`
  2. 90% success Avoid torch.compile for PEFT models with dynamic adapters; use eager mode with `model.eval()` and manual fusion instead. 
    Avoid torch.compile for PEFT models with dynamic adapters; use eager mode with `model.eval()` and manual fusion instead.
  3. 95% success Set environment variable `TORCH_COMPILE_DISABLE=1` and rely on `torch.inference_mode()` for inference. 
    Set environment variable `TORCH_COMPILE_DISABLE=1` and rely on `torch.inference_mode()` for inference.

中文步骤

  1. Use `torch.compile` only after loading all adapters and disable dynamic switching: `model = torch.compile(model, dynamic=False)`
  2. Avoid torch.compile for PEFT models with dynamic adapters; use eager mode with `model.eval()` and manual fusion instead.
  3. Set environment variable `TORCH_COMPILE_DISABLE=1` and rely on `torch.inference_mode()` for inference.

Dead Ends

Common approaches that don't work:

  1. 95% fail

    The issue is not about optimization mode; dynamic adapter switching breaks torch.compile's graph tracing regardless of mode.

  2. 90% fail

    The incompatibility is fundamental to PEFT's dynamic nature, not a torch bug; nightly versions have the same limitation.

  3. 85% fail

    torch.compile captures the graph at compile time; gradient disabling does not prevent graph changes from dynamic adapters.