pytorch module_error ai_generated true

运行时错误：FSDP 检查点加载失败：状态字典中存在意外键："module._fsdp_wrapped_module.flat_param"

RuntimeError: FSDP checkpoint loading failed: Unexpected key(s) in state_dict: "module._fsdp_wrapped_module.flat_param"

ID: pytorch/fsdp-unexpected-key

其他格式: JSON · Markdown 中文 · English
82%修复率
87%置信度
1证据数
2023-09-12首次发现

版本兼容性

版本状态引入弃用备注
torch>=2.0.0 active — — —
FSDP>=1.12 active — — —

根因分析

状态字典包含 FSDP 内部键（例如 flat_param），加载到非 FSDP 模型时这些键是意外的，或者保存和加载时的 FSDP 包装层次不匹配。

English

The state_dict contains FSDP internal keys (e.g., flat_param) that are not expected when loading into a non-FSDP model, or the FSDP wrapping hierarchy mismatches between save and load.

generic

官方文档

https://pytorch.org/docs/stable/fsdp.html

解决方案

  1. Save the model state_dict with `state_dict` method instead of the FSDP wrapped module's state_dict, using `model.state_dict()` after unwrapping.
  2. Use `torch.distributed.fsdp.FullyShardedDataParallel.summon_full_params(model)` to get a full state_dict without FSDP keys.

无效尝试

常见但无效的做法:

  1. 80% 失败

    Simply ignoring the unexpected keys with strict=False may lead to incorrect model weights.

  2. 90% 失败

    Renaming state_dict keys manually often introduces errors and is not scalable.