pytorch config_error ai_generated true

RuntimeError: FSDP 参数在多个 rank 间不匹配。期望参数 'model.fc.weight' 的形状为 [100, 256]，但在 rank 2 上得到 [100, 128]。

RuntimeError: FSDP parameter mismatch across ranks. Expected parameter 'model.fc.weight' to have shape [100, 256] but got [100, 128] on rank 2.

ID: pytorch/fsdp-param-mismatch

其他格式: JSON · Markdown 中文 · English
82%修复率
87%置信度
1证据数
2023-08-20首次发现

版本兼容性

版本状态引入弃用备注
torch>=1.12 active — — —
torch>=2.0 active — — —
CUDA 11.x+ active — — —
NCCL 2.10+ active — — —

根因分析

完全分片数据并行（FSDP）期望所有 rank 上的模型架构相同；层维度不匹配表示模型初始化不一致或不同进程加载了不同的模型文件。

English

Fully Sharded Data Parallel (FSDP) expects identical model architectures across all ranks; a mismatch in layer dimensions indicates inconsistent model initialization or different model files loaded on different processes.

generic

官方文档

https://pytorch.org/docs/stable/fsdp.html

解决方案

  1. Ensure identical model definition across all ranks by loading the same checkpoint: if rank == 0: torch.save(model.state_dict(), 'init.pt'); torch.distributed.barrier(); model.load_state_dict(torch.load('init.pt', map_location='cpu'))
  2. Use torch.distributed.barrier() after model creation to synchronize before FSDP wrapping: model = MyModel(); torch.distributed.barrier(); fsdp_model = FullyShardedDataParallel(model)

无效尝试

常见但无效的做法:

  1. Setting FSDP sync_module_states=True without fixing architecture 95% 失败

    sync_module_states broadcasts parameters from rank 0 but does not resolve shape mismatches; it will fail with the same error.

  2. Using different random seeds on each rank 80% 失败

    Random seeds affect weight initialization but not model architecture; shape mismatch persists regardless of seed.