pytorch config_error ai_generated true

RuntimeError: FSDP parameter mismatch across ranks. Expected parameter 'model.fc.weight' to have shape [100, 256] but got [100, 128] on rank 2.

ID: pytorch/fsdp-param-mismatch

Also available as: JSON · Markdown · 中文
82%Fix Rate
87%Confidence
1Evidence
2023-08-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
torch>=1.12 active — — —
torch>=2.0 active — — —
CUDA 11.x+ active — — —
NCCL 2.10+ active — — —

Root Cause

Fully Sharded Data Parallel (FSDP) expects identical model architectures across all ranks; a mismatch in layer dimensions indicates inconsistent model initialization or different model files loaded on different processes.

generic

中文

完全分片数据并行（FSDP）期望所有 rank 上的模型架构相同；层维度不匹配表示模型初始化不一致或不同进程加载了不同的模型文件。

Official Documentation

https://pytorch.org/docs/stable/fsdp.html

Workarounds

  1. 90% success Ensure identical model definition across all ranks by loading the same checkpoint: if rank == 0: torch.save(model.state_dict(), 'init.pt'); torch.distributed.barrier(); model.load_state_dict(torch.load('init.pt', map_location='cpu')) 
    Ensure identical model definition across all ranks by loading the same checkpoint: if rank == 0: torch.save(model.state_dict(), 'init.pt'); torch.distributed.barrier(); model.load_state_dict(torch.load('init.pt', map_location='cpu'))
  2. 85% success Use torch.distributed.barrier() after model creation to synchronize before FSDP wrapping: model = MyModel(); torch.distributed.barrier(); fsdp_model = FullyShardedDataParallel(model) 
    Use torch.distributed.barrier() after model creation to synchronize before FSDP wrapping: model = MyModel(); torch.distributed.barrier(); fsdp_model = FullyShardedDataParallel(model)

中文步骤

  1. Ensure identical model definition across all ranks by loading the same checkpoint: if rank == 0: torch.save(model.state_dict(), 'init.pt'); torch.distributed.barrier(); model.load_state_dict(torch.load('init.pt', map_location='cpu'))
  2. Use torch.distributed.barrier() after model creation to synchronize before FSDP wrapping: model = MyModel(); torch.distributed.barrier(); fsdp_model = FullyShardedDataParallel(model)

Dead Ends

Common approaches that don't work:

  1. Setting FSDP sync_module_states=True without fixing architecture 95% fail

    sync_module_states broadcasts parameters from rank 0 but does not resolve shape mismatches; it will fail with the same error.

  2. Using different random seeds on each rank 80% fail

    Random seeds affect weight initialization but not model architecture; shape mismatch persists regardless of seed.