pytorch runtime_error ai_generated true

RuntimeError: FSDP parameter mismatch across ranks: expected parameter 'layer.0.weight' but got parameter 'module.layer.0.weight'

ID: pytorch/fsdp-parameter-mismatch

Also available as: JSON · Markdown · 中文
90%Fix Rate
86%Confidence
1Evidence
2023-08-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
pytorch>=2.0 active — — —
torchvision>=0.15 active — — —
cuda>=11.7 active — — —
nccl>=2.14 active — — —

Root Cause

Fully Sharded Data Parallel (FSDP) detected that model parameters have different names across distributed ranks, often due to wrapping the model in an extra module (e.g., nn.DataParallel or DDP) before FSDP, or inconsistent model initialization.

generic

中文

全分片数据并行（FSDP）检测到各分布式rank上的模型参数名称不同，通常由于在FSDP之前将模型包装在额外模块中（例如nn.DataParallel或DDP），或模型初始化不一致。

Official Documentation

https://pytorch.org/docs/stable/fsdp.html

Workarounds

  1. 95% success Remove any extra wrappers (e.g., DataParallel, DDP) before applying FSDP: model = MyModel(); model = FSDP(model) directly, not model = FSDP(DataParallel(model)) 
    Remove any extra wrappers (e.g., DataParallel, DDP) before applying FSDP: model = MyModel(); model = FSDP(model) directly, not model = FSDP(DataParallel(model))
  2. 85% success Ensure consistent model initialization across ranks by using the same random seed and loading the same checkpoint: torch.manual_seed(42); model.load_state_dict(torch.load('checkpoint.pt')) 
    Ensure consistent model initialization across ranks by using the same random seed and loading the same checkpoint: torch.manual_seed(42); model.load_state_dict(torch.load('checkpoint.pt'))
  3. 80% success Use a flat parameter naming scheme by setting param_init_fn in FSDP to avoid prefixes: FSDP(model, param_init_fn=lambda m: m.to_empty(device=torch.cuda.current_device())) 
    Use a flat parameter naming scheme by setting param_init_fn in FSDP to avoid prefixes: FSDP(model, param_init_fn=lambda m: m.to_empty(device=torch.cuda.current_device()))

中文步骤

  1. Remove any extra wrappers (e.g., DataParallel, DDP) before applying FSDP: model = MyModel(); model = FSDP(model) directly, not model = FSDP(DataParallel(model))
  2. Ensure consistent model initialization across ranks by using the same random seed and loading the same checkpoint: torch.manual_seed(42); model.load_state_dict(torch.load('checkpoint.pt'))
  3. Use a flat parameter naming scheme by setting param_init_fn in FSDP to avoid prefixes: FSDP(model, param_init_fn=lambda m: m.to_empty(device=torch.cuda.current_device()))

Dead Ends

Common approaches that don't work:

  1. Adding more FSDP wrapping layers to fix the mismatch 95% fail

    Extra wrapping adds more prefixes to parameter names, worsening the mismatch.

  2. Ignoring the error and continuing training with torch.distributed.barrier() 100% fail

    Parameter mismatch leads to incorrect gradient synchronization and model corruption.