# RuntimeError: FSDP parameter mismatch across ranks: expected parameter 'layer.0.weight' but got parameter 'module.layer.0.weight'

- **ID:** `pytorch/fsdp-parameter-mismatch`
- **Domain:** pytorch
- **Category:** runtime_error
- **Verification:** ai_generated
- **Fix Rate:** 90%

## Root Cause

Fully Sharded Data Parallel (FSDP) detected that model parameters have different names across distributed ranks, often due to wrapping the model in an extra module (e.g., nn.DataParallel or DDP) before FSDP, or inconsistent model initialization.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| pytorch>=2.0 | active | — | — |
| torchvision>=0.15 | active | — | — |
| cuda>=11.7 | active | — | — |
| nccl>=2.14 | active | — | — |

## Workarounds

1. **Remove any extra wrappers (e.g., DataParallel, DDP) before applying FSDP: model = MyModel(); model = FSDP(model) directly, not model = FSDP(DataParallel(model))** (95% success)
   ```
   Remove any extra wrappers (e.g., DataParallel, DDP) before applying FSDP: model = MyModel(); model = FSDP(model) directly, not model = FSDP(DataParallel(model))
   ```
2. **Ensure consistent model initialization across ranks by using the same random seed and loading the same checkpoint: torch.manual_seed(42); model.load_state_dict(torch.load('checkpoint.pt'))** (85% success)
   ```
   Ensure consistent model initialization across ranks by using the same random seed and loading the same checkpoint: torch.manual_seed(42); model.load_state_dict(torch.load('checkpoint.pt'))
   ```
3. **Use a flat parameter naming scheme by setting param_init_fn in FSDP to avoid prefixes: FSDP(model, param_init_fn=lambda m: m.to_empty(device=torch.cuda.current_device()))** (80% success)
   ```
   Use a flat parameter naming scheme by setting param_init_fn in FSDP to avoid prefixes: FSDP(model, param_init_fn=lambda m: m.to_empty(device=torch.cuda.current_device()))
   ```

## Dead Ends

- **Adding more FSDP wrapping layers to fix the mismatch** — Extra wrapping adds more prefixes to parameter names, worsening the mismatch. (95% fail)
- **Ignoring the error and continuing training with torch.distributed.barrier()** — Parameter mismatch leads to incorrect gradient synchronization and model corruption. (100% fail)