# RuntimeError: FSDP parameter mismatch across ranks. Expected parameter 'model.fc.weight' to have shape [100, 256] but got [100, 128] on rank 2.

- **ID:** `pytorch/fsdp-param-mismatch`
- **Domain:** pytorch
- **Category:** config_error
- **Verification:** ai_generated
- **Fix Rate:** 82%

## Root Cause

Fully Sharded Data Parallel (FSDP) expects identical model architectures across all ranks; a mismatch in layer dimensions indicates inconsistent model initialization or different model files loaded on different processes.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| torch>=1.12 | active | — | — |
| torch>=2.0 | active | — | — |
| CUDA 11.x+ | active | — | — |
| NCCL 2.10+ | active | — | — |

## Workarounds

1. **Ensure identical model definition across all ranks by loading the same checkpoint: if rank == 0: torch.save(model.state_dict(), 'init.pt'); torch.distributed.barrier(); model.load_state_dict(torch.load('init.pt', map_location='cpu'))** (90% success)
   ```
   Ensure identical model definition across all ranks by loading the same checkpoint: if rank == 0: torch.save(model.state_dict(), 'init.pt'); torch.distributed.barrier(); model.load_state_dict(torch.load('init.pt', map_location='cpu'))
   ```
2. **Use torch.distributed.barrier() after model creation to synchronize before FSDP wrapping: model = MyModel(); torch.distributed.barrier(); fsdp_model = FullyShardedDataParallel(model)** (85% success)
   ```
   Use torch.distributed.barrier() after model creation to synchronize before FSDP wrapping: model = MyModel(); torch.distributed.barrier(); fsdp_model = FullyShardedDataParallel(model)
   ```

## Dead Ends

- **Setting FSDP sync_module_states=True without fixing architecture** — sync_module_states broadcasts parameters from rank 0 but does not resolve shape mismatches; it will fail with the same error. (95% fail)
- **Using different random seeds on each rank** — Random seeds affect weight initialization but not model architecture; shape mismatch persists regardless of seed. (80% fail)