# RuntimeError: expected all tensors to be on the same device, but model parameters have been offloaded to CPU

- **ID:** `huggingface/model-offloaded-to-cpu`
- **Domain:** huggingface
- **Category:** runtime_error
- **Verification:** ai_generated
- **Fix Rate:** 80%

## Root Cause

Model was loaded with device_map='auto' or similar offloading, but an operation (e.g., forward pass) is being performed on a different device without proper device placement.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| transformers>=4.36.0 | active | — | — |
| accelerate>=0.25.0 | active | — | — |
| PyTorch>=2.1.0 | active | — | — |

## Workarounds

1. **Ensure all operations use the same device context. Use accelerate's context manager: from accelerate import dispatch_model; dispatch_model(model, device_map='auto') and then run inference within a with torch.no_grad() block on the same device.** (85% success)
   ```
   Ensure all operations use the same device context. Use accelerate's context manager: from accelerate import dispatch_model; dispatch_model(model, device_map='auto') and then run inference within a with torch.no_grad() block on the same device.
   ```
2. **Set device_map to a specific device (e.g., device_map='cuda:0') instead of 'auto' to force all layers onto GPU if memory permits.** (75% success)
   ```
   Set device_map to a specific device (e.g., device_map='cuda:0') instead of 'auto' to force all layers onto GPU if memory permits.
   ```

## Dead Ends

- **Manually moving model to GPU with model.to('cuda') after loading with device_map** — device_map='auto' already handles placement; model.to() overrides it and may cause offloaded layers to be lost or cause memory issues on large models. (70% fail)
- **Setting device_map=None and manually placing model on GPU** — For very large models that require offloading, disabling device_map may cause OOM because the entire model won't fit on GPU. (85% fail)