# RuntimeError: The base model dtype is torch.float32 but the LoRA adapter was trained with torch.bfloat16. This may cause numerical instability.

- **ID:** `huggingface/peft-adapter-torch-dtype-mismatch`
- **Domain:** huggingface
- **Category:** type_error
- **Verification:** ai_generated
- **Fix Rate:** 85%

## Root Cause

PEFT adapter weights are loaded onto a base model with a different torch dtype, causing a dtype mismatch that can lead to silent accuracy degradation or NaN outputs.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| peft>=0.7.0 | active | — | — |
| transformers>=4.36.0 | active | — | — |
| torch>=2.1.0 | active | — | — |

## Workarounds

1. **Load the base model with the same dtype as the adapter (bfloat16 in this case):
from transformers import AutoModel
import torch
model = AutoModel.from_pretrained('base-model', torch_dtype=torch.bfloat16)
model.load_adapter('adapter-path')
# Verify dtype
print(model.dtype)  # Should be torch.bfloat16** (85% success)
   ```
   Load the base model with the same dtype as the adapter (bfloat16 in this case):
from transformers import AutoModel
import torch
model = AutoModel.from_pretrained('base-model', torch_dtype=torch.bfloat16)
model.load_adapter('adapter-path')
# Verify dtype
print(model.dtype)  # Should be torch.bfloat16
   ```
2. **Convert the adapter weights to the base model's dtype after loading using PeftModel.from_pretrained with dtype argument:
from peft import PeftModel
base_model = AutoModel.from_pretrained('base-model', torch_dtype=torch.float32)
peft_model = PeftModel.from_pretrained(base_model, 'adapter-path', torch_dtype=torch.float32)** (80% success)
   ```
   Convert the adapter weights to the base model's dtype after loading using PeftModel.from_pretrained with dtype argument:
from peft import PeftModel
base_model = AutoModel.from_pretrained('base-model', torch_dtype=torch.float32)
peft_model = PeftModel.from_pretrained(base_model, 'adapter-path', torch_dtype=torch.float32)
   ```

## Dead Ends

- **** — Forcing model.to(torch.bfloat16) after loading the adapter does not change the adapter's internal dtype and may cause a separate device mismatch error. (80% fail)
- **** — Setting torch_dtype='auto' in from_pretrained may load the base model in float16, which still mismatches if the adapter expects bfloat16. (70% fail)