huggingface type_error ai_generated true

RuntimeError: The base model dtype is torch.float32 but the LoRA adapter was trained with torch.bfloat16. This may cause numerical instability.

ID: huggingface/peft-adapter-torch-dtype-mismatch

Also available as: JSON · Markdown · 中文

85%Fix Rate

87%Confidence

1Evidence

2024-01-20First Seen

Version Compatibility

Version	Status	Introduced	Deprecated	Notes
peft>=0.7.0	active	—	—	—
transformers>=4.36.0	active	—	—	—
torch>=2.1.0	active	—	—	—

Root Cause

PEFT adapter weights are loaded onto a base model with a different torch dtype, causing a dtype mismatch that can lead to silent accuracy degradation or NaN outputs.

generic

中文

PEFT 适配器权重加载到具有不同 torch 数据类型的基础模型上，导致数据类型不匹配，可能引发精度下降或 NaN 输出。

Official Documentation

https://huggingface.co/docs/peft/en/developer_guides/quantization

Workarounds

85% success Load the base model with the same dtype as the adapter (bfloat16 in this case): from transformers import AutoModel import torch model = AutoModel.from_pretrained('base-model', torch_dtype=torch.bfloat16) model.load_adapter('adapter-path') # Verify dtype print(model.dtype) # Should be torch.bfloat16
```
Load the base model with the same dtype as the adapter (bfloat16 in this case):
from transformers import AutoModel
import torch
model = AutoModel.from_pretrained('base-model', torch_dtype=torch.bfloat16)
model.load_adapter('adapter-path')
# Verify dtype
print(model.dtype)  # Should be torch.bfloat16
```
80% success Convert the adapter weights to the base model's dtype after loading using PeftModel.from_pretrained with dtype argument: from peft import PeftModel base_model = AutoModel.from_pretrained('base-model', torch_dtype=torch.float32) peft_model = PeftModel.from_pretrained(base_model, 'adapter-path', torch_dtype=torch.float32)
```
Convert the adapter weights to the base model's dtype after loading using PeftModel.from_pretrained with dtype argument:
from peft import PeftModel
base_model = AutoModel.from_pretrained('base-model', torch_dtype=torch.float32)
peft_model = PeftModel.from_pretrained(base_model, 'adapter-path', torch_dtype=torch.float32)
```

中文步骤

Load the base model with the same dtype as the adapter (bfloat16 in this case):
from transformers import AutoModel
import torch
model = AutoModel.from_pretrained('base-model', torch_dtype=torch.bfloat16)
model.load_adapter('adapter-path')
# Verify dtype
print(model.dtype)  # Should be torch.bfloat16

Convert the adapter weights to the base model's dtype after loading using PeftModel.from_pretrained with dtype argument:
from peft import PeftModel
base_model = AutoModel.from_pretrained('base-model', torch_dtype=torch.float32)
peft_model = PeftModel.from_pretrained(base_model, 'adapter-path', torch_dtype=torch.float32)

Dead Ends

Common approaches that don't work:

80% fail
Forcing model.to(torch.bfloat16) after loading the adapter does not change the adapter's internal dtype and may cause a separate device mismatch error.
70% fail
Setting torch_dtype='auto' in from_pretrained may load the base model in float16, which still mismatches if the adapter expects bfloat16.