huggingface type_error ai_generated true

运行时错误：基础模型数据类型为 torch.float32，但 LoRA 适配器使用 torch.bfloat16 训练。这可能导致数值不稳定。

RuntimeError: The base model dtype is torch.float32 but the LoRA adapter was trained with torch.bfloat16. This may cause numerical instability.

ID: huggingface/peft-adapter-torch-dtype-mismatch

其他格式: JSON · Markdown 中文 · English
85%修复率
87%置信度
1证据数
2024-01-20首次发现

版本兼容性

版本状态引入弃用备注
peft>=0.7.0 active — — —
transformers>=4.36.0 active — — —
torch>=2.1.0 active — — —

根因分析

PEFT 适配器权重加载到具有不同 torch 数据类型的基础模型上，导致数据类型不匹配，可能引发精度下降或 NaN 输出。

English

PEFT adapter weights are loaded onto a base model with a different torch dtype, causing a dtype mismatch that can lead to silent accuracy degradation or NaN outputs.

generic

官方文档

https://huggingface.co/docs/peft/en/developer_guides/quantization

解决方案

  1. Load the base model with the same dtype as the adapter (bfloat16 in this case):
from transformers import AutoModel
import torch
model = AutoModel.from_pretrained('base-model', torch_dtype=torch.bfloat16)
model.load_adapter('adapter-path')
# Verify dtype
print(model.dtype)  # Should be torch.bfloat16
  2. Convert the adapter weights to the base model's dtype after loading using PeftModel.from_pretrained with dtype argument:
from peft import PeftModel
base_model = AutoModel.from_pretrained('base-model', torch_dtype=torch.float32)
peft_model = PeftModel.from_pretrained(base_model, 'adapter-path', torch_dtype=torch.float32)

无效尝试

常见但无效的做法:

  1. 80% 失败

    Forcing model.to(torch.bfloat16) after loading the adapter does not change the adapter's internal dtype and may cause a separate device mismatch error.

  2. 70% 失败

    Setting torch_dtype='auto' in from_pretrained may load the base model in float16, which still mismatches if the adapter expects bfloat16.