huggingface config_error ai_generated true

ValueError：您传递了 `bnb_4bit_compute_dtype=torch.float16` 的量化配置，但模型权重以 `torch.bfloat16` 加载。两者不兼容。

ValueError: You passed `quantization_config` with `bnb_4bit_compute_dtype=torch.float16` but the model weights are loaded in `torch.bfloat16`. This is incompatible.

ID: huggingface/quantization-dtype-mismatch-bnb

其他格式: JSON · Markdown 中文 · English
90%修复率
87%置信度
1证据数
2024-03-10首次发现

版本兼容性

版本状态引入弃用备注
bitsandbytes 0.43.0 active — — —
transformers 4.44.0 active — — —
torch 2.3.0 active — — —

根因分析

Bitsandbytes 量化配置的计算数据类型与模型的原始权重数据类型不匹配，导致前向传播时类型转换失败。

English

Bitsandbytes quantization config compute dtype does not match the model's native weight dtype, causing a type cast failure during forward pass.

generic

官方文档

https://huggingface.co/docs/bitsandbytes/en/quantization#dtype-mismatch

解决方案

  1. Set `bnb_4bit_compute_dtype=torch.bfloat16` to match the model weight dtype: `quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)`.
  2. Load model with `torch_dtype=torch.float16` and set `bnb_4bit_compute_dtype=torch.float16`.

无效尝试

常见但无效的做法:

  1. 50% 失败

    If the model was saved in bfloat16, casting to float16 may cause overflow or underflow; the root mismatch remains if model weights are bfloat16.

  2. 100% 失败

    Bitsandbytes checks for exact dtype match; this combination will still raise the same error.