huggingface config_error ai_generated true

ValueError：您传递了`bnb_4bit_compute_dtype=torch.float16`的`quantization_config`，但模型权重以torch.float32加载。将`bnb_4bit_compute_dtype`设置为与模型权重匹配或转换模型。

ValueError: You passed `quantization_config` with `bnb_4bit_compute_dtype=torch.float16` but the model weights are loaded in torch.float32. Set `bnb_4bit_compute_dtype` to match the model weights or convert the model.

ID: huggingface/quantization-dtype-mismatch

其他格式: JSON · Markdown 中文 · English
88%修复率
86%置信度
1证据数
2024-04-08首次发现

版本兼容性

版本状态引入弃用备注
transformers>=4.32.0 active — — —
bitsandbytes>=0.41.0 active — — —
accelerate>=0.24.0 active — — —

根因分析

使用bitsandbytes 4位量化时，量化配置中指定的计算数据类型与模型权重数据类型不匹配，导致混合精度操作中的类型不匹配。

English

When using bitsandbytes 4-bit quantization, the compute dtype specified in the quantization config does not match the model's weight dtype, causing a type mismatch in mixed-precision operations.

generic

官方文档

https://huggingface.co/docs/transformers/main/en/quantization#bitsandbytes

解决方案

  1. Set quantization config to match model weights: from transformers import BitsAndBytesConfig; bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=model.dtype) and then model = AutoModelForCausalLM.from_pretrained('model-name', quantization_config=bnb_config).
  2. Load model with torch_dtype matching the compute dtype: model = AutoModelForCausalLM.from_pretrained('model-name', torch_dtype=torch.float16, quantization_config=bnb_config).

无效尝试

常见但无效的做法:

  1. Setting bnb_4bit_compute_dtype to torch.float32 without changing model weights 50% 失败

    If the model was loaded with torch_dtype=torch.float16, the compute dtype float32 may cause performance degradation or OOM, but the error might be suppressed at the cost of incorrect computation.

  2. Ignoring the error and continuing with default dtype 100% 失败

    The error is raised as a ValueError; the code will stop execution unless caught, so ignoring is not possible.