huggingface config_error ai_generated true

ValueError: You passed `quantization_config` with `bnb_4bit_compute_dtype=torch.float16` but the model weights are loaded in torch.float32. Set `bnb_4bit_compute_dtype` to match the model weights or convert the model.

ID: huggingface/quantization-dtype-mismatch

Also available as: JSON · Markdown · 中文
88%Fix Rate
86%Confidence
1Evidence
2024-04-08First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
transformers>=4.32.0 active — — —
bitsandbytes>=0.41.0 active — — —
accelerate>=0.24.0 active — — —

Root Cause

When using bitsandbytes 4-bit quantization, the compute dtype specified in the quantization config does not match the model's weight dtype, causing a type mismatch in mixed-precision operations.

generic

中文

使用bitsandbytes 4位量化时，量化配置中指定的计算数据类型与模型权重数据类型不匹配，导致混合精度操作中的类型不匹配。

Official Documentation

https://huggingface.co/docs/transformers/main/en/quantization#bitsandbytes

Workarounds

  1. 90% success Set quantization config to match model weights: from transformers import BitsAndBytesConfig; bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=model.dtype) and then model = AutoModelForCausalLM.from_pretrained('model-name', quantization_config=bnb_config). 
    Set quantization config to match model weights: from transformers import BitsAndBytesConfig; bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=model.dtype) and then model = AutoModelForCausalLM.from_pretrained('model-name', quantization_config=bnb_config).
  2. 85% success Load model with torch_dtype matching the compute dtype: model = AutoModelForCausalLM.from_pretrained('model-name', torch_dtype=torch.float16, quantization_config=bnb_config). 
    Load model with torch_dtype matching the compute dtype: model = AutoModelForCausalLM.from_pretrained('model-name', torch_dtype=torch.float16, quantization_config=bnb_config).

中文步骤

  1. Set quantization config to match model weights: from transformers import BitsAndBytesConfig; bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=model.dtype) and then model = AutoModelForCausalLM.from_pretrained('model-name', quantization_config=bnb_config).
  2. Load model with torch_dtype matching the compute dtype: model = AutoModelForCausalLM.from_pretrained('model-name', torch_dtype=torch.float16, quantization_config=bnb_config).

Dead Ends

Common approaches that don't work:

  1. Setting bnb_4bit_compute_dtype to torch.float32 without changing model weights 50% fail

    If the model was loaded with torch_dtype=torch.float16, the compute dtype float32 may cause performance degradation or OOM, but the error might be suppressed at the cost of incorrect computation.

  2. Ignoring the error and continuing with default dtype 100% fail

    The error is raised as a ValueError; the code will stop execution unless caught, so ignoring is not possible.