# ValueError: You passed `quantization_config` with `bnb_4bit_compute_dtype=torch.float16` but the model weights are loaded in `torch.bfloat16`. This is incompatible.

- **ID:** `huggingface/quantization-dtype-mismatch-bnb`
- **Domain:** huggingface
- **Category:** config_error
- **Verification:** ai_generated
- **Fix Rate:** 90%

## Root Cause

Bitsandbytes quantization config compute dtype does not match the model's native weight dtype, causing a type cast failure during forward pass.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| bitsandbytes 0.43.0 | active | — | — |
| transformers 4.44.0 | active | — | — |
| torch 2.3.0 | active | — | — |

## Workarounds

1. **Set `bnb_4bit_compute_dtype=torch.bfloat16` to match the model weight dtype: `quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)`.** (95% success)
   ```
   Set `bnb_4bit_compute_dtype=torch.bfloat16` to match the model weight dtype: `quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)`.
   ```
2. **Load model with `torch_dtype=torch.float16` and set `bnb_4bit_compute_dtype=torch.float16`.** (80% success)
   ```
   Load model with `torch_dtype=torch.float16` and set `bnb_4bit_compute_dtype=torch.float16`.
   ```

## Dead Ends

- **** — If the model was saved in bfloat16, casting to float16 may cause overflow or underflow; the root mismatch remains if model weights are bfloat16. (50% fail)
- **** — Bitsandbytes checks for exact dtype match; this combination will still raise the same error. (100% fail)