# UserWarning：分词器是'PreTrainedTokenizerFast'，但模型是用慢速分词器加载的。这可能导致编码不一致。

- **ID:** `huggingface/tokenizer-slow-fast-mismatch`
- **领域:** huggingface
- **类别:** config_error
- **验证级别:** ai_generated
- **修复率:** 90%

## 根因

加载模型时，模型配置与实际使用的分词器之间的分词器类型（快速 vs 慢速）不一致，通常是由于AutoTokenizer与AutoModel加载不匹配。

## 版本兼容性

| 版本 | 状态 | 引入 | 弃用 |
|------|------|------|------|
| transformers>=4.28.0 | active | — | — |

## 解决方案

1. ```
   Force loading a fast tokenizer by using use_fast=True: tokenizer = AutoTokenizer.from_pretrained('model-name', use_fast=True). This ensures consistency.
   ```
2. ```
   If you must use a slow tokenizer, explicitly load it: tokenizer = AutoTokenizer.from_pretrained('model-name', use_fast=False) and ensure the model was also loaded with a slow tokenizer.
   ```

## 无效尝试

- **Ignoring the warning and proceeding with training** — The inconsistency can cause subtle bugs in tokenization (e.g., different special token handling, whitespace behavior) leading to incorrect model outputs or training divergence. (60% 失败率)
- **Reinstalling transformers to fix the warning** — The warning is not due to installation issues but to inconsistent tokenizer selection during model loading. (95% 失败率)