huggingface config_error ai_generated true

UserWarning：分词器是'PreTrainedTokenizerFast'，但模型是用慢速分词器加载的。这可能导致编码不一致。

UserWarning: The tokenizer is a 'PreTrainedTokenizerFast' but the model was loaded with a slow tokenizer. This may cause encoding inconsistencies.

ID: huggingface/tokenizer-slow-fast-mismatch

其他格式: JSON · Markdown 中文 · English
90%修复率
82%置信度
1证据数
2023-11-05首次发现

版本兼容性

版本状态引入弃用备注
transformers>=4.28.0 active — — —

根因分析

加载模型时，模型配置与实际使用的分词器之间的分词器类型（快速 vs 慢速）不一致，通常是由于AutoTokenizer与AutoModel加载不匹配。

English

When loading a model, the tokenizer type (fast vs slow) is inconsistent between the model configuration and the actual tokenizer used, often due to mismatched AutoTokenizer vs AutoModel loading.

generic

官方文档

https://huggingface.co/docs/transformers/v4.37.0/en/tokenizer_summary#fast-tokenizers

解决方案

  1. Force loading a fast tokenizer by using use_fast=True: tokenizer = AutoTokenizer.from_pretrained('model-name', use_fast=True). This ensures consistency.
  2. If you must use a slow tokenizer, explicitly load it: tokenizer = AutoTokenizer.from_pretrained('model-name', use_fast=False) and ensure the model was also loaded with a slow tokenizer.

无效尝试

常见但无效的做法:

  1. Ignoring the warning and proceeding with training 60% 失败

    The inconsistency can cause subtle bugs in tokenization (e.g., different special token handling, whitespace behavior) leading to incorrect model outputs or training divergence.

  2. Reinstalling transformers to fix the warning 95% 失败

    The warning is not due to installation issues but to inconsistent tokenizer selection during model loading.