huggingface config_error ai_generated true

UserWarning: The tokenizer is a 'PreTrainedTokenizerFast' but the model was loaded with a slow tokenizer. This may cause encoding inconsistencies.

ID: huggingface/tokenizer-slow-fast-mismatch

Also available as: JSON · Markdown · 中文
90%Fix Rate
82%Confidence
1Evidence
2023-11-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
transformers>=4.28.0 active — — —

Root Cause

When loading a model, the tokenizer type (fast vs slow) is inconsistent between the model configuration and the actual tokenizer used, often due to mismatched AutoTokenizer vs AutoModel loading.

generic

中文

加载模型时，模型配置与实际使用的分词器之间的分词器类型（快速 vs 慢速）不一致，通常是由于AutoTokenizer与AutoModel加载不匹配。

Official Documentation

https://huggingface.co/docs/transformers/v4.37.0/en/tokenizer_summary#fast-tokenizers

Workarounds

  1. 95% success Force loading a fast tokenizer by using use_fast=True: tokenizer = AutoTokenizer.from_pretrained('model-name', use_fast=True). This ensures consistency. 
    Force loading a fast tokenizer by using use_fast=True: tokenizer = AutoTokenizer.from_pretrained('model-name', use_fast=True). This ensures consistency.
  2. 85% success If you must use a slow tokenizer, explicitly load it: tokenizer = AutoTokenizer.from_pretrained('model-name', use_fast=False) and ensure the model was also loaded with a slow tokenizer. 
    If you must use a slow tokenizer, explicitly load it: tokenizer = AutoTokenizer.from_pretrained('model-name', use_fast=False) and ensure the model was also loaded with a slow tokenizer.

中文步骤

  1. Force loading a fast tokenizer by using use_fast=True: tokenizer = AutoTokenizer.from_pretrained('model-name', use_fast=True). This ensures consistency.
  2. If you must use a slow tokenizer, explicitly load it: tokenizer = AutoTokenizer.from_pretrained('model-name', use_fast=False) and ensure the model was also loaded with a slow tokenizer.

Dead Ends

Common approaches that don't work:

  1. Ignoring the warning and proceeding with training 60% fail

    The inconsistency can cause subtle bugs in tokenization (e.g., different special token handling, whitespace behavior) leading to incorrect model outputs or training divergence.

  2. Reinstalling transformers to fix the warning 95% fail

    The warning is not due to installation issues but to inconsistent tokenizer selection during model loading.