llm data_error ai_generated partial

ValueError: Encountered unknown token ID 100000 in cached sequence. Tokenizer vocabulary mismatch.

ID: llm/tokenizer-mismatch-cache-miss

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2024-03-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
transformers 4.35.0 active — — —
tokenizers 0.14.0 active — — —
vLLM 0.3.0 active — — —
Llama 2 70B active — — —
Mistral 7B active — — —

Root Cause

Cached tokenized sequences were generated with a different tokenizer version or model, and the current tokenizer does not recognize token IDs from the cache.

generic

中文

缓存的 token 化序列是由不同版本或模型的分词器生成的，当前分词器无法识别缓存中的 token ID。

Official Documentation

https://huggingface.co/docs/tokenizers/main/en/api/reference#tokenizers.Tokenizer.decode

Workarounds

  1. 90% success Invalidate the cache and regenerate it with the current tokenizer. For Hugging Face datasets, run: `dataset.cleanup_cache_files()` and reload with `load_from_disk(new_path)`. 
    Invalidate the cache and regenerate it with the current tokenizer. For Hugging Face datasets, run: `dataset.cleanup_cache_files()` and reload with `load_from_disk(new_path)`.
  2. 75% success If using vLLM or similar inference engines, set environment variable `VLLM_USE_PREEMPTIVE_MODE=1` to disable token caching at the cost of slightly slower inference. 
    If using vLLM or similar inference engines, set environment variable `VLLM_USE_PREEMPTIVE_MODE=1` to disable token caching at the cost of slightly slower inference.
  3. 85% success Pin the tokenizer version in your environment by specifying `tokenizers==0.14.0` in requirements.txt and using a consistent model snapshot (e.g., `model_name_or_path='meta-llama/Llama-2-7b-hf'`). 
    Pin the tokenizer version in your environment by specifying `tokenizers==0.14.0` in requirements.txt and using a consistent model snapshot (e.g., `model_name_or_path='meta-llama/Llama-2-7b-hf'`).

中文步骤

  1. 使缓存失效并使用当前分词器重新生成。对于 Hugging Face 数据集，运行：`dataset.cleanup_cache_files()` 并使用 `load_from_disk(new_path)` 重新加载。
  2. 如果使用 vLLM 或类似推理引擎，设置环境变量 `VLLM_USE_PREEMPTIVE_MODE=1` 以禁用 token 缓存，代价是推理速度稍慢。
  3. 在 requirements.txt 中固定分词器版本（例如 `tokenizers==0.14.0`），并使用一致的模型快照（例如 `model_name_or_path='meta-llama/Llama-2-7b-hf'`）。

Dead Ends

Common approaches that don't work:

  1. 60% fail

    The cache will be regenerated using the same mismatched tokenizer, and the error will reappear when the tokenizer or model is updated again.

  2. 80% fail

    Mapping unknown IDs to [UNK] discards semantic meaning and degrades model output quality, and the tokenizer may still reject the IDs during decoding.

  3. 90% fail

    The error is not related to fast vs slow tokenizer mode; it is a vocabulary mismatch that persists across both modes.