huggingface data_error ai_generated true

JSONDecodeError: Expecting value: line 1 column 1 (char 0) while loading tokenizer config from 'tokenizer.json'

ID: huggingface/tokenizer-json-parse-error

Also available as: JSON · Markdown · 中文
90%Fix Rate
87%Confidence
1Evidence
2023-11-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
huggingface/tokenizers 0.15.0 active — — —
huggingface/transformers 4.36.0 active — — —
huggingface/tokenizers 0.16.0 active — — —

Root Cause

The tokenizer.json file is empty, truncated, or not valid JSON, often due to incomplete download or disk corruption.

generic

中文

tokenizer.json 文件为空、截断或不是有效的 JSON，通常由于下载不完整或磁盘损坏。

Official Documentation

https://huggingface.co/docs/tokenizers/en/quicktour#loading-a-tokenizer

Workarounds

  1. 90% success Delete the cached tokenizer file and re-download: `rm ~/.cache/huggingface/hub/models--org--model-name/snapshots/*/tokenizer.json` then re-run `AutoTokenizer.from_pretrained('org/model-name')` 
    Delete the cached tokenizer file and re-download: `rm ~/.cache/huggingface/hub/models--org--model-name/snapshots/*/tokenizer.json` then re-run `AutoTokenizer.from_pretrained('org/model-name')`
  2. 85% success Force re-download by setting `force_download=True` in `from_pretrained`: `AutoTokenizer.from_pretrained('org/model-name', force_download=True)` 
    Force re-download by setting `force_download=True` in `from_pretrained`: `AutoTokenizer.from_pretrained('org/model-name', force_download=True)`

中文步骤

  1. Delete the cached tokenizer file and re-download: `rm ~/.cache/huggingface/hub/models--org--model-name/snapshots/*/tokenizer.json` then re-run `AutoTokenizer.from_pretrained('org/model-name')`
  2. Force re-download by setting `force_download=True` in `from_pretrained`: `AutoTokenizer.from_pretrained('org/model-name', force_download=True)`

Dead Ends

Common approaches that don't work:

  1. Manually edit tokenizer.json to fix the JSON syntax 95% fail

    The file is often beyond repair (empty or truncated); manual editing rarely fixes the underlying structural corruption.

  2. Use `from_pretrained(..., trust_remote_code=True)` 85% fail

    This flag bypasses config validation but doesn't fix the corrupted tokenizer.json; the JSON decode error still occurs.