huggingface data_error ai_generated true

JSONDecodeError: 从 'tokenizer.json' 加载分词器配置时，期望值：第 1 行第 1 列（字符 0）

JSONDecodeError: Expecting value: line 1 column 1 (char 0) while loading tokenizer config from 'tokenizer.json'

ID: huggingface/tokenizer-json-parse-error

其他格式: JSON · Markdown 中文 · English
90%修复率
87%置信度
1证据数
2023-11-05首次发现

版本兼容性

版本状态引入弃用备注
huggingface/tokenizers 0.15.0 active — — —
huggingface/transformers 4.36.0 active — — —
huggingface/tokenizers 0.16.0 active — — —

根因分析

tokenizer.json 文件为空、截断或不是有效的 JSON，通常由于下载不完整或磁盘损坏。

English

The tokenizer.json file is empty, truncated, or not valid JSON, often due to incomplete download or disk corruption.

generic

官方文档

https://huggingface.co/docs/tokenizers/en/quicktour#loading-a-tokenizer

解决方案

  1. Delete the cached tokenizer file and re-download: `rm ~/.cache/huggingface/hub/models--org--model-name/snapshots/*/tokenizer.json` then re-run `AutoTokenizer.from_pretrained('org/model-name')`
  2. Force re-download by setting `force_download=True` in `from_pretrained`: `AutoTokenizer.from_pretrained('org/model-name', force_download=True)`

无效尝试

常见但无效的做法:

  1. Manually edit tokenizer.json to fix the JSON syntax 95% 失败

    The file is often beyond repair (empty or truncated); manual editing rarely fixes the underlying structural corruption.

  2. Use `from_pretrained(..., trust_remote_code=True)` 85% 失败

    This flag bypasses config validation but doesn't fix the corrupted tokenizer.json; the JSON decode error still occurs.