# JSONDecodeError: 从 'tokenizer.json' 加载分词器配置时，期望值：第 1 行第 1 列（字符 0）

- **ID:** `huggingface/tokenizer-json-parse-error`
- **领域:** huggingface
- **类别:** data_error
- **验证级别:** ai_generated
- **修复率:** 90%

## 根因

tokenizer.json 文件为空、截断或不是有效的 JSON，通常由于下载不完整或磁盘损坏。

## 版本兼容性

| 版本 | 状态 | 引入 | 弃用 |
|------|------|------|------|
| huggingface/tokenizers 0.15.0 | active | — | — |
| huggingface/transformers 4.36.0 | active | — | — |
| huggingface/tokenizers 0.16.0 | active | — | — |

## 解决方案

1. ```
   Delete the cached tokenizer file and re-download: `rm ~/.cache/huggingface/hub/models--org--model-name/snapshots/*/tokenizer.json` then re-run `AutoTokenizer.from_pretrained('org/model-name')`
   ```
2. ```
   Force re-download by setting `force_download=True` in `from_pretrained`: `AutoTokenizer.from_pretrained('org/model-name', force_download=True)`
   ```

## 无效尝试

- **Manually edit tokenizer.json to fix the JSON syntax** — The file is often beyond repair (empty or truncated); manual editing rarely fixes the underlying structural corruption. (95% 失败率)
- **Use `from_pretrained(..., trust_remote_code=True)`** — This flag bypasses config validation but doesn't fix the corrupted tokenizer.json; the JSON decode error still occurs. (85% 失败率)