# JSONDecodeError: Expecting value: line 1 column 1 (char 0) while loading tokenizer config from 'tokenizer.json'

- **ID:** `huggingface/tokenizer-json-parse-error`
- **Domain:** huggingface
- **Category:** data_error
- **Verification:** ai_generated
- **Fix Rate:** 90%

## Root Cause

The tokenizer.json file is empty, truncated, or not valid JSON, often due to incomplete download or disk corruption.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| huggingface/tokenizers 0.15.0 | active | — | — |
| huggingface/transformers 4.36.0 | active | — | — |
| huggingface/tokenizers 0.16.0 | active | — | — |

## Workarounds

1. **Delete the cached tokenizer file and re-download: `rm ~/.cache/huggingface/hub/models--org--model-name/snapshots/*/tokenizer.json` then re-run `AutoTokenizer.from_pretrained('org/model-name')`** (90% success)
   ```
   Delete the cached tokenizer file and re-download: `rm ~/.cache/huggingface/hub/models--org--model-name/snapshots/*/tokenizer.json` then re-run `AutoTokenizer.from_pretrained('org/model-name')`
   ```
2. **Force re-download by setting `force_download=True` in `from_pretrained`: `AutoTokenizer.from_pretrained('org/model-name', force_download=True)`** (85% success)
   ```
   Force re-download by setting `force_download=True` in `from_pretrained`: `AutoTokenizer.from_pretrained('org/model-name', force_download=True)`
   ```

## Dead Ends

- **Manually edit tokenizer.json to fix the JSON syntax** — The file is often beyond repair (empty or truncated); manual editing rarely fixes the underlying structural corruption. (95% fail)
- **Use `from_pretrained(..., trust_remote_code=True)`** — This flag bypasses config validation but doesn't fix the corrupted tokenizer.json; the JSON decode error still occurs. (85% fail)