# ValueError: Could not load tokenizer cache from /home/user/.cache/huggingface/hub — file is corrupted or truncated

- **ID:** `llm/tokenizer-cache-corruption-multiprocessing`
- **Domain:** llm
- **Category:** resource_error
- **Verification:** ai_generated
- **Fix Rate:** 88%

## Root Cause

Multiple processes (e.g., DataLoader workers with num_workers > 1) concurrently download or write to the Hugging Face tokenizer cache, causing race conditions that corrupt the cached tokenizer files.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| transformers==4.38.0 | active | — | — |
| tokenizers==0.15.2 | active | — | — |
| torch==2.2.0 | active | — | — |

## Workarounds

1. **Set the environment variable HF_HUB_ENABLE_HF_TRANSFER=1 and use a local lock file to serialize cache writes. Example:

import os
os.environ['HF_HUB_ENABLE_HF_TRANSFER'] = '1'
# Use a file lock for cache directory
import filelock
lock = filelock.FileLock('/tmp/hf_cache.lock')
with lock:
    tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')** (95% success)
   ```
   Set the environment variable HF_HUB_ENABLE_HF_TRANSFER=1 and use a local lock file to serialize cache writes. Example:

import os
os.environ['HF_HUB_ENABLE_HF_TRANSFER'] = '1'
# Use a file lock for cache directory
import filelock
lock = filelock.FileLock('/tmp/hf_cache.lock')
with lock:
    tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
   ```
2. **Set num_workers=0 in DataLoader to disable multiprocessing for tokenization, forcing all tokenization to happen in the main process.** (85% success)
   ```
   Set num_workers=0 in DataLoader to disable multiprocessing for tokenization, forcing all tokenization to happen in the main process.
   ```
3. **Pre-download the tokenizer before spawning workers: run AutoTokenizer.from_pretrained() once in the main process before creating the DataLoader. This ensures the cache is populated before workers access it.** (90% success)
   ```
   Pre-download the tokenizer before spawning workers: run AutoTokenizer.from_pretrained() once in the main process before creating the DataLoader. This ensures the cache is populated before workers access it.
   ```

## Dead Ends

- **Setting TOKENIZERS_PARALLELISM=false in environment** — This only disables tokenizer parallelism within a single process, it does not prevent cache corruption from multiple processes writing to the same cache directory. (70% fail)
- **Manually deleting the cache file and re-running** — The corruption will recur if the root cause (concurrent writes) is not addressed. The cache will be corrupted again on the next run with multiple workers. (90% fail)
- **Increasing HF_HUB_DOWNLOAD_TIMEOUT** — Timeout is not the issue; the issue is concurrent writes. Increasing timeout does not prevent race conditions. (95% fail)