llm resource_error ai_generated true

OSError: Unable to load weights from huggingface checkpoint. Error: file is corrupted or truncated

ID: llm/huggingface-model-load-cache-corruption

Also available as: JSON · Markdown · 中文
90%Fix Rate
82%Confidence
1Evidence
2023-09-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
transformers==4.41.0 active — — —
huggingface_hub==0.23.0 active — — —
torch==2.3.0 active — — —

Root Cause

Partial download or disk corruption of Hugging Face model cache files, often due to interrupted downloads or concurrent write access to the cache directory.

generic

中文

Hugging Face 模型缓存文件的部分下载或磁盘损坏，通常是由于下载中断或对缓存目录的并发写入访问导致。

Official Documentation

https://huggingface.co/docs/huggingface_hub/en/guides/download#corrupted-files

Workarounds

  1. 95% success Clear the specific model cache and re-download: `rm -rf ~/.cache/huggingface/hub/models--bert-base-uncased` then reload the model. 
    Clear the specific model cache and re-download: `rm -rf ~/.cache/huggingface/hub/models--bert-base-uncased` then reload the model.
  2. 90% success Use `snapshot_download` with `resume_download=False` to force a fresh download: `from huggingface_hub import snapshot_download; snapshot_download('bert-base-uncased', resume_download=False)` 
    Use `snapshot_download` with `resume_download=False` to force a fresh download: `from huggingface_hub import snapshot_download; snapshot_download('bert-base-uncased', resume_download=False)`
  3. 85% success Set `HF_HUB_ENABLE_HF_TRANSFER=1` to use the faster download library which includes integrity checks by default. 
    Set `HF_HUB_ENABLE_HF_TRANSFER=1` to use the faster download library which includes integrity checks by default.

中文步骤

  1. Clear the specific model cache and re-download: `rm -rf ~/.cache/huggingface/hub/models--bert-base-uncased` then reload the model.
  2. Use `snapshot_download` with `resume_download=False` to force a fresh download: `from huggingface_hub import snapshot_download; snapshot_download('bert-base-uncased', resume_download=False)`
  3. Set `HF_HUB_ENABLE_HF_TRANSFER=1` to use the faster download library which includes integrity checks by default.

Dead Ends

Common approaches that don't work:

  1. 100% fail

    The corrupted file remains in the cache; the loader will attempt to read the same broken file and fail again.

  2. 80% fail

    The error is about file integrity, not capacity; more space won't fix a truncated file.

  3. 90% fail

    This changes the model entirely and doesn't address the cache corruption for the original model.