llm resource_error ai_generated true

OSError：无法从 Hugging Face 检查点加载权重。错误：文件已损坏或截断

OSError: Unable to load weights from huggingface checkpoint. Error: file is corrupted or truncated

ID: llm/huggingface-model-load-cache-corruption

其他格式: JSON · Markdown 中文 · English
90%修复率
82%置信度
1证据数
2023-09-20首次发现

版本兼容性

版本状态引入弃用备注
transformers==4.41.0 active — — —
huggingface_hub==0.23.0 active — — —
torch==2.3.0 active — — —

根因分析

Hugging Face 模型缓存文件的部分下载或磁盘损坏，通常是由于下载中断或对缓存目录的并发写入访问导致。

English

Partial download or disk corruption of Hugging Face model cache files, often due to interrupted downloads or concurrent write access to the cache directory.

generic

官方文档

https://huggingface.co/docs/huggingface_hub/en/guides/download#corrupted-files

解决方案

  1. Clear the specific model cache and re-download: `rm -rf ~/.cache/huggingface/hub/models--bert-base-uncased` then reload the model.
  2. Use `snapshot_download` with `resume_download=False` to force a fresh download: `from huggingface_hub import snapshot_download; snapshot_download('bert-base-uncased', resume_download=False)`
  3. Set `HF_HUB_ENABLE_HF_TRANSFER=1` to use the faster download library which includes integrity checks by default.

无效尝试

常见但无效的做法:

  1. 100% 失败

    The corrupted file remains in the cache; the loader will attempt to read the same broken file and fail again.

  2. 80% 失败

    The error is about file integrity, not capacity; more space won't fix a truncated file.

  3. 90% 失败

    This changes the model entirely and doesn't address the cache corruption for the original model.