huggingface system_error ai_generated true

OSError: Error while deserializing header: HeaderTooLarge

ID: huggingface/cache-corruption-safetensors

Also available as: JSON · Markdown · 中文

80%Fix Rate

85%Confidence

1Evidence

2024-06-10First Seen

Version Compatibility

Version	Status	Introduced	Deprecated	Notes
safetensors 0.4.0	active	—	—	—
transformers 4.42.0	active	—	—	—
huggingface-hub 0.23.0	active	—	—	—

Root Cause

The safetensors file in the Hugging Face cache is corrupted, often due to incomplete download, disk write errors, or concurrent access by multiple processes.

generic

中文

Hugging Face缓存中的safetensors文件损坏，通常由下载不完整、磁盘写入错误或多个进程并发访问导致。

Official Documentation

https://huggingface.co/docs/safetensors/en/faq#corrupted-files

Workarounds

85% success Clear the Hugging Face cache for the specific model and re-download: import shutil from pathlib import Path cache_dir = Path.home() / '.cache' / 'huggingface' / 'hub' # Find and remove the corrupted model folder (e.g., models--bert-base-uncased) shutil.rmtree(cache_dir / 'models--bert-base-uncased') # Then reload the model from transformers import AutoModel model = AutoModel.from_pretrained('bert-base-uncased')
```
Clear the Hugging Face cache for the specific model and re-download:

import shutil
from pathlib import Path
cache_dir = Path.home() / '.cache' / 'huggingface' / 'hub'
# Find and remove the corrupted model folder (e.g., models--bert-base-uncased)
shutil.rmtree(cache_dir / 'models--bert-base-uncased')
# Then reload the model
from transformers import AutoModel
model = AutoModel.from_pretrained('bert-base-uncased')
```
80% success Use `huggingface-cli` to delete the model from cache: `huggingface-cli delete-cache` and select the corrupted model, then re-download with `from_pretrained`.
```
Use `huggingface-cli` to delete the model from cache: `huggingface-cli delete-cache` and select the corrupted model, then re-download with `from_pretrained`.
```

中文步骤

清除特定模型的Hugging Face缓存并重新下载：

import shutil
from pathlib import Path
cache_dir = Path.home() / '.cache' / 'huggingface' / 'hub'
# 找到并删除损坏的模型文件夹（例如 models--bert-base-uncased）
shutil.rmtree(cache_dir / 'models--bert-base-uncased')
# 然后重新加载模型
from transformers import AutoModel
model = AutoModel.from_pretrained('bert-base-uncased')

使用 `huggingface-cli` 从缓存中删除模型：`huggingface-cli delete-cache` 并选择损坏的模型，然后使用 `from_pretrained` 重新下载。

Dead Ends

Common approaches that don't work:

95% fail
The file on disk is corrupted, not the library; reinstalling doesn't fix the cache.
90% fail
The error is about a corrupted file header, not resource exhaustion.