huggingface data_error ai_generated true

ValueError: safetensors metadata mismatched: expected 1024 tensors, got 1023

ID: huggingface/safetensors-metadata-mismatch

Also available as: JSON · Markdown · 中文
85%Fix Rate
85%Confidence
1Evidence
2023-11-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
safetensors>=0.3.0 active — — —
transformers>=4.30.0 active — — —
huggingface-hub>=0.16.0 active — — —

Root Cause

The safetensors file header metadata declares a different number of tensors than are actually stored in the file, typically due to incomplete download or corrupted checkpoint.

generic

中文

safetensors 文件头部元数据声明的张量数量与实际文件中存储的张量数量不一致，通常是由于下载不完整或检查点损坏导致。

Official Documentation

https://huggingface.co/docs/safetensors/en/troubleshooting#metadata-mismatch

Workarounds

  1. 90% success Clear the Hugging Face cache and re-download the model: `rm -rf ~/.cache/huggingface/hub/models--org--model-name/` then `from transformers import AutoModel; AutoModel.from_pretrained('org/model-name')` 
    Clear the Hugging Face cache and re-download the model: `rm -rf ~/.cache/huggingface/hub/models--org--model-name/` then `from transformers import AutoModel; AutoModel.from_pretrained('org/model-name')`
  2. 85% success Use the huggingface_hub CLI to force re-download: `huggingface-cli download org/model-name --cache-dir /tmp/new-cache --local-dir ./model --force-download` 
    Use the huggingface_hub CLI to force re-download: `huggingface-cli download org/model-name --cache-dir /tmp/new-cache --local-dir ./model --force-download`
  3. 75% success If the file is partially downloaded, use `wget -c` to resume: `wget -c https://huggingface.co/org/model-name/resolve/main/model.safetensors` 
    If the file is partially downloaded, use `wget -c` to resume: `wget -c https://huggingface.co/org/model-name/resolve/main/model.safetensors`

中文步骤

  1. Clear the Hugging Face cache and re-download the model: `rm -rf ~/.cache/huggingface/hub/models--org--model-name/` then `from transformers import AutoModel; AutoModel.from_pretrained('org/model-name')`
  2. Use the huggingface_hub CLI to force re-download: `huggingface-cli download org/model-name --cache-dir /tmp/new-cache --local-dir ./model --force-download`
  3. If the file is partially downloaded, use `wget -c` to resume: `wget -c https://huggingface.co/org/model-name/resolve/main/model.safetensors`

Dead Ends

Common approaches that don't work:

  1. Re-download the same file without clearing cache 70% fail

    If the cached file is corrupted, re-downloading to the same cache path without clearing will reuse the broken file.

  2. Manually editing the safetensors metadata with a hex editor 95% fail

    Safetensors files are integrity-checked; any manual tampering causes a checksum error and breaks loading entirely.

  3. Upgrading transformers to the latest version 60% fail

    The error originates from the safetensors library, not transformers; upgrading transformers alone does not fix the corrupted file.