huggingface config_error ai_generated true

值错误：pad_token_id (0) 必须在词汇表大小 (32000) 范围内，但大于或等于词汇表大小

ValueError: pad_token_id (0) must be within the vocabulary size (32000) but is greater than or equal to the vocab size

ID: huggingface/tokenizer-pad-token-id-mismatch

其他格式: JSON · Markdown 中文 · English
85%修复率
88%置信度
1证据数
2023-11-20首次发现

版本兼容性

版本状态引入弃用备注
transformers>=4.30.0 active — — —
tokenizers>=0.14.0 active — — —

根因分析

分词器的pad_token_id设置为超出模型词汇表大小的值，通常是由于使用了与模型词汇表不同的分词器。

English

The tokenizer's pad_token_id is set to a value that exceeds the model's vocabulary size, often due to using a tokenizer with a different vocabulary than the model.

generic

官方文档

https://huggingface.co/docs/transformers/main/en/troubleshooting

解决方案

  1. Resize model embeddings to match tokenizer vocabulary: `model.resize_token_embeddings(len(tokenizer))`
  2. Set pad_token to an existing token within vocab: `tokenizer.pad_token = tokenizer.eos_token` if eos_token_id is valid, or `tokenizer.add_special_tokens({'pad_token': '[PAD]'})` then resize.
  3. Load tokenizer with explicit pad_token_id: `AutoTokenizer.from_pretrained('model-name', pad_token_id=0)` ensuring 0 is within vocab size.

无效尝试

常见但无效的做法:

  1. 95% 失败

    The pad_token_id must be a valid token index within the model's vocabulary; values outside cause index errors in embedding layers.

  2. 80% 失败

    Adding tokens changes the embedding layer size, which may not match the pre-trained model's weights, leading to random initialization.

  3. 70% 失败

    The eos_token_id may also be out of range or not set, causing the same error.