huggingface config_error ai_generated true

ValueError: pad_token_id (0) must be within the vocabulary size (32000) but is greater than or equal to the vocab size

ID: huggingface/tokenizer-pad-token-id-mismatch

Also available as: JSON · Markdown · 中文

85%Fix Rate

88%Confidence

1Evidence

2023-11-20First Seen

Version Compatibility

Version	Status	Introduced	Deprecated	Notes
transformers>=4.30.0	active	—	—	—
tokenizers>=0.14.0	active	—	—	—

Root Cause

The tokenizer's pad_token_id is set to a value that exceeds the model's vocabulary size, often due to using a tokenizer with a different vocabulary than the model.

generic

中文

分词器的pad_token_id设置为超出模型词汇表大小的值，通常是由于使用了与模型词汇表不同的分词器。

Official Documentation

https://huggingface.co/docs/transformers/main/en/troubleshooting

Workarounds

95% success Resize model embeddings to match tokenizer vocabulary: `model.resize_token_embeddings(len(tokenizer))`
```
Resize model embeddings to match tokenizer vocabulary: `model.resize_token_embeddings(len(tokenizer))`
```
90% success Set pad_token to an existing token within vocab: `tokenizer.pad_token = tokenizer.eos_token` if eos_token_id is valid, or `tokenizer.add_special_tokens({'pad_token': '[PAD]'})` then resize.
```
Set pad_token to an existing token within vocab: `tokenizer.pad_token = tokenizer.eos_token` if eos_token_id is valid, or `tokenizer.add_special_tokens({'pad_token': '[PAD]'})` then resize.
```
85% success Load tokenizer with explicit pad_token_id: `AutoTokenizer.from_pretrained('model-name', pad_token_id=0)` ensuring 0 is within vocab size.
```
Load tokenizer with explicit pad_token_id: `AutoTokenizer.from_pretrained('model-name', pad_token_id=0)` ensuring 0 is within vocab size.
```

中文步骤

Resize model embeddings to match tokenizer vocabulary: `model.resize_token_embeddings(len(tokenizer))`

Set pad_token to an existing token within vocab: `tokenizer.pad_token = tokenizer.eos_token` if eos_token_id is valid, or `tokenizer.add_special_tokens({'pad_token': '[PAD]'})` then resize.

Load tokenizer with explicit pad_token_id: `AutoTokenizer.from_pretrained('model-name', pad_token_id=0)` ensuring 0 is within vocab size.

Dead Ends

Common approaches that don't work:

95% fail
The pad_token_id must be a valid token index within the model's vocabulary; values outside cause index errors in embedding layers.
80% fail
Adding tokens changes the embedding layer size, which may not match the pre-trained model's weights, leading to random initialization.
70% fail
The eos_token_id may also be out of range or not set, causing the same error.