huggingface config_error ai_generated partial

UserWarning: The tokenizer's unknown token (unk_token) is set to '<unk>', but the model's embedding layer has 0 embeddings for it. This may cause issues during generation.

ID: huggingface/tokenizer-unk-token-mismatch

Also available as: JSON · Markdown · 中文

75%Fix Rate

82%Confidence

1Evidence

2024-01-20First Seen

Version Compatibility

Version	Status	Introduced	Deprecated	Notes
transformers 4.38.0	active	—	—	—
tokenizers 0.15.0	active	—	—	—
Python 3.10	active	—	—	—

Root Cause

The tokenizer's vocabulary contains an unknown token (e.g., '<unk>') but the model's embedding layer does not have a corresponding embedding vector, often due to a mismatch between the tokenizer and model configuration.

generic

中文

分词器的词汇表中包含未知令牌（如'<unk>'），但模型的嵌入层没有对应的嵌入向量，通常由分词器和模型配置之间的不匹配导致。

Official Documentation

https://huggingface.co/docs/transformers/troubleshooting#tokenizer-model-mismatch

Workarounds

75% success Resize the model's token embeddings to match the tokenizer vocabulary and initialize the new embedding for the unk_token: from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained('gpt2') tokenizer = AutoTokenizer.from_pretrained('gpt2') model.resize_token_embeddings(len(tokenizer)) # Ensure the unk_token embedding is initialized (e.g., using mean of existing embeddings) import torch unk_id = tokenizer.unk_token_id if unk_id is not None and model.config.vocab_size <= unk_id: model.lm_head.weight.data[unk_id] = model.lm_head.weight.data.mean(dim=0)
```
Resize the model's token embeddings to match the tokenizer vocabulary and initialize the new embedding for the unk_token:

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('gpt2')
tokenizer = AutoTokenizer.from_pretrained('gpt2')
model.resize_token_embeddings(len(tokenizer))
# Ensure the unk_token embedding is initialized (e.g., using mean of existing embeddings)
import torch
unk_id = tokenizer.unk_token_id
if unk_id is not None and model.config.vocab_size <= unk_id:
    model.lm_head.weight.data[unk_id] = model.lm_head.weight.data.mean(dim=0)
```
80% success Use a tokenizer that exactly matches the model's original vocabulary by downloading the correct tokenizer from the model hub (e.g., using `AutoTokenizer.from_pretrained('gpt2')` instead of a custom one).
```
Use a tokenizer that exactly matches the model's original vocabulary by downloading the correct tokenizer from the model hub (e.g., using `AutoTokenizer.from_pretrained('gpt2')` instead of a custom one).
```

中文步骤

调整模型的令牌嵌入大小以匹配分词器词汇表，并为unk_token初始化新的嵌入：

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('gpt2')
tokenizer = AutoTokenizer.from_pretrained('gpt2')
model.resize_token_embeddings(len(tokenizer))
# 确保unk_token嵌入已初始化（例如，使用现有嵌入的均值）
import torch
unk_id = tokenizer.unk_token_id
if unk_id is not None and model.config.vocab_size <= unk_id:
    model.lm_head.weight.data[unk_id] = model.lm_head.weight.data.mean(dim=0)

通过从模型中心下载正确的分词器（例如使用 `AutoTokenizer.from_pretrained('gpt2')` 而不是自定义的）来使用与模型原始词汇表完全匹配的分词器。

Dead Ends

Common approaches that don't work:

85% fail
The model's embedding layer size is fixed; changing the tokenizer's unk_token doesn't add a new embedding unless the model is resized.
70% fail
The model will produce incorrect logits for unknown tokens, leading to degraded performance or gibberish generation.