huggingface config_error ai_generated partial

UserWarning: 分词器的未知令牌（unk_token）设置为 '<unk>'，但模型的嵌入层有0个嵌入用于它。这可能在生成过程中导致问题。

UserWarning: The tokenizer's unknown token (unk_token) is set to '<unk>', but the model's embedding layer has 0 embeddings for it. This may cause issues during generation.

ID: huggingface/tokenizer-unk-token-mismatch

其他格式: JSON · Markdown 中文 · English
75%修复率
82%置信度
1证据数
2024-01-20首次发现

版本兼容性

版本状态引入弃用备注
transformers 4.38.0 active — — —
tokenizers 0.15.0 active — — —
Python 3.10 active — — —

根因分析

分词器的词汇表中包含未知令牌（如'<unk>'），但模型的嵌入层没有对应的嵌入向量，通常由分词器和模型配置之间的不匹配导致。

English

The tokenizer's vocabulary contains an unknown token (e.g., '<unk>') but the model's embedding layer does not have a corresponding embedding vector, often due to a mismatch between the tokenizer and model configuration.

generic

官方文档

https://huggingface.co/docs/transformers/troubleshooting#tokenizer-model-mismatch

解决方案

  1. 调整模型的令牌嵌入大小以匹配分词器词汇表，并为unk_token初始化新的嵌入：

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('gpt2')
tokenizer = AutoTokenizer.from_pretrained('gpt2')
model.resize_token_embeddings(len(tokenizer))
# 确保unk_token嵌入已初始化（例如，使用现有嵌入的均值）
import torch
unk_id = tokenizer.unk_token_id
if unk_id is not None and model.config.vocab_size <= unk_id:
    model.lm_head.weight.data[unk_id] = model.lm_head.weight.data.mean(dim=0)
  2. 通过从模型中心下载正确的分词器（例如使用 `AutoTokenizer.from_pretrained('gpt2')` 而不是自定义的）来使用与模型原始词汇表完全匹配的分词器。

无效尝试

常见但无效的做法:

  1. 85% 失败

    The model's embedding layer size is fixed; changing the tokenizer's unk_token doesn't add a new embedding unless the model is resized.

  2. 70% 失败

    The model will produce incorrect logits for unknown tokens, leading to degraded performance or gibberish generation.