{
  "id": "huggingface/tokenizer-unk-token-mismatch",
  "signature": "UserWarning: The tokenizer's unknown token (unk_token) is set to '<unk>', but the model's embedding layer has 0 embeddings for it. This may cause issues during generation.",
  "signature_zh": "UserWarning: 分词器的未知令牌（unk_token）设置为 '<unk>'，但模型的嵌入层有0个嵌入用于它。这可能在生成过程中导致问题。",
  "regex": "UserWarning: The tokenizer's unknown token \\(unk_token\\) is set to '[^']+', but the model's embedding layer has 0 embeddings for it\\.",
  "domain": "huggingface",
  "category": "config_error",
  "subcategory": null,
  "root_cause": "The tokenizer's vocabulary contains an unknown token (e.g., '<unk>') but the model's embedding layer does not have a corresponding embedding vector, often due to a mismatch between the tokenizer and model configuration.",
  "root_cause_type": "generic",
  "root_cause_zh": "分词器的词汇表中包含未知令牌（如'<unk>'），但模型的嵌入层没有对应的嵌入向量，通常由分词器和模型配置之间的不匹配导致。",
  "versions": [
    {
      "version": "transformers 4.38.0",
      "introduced": null,
      "deprecated": null,
      "removed": null,
      "behavior_change": null,
      "status": "active"
    },
    {
      "version": "tokenizers 0.15.0",
      "introduced": null,
      "deprecated": null,
      "removed": null,
      "behavior_change": null,
      "status": "active"
    },
    {
      "version": "Python 3.10",
      "introduced": null,
      "deprecated": null,
      "removed": null,
      "behavior_change": null,
      "status": "active"
    }
  ],
  "os_specific": {},
  "dead_ends": [
    {
      "action": "",
      "why_fails": "The model's embedding layer size is fixed; changing the tokenizer's unk_token doesn't add a new embedding unless the model is resized.",
      "fail_rate": 0.85,
      "condition": "",
      "sources": []
    },
    {
      "action": "",
      "why_fails": "The model will produce incorrect logits for unknown tokens, leading to degraded performance or gibberish generation.",
      "fail_rate": 0.7,
      "condition": "",
      "sources": []
    }
  ],
  "workarounds": [
    {
      "action": "Resize the model's token embeddings to match the tokenizer vocabulary and initialize the new embedding for the unk_token:\n\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nmodel = AutoModelForCausalLM.from_pretrained('gpt2')\ntokenizer = AutoTokenizer.from_pretrained('gpt2')\nmodel.resize_token_embeddings(len(tokenizer))\n# Ensure the unk_token embedding is initialized (e.g., using mean of existing embeddings)\nimport torch\nunk_id = tokenizer.unk_token_id\nif unk_id is not None and model.config.vocab_size <= unk_id:\n    model.lm_head.weight.data[unk_id] = model.lm_head.weight.data.mean(dim=0)",
      "success_rate": 0.75,
      "how": "Resize the model's token embeddings to match the tokenizer vocabulary and initialize the new embedding for the unk_token:\n\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nmodel = AutoModelForCausalLM.from_pretrained('gpt2')\ntokenizer = AutoTokenizer.from_pretrained('gpt2')\nmodel.resize_token_embeddings(len(tokenizer))\n# Ensure the unk_token embedding is initialized (e.g., using mean of existing embeddings)\nimport torch\nunk_id = tokenizer.unk_token_id\nif unk_id is not None and model.config.vocab_size <= unk_id:\n    model.lm_head.weight.data[unk_id] = model.lm_head.weight.data.mean(dim=0)",
      "condition": "",
      "sources": []
    },
    {
      "action": "Use a tokenizer that exactly matches the model's original vocabulary by downloading the correct tokenizer from the model hub (e.g., using `AutoTokenizer.from_pretrained('gpt2')` instead of a custom one).",
      "success_rate": 0.8,
      "how": "Use a tokenizer that exactly matches the model's original vocabulary by downloading the correct tokenizer from the model hub (e.g., using `AutoTokenizer.from_pretrained('gpt2')` instead of a custom one).",
      "condition": "",
      "sources": []
    }
  ],
  "workarounds_zh": [
    "调整模型的令牌嵌入大小以匹配分词器词汇表，并为unk_token初始化新的嵌入：\n\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nmodel = AutoModelForCausalLM.from_pretrained('gpt2')\ntokenizer = AutoTokenizer.from_pretrained('gpt2')\nmodel.resize_token_embeddings(len(tokenizer))\n# 确保unk_token嵌入已初始化（例如，使用现有嵌入的均值）\nimport torch\nunk_id = tokenizer.unk_token_id\nif unk_id is not None and model.config.vocab_size <= unk_id:\n    model.lm_head.weight.data[unk_id] = model.lm_head.weight.data.mean(dim=0)",
    "通过从模型中心下载正确的分词器（例如使用 `AutoTokenizer.from_pretrained('gpt2')` 而不是自定义的）来使用与模型原始词汇表完全匹配的分词器。"
  ],
  "transition_graph": {
    "leads_to": [],
    "preceded_by": [],
    "frequently_confused_with": []
  },
  "official_doc_url": "https://huggingface.co/docs/transformers/troubleshooting#tokenizer-model-mismatch",
  "official_doc_section": null,
  "error_code": null,
  "verification_tier": "ai_generated",
  "confidence": 0.82,
  "fix_success_rate": 0.75,
  "resolvable": "partial",
  "first_seen": "2024-01-20",
  "last_confirmed": "2024-06-01",
  "last_updated": "2024-06-01",
  "evidence_count": 1,
  "tags": [],
  "locale": "en",
  "aliases": []
}