huggingface config_error ai_generated true

UserWarning: 使用了 `pad_token_id` 但未设置 `padding_side`。分词器将默认使用 'right' 填充，这对于仅解码器模型可能不是最优的。

UserWarning: Using `pad_token_id` but `padding_side` is not set. The tokenizer will use 'right' padding by default, which may not be optimal for decoder-only models.

ID: huggingface/tokenizer-fast-vs-slow-padding

其他格式: JSON · Markdown 中文 · English
95%修复率
82%置信度
1证据数
2023-11-05首次发现

版本兼容性

版本状态引入弃用备注
transformers>=4.34.0 active — — —
tokenizers>=0.14.0 active — — —

根因分析

当分词器的 `padding_side` 未显式设置时，默认为 'right'，这会导致仅解码器模型（如 GPT 或 LLaMA）中的因果掩码不正确，从而降低生成质量。

English

When a tokenizer's `padding_side` is not explicitly set, it defaults to 'right', which causes incorrect causal masking in decoder-only models like GPT or LLaMA, leading to degraded generation quality.

generic

官方文档

https://huggingface.co/docs/transformers/pad_truncation#padding-and-truncation

解决方案

  1. Set `tokenizer.padding_side = 'left'` before tokenization for decoder-only models. This ensures padding tokens are on the left and do not interfere with causal attention.
  2. Use the `pad_token_id` and explicitly pass `attention_mask` to the model to avoid reliance on padding_side defaults.

无效尝试

常见但无效的做法:

  1. 60% 失败

    Setting `padding_side='right'` explicitly silences the warning but does not fix the causal masking issue for decoder-only models.

  2. 40% 失败

    Ignoring the warning and proceeding with training often leads to subtle quality degradation that is hard to detect until evaluation.