llm config_error ai_generated true

ValueError: You have to provide either 'max_length' or 'padding' and 'truncation' to use padding side 'left' with batch size > 1

ID: llm/huggingface-tokenizer-padding-side-mismatch

Also available as: JSON · Markdown · 中文

85%Fix Rate

88%Confidence

1Evidence

2023-08-15First Seen

Version Compatibility

Version	Status	Introduced	Deprecated	Notes
transformers 4.30.0	active	—	—	—
transformers 4.31.0	active	—	—	—
transformers 4.35.0	active	—	—	—
Python 3.9	active	—	—	—
Python 3.10	active	—	—	—
Python 3.11	active	—	—	—

Root Cause

Hugging Face tokenizer requires explicit padding and truncation configuration when using left padding (common for decoder-only LLMs) in batched inference, otherwise it cannot determine the sequence length alignment.

generic

中文

Hugging Face分词器在使用左侧填充（解码器专用LLM常见设置）进行批量推理时，需要显式配置填充和截断参数，否则无法确定序列长度对齐方式。

Official Documentation

https://huggingface.co/docs/transformers/main/en/main_classes/tokenizer#transformers.PreTrainedTokenizer.__call__.padding

Workarounds

95% success Configure tokenizer with explicit padding=True, truncation=True, and max_length=model_max_length before tokenizing batched inputs: tokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf') tokenizer.pad_token = tokenizer.eos_token tokenizer.padding_side = 'left' encoded = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors='pt')
```
Configure tokenizer with explicit padding=True, truncation=True, and max_length=model_max_length before tokenizing batched inputs:

tokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf')
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = 'left'
encoded = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors='pt')
```
90% success Set tokenizer.pad_token_id explicitly and use DataCollatorWithPadding from transformers for dynamic batching: from transformers import DataCollatorWithPadding data_collator = DataCollatorWithPadding(tokenizer, padding='max_length', max_length=512)
```
Set tokenizer.pad_token_id explicitly and use DataCollatorWithPadding from transformers for dynamic batching:

from transformers import DataCollatorWithPadding
data_collator = DataCollatorWithPadding(tokenizer, padding='max_length', max_length=512)
```

中文步骤

在分词前显式配置padding=True, truncation=True和max_length=model_max_length：

tokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf')
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = 'left'
encoded = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors='pt')

显式设置tokenizer.pad_token_id，并使用transformers的DataCollatorWithPadding进行动态批处理：

from transformers import DataCollatorWithPadding
data_collator = DataCollatorWithPadding(tokenizer, padding='max_length', max_length=512)

Dead Ends

Common approaches that don't work:

70% fail
Decoder-only models (e.g., LLaMA, GPT) expect left padding for causal attention masking; right padding causes misalignment and incorrect generation.
50% fail
Bypasses the error but prevents batching, leading to poor throughput and increased latency in production.