huggingface config_error ai_generated true

ValueError: The sum of `max_new_tokens` (1000) and input length (512) exceeds the model's maximum position embeddings (2048). Reduce `max_new_tokens` or truncate input.

ID: huggingface/text-generation-pipeline-max-new-tokens-exceeded

Also available as: JSON · Markdown · 中文

93%Fix Rate

89%Confidence

1Evidence

2023-10-01First Seen

Version Compatibility

Version	Status	Introduced	Deprecated	Notes
transformers>=4.38.0	active	—	—	—
torch>=2.0.0	active	—	—	—
Python>=3.8	active	—	—	—

Root Cause

The total sequence length (input tokens + new tokens to generate) exceeds the model's maximum position embedding size, which is a hard limit for positional encodings.

generic

中文

总序列长度（输入标记数 + 要生成的新标记数）超过了模型的最大位置嵌入大小，这是位置编码的硬性限制。

Official Documentation

https://huggingface.co/docs/transformers/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens

Workarounds

93% success Reduce max_new_tokens so that input_length + max_new_tokens <= model_max_length: from transformers import pipeline generator = pipeline('text-generation', model='gpt2') model_max_length = generator.model.config.max_position_embeddings # 1024 for gpt2 input_length = len(generator.tokenizer(prompt)['input_ids']) max_new_tokens = min(500, model_max_length - input_length - 10) # safety margin result = generator(prompt, max_new_tokens=max_new_tokens)
```
Reduce max_new_tokens so that input_length + max_new_tokens <= model_max_length:
from transformers import pipeline
generator = pipeline('text-generation', model='gpt2')
model_max_length = generator.model.config.max_position_embeddings  # 1024 for gpt2
input_length = len(generator.tokenizer(prompt)['input_ids'])
max_new_tokens = min(500, model_max_length - input_length - 10)  # safety margin
result = generator(prompt, max_new_tokens=max_new_tokens)
```
85% success Truncate the input to allow more generation tokens: truncated_prompt = generator.tokenizer.decode(generator.tokenizer(prompt, truncation=True, max_length=512)['input_ids']) result = generator(truncated_prompt, max_new_tokens=1000)
```
Truncate the input to allow more generation tokens:
truncated_prompt = generator.tokenizer.decode(generator.tokenizer(prompt, truncation=True, max_length=512)['input_ids'])
result = generator(truncated_prompt, max_new_tokens=1000)
```

中文步骤

Reduce max_new_tokens so that input_length + max_new_tokens <= model_max_length:
from transformers import pipeline
generator = pipeline('text-generation', model='gpt2')
model_max_length = generator.model.config.max_position_embeddings  # 1024 for gpt2
input_length = len(generator.tokenizer(prompt)['input_ids'])
max_new_tokens = min(500, model_max_length - input_length - 10)  # safety margin
result = generator(prompt, max_new_tokens=max_new_tokens)

Truncate the input to allow more generation tokens:
truncated_prompt = generator.tokenizer.decode(generator.tokenizer(prompt, truncation=True, max_length=512)['input_ids'])
result = generator(truncated_prompt, max_new_tokens=1000)

Dead Ends

Common approaches that don't work:

100% fail
Setting max_new_tokens to a very large value (e.g., 10000) expecting the model to handle it, which causes the same error.
50% fail
Using max_length instead of max_new_tokens may truncate input silently and produce incomplete outputs.