llm api_error ai_generated true

openai.BadRequestError: This model's maximum context length is 128000 tokens

ID: llm/context-length-exceeded

Also available as: JSON · Markdown
88%Fix Rate
90%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

Input + output tokens exceed the model's context window. Long conversations, large documents, or system prompts.

generic

Workarounds

  1. 88% success Implement conversation summarization/windowing 
    Keep last N messages + summary of older messages; summarize when approaching limit
  2. 92% success Use tiktoken to count tokens before sending 
    import tiktoken; enc = tiktoken.encoding_for_model('gpt-4'); len(enc.encode(text))
  3. 85% success Chunk large documents and process separately 
    Split into overlapping chunks of ~4000 tokens; process each; combine results

Dead Ends

Common approaches that don't work:

  1. Truncate the prompt from the beginning 72% fail

    Cutting the start loses system prompts and instructions. Truncate middle conversation, keep start and end.

  2. Switch to a model with larger context window 60% fail

    Larger context = higher cost and latency. Fix the prompt design first.