llm context_management ai_generated true

Input silently truncated — total tokens exceed model context window

ID: llm/token-limit-silent-input-truncation

Also available as: JSON · Markdown
88%Fix Rate
90%Confidence
4Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

Some APIs silently truncate input that exceeds the context window.

generic

Workarounds

  1. 92% success Count tokens with tiktoken before sending and chunk within limits 
    import tiktoken; enc = tiktoken.encoding_for_model('gpt-4'); len(enc.encode(text))

    Sources: https://github.com/openai/tiktoken

  2. 85% success Use a sliding window or map-reduce pattern for long documents 
    Split into overlapping chunks, process each, then combine

    Sources: https://python.langchain.com/docs/how_to/summarize_stuff/

Dead Ends

Common approaches that don't work:

  1. Just increase max_tokens parameter 90% fail

    max_tokens controls output length, not input

  2. Split input into multiple sequential API calls 70% fail

    Sequential calls do not share context