llm token_counting ai_generated partial

tiktoken token count doesn't match actual API token usage in response

ID: llm/tiktoken-count-mismatch-api

Also available as: JSON · Markdown
70%Fix Rate
82%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

tiktoken counts raw text tokens but doesn't account for chat message formatting overhead. Each message has ~4 tokens of overhead (role, delimiters). System messages, function definitions, and tool schemas add significant hidden tokens.

generic

Workarounds

  1. 88% success Use tiktoken with chat message overhead formula 
    tokens = sum(4 + len(enc.encode(m['content'])) for m in messages) + 2  # +2 for assistant reply priming
  2. 90% success Use the API's actual token count from response.usage as ground truth for future estimates 
    actual = response.usage.prompt_tokens  # compare with your estimate to calibrate
  3. 85% success Reserve 10-15% token budget as safety margin for overhead 
    safe_budget = int(model_max_tokens * 0.85) - estimated_prompt_tokens

Dead Ends

Common approaches that don't work:

  1. Use tiktoken.encode(prompt_text) to count total prompt tokens 88% fail

    Chat completions add ~4 tokens per message for role/delimiters, plus tokens for function schemas that aren't in your message text. Real count is always higher.

  2. Calculate context budget as model_max_tokens - tiktoken_count 82% fail

    Underestimating prompt tokens means overestimating available completion tokens, leading to context length exceeded errors.