# Warning: Token count mismatch — prompt tokens (4500) + completion tokens (1200) = 5700, but API reports total_tokens=5800

- **ID:** `llm/token-count-mismatch-streaming`
- **Domain:** llm
- **Category:** data_error
- **Verification:** ai_generated
- **Fix Rate:** 75%

## Root Cause

The tiktoken tokenizer counts tokens differently than the OpenAI API's internal tokenizer, especially for multi-byte characters, special tokens, or streaming overhead tokens like role markers and stop sequences.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| tiktoken==0.7.0 | active | — | — |
| openai==1.30.0 | active | — | — |
| python==3.11 | active | — | — |

## Workarounds

1. **Use the API's reported `total_tokens` as the source of truth for billing and truncation, and only use tiktoken for approximate pre-flight checks: `response = client.chat.completions.create(model='gpt-4', messages=messages, stream=True); total_tokens = response.usage.total_tokens if hasattr(response, 'usage') else None`** (95% success)
   ```
   Use the API's reported `total_tokens` as the source of truth for billing and truncation, and only use tiktoken for approximate pre-flight checks: `response = client.chat.completions.create(model='gpt-4', messages=messages, stream=True); total_tokens = response.usage.total_tokens if hasattr(response, 'usage') else None`
   ```
2. **Implement a buffer of 10% when using tiktoken to estimate context window usage: `max_tokens_allowed = int(model_max_tokens * 0.9) - tiktoken_count(prompt)`** (80% success)
   ```
   Implement a buffer of 10% when using tiktoken to estimate context window usage: `max_tokens_allowed = int(model_max_tokens * 0.9) - tiktoken_count(prompt)`
   ```

## Dead Ends

- **** — The discrepancy is inherent because the API uses a different tokenization pipeline that includes service-side tokens (e.g., role markers, stop sequences) not counted by tiktoken. (90% fail)
- **** — The mismatch varies with prompt structure and completion length; a fixed fudge factor is unreliable and may cause budget overruns or early truncation. (70% fail)
- **** — While this avoids the mismatch warning, it doesn't fix the root cause; the token count reported by the API still uses its internal tokenizer, and the mismatch persists in non-streaming mode too. (50% fail)