api timeout ai_generated true

openai.APITimeoutError: Request timed out. or unexpected truncated response

ID: api/openai-timeout-vs-max-tokens

Also available as: JSON · Markdown
85%Fix Rate
90%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

OpenAI API timeout is client-side (default 600s). A long response can timeout before completion. Separately, max_tokens silently truncates without error - finish_reason becomes 'length' not 'stop'.

generic

Workarounds

  1. 95% success Always check response.choices[0].finish_reason == 'stop' 
    if response.choices[0].finish_reason == 'length': # response was truncated, need continuation
  2. 88% success Implement streaming for long responses to avoid timeout 
    stream = client.chat.completions.create(stream=True, ...); for chunk in stream: ...

Dead Ends

Common approaches that don't work:

  1. Increase client timeout to avoid truncation 82% fail

    Timeout and truncation are different issues. Timeout = client gave up waiting. Truncation = max_tokens limit hit. You may have both problems simultaneously.

  2. Assume response is complete if no error was raised 88% fail

    finish_reason='length' means output was truncated but no exception is raised. You must check finish_reason explicitly.