llm streaming ai_generated true

json.decoder.JSONDecodeError when parsing OpenAI function call arguments in streaming mode

ID: llm/openai-function-calling-json-parse-stream

Also available as: JSON · Markdown
88%Fix Rate
90%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

In streaming mode, function/tool call arguments arrive as partial JSON chunks. Each delta.tool_calls[].function.arguments is a string fragment, not valid JSON. You must concatenate all chunks before parsing.

generic

Workarounds

  1. 95% success Accumulate argument string chunks, parse only after finish_reason='tool_calls' 
    args_str = ''; for chunk in stream: delta = chunk.choices[0].delta; if delta.tool_calls: args_str += delta.tool_calls[0].function.arguments; # parse after stream ends
  2. 85% success Use non-streaming mode for function calling if latency permits 
    response = client.chat.completions.create(stream=False, tools=tools)  # complete JSON in one response
  3. 82% success Use structured output mode (response_format) instead of function calling for complex schemas 
    response_format={'type': 'json_schema', 'json_schema': {...}}  # enforced valid JSON output

Dead Ends

Common approaches that don't work:

  1. Parse delta.tool_calls[0].function.arguments as JSON per chunk 95% fail

    Each streaming chunk contains a partial JSON string fragment like '{"na' or 'me": "'. Parsing individual chunks always fails.

  2. Wait for finish_reason='function_call' then parse the last chunk 85% fail

    The last chunk doesn't contain the full arguments. You need ALL chunks concatenated. finish_reason='tool_calls' signals completion.