llm api_error ai_generated true

openai.BadRequestError: logprobs is not supported when using response_format parameter

ID: llm/logprobs-request-fails-on-structured-output

Also available as: JSON · Markdown · 中文
95%Fix Rate
88%Confidence
1Evidence
2024-10-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
openai-python>=1.30.0 active — — —
gpt-4o-2024-08-06 active — — —
gpt-4o-mini-2024-07-18 active — — —

Root Cause

OpenAI API does not allow requesting logprobs when structured output (response_format) is enabled, as the constrained decoding mechanism cannot produce token-level probabilities.

generic

中文

OpenAI API 不允许在启用结构化输出（response_format）时请求 logprobs，因为约束解码机制无法生成 token 级别的概率。

Official Documentation

https://platform.openai.com/docs/guides/structured-outputs/supported-models

Workarounds

  1. 95% success Remove logprobs from the request entirely when using response_format. Example: `response = client.chat.completions.create(model="gpt-4o", messages=messages, response_format={ "type": "json_object" })` without `logprobs`. 
    Remove logprobs from the request entirely when using response_format. Example: `response = client.chat.completions.create(model="gpt-4o", messages=messages, response_format={ "type": "json_object" })` without `logprobs`.
  2. 80% success If logprobs are needed, use a separate API call without response_format and parse JSON manually with error handling. 
    If logprobs are needed, use a separate API call without response_format and parse JSON manually with error handling.
  3. 60% success Switch to a model or provider that supports both features simultaneously (e.g., some open-source models with guided decoding). 
    Switch to a model or provider that supports both features simultaneously (e.g., some open-source models with guided decoding).

中文步骤

  1. Remove logprobs from the request entirely when using response_format. Example: `response = client.chat.completions.create(model="gpt-4o", messages=messages, response_format={ "type": "json_object" })` without `logprobs`.
  2. If logprobs are needed, use a separate API call without response_format and parse JSON manually with error handling.
  3. Switch to a model or provider that supports both features simultaneously (e.g., some open-source models with guided decoding).

Dead Ends

Common approaches that don't work:

  1. 70% fail

    The error occurs when logprobs is present at all, even if set to false; the API still validates the field presence.

  2. 90% fail

    Older API versions may not support structured output at all, or have different bugs; the restriction is intentional and consistent across versions.

  3. 50% fail

    This works functionally but defeats the purpose of using structured output for guaranteed JSON.