llm reproducibility ai_generated partial

LLM produces different outputs for identical prompts even with temperature=0

ID: llm/temperature-zero-not-deterministic

Also available as: JSON · Markdown
60%Fix Rate
85%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

temperature=0 does NOT guarantee deterministic output. GPU floating point operations are non-deterministic, model weights may be updated, and batching/routing can affect results. OpenAI introduced 'seed' parameter but it's still best-effort.

generic

Workarounds

  1. 85% success Use seed parameter AND check system_fingerprint for change detection 
    response = client.chat.completions.create(seed=42, ...); print(response.system_fingerprint)  # track for changes
  2. 90% success Implement output caching with (prompt_hash, model, system_fingerprint) as key 
    Cache responses keyed on hash(messages + model + system_fingerprint). Re-generate only on fingerprint change.
  3. 92% success Design systems to tolerate non-deterministic LLM output instead of depending on reproducibility 
    Use structured output validation, retry with parsing, and fuzzy matching instead of exact output comparison

Dead Ends

Common approaches that don't work:

  1. Set temperature=0 and expect identical outputs every time 88% fail

    GPU parallel float operations have non-deterministic rounding. Different GPUs, batch sizes, or model shards produce different softmax distributions.

  2. Use seed parameter and assume perfect reproducibility 78% fail

    OpenAI's seed parameter is 'best-effort'. system_fingerprint changes when infrastructure updates happen, breaking reproducibility.