llm training_error ai_generated true

Training loss diverging: NaN or increasing

ID: llm/fine-tuning-loss-divergence

Also available as: JSON · Markdown
75%Fix Rate
80%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

Fine-tuning loss not converging or diverging.

generic

Workarounds

  1. 88% success Reduce learning rate by 10x and use warmup steps
  2. 90% success Check training data for quality issues: duplicates, noise, format errors

Dead Ends

Common approaches that don't work:

  1. Increase learning rate to converge faster 85% fail

    Worsens divergence

  2. Train for more epochs without investigation 78% fail

    Wastes compute, same divergence