tensorflow training_error ai_generated true

Loss scaling resulted in NaN: dynamic loss scale too high

ID: tensorflow/mixed-precision-nan

Also available as: JSON · Markdown
78%Fix Rate
82%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —

Root Cause

Mixed precision training producing NaN from loss scale overflow.

generic

Workarounds

  1. 88% success Use dynamic loss scaling (default in mixed_float16 policy)
  2. 85% success Reduce learning rate or check data for extreme values

Dead Ends

Common approaches that don't work:

  1. Disable loss scaling 78% fail

    Gradient underflow in float16

  2. Set fixed loss scale 72% fail

    May be too high or too low