pytorch training_error ai_generated true

Loss is nan at step N

ID: pytorch/loss-nan

Also available as: JSON · Markdown
75%Fix Rate
80%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —

Root Cause

Training loss became NaN. LR too high, bad data, or numerical instability.

generic

Workarounds

  1. 88% success Check data for NaN/Inf values in dataset
  2. 90% success Use anomaly detection: torch.autograd.set_detect_anomaly(True)

Dead Ends

Common approaches that don't work:

  1. Replace NaN with 0 and continue training 85% fail

    Model already diverged, garbage weights

  2. Remove batch normalization 78% fail

    BN is usually stabilizing