pytorch training_error ai_generated true

RuntimeError: Loss is NaN or Inf

ID: pytorch/gradient-explosion

Also available as: JSON · Markdown
75%Fix Rate
80%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —

Root Cause

Gradient explosion causing NaN loss.

generic

Workarounds

  1. 90% success Use gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(),1.0)
  2. 88% success Reduce learning rate and use learning rate warmup

Dead Ends

Common approaches that don't work:

  1. Set learning rate to near zero 72% fail

    Training never converges

  2. Ignore NaN batches and continue 82% fail

    Model weights already corrupted