pytorch assertion_error ai_generated true

RuntimeError: Loss is NaN or Inf at step 500. Consider reducing learning rate, adding gradient clipping, or checking input data for outliers.

ID: pytorch/nan-loss-step

Also available as: JSON · Markdown · 中文
80%Fix Rate
90%Confidence
1Evidence
2023-02-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
torch>=1.8 active — — —
torch>=2.0 active — — —

Root Cause

Training loss becomes NaN or Inf due to exploding gradients, numerical instability (e.g., log of zero), or corrupted input data with extreme values.

generic

中文

训练损失变为 NaN 或 Inf，原因包括梯度爆炸、数值不稳定（例如，对零取对数）或输入数据中存在极端异常值。

Official Documentation

https://pytorch.org/docs/stable/notes/numerical_stability.html

Workarounds

  1. 85% success Add gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) after loss.backward() and before optimizer.step() 
    Add gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) after loss.backward() and before optimizer.step()
  2. 75% success Reduce learning rate by factor of 10: optimizer = torch.optim.Adam(model.parameters(), lr=1e-4) instead of 1e-3 
    Reduce learning rate by factor of 10: optimizer = torch.optim.Adam(model.parameters(), lr=1e-4) instead of 1e-3
  3. 80% success Add a small epsilon to log inputs: loss = F.cross_entropy(logits, targets) inside a try block with torch.clamp(logits, min=-100, max=100) 
    Add a small epsilon to log inputs: loss = F.cross_entropy(logits, targets) inside a try block with torch.clamp(logits, min=-100, max=100)

中文步骤

  1. Add gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) after loss.backward() and before optimizer.step()
  2. Reduce learning rate by factor of 10: optimizer = torch.optim.Adam(model.parameters(), lr=1e-4) instead of 1e-3
  3. Add a small epsilon to log inputs: loss = F.cross_entropy(logits, targets) inside a try block with torch.clamp(logits, min=-100, max=100)

Dead Ends

Common approaches that don't work:

  1. Setting torch.set_default_dtype(torch.float64) to increase precision 65% fail

    Float64 may delay but not prevent NaN if the root cause is exploding gradients or log(0); also slows training significantly.

  2. Restarting training from scratch without changes 95% fail

    The same hyperparameters and data will produce the same NaN at the same step; the issue is deterministic.