pytorch assertion_error ai_generated true

RuntimeError: 在第 500 步损失为 NaN 或 Inf。考虑降低学习率、添加梯度裁剪或检查输入数据中的异常值。

RuntimeError: Loss is NaN or Inf at step 500. Consider reducing learning rate, adding gradient clipping, or checking input data for outliers.

ID: pytorch/nan-loss-step

其他格式: JSON · Markdown 中文 · English
80%修复率
90%置信度
1证据数
2023-02-10首次发现

版本兼容性

版本状态引入弃用备注
torch>=1.8 active — — —
torch>=2.0 active — — —

根因分析

训练损失变为 NaN 或 Inf，原因包括梯度爆炸、数值不稳定（例如，对零取对数）或输入数据中存在极端异常值。

English

Training loss becomes NaN or Inf due to exploding gradients, numerical instability (e.g., log of zero), or corrupted input data with extreme values.

generic

官方文档

https://pytorch.org/docs/stable/notes/numerical_stability.html

解决方案

  1. Add gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) after loss.backward() and before optimizer.step()
  2. Reduce learning rate by factor of 10: optimizer = torch.optim.Adam(model.parameters(), lr=1e-4) instead of 1e-3
  3. Add a small epsilon to log inputs: loss = F.cross_entropy(logits, targets) inside a try block with torch.clamp(logits, min=-100, max=100)

无效尝试

常见但无效的做法:

  1. Setting torch.set_default_dtype(torch.float64) to increase precision 65% 失败

    Float64 may delay but not prevent NaN if the root cause is exploding gradients or log(0); also slows training significantly.

  2. Restarting training from scratch without changes 95% 失败

    The same hyperparameters and data will produce the same NaN at the same step; the issue is deterministic.