pytorch runtime_error ai_generated true

运行时错误：在第1000步时损失为NaN或Inf

RuntimeError: Loss is NaN or Inf at step 1000

ID: pytorch/loss-nan-step

其他格式: JSON · Markdown 中文 · English
85%修复率
85%置信度
1证据数
2023-03-15首次发现

版本兼容性

版本状态引入弃用备注
pytorch>=1.12 active — — —
torchvision>=0.13 active — — —
cuda>=11.6 active — — —

根因分析

损失函数输出因数值不稳定变为NaN或Inf，通常由梯度爆炸、除零、对零取对数或输入缩放不正确导致。

English

The loss function output becomes NaN or Inf due to numerical instability, often caused by exploding gradients, division by zero, log of zero, or incorrect input scaling.

generic

官方文档

https://pytorch.org/docs/stable/notes/faq.html#my-loss-is-nan

解决方案

  1. Add gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) before optimizer.step()
  2. Use a smaller learning rate, e.g., optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
  3. Ensure input data is normalized (e.g., mean=0, std=1) and avoid log(0) by adding a small epsilon like 1e-8

无效尝试

常见但无效的做法:

  1. Increasing learning rate to escape local minima 95% 失败

    Higher learning rate exacerbates gradient explosion, making NaN more likely.

  2. Removing gradient clipping completely 90% 失败

    Without clipping, unchecked large gradients cause overflow, leading to NaN.