pytorch runtime_error ai_generated true

RuntimeError: Loss is NaN or Inf at step 1000

ID: pytorch/loss-nan-step

Also available as: JSON · Markdown · 中文

85%Fix Rate

85%Confidence

1Evidence

2023-03-15First Seen

Version Compatibility

Version	Status	Introduced	Deprecated	Notes
pytorch>=1.12	active	—	—	—
torchvision>=0.13	active	—	—	—
cuda>=11.6	active	—	—	—

Root Cause

The loss function output becomes NaN or Inf due to numerical instability, often caused by exploding gradients, division by zero, log of zero, or incorrect input scaling.

generic

中文

损失函数输出因数值不稳定变为NaN或Inf，通常由梯度爆炸、除零、对零取对数或输入缩放不正确导致。

Official Documentation

https://pytorch.org/docs/stable/notes/faq.html#my-loss-is-nan

Workarounds

85% success Add gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) before optimizer.step()
```
Add gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) before optimizer.step()
```
80% success Use a smaller learning rate, e.g., optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
```
Use a smaller learning rate, e.g., optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
```
90% success Ensure input data is normalized (e.g., mean=0, std=1) and avoid log(0) by adding a small epsilon like 1e-8
```
Ensure input data is normalized (e.g., mean=0, std=1) and avoid log(0) by adding a small epsilon like 1e-8
```

中文步骤

Add gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) before optimizer.step()

Use a smaller learning rate, e.g., optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)

Ensure input data is normalized (e.g., mean=0, std=1) and avoid log(0) by adding a small epsilon like 1e-8

Dead Ends

Common approaches that don't work:

Increasing learning rate to escape local minima 95% fail
Higher learning rate exacerbates gradient explosion, making NaN more likely.
Removing gradient clipping completely 90% fail
Without clipping, unchecked large gradients cause overflow, leading to NaN.