pytorch runtime_error ai_generated true

RuntimeError: GradScaler unscale_() encountered None gradient. Ensure loss.backward() and optimizer.step() are called correctly.

ID: pytorch/grad-scale-unscale-error

Also available as: JSON · Markdown · 中文
88%Fix Rate
85%Confidence
1Evidence
2023-07-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
torch>=1.10 active — — —
torch<=2.5.1 active — — —

Root Cause

GradScaler's unscale_() method was called on a parameter whose gradient is None, typically because the parameter was not used in the forward pass or was excluded from gradient computation.

generic

中文

GradScaler 的 unscale_() 方法被调用时某个参数的梯度为 None，通常是因为该参数在前向传播中未被使用或被排除在梯度计算之外。

Official Documentation

https://pytorch.org/docs/stable/amp.html#torch.cuda.amp.GradScaler.unscale_

Workarounds

  1. 90% success Exclude parameters that are not used in the forward pass from the optimizer. For example, if a submodule is conditionally used, filter its parameters: optimizer = torch.optim.SGD([p for p in model.parameters() if p.requires_grad and p.grad is not None], lr=0.01) 
    Exclude parameters that are not used in the forward pass from the optimizer. For example, if a submodule is conditionally used, filter its parameters: optimizer = torch.optim.SGD([p for p in model.parameters() if p.requires_grad and p.grad is not None], lr=0.01)
  2. 85% success Use torch.no_grad() context or set requires_grad=False on parameters that are not part of the computation graph to avoid gradient computation. 
    Use torch.no_grad() context or set requires_grad=False on parameters that are not part of the computation graph to avoid gradient computation.

中文步骤

  1. Exclude parameters that are not used in the forward pass from the optimizer. For example, if a submodule is conditionally used, filter its parameters: optimizer = torch.optim.SGD([p for p in model.parameters() if p.requires_grad and p.grad is not None], lr=0.01)
  2. Use torch.no_grad() context or set requires_grad=False on parameters that are not part of the computation graph to avoid gradient computation.

Dead Ends

Common approaches that don't work:

  1. Setting all parameters to require grad=True 60% fail

    If a parameter is truly unused in the forward pass, setting requires_grad=True will still result in a None gradient after backward, because no gradient flows through it.

  2. Calling unscale_() multiple times to clear None gradients 80% fail

    The GradScaler's internal state does not allow multiple unscale_() calls per optimizer step; it raises a different error about double unscale.