pytorch autograd_error ai_generated true

RuntimeError: Trying to backward through the graph a second time, but the saved intermediate values have already been freed. Specify retain_graph=True when calling backward the first time.

ID: pytorch/strided-gradient-issue

Also available as: JSON · Markdown · 中文
95%Fix Rate
88%Confidence
1Evidence
2023-06-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
torch>=1.0.0 active — — —

Root Cause

Calling backward() twice on the same computation graph without setting retain_graph=True, causing the intermediate buffers to be freed after the first backward pass.

generic

中文

在同一个计算图上两次调用backward()，而没有设置retain_graph=True，导致第一次反向传播后中间缓冲区被释放。

Official Documentation

https://pytorch.org/docs/stable/autograd.html#torch.autograd.backward

Workarounds

  1. 95% success loss.backward(retain_graph=True) # Then later loss.backward() # This now works 
    loss.backward(retain_graph=True)
# Then later
loss.backward()  # This now works
  2. 85% success loss1.backward(retain_graph=True) loss2.backward() optimizer.step() # Or compute both losses together 
    loss1.backward(retain_graph=True)
loss2.backward()
optimizer.step()
# Or compute both losses together

中文步骤

  1. loss.backward(retain_graph=True)
# Then later
loss.backward()  # This now works
  2. loss1.backward(retain_graph=True)
loss2.backward()
optimizer.step()
# Or compute both losses together

Dead Ends

Common approaches that don't work:

  1. Setting requires_grad=False on all tensors 90% fail

    This disables gradient computation entirely, which is not the intended fix. The error is about graph retention, not about needing gradients.

  2. Calling zero_grad() between backward passes 70% fail

    zero_grad() only resets gradients to zero, it does not retain the computation graph. The intermediate values are still freed after the first backward.

  3. Recreating the model and optimizer after each backward 80% fail

    This is extremely inefficient and does not address the root cause. The graph is still freed after the first backward unless retain_graph is set.