PyTorch Autograd and Gradients

PyTorch autograd is the automatic differentiation engine that makes training possible. When a tensor has requires_grad=True, PyTorch records every operation applied to it. Calling .backward() on a result then computes the gradient, the direction in which each input should change to increase that result.

A First Gradient

Take y = x squared plus 3x. Calculus says dy/dx = 2x + 3, so at x = 2 the gradient is 7. Autograd agrees:

import torch

x = torch.tensor(2.0, requires_grad=True)
y = x ** 2 + 3 * x
y.backward()
print(y.item(), x.grad)

10.0 tensor(7.)

The Computation Graph

Every result keeps a grad_fn that points back to the operation that created it. Together these form a graph that backward walks in reverse, applying the chain rule.

a = torch.tensor([1.0, 2.0], requires_grad=True)
b = (a * 3).sum()
print(b.grad_fn)
b.backward()
print(a.grad)

<SumBackward0 object at 0x...>
tensor([3., 3.])

backward can only be called directly on a scalar (a single number), which is why training always reduces the loss to one value with mean or sum.

Gradients Accumulate

Calling backward again adds to .grad instead of replacing it. That is useful for some tricks, but in a normal training loop you must reset gradients each step:

w = torch.tensor(1.0, requires_grad=True)
for _ in range(2):
    (w * 5).backward()
    print(w.grad)
w.grad = None   # or optimizer.zero_grad()

tensor(5.)
tensor(10.)

Gradient Descent by Hand

Here autograd finds the w that makes w times 3 close to 12. This is the same idea every optimizer uses, just written manually:

w = torch.tensor(0.0, requires_grad=True)
lr = 0.05
for step in range(30):
    loss = (w * 3 - 12) ** 2
    loss.backward()
    with torch.no_grad():
        w -= lr * w.grad
    w.grad = None
print(round(w.item(), 3))

4.0

The update is wrapped in torch.no_grad() because we do not want autograd to record the weight update itself as part of the graph.

Turning Gradients Off

• with torch.no_grad(): disables tracking, used for evaluation and manual updates.
• with torch.inference_mode(): an even faster variant for pure inference.
• t.detach() returns a tensor that shares data but is cut off from the graph.
• param.requires_grad_(False) freezes a parameter, as in transfer learning.

Frequently asked questions

What does loss.backward() do in PyTorch?

It computes the gradient of the loss with respect to every tensor that has requires_grad=True and stores it in each tensor's .grad attribute.

Why do I need to call zero_grad in PyTorch?

Gradients accumulate by default. Without zeroing them each step, gradients from previous batches add up and the updates become wrong.

What is the difference between torch.no_grad and torch.inference_mode?

Both stop gradient tracking. inference_mode is slightly faster but tensors created inside it cannot be used later in autograd; no_grad is more flexible.