Who's to blame when a network gets it wrong?

Backpropagation, step by step: how one backward pass hands every weight its share of the blame.

When a neural network makes a wrong prediction, training has to fix it. But a network has many weights, and each one played some part in the mistake. Before we can change anything, we need to know how much each weight is to blame.

That's what backpropagation does. After the network makes its prediction (the forward pass), it sends the error back through the network (the backward pass). On the way, every weight picks up a number, its gradient, that says how much the loss would change if that weight moved a little. That number is the weight's share of the blame.

The examples below are the ones from the CS878 Deep Learning lecture slides, with the same numbers.

A tiny graph, forward

Start with something small enough to see all at once:

f(x, y, z) = (x + y) · z

Draw it as a graph with two steps. An add node computes d = x + y, and a multiply node computes f = d · z.

Put in x = 1, y = −3 and z = 4. The add node gives d = 1 + (−3) = −2. The multiply node gives f = −2 · 4 = −8. That's the forward pass: values flow from the inputs to the output.

The forward pass. The add node turns 1 and −3 into −2, and the multiply node turns −2 and 4 into −8.

The same graph, backward

Now go the other way. We want to know how f changes when each input changes: ∂f/∂x, ∂f/∂y and ∂f/∂z.

Start at the output with a gradient of 1, because f changes exactly as much as f does.

A multiply node swaps: each input gets the upstream gradient times the other input's value. For f = d · z, the local gradients are ∂f/∂z = d and ∂f/∂d = z. So z gets 1 × (−2) = −2 (the value of d) and the sum gets 1 × 4 = 4 (the value of z).

An add node copies: it passes the gradient straight through to every input. For d = x + y, both local gradients are 1, so x and y each get 4 × 1 = 4.

Check one of them with the chain rule: ∂f/∂x = ∂f/∂d · ∂d/∂x = 4 · 1 = 4.

The backward pass. Starting from 1 at the output, the multiply node sends −2 to z and 4 to the sum; the add node passes 4 to both x and y.

Every node does the same job

Look at what each node did. It took the gradient arriving from above, the upstream gradient, and multiplied it by its own local gradient. For any node, with in one of its inputs and out its output:

∂L/∂in = ∂L/∂out · ∂out/∂in (upstream times local)

That's the whole algorithm. Each node only needs to know its own small derivative. It never has to see the rest of the network. The chain rule from the last episode, applied node by node, from the output back to the inputs.

A real network

Now a tiny network: one input x = 1, a first weight w₁ = 1/3, a ReLU, a second weight w₂ = 2, and a squared-error loss against a target of y = 0.

Forward:

• z = x · w₁ = 1/3
• s = max(0, z) = 1/3 (the ReLU's output; σ usually means the sigmoid)
• ŷ = s · w₂ = 2/3
• L = (ŷ − y)² = 4/9

The network predicts 2/3 when the answer is 0. That's the mistake, and the loss of 4/9 measures it.

The network after its forward pass. It predicts 2/3 instead of 0, a loss of 4/9.

Handing out the blame

Backward, from the loss, one node at a time:

• At the output: ∂L/∂ŷ = 2(ŷ − y) = 4/3.
• The multiply node ŷ = s · w₂ swaps. The second weight gets 4/3 × s = 4/3 × 1/3 = 4/9, and the other branch gets 4/3 × w₂ = 4/3 × 2 = 8/3.
• The ReLU's input was positive (z = 1/3), so the gate is open and its local gradient is 1: z gets 8/3 × 1 = 8/3. Had z been negative, the gate would be shut and pass back nothing.
• The multiply node z = x · w₁ swaps too. The first weight gets 8/3 × x = 8/3 × 1 = 8/3.

Multiply the factors along the path and you get the slides' chain rule, ∂L/∂w₁ = ∂L/∂ŷ · ∂ŷ/∂s · ∂s/∂z · ∂z/∂w₁ = 4/3 · 2 · 1 · 1.

So the first weight gets six times as much of the blame as the second: 8/3 = 6 × 4/9. One backward pass found both numbers.

The backward pass through the network. The first weight's gradient is 8/3, the second weight's is 4/9.

One step downhill

Gradient descent then moves every weight at once, each against its own gradient, scaled by a learning rate α:

w ← w − α · ∂L/∂w

With α = 0.05, the first weight moves six times as far as the second: w₁ goes from 1/3 to 1/3 − 0.05 · 8/3 = 0.20, and w₂ from 2 to 2 − 0.05 · 4/9 ≈ 1.98. Run the forward pass again with the new weights: z = 0.20, s = 0.20, ŷ ≈ 0.40, and the loss drops from 0.44 to about 0.16.

One gradient step with α = 0.05. The weights become 0.20 and 1.98, and the loss falls from 0.44 to 0.16.

Why it matters

A real network has millions of weights. Nudging them one at a time would take millions of forward passes per step. Backpropagation gets all of them in one backward pass that costs about as much as the forward pass. It does this by reusing work: the 4/3 at the output was computed once and shared by both weights.

It became the standard way to train neural networks after Rumelhart, Hinton and Williams showed in 1986 that it lets hidden layers learn useful features.

Try it in PyTorch

PyTorch runs backpropagation for you. Build the loss, call backward(), and every weight's gradient appears in .grad:

import torch
w = torch.tensor([1/3, 2.],
  requires_grad=True)
L = (w[1] * w[0].relu())**2
L.backward()
print(w.grad)

It prints tensor([2.6667, 0.4444]): 8/3 and 4/9, the same numbers we found by hand. (Here x = 1 and y = 0, so they drop out of the formula.) Change 1/3 to −1/3 and both gradients become zero: the ReLU outputs 0, so no blame gets through.

Key takeaways

• The forward pass computes values; the backward pass sends gradients from the loss back to every weight.
• Each node multiplies the gradient from above by its own local gradient.
• A multiply node swaps: each input gets the gradient times the other's value. An add node copies: it passes the gradient straight through. An open ReLU multiplies by 1.
• In the slides' network, one backward pass gives the first weight 8/3 and the second 4/9, and one step with α = 0.05 takes the loss from 0.44 to 0.16.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.