Backpropagation: The Intuition
A neural network can have millions of weights, but when it makes a mistake you only get back one number — the loss, a single score that says “how wrong were you overall.” So here is the puzzle that stumped researchers for years: if a million tiny knobs together produced one bad answer, which knobs deserve the blame, and how much? Backpropagation is the beautifully simple answer. It sends the error backwards through the network, layer by layer, handing every single weight its exact share of responsibility — and it does this for all weights at once, in a single sweep. This lesson builds that idea from the ground up, with real numbers you can check by hand.
What you will be able to do after this lesson
- Explain the credit assignment problem: why turning one loss number into millions of per-weight corrections is hard.
- State the chain rule in plain words — if A affects B and B affects C, then A affects C by the product of the two local effects.
- Trace a full forward pass and backward pass through a tiny network and end with real gradient numbers and a weight update.
- Say why backprop is efficient: one forward pass plus one backward pass gives every gradient, instead of wiggling each weight separately.
- Recognise vanishing and exploding gradients, why they wrecked early deep networks, and the intuition behind the fixes.
- Know clearly what backprop is not — it is not how the brain learns, and it is not memorising.
Before we start — the words we need. A weight is one adjustable number inside the network (a knob). The forward pass is running an input through the network to get a prediction. The loss is a single number measuring how far the prediction is from the correct answer — smaller is better. A gradient