Backpropagation

Backpropagation is the algorithm that computes the gradient of a neural network's loss with respect to every weight in a single backward sweep. It applies the chain rule layer by layer, reusing each layer's error signal to compute the one before it — so all the gradients together cost about as much as one extra forward pass.

The prediction comes out wrong, and now every weight in the building owes a share of the blame. The naive audit re-runs the whole network once per weight to see how the loss moves — ruinously expensive. Backprop is the accountant who walks the bill backward through the org chart instead: the output's error is itemized to the last layer, each layer deducts its local rate and forwards the remainder, and one trip through the building leaves every weight holding an exact invoice.

How Backpropagation Works

For a single weight w, the loss depends on w only through a chain — w moves a pre-activation z, which moves an activation a, which eventually moves the loss L. The chain rule prices that chain as a product of local rates: ∂L/∂w = ∂L/∂a · ∂a/∂z · ∂z/∂w.

The accounting trick is grouping. Define each neuron's error signal δ = ∂L/∂z — how sensitive the loss is to that neuron's pre-activation. Two facts then carry the whole algorithm. First, the gradient of any weight feeding a neuron is just δ × (the input on that wire) — once a neuron's blame is known, its incoming invoices are one multiplication each. Second, an earlier layer's δ comes from the later layer's by sending it back through the connecting weights and multiplying by the activation's local derivative: δ_prev = (Wᵀ·δ_next) ⊙ σ′(z), where ⊙ is element-wise. The transpose routes each error back along the wire it arrived on. For sigmoid, if s = σ(z) is the cached activation, the derivative with respect to z is σ′(z) = s·(1 − s), computable from that activation — which is why the forward pass saves its receipts.

So the backward pass is a short loop: compute δ at the output, invoice the output weights, push δ back one layer, invoice those weights, repeat. Then every weight takes one gradient-descent step, w ← w − η·∂L/∂w — the exact rule from the Gradient Descent lesson, now fed by gradients backprop produced.

Backpropagation Step by Step

The default run is one training step on a toy 2 → 2 → 1 sigmoid net with squared-error loss L = ½(ŷ − y)² — small enough that you can check every number by hand, and the canvas shows them all. Input x = [0.50, 0.90], target y = 1, learning rate η = 0.5. The starting parameters: W₁ = [0.15, 0.20; 0.25, 0.30] (one row per hidden neuron), b₁ = [0.35, 0.35], W₂ = [0.40, 0.45], b₂ = 0.60. The biases are held fixed to keep the picture to six weights; in a real step they move by the same rule, since ∂L/∂b = δ.

1. Forward · hidden. a = σ(W₁·x + b₁) = [0.647, 0.678].
2. Forward · output. ŷ = σ(W₂·a + b₂) = 0.762.
3. Loss. L = ½(0.762 − 1)² = 0.028. The prediction is short of the target.
4. δ at the output. δ_o = (ŷ − y)·ŷ(1−ŷ) = (−0.238)·(0.762·0.238) = −0.043. Negative: raising ŷ would lower the loss.
5. Output invoices. ∂L/∂W₂ = δ_o·a = [−0.028, −0.029] — one gradient per hidden→output wire.
6. δ back to the hidden layer. δ_h = (W₂·δ_o) ⊙ a(1−a) = [−0.004, −0.004]. The output's blame just became the hidden layer's blame — this is the chain rule's one move.
7. Input invoices. ∂L/∂W₁ = δ_h ⊗ x = [−0.0020, −0.0035, −0.0021, −0.0038] (the 2×2 matrix, row-major) — tiny, because each carries two sigmoid derivatives.
8. Update. w ← w − η·∂L/∂w with η = 0.5. All six gradients are negative, so all six weights nudge up, pushing ŷ toward the target.
9. Verify. Re-running the forward pass on the updated weights gives ŷ = 0.766 (up from 0.762) and L = 0.027 (down from 0.028). One step, one small dent; training repeats this loop thousands of times.

On the canvas, amber sweeps the neurons left to right on the forward beats, then orange sweeps right to left as each δ appears, and the header flips from forward → to ← backward and back for the closing re-run. That direction reversal is the lesson. The second input example (x = [0.1, 0.8]) runs the identical choreography with every number recomputed — same bookkeeping, different bill.

The learning rate is a knob too: η ∈ {0.1, 0.5, 0.9} in the Inputs panel. The gradients never change with it — backprop's bill is the same — but the step taken along them scales, and so does the verify beat's loss drop (0.0002 at η = 0.1, 0.0008 at 0.5, 0.0015 at 0.9 on the default input). This is the single-step view of the dial whose whole convergence story the Gradient Descent lesson tells: that lesson shows where η breaks; this one shows what one step of it buys.

Why the Chain Rule Runs Backward

Run the audit forward and you are asking, per weight, "how does this one number move the loss?" — a separate full pass for each of the network's weights, O(weights) passes, O(weights²) work. Run it backward and you ask each neuron once, "how does the loss move with you?" — and every weight's gradient falls out of its neuron's answer for the price of one multiplication.

That asymmetry exists because the network has many parameters and one loss. Derivatives of one output with respect to many inputs are cheapest computed output-to-input — the shared factors (δ) get reused instead of recomputed. The bill: forward pass O(weights), backward pass O(weights), additional cached-forward memory O(activations), excluding parameters and stored parameter gradients. All the gradients for roughly one extra forward pass, at any scale — this is the fact that makes training billion-weight networks feasible rather than fantastical. The 784 → 25 → 25 → 10 model in Forward Propagation was trained by exactly this δ-bookkeeping over its 20,475 weights, averaged over mini-batches (its update step used Adam, an adaptive variant of gradient descent — the gradients themselves come from the backprop you just watched).

Edge Cases

• Already correct. If ŷ = y then δ_o = 0, every invoice is zero, and no weight moves — for this example, you are at a loss minimum.
• Saturated neurons. Where a sigmoid sits near 0 or 1, σ′ = a(1−a) is nearly zero and the δ passing through is throttled — the vanishing gradient.
• Depth compounds it. Each layer multiplies in another σ′ ≤ 0.25, so in deep sigmoid chains δ shrinks geometrically (or, with large weights, explodes). This 2→2→1 net is too shallow to show it — the sigmoids here are a visual-clarity choice, not a recommendation; real networks reach for ReLU largely for this reason.

Common Mistakes

• Dropping the σ′ term. A neuron's δ is the incoming error times its local derivative. Skip the deduction and every invoice upstream is wrong.
• Updating weights mid-backward. The backward pass must use the weights from the forward pass. Change them early and later layers compute δ against a network that no longer exists.
• Flipping a sign. δ_o = (ŷ − y)·σ′ and the update subtracts η·∂L/∂w. One sign error converts descent into ascent.
• Recomputing instead of caching. The backward pass reuses every forward activation. Without the cached values you must recompute what the backward pass needs. Checkpointing deliberately trades recomputation for memory; it does not necessarily revert to one full pass per weight.