Forward Propagation

Forward propagation is the computation a neural network runs to turn an input into a prediction: each layer multiplies the incoming values by its weights, adds a bias, and applies a nonlinearity, passing the result left to right until the output layer reports a score for every possible answer. It is inference — the network's only job once training ends.

Think of each layer as a panel of judges. Every judge reads all the evidence coming in, but each has a fixed scoring sheet — the weights — that says how much each piece of evidence counts, and in which direction. The judges never confer; they just total their sheets and pass their verdicts to the next panel. The network here recognizes handwritten digits: 784 pixel brightnesses go in, ten digit scores come out, and the largest score is the guess.

How Forward Propagation Works

The architecture is 784 → 25 → 25 → 10: the flattened 28×28 image, two hidden panels of 25 judges each, and a final panel of 10 — one per digit.

Every judge performs the same two-part computation, and the animation insists on showing the parts separately because conflating them is where intuition goes wrong. First the weighted sum, the pre-activation z = w·a + b: multiply each input by its weight, add everything up, add a bias. Nothing stops z from coming out negative — a judge can be actively repelled by what it sees. Then the ReLU nonlinearity, max(0, z): negative verdicts are clamped to zero and the judge goes silent; positive ones pass through unchanged. A weight is the strength of one connection — positive weights (drawn green) excite the next judge, negative weights (drawn magenta) inhibit it.

The output panel skips ReLU. Its ten raw scores — the logits — go through softmax, pᵢ = exp(zᵢ) / Σⱼ exp(zⱼ), which exponentiates every score and normalizes so the results sum to 1. The exponential is aggressive: a modest lead in logits becomes a landslide in probability. The prediction is the argmax — whichever digit holds the most probability.

One number worth holding onto: the whole pass costs about 784·25 + 25·25 + 25·10 = 20,475 multiply-adds, one per connection, in a single layer-by-layer sweep. This counts connections, excluding bias additions and activation/softmax work; the implementation still uses loops or matrix operations. And it is the weights' values, not their count, that make this work — random values in the same 20,475 connections would not provide the learned digit recognition; chance-level accuracy is a baseline, not a guarantee for every draw. The count is capacity; training fills it. (The 19,600 connections into the first hidden layer are too dense to draw, so the canvas shows a representative subset between the later layers — a drawing constraint, not a property of the network.)

Forward Propagation Step by Step

The default input is a real MNIST test image of a handwritten 7, run through a real trained network — every number on screen is a genuine forward pass.

1. Input. The 28×28 image flattens into 784 pixel values. Ink is high, background near zero.
2. Layer 1 — weighted sum. Each of the 25 judges computes z₁ = x·W₁ + b₁ over all 784 pixels. Ten of the 25 come out negative (drawn blue); the rest positive (dark).
3. Layer 1 — ReLU. max(0, z₁) silences the ten negative judges — they fade to empty — while the positive ones keep their values as the activations a₁.
4. Layer 2 — weighted sum. The drawn connections light up and each layer-2 judge shows z₂ = a₁·W₂ + b₂. Nine of 25 are negative this time.
5. Layer 2 — ReLU. Negatives zeroed again, giving a₂.
6. Output — logits. Ten raw scores z₃ = a₂·W₃ + b₃ appear. The judge for 7 posts about 19.0; the runner-up, digit 2, manages roughly 1.2; most of the rest are negative.
7. Predict. Softmax exponentiates that 17.9-point gap until almost all of the probability mass lands on one digit. The network predicts 7 at ~100%.

The visual grammar to read as it plays: blue means a negative pre-activation, and the ReLU beats are where blue cells fade to empty — the two-step rhythm of "raw sum, then nonlinearity" repeats twice in the hidden layers and once more as "logits, then softmax" at the output. Same shape, three times.

Why a Confident Network Can Still Be Wrong

Softmax confidence measures decisiveness, not truth. The exponential rewards whichever logit leads, however the lead was earned — and the judges' scoring sheets are frozen, so an unusual image that happens to excite the wrong judges produces a confidently wrong verdict by the same honest arithmetic that nailed the 7.

The lesson ships this failure as a first-class input. The Misclassified variant loads, for each digit, the test image this exact network gets wrong with the highest confidence — real images, predictions above 90% on the wrong answer. The seven beats are identical; only the input changed. The Ambiguous variant sits in between: correctly classified, but at ~75%, so softmax leaves a visible runner-up. Run all three for one digit and you get the lesson's real takeaway — the network never changes, so where the probability lands is a property of the input-network pair. This network gets about 3.7% of the MNIST test set wrong, and on some of those it is more confident than it was on your clean 7.

Where the scoring sheets came from is the next lesson: Gradient Descent shows the update rule and Backpropagation shows how every weight learns its share of the blame.

Edge Cases

• Dead-quiet judges. ReLU outputs exactly zero whenever a weighted sum is negative, so many hidden neurons stay dark for any given image. Normal, not a bug — different images wake different subsets.
• Saturated vs split softmax. A clean digit collapses the distribution to ~100% on one class; an ambiguous one leaves real mass on a runner-up. Both are correct forward passes — only the logit gap differs.
• The all-zero input. With no ink, the first layer’s weighted sums reduce to its biases; later layers also receive the activations those biases produce. the network still produces a full probability distribution. It always answers, even when there is no question.

Common Mistakes

• Confusing logits with probabilities. Logits are unbounded raw scores — 19.0 is not "1900% sure". Only after softmax do the numbers form a distribution you may read as confidence.
• Reading meaning into hidden neurons. Individual hidden judges rarely correspond to anything a human would name — "loop detector" stories are mostly retrofitted. Only the output layer has assigned meanings.
• Dropping the nonlinearity. Without ReLU between them, two linear layers multiply out to one linear layer — the hidden panels would buy you nothing.
• Treating confidence as accuracy. The Misclassified variant exists precisely because >90% softmax on a wrong answer is not an anomaly; it is what the exponential does to any decisive logit gap, deserved or not.