Gradient Descent

Gradient descent is the optimization algorithm that trains nearly every modern neural network: measure the slope of the loss function at the current parameter value, then step in the opposite direction, scaled by a learning rate η. Repeat. The update rule is one line — θ ← θ − η·f′(θ) — and the learning rate decides whether it converges, crawls, or explodes.

Picture hiking down a mountain in fog so thick you can see exactly nothing. You can't plan a route to the valley — you can only feel which way the ground tilts under your boots and commit to a stride. Slope under foot, step downhill, repeat. That's the entire algorithm. The catch, and the actual subject of this lesson, is your stride length: too timid and you'll die of old age on the slope, too bold and you'll leap clear over the valley onto the opposite face — possibly higher than where you started.

How Gradient Descent Works

This lesson shrinks the mountain to one parameter and one curve: the convex bowl f(θ) = θ². The height of the curve is the loss — how wrong you are — and your position along it is the parameter θ you're allowed to change. The slope at any point is the gradient, f′(θ) = 2θ: positive when the bowl rises to your right, negative when it rises to your left. The slope points uphill, so you walk the other way. That minus sign in θ ← θ − η·f′(θ) is the whole "descent" part.

Notice what the rule does not require: you never see the bottom. You never solve for it. Like the hiker, you only know the tilt where you stand — and that's enough, because the gradient carries distance information for free. Far from the minimum the bowl is steep and the steps are long; near the bottom it flattens and the steps shrink with it. The algorithm decelerates into the answer without ever being told where the answer is.

On this particular bowl the update collapses into something beautiful. Substitute f′(θ) = 2θ and you get θ ← θ·(1 − 2η): every step multiplies θ by the same constant. The entire fate of the run — converge, crawl, bounce, or blow up — is one number. If |1 − 2η| < 1 the parameter shrinks toward zero; if |1 − 2η| > 1 every single step makes things worse. The canvas prints this multiplier under the headline, because once you can read it, nothing that follows will surprise you.

Gradient Descent Step by Step

The default run uses the well-tuned rate η = 0.25 (multiplier 1 − 2·0.25 = 0.5), starting at θ = 4, for exactly six iterations. Each iteration is two beats: measure the slope, then step.

1. Start. θ = 4, loss f(4) = 16. The ball sits high on the right arm of the bowl.
2. Measure. The tangent line appears: f′(4) = 2·4 = 8. Steep and positive — downhill is to the left.
3. Step. θ ← 4 − 0.25·8 = 2. The ball rolls left and down; loss drops to 4. The step was long because the slope was long — the hiker strides confidently on steep ground.
4. Measure. f′(2) = 2·2 = 4. Still positive, half as steep.
5. Step. θ ← 2 − 0.25·4 = 1. Loss falls to 1.

Iterations 3 through 6 repeat the same two-beat rhythm, and because the multiplier is 0.5, each step simply halves θ: 1 → 0.5 → 0.25 → 0.125 → 0.0625. The loss collapses toward zero, the tangent flattens to nearly horizontal, and the step arrows under the axis shrink in lockstep with it. After step 6, θ = 0.0625 lands inside the ±0.15 window the visualization treats as "converged," and the run ends there.

One honesty note about that ending. The ±0.15 window is a display threshold so the story closes cleanly — real training never checks "is the parameter near its final value," because you don't know the final value. Real stopping rules are a gradient-norm tolerance, a step budget, or validation loss going stale. The animated trace itself runs a fixed six steps with no convergence branch.

Why the Learning Rate Decides Everything

The config panel gives you η itself — a slider from 0.05 to 1.15 in steps of 0.05, on the same bowl, same start. Every position is one value of the multiplier 1 − 2η, which is the only thing that actually changes, and the panel prints it live as you drag. Four positions are worth knowing as landmarks (they're the recipe buttons):

• η = 0.10, multiplier 0.8. Converges in theory, crawls in practice. After six steps θ = 4·0.8⁶ ≈ 1.05 — the hiker is taking ankle-deep shuffles and runs out the step budget still well up the slope. The run ends stalled. Safe is not the same as good.
• η = 0.25, multiplier 0.5. The default. Each step halves the remaining distance; six steps is plenty.
• η = 0.85, multiplier −0.7. Now the multiplier is negative: every step jumps past the minimum to the opposite arm. The hiker leaps clear over the valley floor each stride — but |−0.7| < 1, so each landing is lower than the last and the zig-zag decays. Wasteful, ugly, still stable.
• η = 1.05, multiplier −1.1. Past the stability bound. Every step jumps over the minimum and lands higher: |θ| grows to 4·1.1⁶ ≈ 7.09 after six steps, the loss climbs instead of falling, and the plot literally widens to fit the escaping trajectory. This is divergence, and it is not a rounding problem you can wait out — it compounds. (Push the slider to 1.15 and the trajectory escapes even the widened plot — the ball pins at the edge with an honest "off the chart" callout and the true θ printed beside it.)

But the slider's real job is the two experiments no preset can hand you. First, sweep slowly across η = 1.00 and find the boundary yourself: at 0.95 the bounces still decay, at 1.00 exactly the ball oscillates between ±4 forever (multiplier −1, the knife edge), and at 1.05 every bounce lands higher. The stability condition |1 − 2η| < 1 stops being a line you read and becomes a cliff you've stood on. Second, park at η = 0.50: the multiplier is exactly 0, and the ball drops from θ = 4 to the minimum in a single step — the most surprising stop on the slider, and a one-step proof that the multiplier, not effort, decides the outcome.

The behavior flips from converging to diverging exactly where the multiplier's absolute value crosses 1. On this bowl that boundary sits at η = 1. But the boundary is a property of this bowl's curvature — f″ = 2 here. A sharper bowl has a smaller safe range, which is why no single learning rate is safe for every problem, and why tuning η is the first real decision in training anything.

Worth saying plainly: this is the one-parameter cartoon of the real thing. Production training pushes millions of parameters at once, estimates the gradient from noisy mini-batches, navigates non-convex terrain, and wraps the raw update in momentum or Adam plus a learning-rate schedule. But the core move — feel the slope, step the other way — is genuinely identical.

Edge Cases

• Already at the minimum. At θ = 0 the gradient is 0, so the step is zero. The hiker stands on flat ground; there's no downhill to follow.
• Flat regions. Near the bottom the slope is tiny and progress slows to a crawl. Here that's convergence behaving itself; on real loss surfaces, wide plateaus produce the same tiny gradients far from any minimum and stall training.
• Exactly at the stability edge. η = 1 gives multiplier −1: a perfect permanent oscillation between ±4. Any η above it diverges from a nonzero starting point; θ = 0 remains fixed, because the multiplier acts on θ every step regardless of where you began.
• Non-convex surfaces. This bowl has one minimum on purpose. Real loss curves have several valleys, and gradient descent — feeling only the local tilt — can approach a local minimum, stall near a saddle or plateau, or fail to converge if the update settings are unsuitable.

Common Mistakes

• Dropping the minus sign. θ ← θ + η·f′(θ) walks uphill — gradient ascent. The minus sign is the algorithm.
• Treating one learning rate as universal. η = 0.25 is "just right" on this bowl and could be catastrophic on a sharper one. The safe range depends on curvature; that's why practitioners tune η and usually decay it over time.
• Reading a large gradient as "almost done." On this quadratic, a large gradient means large distance from the minimum. On a general loss surface, gradient magnitude does not directly measure that distance. The gradient shrinks as you arrive.
• Confusing the loss with the gradient. The loss is the height of the curve; the gradient is its slope. You minimize the first by following the second.
• Assuming overshoot means failure. The η = 0.85 run crosses the minimum every single step and still converges, because the bounces decay. Divergence only begins when each bounce lands higher than the last — a few ticks of the slider, and the whole point of sweeping it across 1.0 yourself.