Optimizers — SGD, Momentum, Adam

Optimizers turn gradients into parameter updates, but no optimizer wins every landscape. In this fixed 20-update race, SGD wins the round clean bowl, Momentum wins the narrow ravine, and Adam wins the high-noise case. The runs share their start, seeded batches, budget, and evaluation metric; each optimizer uses the lesson's declared reference hyperparameters.

The useful question is not "Which optimizer is best?" It is "Which failure mode does its state correct?" SGD reacts to the gradient in front of it, Momentum carries velocity across batches, and Adam also tracks a coordinate-wise RMS scale. Geometry and noise decide which memory helps and which merely overshoots.

How SGD, Momentum, and Adam Work

All three optimizers receive a gradient, but they keep different amounts of history.

• SGD uses only the current gradient: θₜ = θₜ₋₁ − αgₜ. It reacts immediately and carries no optimizer state into the next batch.
• Momentum keeps a velocity: uₜ = 0.8uₜ₋₁ − αgₜ, then θₜ = θₜ₋₁ + uₜ. Gradients that keep pointing the same way build speed. Gradients that reverse across a narrow ravine partly cancel.
• Adam keeps a moving average of gradients, m, and squared gradients, v. After bias correction, it divides m̂ by √v̂ + ε. This gives each coordinate its own effective scale instead of letting the steepest coordinate set the whole update.

The learning rates in this visualization are fixed for its quadratic landscape: SGD uses 1.5 / κ, Momentum uses 0.45 / κ, and Adam uses 0.2. Those values make the comparison readable; they are not universal defaults for real models.

Optimizers Step by Step

The default run sets κ = 20 and mini-batch noise σ = 0.08. The true loss is L(x,y) = ½(x² + 20y²), so the y direction is twenty times steeper than the x direction. Every optimizer starts at (4, 1) and receives the same twenty seeded batch shifts.

1. The first batch moves the temporary batch optimum slightly away from the true minimum. Each optimizer measures its own gradient against that same shifted bowl.
2. The rule beats preview the first displacement. SGD follows the present gradient, and Momentum creates its first velocity. Adam takes two beats: the first builds and bias-corrects its direction estimate m̂; the second builds v̂ and divides by its coordinate-wise RMS scale.
3. Update 1 commits all three displacements. The next batch then arrives while the parameter points remain fixed, making the memory boundary visible: SGD carries nothing, Momentum carries u₁, and Adam carries m₁ and v₁.
4. Updates 2–20 run as a compact race, with checkpoints at updates 5, 10, and 20. The contour plane keeps equal geometric scale on x and y, so a long cross-ravine move really is longer rather than a chart distortion.
5. The comparison uses mean true loss over updates 16–20, not a single final sample. On the default narrow ravine, the means are SGD 0.7203, Momentum 0.0585, and Adam 0.3854. Momentum is the most stable late runner here.
6. Three short proof beats then replay core-generated paths for the round, narrow, and high-noise recipes. They show path length, cross-ravine sign changes, and Adam's coordinate RMS scale before the final winner grid appears.

Why the Winner Changes

The final proof sequence repeats the exact experiment under three fixed optimizer recipes. The initialization, seeded batch sequence, 20-update budget, and evaluation metric stay fixed. Each optimizer keeps its declared reference learning rate from above.

This is the main result: an optimizer encodes assumptions about useful memory and scale. The result is evidence for those three landscapes, not a universal ranking of optimizer names.

Time and Space Cost

For d parameters, every rule must process a d-dimensional gradient, so the optimizer update itself costs O(d) time. SGD adds O(1) optimizer state beyond the parameters. Momentum stores one velocity vector, or O(d) state. Adam stores first- and second-moment vectors, or O(d) state with a larger constant. In real training, the forward and backward passes usually dominate the arithmetic, while optimizer state can still be a major memory cost for very large models.

Optimizer Edge Cases

• With σ = 0, every mini-batch optimum is the true minimum. The traces become fully smooth, exposing overshoot without noise as an excuse.
• With κ = 1, the bowl is round. There is no steep coordinate for Momentum to damp or Adam to normalize away.
• Increasing σ does not change the true loss surface. It changes only the temporary batch optimum used to produce each gradient.
• A final point can be lucky or unlucky under noise. That is why the lesson compares the last five true losses instead of declaring a winner from update 20 alone.

Optimizer Common Mistakes

• Comparing optimizers on different batches. A fair race must share the initialization, batch order, update budget, and evaluation loss.
• Treating Momentum as a larger learning rate. Its defining feature is the carried velocity. Reset that velocity before every update and it collapses to a rescaled current-gradient method.
• Calling Adam “Momentum with two averages.” The second moment is not merely extra smoothing; its square-root denominator changes the scale of each coordinate.
• Reading one toy win as a production recommendation. Real optimizer choices interact with schedules, weight decay, batch size, normalization, precision, and model architecture.

Use Optimizer Traces as a Diagnostic

A training curve does not name the fix for you, but it can narrow the question. Persistent zig-zag across one direction points first to conditioning, feature scale, and learning rate. Check those before treating an optimizer swap as a cure. If progress along the shallow direction repeatedly disappears when the steep gradient reverses, ask whether carried velocity would preserve it.

When late losses bounce, compare a window rather than one endpoint. Then inspect batch size, gradient clipping, and the learning-rate schedule: optimizer state can reshape noise, but it cannot remove a noisy gradient source. Adam's two moment vectors also cost memory, and a resumed training run must restore optimizer state along with model parameters. Resetting u, m, or v changes the next update even when the weights are identical.

A Note on Simplification

This lesson uses two parameters and seeded quadratic mini-batches so update geometry remains visible and every result is reproducible. Real neural networks use millions or billions of parameters, non-convex losses, learning-rate schedules, weight decay, gradient clipping, mixed precision, and optimizer-specific tuning. The update equations are real; the landscape is deliberately small enough to inspect.