--- title: "Overfitting & Validation" description: "Fit polynomials of rising degree and watch train and test error split into a U-curve." source: https://vizlearn.app/ai/ml-evaluation/overfitting-and-validation --- # Overfitting & Validation Overfitting is what happens when a model stops learning the pattern in your data and starts memorizing the sample you happened to collect. The cheapest way to catch it is validation: hold some data back, never let the model study it, and score the model on that instead. The gap between the two scores is the diagnostic. Here is the uncomfortable part. Training error — the number your fit actually minimizes — cannot increase at the exact optimum as you enlarge a nested, unregularized hypothesis class, all the way past the point where the model has become useless. It is not a warning system. It is a thermometer that only reads "fine." In this lesson you fit nine polynomials of growing degree to the same ten noisy points, and by the end the model with a *perfect* training score is 899 times worse on data it has not seen than the model validation would have shipped. ## How Overfitting Happens The data comes from a process: `y = 0.8·sin(πx)` over x ∈ [−1, 1], plus a bit of Gaussian noise on every measurement. That is a real distinction and the lesson turns on it. There is a **pattern**, which repeats — every future sample from this process will still bend the same way. And there is **noise**, which does not — the particular wobble on point 6 is an accident of this one sample and need not recur in another sample. A polynomial of degree *d* has *d+1* coefficients to spend, and degree is this lesson's capacity knob — the same knob that in a real model is tree depth, layer width, or how long you train. At degree 1 it can only draw a straight line, so it spends everything trying to average out a curve and misses the shape entirely: that is underfitting, and both errors are bad. The interval holds one full period of the sine, so the truth falls, crosses zero at the origin, and rises again — an S with two turning points, odd about the middle. The lowest-degree polynomial that can draw that S is the cubic. Degree 3 has exactly enough capacity to describe the pattern and not much left over. Then you keep going. Degree 5, 6, 7 — the extra coefficients have no pattern left to describe, so they go looking for anything else that will reduce the training error, and the only thing left in the training set is noise. The curve starts detouring toward individual points. Training error keeps falling, because that detour genuinely does reduce the distance to *those* points. Held-out error starts rising, because the detour is aimed at wobbles the held-out points do not share. The endgame is exact. With ten training points and a degree-9 polynomial, you have ten coefficients for ten constraints, and there is a curve that passes through every single point. Least squares finds it. Training error: `0.0000`. To hit all ten exactly, that curve swings to a peak of ±353 between them — off the top and bottom of the plot — and the held-out points sit in exactly those gaps. This is worth saying plainly, because the wrong lesson is easy to take away: **the curve is not being punished for being wiggly.** It is being punished for being wiggly *in a way that only fits this sample*. Turn the noise off and degree 9 is not penalized at all. ## Overfitting and Validation Step by Step The default run is ten training points and ten held-out points, both drawn from the same process with σ = 0.25. Every degree gets two beats: fit it on the training half, then score that same curve on the validation half. The stems dropped from each point to the curve are the residuals being squared and averaged, and they are **vertical**, not perpendicular — the model predicts y from x, so the only miss that counts is the miss in y. (Perpendicular distance is a different objective belonging to a different tool; that is the PCA lesson's argument, not this one's.) **Degree 1.** A straight line. Training error 0.2711, held-out 0.1367. Both bad, and both bad for the same reason — the model cannot bend, so it is wrong about the process itself, not about this sample. **Degree 2.** Barely moves: 0.2482 and 0.1335. A parabola is symmetric about its vertex and this S is antisymmetric about the origin, so this sample gets little benefit from the extra coefficient; a quadratic term can still affect a noisy fit. **Degree 3.** The first real jump: training 0.0703, held-out **0.0305**. This is the bottom of the held-out curve and the model the lesson is going to ship. Notice the shape of it — the fit follows the process, not the individual points. **Degree 4.** Training error falls again, to 0.0665. It has to; a degree-4 polynomial can reproduce every degree-3 polynomial exactly, so the best it can do is never worse. Held-out error *rises*, to 0.0475. That is the turn. From here every improvement on the training set is bought with a loss on the held-out set. **Degrees 5 through 7.** The pattern repeats and steepens. Training: 0.0543, 0.0501, 0.0483. Held-out: 0.0583, 0.0860, 0.1549 — already five times worse than degree 3, and the training score still looks like progress. **Degree 8.** Training 0.0437. Held-out **1.2059**. The curve now peaks near ±29; it is leaving the plot to reach individual points. **Degree 9.** Ten coefficients, ten points. Training error **0.0000** — perfect, exact, every point threaded. Held-out error **27.4512**. **The verdict.** Selecting on validation ships degree 3 at 0.0305. Selecting on training error ships degree 9 at 27.4512 held out. Same data, same twenty lines of code, one decision different: which of the two numbers you read. The second model is 899× worse on data neither has seen — and from inside the training set, it looked like the best one you ever built. ## Why Training Error Cannot Choose the Model The monotonicity is not an empirical tendency, it is an algebraic fact. The degree-*d* polynomials are a subset of the degree-*(d+1)* polynomials — set the leading coefficient to zero and you are back where you started. So the best degree-*(d+1)* fit is at least as good as the best degree-*d* fit on whatever data you fit it to. Always. On every dataset, for every noise level. A number that can only move one direction carries no information about which direction is right. That is the whole argument. "My training loss went down" is evidence that your optimizer is working and nothing more — not that the model improved, not that this model beats the last one — in this lesson or in a neural network with a hundred million parameters. Held-out error has no such guarantee, and that is precisely what makes it useful. The validation points were never in the objective, so nothing about the fitting procedure is trying to make them look good. When the score gets worse, it means something. The cost is data. You gave up ten points you could have trained on to buy a number you can trust — which is why the next thing anyone reaches for is k-fold cross-validation, where every point takes a turn being held out and you never permanently sacrifice the points you held out. ## Three Experiments, Three Misreadings Three of the recipes in the config panel break the result in different ways, and each one kills a different misreading. **Tiny sample (8 points).** The training error hits zero at degree 7 now, not 9, and that model scores 0.8869 held out — fifteen times the best. The wall is at *n − 1*, one less than the number of points you have. "Degree 9 is too much" was never the lesson; there is no such thing as too much capacity in the abstract. **More data (20 points).** Same noise, same degree 9, and now it is only 1.5× worse than the best model instead of 899×, with the curve never leaving the plot. Ten coefficients cannot thread twenty points, so the fit has no choice but to describe the trend. The same model went from catastrophic to nearly free, because capacity is only excessive *relative to the data you have*. **Noise-free (σ = 0).** Every point sits exactly on the true curve. The U vanishes: held-out error reaches the floor at degree 5 and stays there, and the top degree is not punished at all. In this experiment, overfitting is the memorization of noise — remove the noise and there is nothing left to memorize. Do not promote that to a theorem. Noise is one way to overfit, not the only one. A high-degree curve fitted to a handful of perfectly clean points can still swing violently between them — that is Runge's phenomenon, and it is a property of where the points sit, not of any measurement error. It stays tame for this sine, these sample locations, and the limited degree range. Evenly spaced nodes do not generally prevent Runge's phenomenon. What transfers out of this experiment is the diagnostic, not the diagnosis: held-out error is what tells you which case you are in. ## Edge Cases - **Degree = n − 1 (exact interpolation).** The fit passes through every training point and the training error is 0 to floating-point precision. For distinct input x values, interpolation can fit any target values. Zero training error therefore cannot distinguish signal fitting from noise fitting on its own. - **Degree ≥ n.** Never generated here. The system is underdetermined — infinitely many curves pass through the points — and which one you get depends on the solver, not the data. The sweep stops at `min(9, n − 1)`. - **Zero noise.** The held-out curve has no U — the top degree scores best held out, so validation picks it — and the lesson's headline result correctly does not appear. The animation says so rather than pretending to find a U. - **The chart's floor.** Errors are plotted on a log axis spanning two to four decades, sized from the spread of the run's own scores, so a training error of exactly 0 has no position on it. Those markers pin to the floor line with an `≈0` tag rather than being dropped. - **A curve that leaves the plot.** The fit really does exit the frame — just past the edge at degrees 2 and 7, by tens to hundreds of units at 8 and 9. It is clipped at the frame, not hidden — the interpolation beat's status line says the curve leaves the plot, because the excursion is the mechanism, not a rendering artifact. ## Common Mistakes **Reading "training loss is falling" as progress.** It falls by construction. The only reason to watch it at all is to confirm your optimizer is working — never to decide whether the model is good. **Tuning against the test set.** Pick a model by looking at held-out scores enough times and that held-out set is no longer held out; you have laundered it into the training procedure through your own choices. This is why serious practice uses three splits: train, validation (which you *are* allowed to select on), and a test set you touch exactly once. **Believing "simpler is safer."** Degree 1 has a held-out error of 0.1367, four times worse than degree 3. Underfitting is a real failure with real cost — it just fails visibly, so it scares people less. The bottom of the U is a bottom, not a slope. **Confusing capacity with parameter count in general.** Degree is this lesson's capacity knob because there is one input feature and a polynomial hypothesis class. In practice capacity is depth, width, tree count, or training time, and it interacts with regularization — you often keep the big model and shrink its coefficients instead of dropping to a smaller one. **Assuming a bad held-out score means the model is too big.** It might mean your dataset is too small, which is the same ratio seen from the other side, and often the cheaper thing to fix. ## A Note on Simplification What runs here is real least-squares fitting (Householder QR in a Chebyshev basis, so the degree-9 interpolation is accurate to floating-point precision rather than exact in symbolic arithmetic) and real held-out scoring. No step is staged. But this is one input feature and a polynomial hypothesis class, where real models have many features and a capacity knob that is not degree. It uses a single train/validation split rather than k-fold cross-validation, and there is no final test set, so the lesson stops one step short of the honest three-way protocol. And there is no regularization anywhere — in practice that is often what you reach for instead of a smaller model.