---
title: "Cross-Validation"
description: "Rotate k folds of held-out data so every point gets scored as unseen."
source: https://vizlearn.app/ai/ml-evaluation/cross-validation
---

# Cross-Validation

Cross-validation estimates how well a model performs on unseen data by splitting the dataset into k nearly equal folds, training k times, holding out a different fold each round, and averaging the k test scores. Every row is held out exactly once, so the score no longer hangs on which rows you happened to draw for a single test set.

That last clause is the whole reason the method exists, and it is easy to nod at without believing. So here is the demonstration, on 20 photos of 20 animals and a model so simple it has nothing to hide: 1-NN, which labels a new photo by copying the label of the closest photo it has already seen. Hold out four of the twenty, fit on the other sixteen, score. Do it once and you get 1.00. Do it again with four different photos hidden and you get 0.25. Same data, same model, same code. The only thing that changed was the draw.

## How Cross-Validation Works

The naive protocol has one step and one hidden assumption. You hold out 20% of your data, fit the model on the remaining 80%, and report the accuracy on the held-out part. The assumption is that this number is a property of the model. It is not. It is a property of the model **and** the particular rows you drew, and on a small dataset the second term dominates.

You can measure that. Over 400 different random 80/20 splits of these same twenty photos, the reported accuracy has a standard deviation of **0.2005** and lands anywhere in {0.25, 0.50, 0.75, 1.00}. On 140 of those 400 splits — 35% of the time — it reports a flawless 1.00 for a dataset whose leave-one-out accuracy is 80%; that is not a known population accuracy. If your evaluation reports a perfect score once every three tries on a model that is wrong one time in five, the estimate has substantial split variability.

Cross-validation's fix is not "average the noise away". Averaging noisy things is what you do when you are stuck with a random process. Here the fix is structural: **stop drawing the test set**. Shuffle the rows once, cut them into k contiguous blocks, and let each block take a turn as the test set. Every row now sits in the test set exactly once, so across the whole run there are exactly n predictions — never more, never fewer, whatever k is — and each one is made by a model that did not have that row in training. The number that comes out is the average of the k fold scores; when the folds are the same size, that is just: of the twenty photos, how many did a model that had not seen them get right?

One residue survives, and it is worth naming before you meet it two paragraphs from now. *Who* gets tested is no longer a draw — everyone does. *What each row is tested against* still is: photo #4's prediction comes from a model trained on the sixteen rows outside #4's fold, and which sixteen those are still depends on the shuffle. A k-fold score is therefore not perfectly split-independent at finite k. In this deterministic example it is steadier than a single split and tends to get steadier as k rises, though not at every step; at k = n the partition-shuffle residue disappears — which the next paragraphs measure rather than assert.

Two consequences worth stating in full, because they are usually collapsed into one.

The **estimate** tends to get steadier as k rises. Over the same 400 splits, the standard deviation of the cross-validation score falls from 0.0685 at k = 2, to 0.0351 at k = 5, to 0.0186 at k = 10, to exactly **0.0000** at k = 20. The trend is not a guarantee for every step: in between it climbs back to 0.0356 at k = 13. At k = 20 there is one photo per fold; the shuffle has nothing left to shuffle, so every split id produces the identical answer. The knob stops responding, which is a stronger proof than any caption.

The **individual fold scores** go the other way. Their standard deviation *rises* over the same range: 0.1009, 0.1837, 0.2769, and 0.4000 at k = 20, where a fold holds one photo and can only score 0.00 or 1.00. So the spread across folds is not the error bar of the cross-validation score. It moves in the opposite direction. Writing `0.80 ± 0.40` from a leave-one-out run is the single most common misuse of these numbers, and it gets *worse* the more folds you use.

One naming hazard, and it is the reason the model here is deliberately called **1-NN** and never "k-NN": the `k` in k-fold counts folds, and the `k` in k-nearest-neighbours counts neighbours. They are unrelated, they appear in the same sentence constantly, and confusing them costs an afternoon.

## Cross-Validation Step by Step

The default run: 20 photos, 10 cats and 10 dogs, 1-NN, `k = 5`, split #8. That split id is not a neutral placeholder. It was chosen because its first fold is perfect and its second is a disaster.

1. **The data.** Twenty labelled photos, ten of each class, plotted as two features. Nothing is hidden yet. 1-NN has no parameters to fit — its "training" is remembering the sixteen photos it is given, which is exactly why nothing on screen competes with the protocol.
2. **Hold out fold 1.** Photos **#2, #12, #13, #15** are hidden; the other sixteen become the model. This is the ordinary train/test split, run once.
3. **Four predictions.** #2 finds #6, a cat, 2.50 away — right. #12 finds #16, right. #13 finds #19, right. #15 finds #9, right.
4. **Report.** Four out of four. **1.00.** Stop here, as most people do, and 1.00 is the number that goes in the report.
5. **Rotate to fold 2.** Those four were one of five possible test sets. Now **#1, #4, #7, #8** are hidden instead and the model is refitted on the other sixteen — including the four just tested.
6. **Four more predictions, and it falls apart.** #1 (a dog) finds #10, a cat, 2.92 away — wrong. #4 (a cat) finds #5, a dog — wrong. #7 (a dog) finds #15, a dog — right. #8 (a cat) finds #9, a dog — wrong.
7. **Score.** One out of four. **0.25.** Nothing changed except which four rows were hidden, and the reported accuracy moved by 0.75.
8. **Folds 3, 4, 5.** The same rotation three more times: 0.75 (only #10 missed), then 1.00, then 1.00.
9. **The spread.** Five folds, five legitimate single-split experiments, priced on one axis: **1.00, 0.25, 0.75, 1.00, 1.00**. Any one of them is a defensible "test accuracy" for this model, and they disagree by 0.75.
10. **The mean.** Every one of the twenty photos was held out exactly once and the five folds are the same size, so the average of those five fold scores is just 16 of 20 = **0.800**. The ledger shows it directly: after the last fold, each of the twenty columns holds exactly one coloured cell — sixteen green, four red.

Now turn one dial. Set the split to **#34** and the folds come out completely differently: the first one now scores **0.25**. The five-fold score is **0.800**, unchanged to the last digit. The naive number moved by three quarters; the honest number did not move at all.

## Why More Folds Cost More and Buy Less Than You Think

The bill is straightforward: k model fits, each on roughly n(1 − 1/k) rows, and exactly n predictions in total. Raising k raises the number of fits linearly and grows each training set slightly. Leave-one-out (k = n) is the extreme — twenty fits here, and for a model that takes an hour to train, twenty hours.

What you buy is less than the price suggests. Going from k = 5 to k = 10 cuts the estimate's standard deviation from 0.0351 to 0.0186 on this data; going to k = 20 gets it to zero, but "zero" here means only that the split id no longer matters — it does not mean the estimate is correct, and the next section is about exactly that. Five and ten folds are the defaults everyone uses because they sit at the knee of that curve.

There is also a subtlety in the averaging. When k does not divide n, the folds are unequal. At k = 3 on twenty photos the folds hold 7, 7 and 6 rows, and the unweighted mean of the three fold scores is **0.754** while the pooled rate — total correct over total rows — is **0.750**. `cross_val_score` hands back the k fold scores as an array, and the number almost everyone reports is `.mean()` of it — the unweighted one. Both numbers are defensible and neither is a bug. They just answer slightly different questions, and the lesson prints both when they differ.

## Edge Cases

- **k does not divide n.** Fold sizes differ by one; the first `n % k` folds take the extra row. The unweighted fold-score mean and the pooled rate can then disagree (0.754 vs 0.750 at k = 3 here), and the customary `cross_val_score(...).mean()` reports the former.
- **k = n (leave-one-out).** Every fold score is 0 or 1, the split id becomes irrelevant, and the estimate stops moving. Cheap-sounding and expensive in practice — it is n model fits.
- **k = 2.** Each model trains on half the data, so each fit has less training data than a final model fitted on all rows; its performance is not guaranteed to be worse in every run. The estimate is pessimistic on average (0.7609 over 400 splits here, against the 0.800 that leave-one-out reports), which is the bias half of the bias–variance trade-off in cross-validation.
- **Grouped data.** When several rows describe the same underlying thing — two photos of one pet, several visits by one patient, consecutive frames of one video — a random split puts siblings on both sides. Use `GroupKFold`, which cuts groups instead of rows. It cannot make more folds than there are groups.
- **Unstratified folds with rare classes.** With a random shuffle, a small fold can end up holding only one class, which makes accuracy on it nearly meaningless. `StratifiedKFold` keeps each fold's class proportions as close to the overall ones as the counts allow — a class with fewer rows than folds still misses some folds — and is the sane default for classification.
- **Time series.** Shuffling destroys the ordering, so a normal k-fold trains on the future to predict the past. Use forward-chaining splits (`TimeSeriesSplit`) instead.

## Choosing a Splitter: KFold vs StratifiedKFold vs GroupKFold

Every one of those cases turns on the same question, and it is worth asking before you pick a `k`: what does one row actually represent?

| One row is | Split with | Because |
|---|---|---|
| an independent observation | `KFold` | there is nothing to preserve |
| an independent observation, one of several classes | `StratifiedKFold` | a fold holding one class scores nothing meaningful — and this is what `cross_val_score` already uses for a classifier |
| one measurement of a subject that appears several times | `GroupKFold` | siblings on both sides of the split is leakage, not evaluation |
| a point in time | `TimeSeriesSplit` | shuffling trains on the future |
| scarce, and you need an uncertainty | repeated k-fold | the honest spread is across runs, not across folds |
| feeding a hyperparameter search | nested CV | the number you report must not be the number you optimised |

## Common Mistakes

- **Reporting a single split.** The most common one, and the whole point of this lesson. One split reports the split. If you must report one number, report the k-fold score, and put the fold range next to it as a spread — not as an error bar, for the reason above.
- **Treating fold spread as an error bar.** `0.80 ± 0.40` from a leave-one-out run is nonsense: those fold scores can only be 0 or 1. Fold variance tends to grow with k while the estimate's variance tends to shrink. If you need uncertainty, use repeated cross-validation — several full runs under different shuffles — and take the spread of the *run scores*.
- **Ignoring leakage.** Try the second dataset: 20 photos of only 10 animals, two shots each. Random 5-fold reports a flawless **1.000**, because 18 of the 20 test photos have their own twin sitting in the training set and 1-NN simply finds it — the paired examples are much closer in the designed features, with the illustrated distance ratio of 2.24. Switch to `GroupKFold` and the same data, same k, same split reports **0.500**. The model is a coin flip. And raising k does not help: leave-one-out on that dataset scores a perfect 1.000, because the twin is *always* in the training set. The protocol everyone calls the most trustworthy is the most deceived.
- **Fitting the preprocessing outside the loop.** Scaling, imputing, or selecting features on the full dataset before splitting leaks the test folds into the training statistics. Everything fitted must go inside a `Pipeline` that is cross-validated as a unit.
- **Tuning on the same folds you report.** Pick the best of thirty hyperparameter settings by cross-validated score and that score is now optimistic — you selected on it. Use a nested loop: an inner cross-validation to choose, an outer one to report.

## A Note on Simplification

What is animated is real k-fold cross-validation — the sklearn fold sizing, the unweighted mean, the group-aware splitter — running on a deliberately small stage: one 20-row two-feature dataset, a 1-NN model, and accuracy as the only metric. Twenty rows is what makes the phenomenon visible; on 100,000 rows a single split is far steadier, and cross-validation earns its keep for a different reason (model selection) rather than because one split is unreliable.

Three things a real evaluation adds. **Stratification**: neither dataset here is stratified, so at k = 10 a fold can hold two photos of the same class. `StratifiedKFold` is the classification default, and it would have muted the very variance this lesson exists to show. **Repeated cross-validation**: running the whole k-fold procedure under several shuffles and reporting the spread of the run scores, which is a measure of split sensitivity rather than a complete confidence interval for population performance; repeated runs share the same data. **Nested cross-validation**: the outer loop reports, the inner loop tunes, so the number you publish was not the one you optimised.

The metric is also a simplification. Accuracy is a useful illustration on this balanced 10/10 dataset, but metric choice also depends on error costs and deployment prevalence; on imbalanced data it is replaced by precision, recall, F1 or ROC-AUC, all of which cross-validate the same way.