---
title: "Classical ML overview"
description: "The arithmetic underneath every model, animated step by step — the dot product that scores an input, the matrix product that runs every such score at once, the linear classifier that turns K scores into a decision, the k-nearest-neighbors vote that never trains at all, the decision tree that greedily carves the plane into rectangles, the support vector machine that picks the boundary with the widest empty corridor, the least-squares fit that shows what a loss function actually measures, the sigmoid that turns a score into a probability cross-entropy can judge, the k-means loop that provably always stops — somewhere the initial guess chose — the PCA budget where every direction's captured variance and discarded error add to the same fixed total, and the Naive Bayes jury where every word votes without hearing the others."
source: https://vizlearn.app/ai/classical-ml
---

# Classical Machine Learning Visualizations

Classical machine learning is where the mathematics is still small enough to watch. One vector meets another and produces a single number; a handful of those numbers become a prediction; a straight line on a plane decides which class a point belongs to. These lessons animate that arithmetic at the size where you can check every step by hand — and then show you exactly which of its properties survive into the deep networks built on top.

Nothing here is a warm-up you throw away later. The dot product in the first lesson is the same operation a transformer runs billions of times per second. The score-then-argmax rule in the second is the last layer of nearly every image classifier ever shipped. What changes at scale is the number of dimensions, not the idea.

## What These Lessons Cover

| Lesson | What you watch |
|---|---|
| Vectors & Dot Product | one multiply-and-add loop turning into a shadow on a line — and the angle deciding the sign |
| Matrix Multiplication | a row of A and a column of B lined up on the cell they compute — and the same two matrices swapped into a grid of a different shape |
| Linear Classifier | three scorers carving a plane into three convex wedges, and failing on data that needs a fourth shape |
| k-Nearest Neighbors | one sorted list of distances and a cut sliding through it — the same query flipping class between k = 1 and k = 3, and a decision map nobody drew |
| Decision Tree | every axis-aligned cut scored by the entropy it removes, the argmax carving rectangles — and one extra depth level giving a noise point its own private region |
| Support Vector Machine | one boundary sweeping to the widest empty corridor between the clusters — and three points out of twelve deciding where it lands |
| Linear Regression | residual squares as literal area, a fit computed in one pass — and one flung point dragging the answer off the crowd |
| Logistic Regression | a score squashed into a probability, a boundary that provably cannot move when (w, b) doubles — and cross-entropy deciding whether sharper confidence was a good idea |
| K-Means Clustering | unlabeled points earning a color, a total that can only fall, and the same data settling at 28.80 from one start and 133.10 from another |
| Principal Component Analysis | one fixed budget of variance that every direction cuts in two, a nine-angle sweep that brackets the answer without ever landing on it — and a change of units that raises the score while wrecking the representation |
| Naive Bayes | one fixed factor per word sliding a marker along a log-odds axis — and the same email with one word repeated twice more coming out 13.6× more confident, because a copy is scored as a second independent witness |

## A Recommended Learning Order

Start with **Vectors & Dot Product**. It is one loop — multiply matching slots, add — and the whole lesson is about why the number that loop returns behaves like a protractor. Two controlled experiments separate the two ingredients baked into it: hold the angle and stretch the vector, and the score scales; hold the length and rotate, and the score traces a cosine from maximum through exactly zero to minimum. That zero is the important one, because it is what makes `w·x = 0` usable as a boundary.

**Matrix Multiplication** scales that loop up to a table. Each cell of C = AB is one dot product, row i of A against column j of B, and the lesson lays the three matrices out so the row and the column literally cross at the cell they produce. That picture explains the shape rule — the inner sizes must match because a dot product needs two lists of the same length — and the cost, m·n·p multiplications. Then A and B trade places: on the default input AB is 2×2 at 12 multiplications while BA is 3×3 at 18, so order is part of the question. A neural layer is this product with names attached: inputs as rows, weight vectors as columns.

Then **Linear Classifier**, which runs that same loop once per class and lets `argmax` read the results. Three scores instead of one changes the geometry completely: a border stops being "where one score hits zero" and becomes "where two scores tie", and the plane comes out cut into convex wedges. Two experiments take the model's ingredients apart the same way — turning a row swings its wedge, moving a bias slides its borders parallel — and a deliberately impossible dataset shows the ceiling: a convex region cannot hold two clusters with somebody else's data in between.

**k-Nearest Neighbors** is the classifier lesson's deliberate counterpoint, and the two close each other's story. Where the linear classifier compresses its data into a matrix and throws the points away, kNN keeps every point and computes nothing until a query arrives — all the work is a distance sort at prediction time, and its one dial, k, moves a cut through that sorted list without recomputing a thing. The payoff for watching them back to back is the `islands` dataset: the two-island class the classifier lesson *proves* impossible for any choice of weights, kNN solves with three votes — and loses again at k = 13, because its ceiling is committee size where the linear model's was shape. The lesson's decision maps also hand you the cleanest picture of overfitting in the collection: k = 1 scores a perfect 100% on the stored points while growing an island of wrong answers around a single mislabeled one.

The **Decision Tree** is the collection's first model that is *built* during its own animation, and it answers the classifier's wall a second way. At every node it tries every axis-aligned cut, scores each by the bits of entropy it removes, keeps the argmax, and recurses — so the decision map comes out as rectangles the questions imply rather than a shape anyone drew. The same two-island dataset the linear classifier proves impossible falls to two greedy cuts here; the revenge match is a dataset one slanted line separates, where the tree builds a staircase of straight cuts and still misses at depth 3. And its depth cap is the collection's sharpest overfitting dial: depth 2 accepts one mis-covered noise point, depth 3 carves that single point a private region — zero training error, nothing learned.

The **Support Vector Machine** answers the same question a third way, and it is the one that finally
says out loud what the earlier lessons leave implicit: on separable data there are infinitely many
correct boundaries, and "correct" cannot choose between them. Its criterion is width. Fix an
orientation, widen an empty band until it hits a point, and you have that orientation's score; take
the best score over every angle and the boundary is decided. The payoff is the collection's most
surprising invariant, and the lesson runs it as an experiment rather than asserting it: slide a
point with α = 0 seven units across the plane, re-solve from scratch, and the answer comes back
bit-for-bit identical — then back one *support* vector two units off its wall and the corridor
widens, tilts, and hands its job to a different point. Twelve points, three of them in the model.
Its failure case is the sharpest argument for soft margin anywhere in the collection: one sample
parked in the gap, still perfectly separable, collapses the corridor from 2.83 to 0.71 — and a
lower price on violations buys the whole thing back for exactly one deliberate mistake.

Then **Linear Regression**, which is the first lesson where a model is *fitted* rather than handed to you. It swaps the classification question for a numeric one and makes the loss visible: every point's error is drawn as a square standing on its residual, so "minimize squared error" becomes "shrink this total area". The answer needs no search — two sums give the slope and the mean point pins the intercept — and the closing experiment is the one worth carrying forward: move a single point far off the trend, re-run the same formula, and watch the optimal line abandon the other eight. That is squared error working exactly as defined, and it is the clearest early warning about what a loss function chooses for you.

**Logistic Regression** closes the loop by joining the two threads. From the classifier lessons it takes the straight boundary; from the regression lesson it takes the visible loss — and then it shows what neither could alone. The score z = w·x + b gets squashed through the sigmoid into a probability, and the lesson's controlled experiment doubles (w, b): the boundary provably does not move by a pixel, the unsure band exactly halves, and cross-entropy delivers the verdict — on data with one wrong-side point the total loss *rises*, because a confidently-wrong point pays without bound, while on separable data the same doubling wins forever and the weights run away toward infinity. Where the model decides and how sure it is are two different dials, and the loss only controls the second.

**K-Means Clustering** takes away the thing every lesson before it assumed: the labels. Nothing in its input says which group anything belongs to, so nothing can be right or wrong — there is only a partition, and a number measuring how tight it is. Two steps take turns (each point joins the nearest centroid; each centroid moves to the mean of the points that chose it), and neither can raise that number, which is the whole proof that the loop halts: no tolerance, no iteration budget, just a round that reproduces its own input. What the lesson spends its second half on is the gap between *converging* and *being right*. It runs the same fifteen points and the same k twice from different starts, and the two runs halt just as firmly at inertia 28.80 and 133.10 — the second one having fused the two left blobs and cut the right one apart. All 455 ways to start from three of the points are then enumerated on screen: 332 find the good answer, 123 do not, and from inside a single run nothing tells you which happened. Its failure cases are the collection's most geometric: k = 4 splits a real group *and lowers the objective* (so inertia can never choose k), and two concentric rings defeat every start there is, because a partition into Voronoi cells is a plane cut by straight lines and no straight line makes a ring.

**Principal Component Analysis** stays unsupervised and changes the question from "which group is this in" to "how few numbers do I need". Its argument is one identity you can check with a right triangle: center the data, pick any direction, and the variance captured along it plus the perpendicular reconstruction error left over always add to the same total. There is one budget, and a direction only chooses where to cut it — which is why "greatest spread" and "least reconstruction error" turn out to be the same request. The lesson prices the naive move first (drop a column: 63.75% kept, 2.30 cm typical miss), sweeps nine directions to show the split sliding inside a bar whose ends never move, then lands the closed form: the best direction is the top eigenvector of a 2 × 2 matrix, at 36.16°, keeping 95.26% — and the nine samples bracketed it without one of them hitting it, which is precisely the argument for not searching. Its two failure cases fail for different reasons and that is why both ship. Record the second feature in millimetres instead of centimetres and the explained variance *rises* to 99.67% while the first component quietly becomes "width, in mm" — a higher score for a worse answer, fixed by scaling both features to unit variance, after which the scale knob provably does nothing at all. The second one has no fix: on twelve samples from two real groups, PC1 takes 86.91% of the variance and fuses the groups completely, while the 13.09% it discards is the only thing that separates them. PCA never sees a label, and variance is not a synonym for information.

**Naive Bayes** is the collection's only *generative* model, and the change of question is the point. Every lesson before it draws a boundary through the input space and asks which side a point falls on. Naive Bayes instead asks what each class would have to be like to have produced this document, multiplies one probability per word, and takes whichever answer is larger. The arithmetic is small enough to audit by hand — eight training emails, twenty-three words, one counting pass — and the animation runs it on a log-scaled odds axis, where a multiplication is a fixed displacement and you literally watch each word slide the answer a fixed distance in its own direction. `free` pushes ×2.46 toward spam, `meeting` pushes ×0.41 back toward ham, and the verdict is wherever the product lands.

What the lesson spends its last three beats on is the price of the word "naive". Because each word votes independently of every other, a repeated word is a second independent witness: append two more copies of the email's strongest token and the odds go from 5.47 : 1 to 74.28 : 1, a model 13.6 times more certain on two tokens that are perfectly correlated with one it already counted. The argmax survives the rewrite; the confidence does not, and the same arithmetic runs on any correlated pair — a word and the bigram containing it, the subject line repeating the body. That is the honest reason to read a Naive Bayes probability as a ranking and not as a calibration. Its failure presets are the two every practitioner meets: turn smoothing off and one word the class never happened to use zeroes its entire product (both classes, on the default email, leaving `argmax` comparing −∞ to −∞), and turn it up to 5 and 115 pseudo-counts per class bury 32 real tokens until the answer is a coin toss.

Those eleven lessons together answer *what a model computes* and *what it is optimizing*. What none of them answers is where the weights came from — k-means moves its centroids by a closed-form mean, not by following a gradient — which is the neural-network collection's subject: [gradient descent](https://vizlearn.app/ai/neural-networks/gradient-descent) measures which way to nudge them, [backpropagation](https://vizlearn.app/ai/neural-networks/backpropagation) works out each weight's share of the blame, and the [training loop](https://vizlearn.app/ai/neural-networks/training-loop) runs the whole cycle live. Gradient descent is the natural sequel to the regression lesson: where least squares solves for its line outright, an optimizer only feels the slope and steps downhill — shown there on a one-parameter bowl rather than a fitted line. The [perceptron](https://vizlearn.app/ai/neural-networks/perceptron) is the other bridge — a single neuron learning the boundary the classifier lesson hands you ready-made.

## All Classical ML Visualizations

Pick any lesson below to open its interactive player — change the input, step through it, and watch the code highlight move in lockstep with the animation.