---
title: "Support Vector Machine"
description: "The boundary with the widest margin between classes, set by its support vectors."
source: https://vizlearn.app/ai/classical-ml/svm
---

# Support Vector Machine

A support vector machine is a linear classifier that chooses its decision boundary by maximizing the margin — the width of the empty corridor it can fit between the two classes. In the hard-margin case, points with nonzero dual coefficients are support vectors and lie on the corridor walls. Every other point could be deleted and nothing would move.

Soft-margin support vectors may also lie inside the corridor or on the wrong side.

That last sentence is the part people find hard to believe, so this lesson is built to make you believe it. A perceptron stops at the first line that classifies everything correctly, and on separable data there are infinitely many such lines — all of them perfect on the training set, most of them terrible on the next point you see. Vapnik's answer was to stop thinking about lines and start thinking about streets: drive the widest street you can between the two crowds, then paint the centre line.

## How a Support Vector Machine Works

Start from the thing zero training error cannot do: choose. On the twelve points in this lesson you could draw a line at 30°, at 45°, at 60° — every one of them separates the A cluster from the B cluster without a single mistake. "Correct" does not rank them. "Widest gap" does.

Fix an orientation and the widest empty band around it is determined: push the two parallel walls apart until each one hits a point. That width is the score for that orientation, and the SVM's entire criterion is to take the orientation with the highest score. Nothing else is going on. The optimization is just a careful way of searching every angle at once.

The algebra is the same idea in coordinates. Scale the weights so the corridor's two walls are exactly `w·x + b = +1` and `w·x + b = −1`. The perpendicular distance between those walls is `2 / ‖w‖`, so **maximizing the margin is minimizing `‖w‖`** subject to every point staying on its own side: `yᵢ(w·xᵢ + b) ≥ 1`. That is a convex quadratic program, and its dual attaches one number `αᵢ ≥ 0` to each training point, with two conditions worth memorizing:

`w = Σ αᵢ yᵢ xᵢ` and `Σ αᵢ yᵢ = 0`.

The weight vector is a weighted sum *of the data itself*. And the optimality conditions force `αᵢ = 0` for every point that is not touching a wall — a point strictly outside the corridor has slack, and slack means the constraint is not binding, and a non-binding constraint contributes nothing. Only nonzero-alpha wall points contribute; redundant wall points can also have alpha zero. Here that is three, out of twelve.

## Support Vector Machine Step by Step

The default run uses twelve labeled points — six A in the upper left, six B in the lower right — and configured `C = 0.1`. For separable data, the opening comparison chapter first solves with `C_HARD = 1000`; the final chapter then uses the configured soft-margin C.

**Five separators.** The animation opens by drawing five different lines through the gap. Every one makes zero training errors. This is the problem statement, not the solution.

**Two orientations, measured.** Take the 30° line and widen its band until it touches a point: it fits **1.27**. Take the 60.25° line: **2.11**. The ledger on the left keeps both, as numbers and as bars, so the comparison is visual before it is arithmetic.

**The maximum.** Sweep to 45° and the band opens to **2.83** — wider than either candidate, and wider than any other angle. Concretely, `w = (−0.50, 0.50)` and `b = 0.00`, which puts the boundary on the line `y = x` and gives `2 / ‖w‖ = 2 / 0.707 = 2.83`. The panel on the right stops that from being a claim you have to take on faith: it plots the corridor's width at every angle that separates this data at all, from 18.5° to 80.5°, and the result is one hump with its apex exactly at 45°. Both measured orientations sit on that curve, visibly below the top.

**Three points hold it, and here is why.** Rings appear on `A(−4, −2)`, `A(1, 3)` and `B(2, 0)` — the only points touching a wall. Each reads `y(w·x + b) = 1.00` exactly, while the nearest point that is not touching reads 1.50. That equality is what makes those three special: a constraint held with equality is one the optimum is pressing against, and `αᵢ · (yᵢ(w·xᵢ + b) − 1) = 0` then says the other factor is free to be non-zero. Everyone else has slack, so their α is pinned at 0. Their alphas are `0.05`, `0.20` and `0.25`, and you can check both dual conditions by hand from the screen. The equality constraint: `0.05 + 0.20 − 0.25 = 0`. And the weight vector:

`0.05·(−4, −2) + 0.20·(1, 3) − 0.25·(2, 0) = (−0.50, 0.50)`

which is exactly the `w` the solver reported. Nine points have `α = 0` and appear nowhere in that sum.

**The experiment, run twice.** The animation now does the same thing to two different points. First it takes `B(5, 0)`, a point with `α = 0`, and slides it **6.58 units** across the plane to `(0.35, −4.65)` — two-thirds of the plot's width — then re-solves the whole quadratic program from scratch. Same `w`, same `b`, same three support vectors, same 2.83, to the last decimal. Then it takes `B(2, 0)`, the support vector with the largest alpha, and backs it **2.00 units** off the wall it was touching. Re-solve: the corridor opens to **3.54**, and the support set changes — `B(0, −3)` takes over the job the moved point was doing. One point that travelled three times as far changed nothing; the one that shifted 2.00 units changed everything, including which other points matter.

**The price of a violation.** Both points go home, and then `C` drops to `0.1`. The corridor widens to **3.61** and tilts slightly to 47.15°, and two points — `A(1, 3)` and `B(2, 0)` — end up *inside* it, still on the correct side but no longer respecting the margin. The model has bought room by paying for two violations, which is the whole soft-margin trade.

## Why the Widest Corridor Wins

The corridor is your error budget. A boundary crammed against the data flips its answer on the smallest perturbation of a single point; a boundary with a 2.83-unit corridor, about 1.415 units on each side needs a much larger surprise before it changes its mind. This is the intuition behind margin-based generalization bounds, and it is a *prior*, not a theorem about your data — max margin is a bet that the test points look roughly like the training points, and the bet can lose.

There is a practical dividend too. Because `w = Σ αᵢ yᵢ xᵢ` runs over support vectors only, the trained model is a handful of stored points plus their alphas. Training costs a quadratic program — solving it runs somewhere between quadratic and cubic in the number of samples, for dense kernel or generic QP approaches; specialized linear SVM solvers can scale much further. Prediction costs one dot product with `w`, or, once you swap in a kernel, one kernel evaluation per support vector.

That shapes both when to reach for an SVM and how to tune one. It is at its best on datasets of thousands to tens of thousands of rows with many features on a comparable scale — text classification, small tabular problems — and a poor fit for million-row logs. Standardize the features first, then cross-validate `C` on a logarithmic grid rather than a linear one; the slider in this lesson runs 0.03 to 100 in log steps for exactly that reason, because the behaviour worth finding spans orders of magnitude and a linear sweep spends every sample in the region where nothing changes. With a kernel, watch the support-vector count as you tune: it is the size of the model and the cost of every prediction, and a fit that keeps most of the training set merits checking prediction cost and validation quality, but is not automatically wrong.

## Soft Margin: What C Buys

Hard margin has a fatal flaw: it assumes a corridor exists. Real data overlaps, and one mislabeled sample can make the constraints unsatisfiable. Soft margin adds a slack term and minimizes `½‖w‖² + C · Σ max(0, 1 − yᵢ(w·xᵢ + b))`. Every point inside the corridor or on the wrong side pays a hinge penalty; `C` is the exchange rate between "narrower corridor" and "one more violation".

Large `C` means violations are unaffordable, so the fit obeys its worst point. Small `C` means they are cheap, so the fit buys room with them. On separable data the knob does nothing until it gets low enough to matter — every `C` above about 0.3 here returns the same 2.83 corridor, and the `Strict (C = 100)` recipe exists to show you exactly that non-effect.

The interesting case is the `One point in the gap` recipe. It adds a single B point at `(−1, 0)`, in the gap but on A's side of it. The data is still perfectly separable, so hard margin still has an answer — and that answer is a **0.71-wide sliver**, a quarter of what the same clusters supported before, with the intruder as one of its three support vectors. One sample now owns a third of the model. Drop `C` to 0.3 and the full 2.83 corridor comes straight back, at the cost of exactly one misclassified training point. That trade — one known mistake for four times the clearance — is what soft margin is for, and it is why `C` is the hyperparameter you actually tune.

## Edge Cases

- **A point exactly on a wall.** A point with `yᵢ(w·xᵢ + b) = 1` may have nonzero alpha, but equality alone does not guarantee it: a redundant wall point can still have alpha zero. "Support vector" is a role in a configuration, not a property of a sample.
- **Non-separable data.** Hard margin has no feasible solution at all — not a bad one, none. Soft margin always has one, which motivates soft-margin formulations in practical implementations. Drag a B point deep into the A cluster on the plane editor and the lesson drops the empty-corridor chapter rather than pretending there is a gap to widen.
- **Every support vector pinned at `α = C`.** When no support vector sits strictly between 0 and `C`, the bias `b` is only determined up to an interval — the corridor's width is unique but its exact position is not. The shipped recipes stay clear of that regime on purpose.
- **Unscaled features.** The margin is measured in whatever units you handed the model. A feature in dollars and a feature in kilometres do not compete fairly, and "maximize the width" quietly becomes "maximize the width along the big-numbers axis". Scale features whose units differ before fitting; standardization is the usual choice, not the only one.
- **Duplicate points.** Two identical training points give the dual a pair with no curvature to work with. The config panel refuses a drop onto an occupied cell for that reason.

## Common Mistakes

- **Thinking more data makes a better boundary.** Adding a thousand points deep inside the A cluster changes nothing at all — they all get `α = 0`. What improves an SVM is data *near the boundary*, which is also why SVMs pair naturally with active learning.
- **Reading `C` as a learning rate.** It is not a step size or an iteration count. It is a price, and the optimum it defines is exact — though a larger `C` can make the solver take longer to reach it.
- **Charging every violation the same price.** `C` multiplies the hinge loss, not a count of violations. A point that slips barely inside the corridor pays a sliver of `C`; one sitting deep inside, or on the wrong side entirely, pays several times as much. That is why dropping `C` buys room with shallow violations first — on the default run, `C = 0.1` pulls two points just inside the walls and misclassifies none.
- **Assuming the corridor is always empty.** With small `C` it is not: the two walls are still `w·x + b = ±1`, but points are allowed to sit inside and pay for it. The width `2/‖w‖` stays meaningful; "no point closer than this" does not.
- **Treating the support vectors as the important data.** They are the *binding constraints*, which is not the same as the most informative or most typical samples. Remove one and another point may take over — as the animation shows when `B(0, −3)` inherits the job — or the remaining ones carry it alone.
- **Expecting max margin to be optimal.** It is a well-motivated default, not a guarantee. On noisy, overlapping data, validate C rather than assuming a widest empty corridor is feasible or best.

## A Note on Simplification

This lesson shows a linear soft-margin SVM on twelve or thirteen points in two dimensions, because that is the version where you can check the arithmetic by eye. Real use adds pieces this animation leaves out: kernels (the same dual with `⟨xᵢ, xⱼ⟩` replaced by a kernel function, which buys curved boundaries at no change to the theory), feature scaling, multi-class schemes such as one-vs-rest, and a cross-validated `C` rather than a slider. The dual is genuinely solved here by SMO, but the solver iterations stay backstage because they are bookkeeping; the geometry of the answer is the lesson. Treat this as an explainer, not a substitute for a course in convex optimization.