---
title: "Logistic Regression"
description: "Squash a score through the sigmoid into a probability, then classify."
source: https://vizlearn.app/ai/classical-ml/logistic-regression
---

# Logistic Regression

Logistic regression is a linear classifier that reports its confidence. It scores an input with the same dot product every linear model uses, z = w·x + b, squashes that score through the sigmoid σ(z) = 1/(1+e^−z) into a probability between 0 and 1, and classifies by whether p ≥ 0.5. Training picks (w, b) by minimizing cross-entropy, also called log loss.

The idea worth keeping is that *where* the model decides and *how sure* it is are two separate dials. Doubling both w and b works like turning up the contrast on a photograph: every gray pixel off the outline snaps toward black or white, but the outline of the picture does not move a millimeter. The decision boundary is that outline. The sigmoid only sets the contrast — and cross-entropy is the judge of whether more contrast was a good idea.

## How Logistic Regression Works

Three lines of arithmetic, then one line of judgment.

The score comes first: z = w·x + b, one multiply-add per feature. This is the identical first move as the linear classifier and the perceptron; z = 1 or z = −3 are perfectly good scores, but they have no scale — 1 out of what?

The sigmoid answers that. σ(z) = 1/(1+e^−z) maps any score into (0, 1): a big positive z lands near 1, a big negative z near 0, and z = 0 lands on exactly ½. The output is read as P(class = + | x). The verdict is then a threshold, [p ≥ 0.5], which fires exactly when z ≥ 0 — so the set of inputs where the verdict flips is the straight line w·x + b = 0. The sigmoid changed the output, not the geometry: logistic regression is still a linear classifier.

Confidence has a geometry too. Because p depends on x only through z, every point at the same *signed* distance from the boundary gets the same probability — mirror-image points on opposite sides get p and 1 − p — and the "unsure zone" where p ∈ [0.25, 0.75] is a straight strip of width 2·ln3/‖w‖ centered on the boundary. Grow ‖w‖ and the strip narrows; the model gets opinionated faster as you walk away from the line.

Finally the price. Cross-entropy charges each training point −ln p if its label is + and −ln(1−p) if its label is −: confidently right costs almost nothing, hesitantly right costs a little, and confidently *wrong* costs without bound — −ln p blows up as p falls toward 0 on a point whose label says it should be near 1. The mean of these bills, the NLL, is the one number training pushes down.

## Logistic Regression Step by Step

The default run: ten labeled points (five +, five −), pose w = (1, 1), b = 0 — so z = x₁ + x₂ — and query x = (1, 0). One + point, at (−1, −2), sits *inside the − cluster* on purpose — not just on the wrong side of this particular line, but placed so that no straight line whatsoever can classify all ten points. Call it the objector.

1. **Setup.** The ten points and the query appear. A linear classifier would answer + or −; this lesson answers with a number in between.
2. **Score.** z = 1×1 + 1×0 + 0 = 1. Same dot product as always, no scale attached.
3. **Squash.** p = σ(1) = 0.73. The sigmoid inset shows the query riding the curve: z on the horizontal axis, p on the vertical, ½ exactly at z = 0.
4. **Threshold.** [p ≥ 0.5] says +, at a stated 73% confidence — an answer with a number attached. (Whether that number deserves trust is a separate question; the caveat below returns to it.)
5. **Boundary.** Every x with p = 0.5 is drawn: the straight line x₁ + x₂ = 0.
6. **Field.** The plane tints by probability, fading to nothing at the boundary. The p ∈ [0.25, 0.75] band is 1.55 units wide. The objector at (−1, −2) gets an error ring: a + point with p(+) = 0.05.
7. **One bill.** The solid + point at (2, −1) has z = 1, p = 0.73, so it pays −ln(0.73) ≈ 0.31.
8. **The objector's bill.** −ln p ≈ 3.05 with p ≈ 0.047 (shown rounded as 0.05) — about ten times the solid point's. This asymmetry is the whole personality of log loss: right answers pay pennies, confidently-wrong answers pay without bound.
9. **The total.** All ten bills average to NLL = 0.374 — and the objector's 3.05 is most of it.
10. **The experiment.** (w, b) → (2w, 2b). Every z doubles, so the boundary — the set where z = 0 — cannot move, and it doesn't. The band halves to 0.78 units; the arrow doubles; every point slides outward along the fixed sigmoid curve. Same picture, more contrast.
11. **The verdict.** Reprice everything. Nine points got cheaper (the solid point falls 0.31 → 0.13, the query's confidence rises 0.73 → 0.88), but the objector's bill jumps 3.05 → 6.00, and the total *rises*: NLL 0.374 → 0.617. Sharper lost. In this scaling experiment, log loss penalizes increasing confidence because the wrong-side point becomes unboundedly expensive.
12. **Done.** And here "can't fit" is a theorem, not a complaint: the objector sits inside the triangle of three − points — (−1, −2) = ¼(−3, 1) + ¼(1, −3) + ½(−1, −3) — so under *any* (w, b) its score is trapped between theirs, and some point is always wrong. The conflicting labels prevent the strictly separating runaway shown next. In general, a finite unregularized optimum also requires ruling out quasi-separation and parameter non-identifiability; nonseparability alone is not enough.

The `separable` preset moves the objector to (1, 1) and reruns the identical experiment. Now nothing objects: NLL falls 0.082 → 0.019 at ×2, falls again to 0.002 at ×4, and the trace ends on the theorem — every margin is positive, so *every* doubling is a pure win, and no finite (w, b) is optimal. The weights run away toward infinity while the boundary never moves. On screen, the tell is that the picture stops changing: band collapsed to a sliver, every point giving its own label a probability of 0.98 or more, and the one line that actually decides sitting exactly where it started.

## Why Scaling (w, b) Cannot Move the Boundary

Replace (w, b) with (kw, kb) for any k > 0 and every score becomes kz. Whatever sign z had, kz has — so the set z = 0 is identical, every verdict is identical, and the only thing that changed is how hard σ commits on each side. This is worth internalizing because it means accuracy cannot tell the poses apart: on the default data, ×1 and ×2 classify every point the same way. Only the loss can rank them, and it ranks them by how they treat the point they get wrong.

That is also why separable data breaks the training objective. With no wrong-side point to object, the loss rewards contrast forever, and gradient descent on this loss would push ‖w‖ toward infinity chasing a minimum that is a limit, never a point. Regularization adds a price on parameter magnitude and can yield a finite optimum under suitable conditions; a solver's numerical stopping rule remains separate. Where (w, b) comes from at all is the gradient-descent lesson's story; this loss surface is exactly the landscape it descends.

One honest caveat the probability deserves: σ's confidence grows with distance from the boundary, not with evidence. The `far-corner` recipe puts the query at (5, 5), far from every training point, and the model reports p ≈ 1.00 without blinking. A linear scorer still has no "none of the above."

## Edge Cases

- **A query exactly on the boundary** (try x = (0, 0), the `on-the-fence` recipe): z = 0, p = 0.50 exactly. The threshold [p ≥ 0.5] answers + purely by convention; the honest reading is "no idea."
- **w = (0, 0)** is rejected by the input panel: with no direction, p = σ(b) everywhere and there is no boundary to draw.
- **A small ‖w‖** can make the unsure band wider than the visible plane — the band lines clip out of frame. That is a legitimate pose, just a very hesitant one.
- **Any pose that strictly separates its data** triggers the runaway ending. The predicate is "every margin positive," a property of pose plus data, not a dataset label — on the `separable` set, poses other than the preset's earn it too. On the default set no pose can: the objector is inside the − cluster, so the runaway is unreachable there by construction.
- **A wrong-side point far from the line** dominates the total loss on its own; one such point can hold the optimum hostage, which is exactly the behavior the default run demonstrates.

## Common Mistakes

- **Expecting a curved boundary because σ is nonlinear.** The sigmoid is nonlinear in z, but z is linear in x. The boundary is w·x + b = 0, a straight line at every scale of (w, b).
- **Reading p as evidence-backed probability.** p = 0.99 means "far from the boundary on the + side," not "the model has seen lots of data here." Distance is all σ sees.
- **Ranking models by accuracy during training.** Accuracy is invariant under scaling (w, b) and flat almost everywhere; log loss is what actually distinguishes poses and provides gradients. Train on the loss, report accuracy afterwards.
- **Using squared error for classification.** MSE on sigmoid outputs can produce very small score gradients on confidently-wrong points when the sigmoid saturates — the ones training most needs to fix. Cross-entropy's unbounded bill is a feature, not a bug.
- **Forgetting regularization on separable data.** Without a penalty on ‖w‖, the optimizer chases the runaway and the weights explode while the decision boundary gains nothing.