---
title: "Naive Bayes"
description: "Classify by multiplying per-feature probabilities — naive, fast, surprisingly strong."
source: https://vizlearn.app/ai/classical-ml/naive-bayes
---

# Naive Bayes

Naive Bayes is a probabilistic classifier that scores a document by multiplying one probability per feature — the chance of seeing that word in each class — and returning whichever class comes out ahead. The "naive" part is an assumption it knows is false: the words are independent given the class. Training is one counting pass; prediction is one multiplication per word.

Think of it as a jury where every juror votes without hearing any of the others. Each word is a juror, and the weight of its vote was fixed during training: `now` votes for spam about 3.7 times as hard as it votes for ham, forever, no matter what the rest of the email says. Nobody confers. That is what makes Naive Bayes fast enough to have run the world's spam filters for a decade — and it is also why, when two of your jurors are the same person in a wig, nothing in the algorithm notices.

## How Naive Bayes Works

Bayes' rule says the probability a document `d` belongs to class `c` is proportional to `P(c) · P(d | c)`. The second term is the hard one: the number of distinct documents is astronomically large, so no corpus is big enough to estimate `P(d | c)` directly. Naive Bayes takes the assumption that lets you estimate it anyway — that each word is drawn independently given the class — and factors it apart:

`P(c | d) ∝ P(c) · P(w₁ | c) · P(w₂ | c) · … · P(wₙ | c)`

Now every term is something you can count. `P(c)` is how often that class appears in the training set. `P(w | c)` is how often word `w` appears among the tokens of class `c`. Both come from one pass over the training data, which makes the counting pass inexpensive relative to iterative training in many settings.

Two practical corrections turn that formula into working code. First, **smoothing**. If a word never appeared in one class, its raw probability there is exactly zero, and a product containing a zero is zero — one word would veto every other word in the document. Laplace (or Lidstone) smoothing adds a pseudo-count `α` to every word in every class: `P(w | c) = (count(w, c) + α) / (total_c + α·|V|)`. Nothing is ever impossible, only unlikely.

Second, **logs**. Long documents multiply many probabilities below one and can underflow; the length at which this happens depends on the probabilities and numeric precision. So real implementations add logarithms instead: `score(c) = log P(c) + Σ log P(w | c)`, and compare scores rather than probabilities. The ranking is identical, because `log` is monotonic.

That log form is what the animation draws. Its axis is the odds of spam against ham on a **log scale**, which turns the multiplication into geometry: each word displaces the marker by a fixed distance in its own direction, and that distance is the same wherever the marker sits, as long as both ends stay on the drawn axis. Watching a word push the marker *is* watching a multiplication.

## Naive Bayes Step by Step

The lesson trains on eight emails — three spam, five ham — and classifies `free money meeting now` with `α = 1`. The vocabulary is 23 distinct words; spam contributes 12 tokens in total, ham contributes 20.

1. **The prior.** Before reading a word, the model already has an opinion: 3 of the 8 training emails are spam, so the odds start at 3 : 5, a factor of ×0.60 *against* spam. The marker lands left of centre. This is not a formality — train on a corpus that is 99% ham and every email begins 99 : 1 in the hole.
2. **`free`.** It appears 3 times among the spam tokens and once among the ham tokens (the eighth training email is `free coffee in the kitchen`, which is about a kitchen). Smoothed: `P(free | spam) = (3 + 1) / (12 + 23) = 0.1143` and `P(free | ham) = (1 + 1) / (20 + 23) = 0.0465`. The ratio is **×2.46** toward spam, and the running odds go from 0.60 to 1.47 : 1. Note what did *not* happen: `free` is not a magic spam word. It is a word with a weight.
3. **`money`.** Three spam emails, one use, zero uses in ham. Smoothed: `0.0571` against `0.0233`, another **×2.46**, and the odds reach 3.62 : 1. A word that never occurs in ham still gets a finite weight — that is smoothing doing its job.
4. **`meeting`.** Zero uses in spam, two in ham. `0.0286` against `0.0698` gives **×0.41**, a factor below 1, so it multiplies the odds *down*: 3.62 : 1 falls back to 1.48 : 1. The marker moves left. This is the beat that shows the axis is a tug-of-war and not a progress bar.
5. **`now`.** Twice in spam, never in ham: `0.0857` against `0.0233`, the strongest factor in the email at **×3.69**. The odds finish at **5.47 : 1**.
6. **The verdict.** `score(spam) = −12.02` beats `score(ham) = −13.72`, so the answer is spam. Converting the odds gives `P(spam | doc) = 0.8454`. The `argmax` only ever needed the *sign* of the gap; the probability is the part that is easy to over-trust — which is what the contrast beats that follow are about.

## Why a Repeated Word Inflates the Confidence

The default run does not stop at the verdict. It rewrites the same email as `free money meeting now now now` — the strongest word repeated, not one new claim about the message — and scores it again.

Each extra copy is a fresh juror casting the same ×3.69 vote, because the model was told the tokens are independent given the class. Two of them shift the odds twice more: **5.47 : 1 becomes 74.28 : 1**, posterior 0.8454 → 0.9867. The spam-to-ham odds are multiplied by about 13.6, and the second and third `now` are perfectly correlated with the first — which is the one thing an independent witness cannot be.

The argmax survived; it was spam before and it is spam now. The confidence did not. This is the concrete price of the word "naive": the independence assumption is what lets you estimate `P(d | c)` at all, and it is also what makes a duplicate count as new evidence. In the animation, watch the two extra `now` chips arrive in the document strip and the marker take two more identical steps right, each exactly as long as the first one.

It generalizes past literal duplicates. Any two features that move together — `free` and `offer`, a word and the bigram containing it, the same term in the subject line and the body — get multiplied in as if each were fresh, and the product runs away from anything a frequency could justify. Correlated evidence can make the model posterior poorly calibrated. Both ranking quality and calibration still need evaluation.

## Time and Space Complexity

| Phase | Time | Notes |
|---|---|---|
| Train | O(N · L) | One pass over N documents of average length L, incrementing counters |
| Predict | O(\|C\| · L) | One lookup and one addition per word per class |
| Space | O(\|C\| · \|V\|) | A count for every word in every class |

Prediction does not depend on the size of the training set at all, only on the length of the document and the number of classes. That is the practical reason Naive Bayes outlived far more accurate models in production spam filtering: a mail server can afford one hash lookup per word, and cannot afford much more.

## Edge Cases

- **A word in neither class.** It is not in the vocabulary, so it is not a feature. The model drops it and the odds do not move — type `urgent invoice attached` into the panel and every token is discarded, leaving the prior alone as the answer. The result looks confident (ham at 62.5%) and contains no information about the email whatsoever.
- **`α = 0`.** Set the slider to zero and the default email breaks the classifier outright: `money` never occurred in ham, so ham's product is zero; `meeting` never occurred in spam, so spam's product is zero too. `argmax` is left comparing `−∞` to `−∞`. Real code does not crash — it returns whichever class it visited first, which is a property of dictionary ordering, not of the email.
- **α too large.** At `α = 5`, each class absorbs 115 pseudo-counts against 12 and 20 real tokens. The same email's odds collapse from 5.47 : 1 to 1.23 : 1, a coin toss. Smoothing is a dial between trusting your data and trusting nothing.
- **An exact tie.** This trace resolves a tie to `ham`, the alphabetically first class. Leaving the tie-break to dictionary order is a real (if rare) source of non-reproducible predictions.
- **A repeated word.** Multinomial Naive Bayes counts every token, so `now now now` votes three times. Scoring each distinct word once (binarized multinomial NB) would treat it identically to one `now`; Bernoulli Naive Bayes also scores presence, but it estimates and scores words differently, so its numbers are not the same.

## Common Mistakes

- **Reading the posterior as a confidence.** `P(spam | doc) = 0.99` from a Naive Bayes model does not mean 99 emails out of 100 like this one are spam. It means the product of independent-ish factors landed at 99 : 1, and correlated features inflate that product without adding evidence. Neither useful ranking nor calibration is guaranteed; evaluate both on held-out data.
- **Smoothing the numerator but not the denominator.** `(count + α) / total_c` is not a probability distribution — it does not sum to 1 over the vocabulary. The denominator has to grow by `α·|V|` too, or every class's "probabilities" quietly sum to more than one and the comparison between classes tilts toward whichever class has fewer training tokens.
- **Adding unseen words as count-0 features.** It looks harmless and symmetric. It is not: an unseen word gets `α / (total_c + α·|V|)` in each class, same numerator over different denominators, so the class with fewer training tokens wins every unknown word. Typing gibberish would then drift the answer in a fixed direction for no reason. Drop out-of-vocabulary tokens instead.
- **Multiplying raw probabilities in production.** It works on the four-word examples in a tutorial and silently underflows to `0.0` on a real email, at which point every class ties at zero. Sum logs.
- **Assuming a low count means low weight.** `money` appears exactly once in the training set and carries the same ×2.46 as `free`, which appears four times (three in spam, once in ham), because that one ham occurrence pulls `free`'s ratio down. The weight is a *ratio* between classes, not a frequency.

## When to Reach for It

Naive Bayes is the right *first* model for a text problem and rarely the right last one. It trains in one pass, predicts in one lookup per word, has a smoothing parameter in this lesson, along with preprocessing and modeling choices in practice, and hands you an accuracy number the next model has to beat. Reach for something else when the number itself has to be trusted.

| Situation | What to use |
|---|---|
| Longer documents where how often a word appears is signal | Multinomial NB — the model in this lesson |
| Short text where repetition is noise: subject lines, chat messages, log entries | Presence features — binarized multinomial NB scores each distinct word once (exactly the fix for the shouting email above); Bernoulli NB also scores presence and absence |
| Continuous features (measurements, sensor readings) | Gaussian NB — same independence assumption, a normal density per feature instead of a count |
| You need the probability, not just the label | Keep NB as a score and calibrate it on held-out data (Platt scaling, isotonic regression) |
| Features are correlated by construction: n-grams, TF-IDF over overlapping vocabularies, the same field twice | Logistic regression — it fits the weights *jointly*, so a redundant feature can be discounted instead of counted twice |

The last row is the one this lesson argues for. Naive Bayes cannot discount a redundant feature, because it never looks at two features at the same time.

## A Note on Simplification

This lesson runs a real multinomial Naive Bayes — every count, probability, and score on screen is computed from the eight emails, with no hand-written numbers anywhere — but eight emails is not a spam filter. A deployed one tokenizes headers and URLs as well as body text, uses character n-grams so that `fr33` does not read as an unseen word, carries tens of thousands of features, weights them with TF-IDF or feature selection, and retrains continuously against an adversary who is reading its outputs. It is an explainer for the algorithm, not a substitute for the engineering around it. The failure modes it shows you — zero probabilities, over-smoothing, and correlated features inflating confidence — are the real ones, at a scale you can check by hand.