BPE Training

BPE training is the procedure that learns a subword vocabulary from raw text. Starting with every word split into single characters, Byte-Pair Encoding repeatedly counts adjacent symbol pairs across the corpus — weighted by each word's frequency — merges the most frequent pair into a new token, and records that merge as a ranked rule. The rules become the tokenizer.

Medieval scribes did this by hand. They wrote the Latin et so often that they fused its letters into a single stroke — the ampersand — and the abbreviation stuck because frequency earned it. BPE training is a scribe with a tally counter: it watches which two symbols keep landing next to each other, mints a ligature for the most frequent pair, and repeats. Nothing is designed; everything is counted. Do this lesson first to see where the rules come from, then switch to the companion Tokenization lesson, which applies a merge table — there, at real GPT-2 scale (50,257 entries) — to encode any text you type. This lesson uses a tiny corpus so the whole table fits on screen; the mechanism is identical at either size.

How BPE Training Works

The corpus is a list of words, each with a frequency. Every word begins as a row of single characters. Each iteration makes three moves:

1. Count every adjacent pair of symbols across all words, weighting by word frequency. A pair inside a word seen 12 times contributes 12 to the tally, not 1.
2. Pick the most frequent pair.
3. Merge it everywhere it appears — across the whole corpus at once — and record the rule.

The recorded rules are ordered: the first merge learned is the highest-priority rule, and that ordering is part of the trained tokenizer, because the encoder later replays the rules by rank. The loop runs for a fixed number of merges here; in production you stop when the vocabulary reaches a target size.

BPE Training Step by Step

The default run learns 4 merges from this corpus:

1. Start. Every word splits into characters: h u g, p u g, p u n, b u n, h u g s.
2. Merge 1. The tally finds (u,g) in hug (10) + pug (5) + hugs (5) = 20, the highest count. The scribe mints ug, and it fuses in every word that contains it: h ug, p ug, p u n, b u n, h ug s.
3. Merge 2. Recount on the new symbols. (u,n) scores pun (12) + bun (4) = 16 and wins. After fusing: h ug, p ug, p un, b un, h ug s. Note what almost happened here — (h,ug) scored 15, one behind. Frequency weighting decided it.
4. Merges 3 and 4 repeat the pattern: (h,ug) wins at 15 (from hug ×10 and hugs ×5), then (p,un) wins at 12 (from pun alone). The corpus now reads hug, p ug, pun, b un, hug s.
5. Done. The learned rules, in priority order: ug, un, hug, pun. Together with the seven base characters b g h n p s u, they form an 11-token vocabulary.

The tally panel mid-canvas is the thing to watch: each count beat ranks the candidate pairs and lights the winner, and on the following merge beat that pair fuses simultaneously in every word that contains it — one rule, corpus-wide. The growing list on the right is a merge table — the same kind of artifact the Tokenization lesson applies, there at real GPT-2 scale.

How Many Merges Should BPE Learn?

The default run learns four — the four whose winners are unambiguous. But set the merge count to 5 and you walk straight into the question every real implementation must answer: the fifth tally is a tie. (hug,s) and (p,ug) both score exactly 5, and "pick the most frequent pair" no longer picks anything. The animation gives the moment its own beat: both candidates light up side by side at 5, and then the rule decides — this implementation breaks ties alphabetically, so (hug,s) wins and hugs joins the table as rule #5. The specific rule is a convention (training implementations can use different fixed tie orders; this demonstration sorts alphabetically); what matters is that it is fixed. Any deterministic rule keeps training reproducible — no rule at all, and two training runs over the same corpus could learn different vocabularies.

Real training answers the question differently: you choose a target vocabulary size — a hyperparameter, typically tens of thousands of tokens — and keep merging until base symbols plus learned merges reach it. "How many merges" falls out of that choice rather than being the knob itself. Too few merges and common words stay shattered into characters; too many and the vocabulary fills with corpus-specific oddities that rarely recur. Production systems may use much larger corpora, byte-level or other base symbols, pre-tokenization boundaries, and reserved special tokens. The loop you watched — count, pick, fuse, record — is the same loop, scaled up.

Edge Cases

• Frequency ties. Set the merge count to 5 and watch one happen: (hug,s) and (p,ug) level at 5, and the fixed alphabetical rule decides. Real tokenizers settle millions of such ties the same deterministic way — every tie needs a tiebreaker, or the table would not be reproducible.
• A pair spans many words. (u,g)'s count of 20 comes from three different words. Counting is corpus-wide, and frequencies add.
• Merging fuses every occurrence. When a rule fires, it replaces the pair in all words at once, rather than stopping at the first match.
• Order is part of the model. Learn the same merges in a different order and the encoder produces different tokenizations. The ranking ships with the rules.
• A merged symbol is a first-class citizen. ug immediately participates in later counts — that is how (h,ug) becomes a candidate at all. Merges compose.

Common Mistakes

• Counting pairs unweighted. Word frequency belongs in the count, not in a tiebreaker. Unweighted, merge 2 is itself a tie — (u,n) and (h,ug) each appear in two words — and only the weighting (16 vs 15) settles it. Drop the weights and the learned table changes.
• Merging inside one word only. A rule is global. Fusing the pair in a single word leaves the corpus inconsistent and the counts wrong from then on.
• Confusing training with encoding. Training produces the ranked rules from a corpus; encoding replays them on new text. Same idea, opposite direction — and the two lessons split exactly along that line.
• Expecting whole words as tokens. With the default four-merge budget, even the corpus word hugs is not a single token; the fifth merge makes it one. The vocabulary holds only the pieces the loop had budget to learn, which is why the encoder later renders hugs as hug + s.

A Note on Simplification

The corpus here is a five-word teaching corpus with 36 total word occurrences, with characters as base symbols and an alphabetical tie-break. Real BPE training uses the same broad count-pick-fuse-record idea with implementation-specific base symbols, boundaries, overlap handling, and reserved tokens — and it stops at a target vocabulary size instead of a merge count. Nothing in the loop itself changes with scale.