Tokenization

Tokenization is the step that turns raw text into the integer tokens a language model actually reads. Byte-Pair Encoding (BPE) — the algorithm behind GPT-style models — starts from the raw bytes of your text, greedily fuses the adjacent pair with the highest-priority learned merge, again and again, and maps each surviving subword to its vocabulary ID. This lesson runs the real GPT-2 (r50k) tokenizer, in your browser, on whatever you type.

Think of subword tokens as LEGO bricks. No model can stock a brick for every word in every language, but give it fifty thousand well-chosen pieces and any text — a typo, a brand name, an emoji, a word coined this morning — snaps together from parts it already owns. Type tokenization and watch it land as token + ization; type something nobody has ever written and it still encodes. There is no "unknown word" — only smaller known pieces, and underneath them, raw bytes.

How BPE Tokenization Works

Before any merging, the tokenizer splits your text with a regular expression into pieces — roughly, words, runs of digits, and runs of punctuation. A leading space stays attached to the word that follows it, which is why the tokenizer for the is different from the; on the canvas that glued space shows up as the marker Ġ. Merges never cross a piece boundary, so two words never fuse into one token.

Inside each piece, encoding starts pessimistically: every byte is its own token. The tokenizer then consults its ranked merge table — fifty thousand rules, each saying "whenever these two symbols sit next to each other, fuse them" — and the ranking is everything: a lower rank was assigned higher priority in the learned table, so it always fires first.

On every step the algorithm finds the single adjacent pair with the lowest rank anywhere in the piece, fuses just that pair, and looks again. When no adjacent pair is a learned merge, encoding stops and each surviving token is looked up to get its integer ID. Those integers — not the letters — are what the model receives.

The property worth tattooing somewhere: BPE is greedy by rank, not left to right. It fuses the highest-priority pair available anywhere, even if that means working in the middle of a word before its start. And the ranks it applies never decrease — once it has moved past a priority it never comes back, because fusing two pieces can only ever create a longer, later-learned (higher-rank) pair. The merge log fills top to bottom with rising ranks.

Tokenization Step by Step

The default input is the word tokenization. It is one regex piece (no spaces), so it begins as twelve raw byte tokens — for plain ASCII each byte is just its letter:

t o k e n i z a t i o n

Now the tokenizer repeatedly fuses the lowest-rank adjacent pair. Watch where it works — not strictly left to right:

t o k e n i z a t i·o·n   →   fuse (o, n)  → on
t o k e n i z a·t i on    →   fuse (a, t)  → at
t o k e·n i z at i on     →   fuse (e, n)  → en
t o k en i z at i·on      →   fuse (i, on) → ion
t o k en i z at·ion       →   fuse (at, ion) → ation
t o·k en i z ation        →   fuse (o, k)  → ok
t ok en i·z ation         →   fuse (i, z)  → iz
t ok en iz·ation          →   fuse (iz, ation) → ization
t ok·en ization           →   fuse (ok, en) → oken
t oken ization            →   fuse (t, oken) → token
token ization

1. Start. tokenization is twelve byte tokens.
2. Merge, ten times. Each step fuses the single lowest-rank pair. Notice the tokenizer builds the suffix ization (via at → ation → ization) and the stem token in interleaved order — ation is finished before token even begins, because rank, not position, drives the choice. The same suffix ization is a chunk the model reuses across organization, realization, and thousands of other words; that reuse is what the merge table buys.
3. Settle. No adjacent pair — (token, ization) — is a learned merge, so encoding stops.
4. Tokens. The result is [token, ization]: two tokens for a twelve-letter word. Token count is not letter count and not word count.
5. IDs. Each token is looked up in the 50,257-entry vocabulary: token → 30001, ization → 1634. The model receives [30001, 1634] — and those five-digit integers, not the text, are all it ever sees.

On the canvas, the cells carry the story in their width: a single byte is a narrow brick, a fused chunk is a wide one, and each merge slides two cells together into a larger piece while the left panel logs the merge and its global rank.

Why Subwords Beat Words and Characters

A word-level vocabulary is a closed club: any word missing at training time becomes a useless <unk> at inference, and language mints new words faster than anyone retrains. A character-level vocabulary never hits an unknown but pays the opposite price — every word becomes a long string of nearly meaningless symbols and the model burns capacity relearning spelling.

Subwords split the difference, and byte-level BPE takes it all the way: because the atoms are the 256 possible bytes, nothing is ever out of vocabulary. Type the made-up word fizzlepuff and it encodes cleanly into real subwords (f + izzle + puff); type an emoji like 🦄 and it becomes the handful of byte tokens its UTF-8 encoding requires. There is no refusal and no <unk> — only known pieces, sometimes expensively spelled out one byte at a time. Byte coverage is useful, but modern language models use different tokenizer families, vocabularies, and special-token policies.

The cost side is just as real, and the input box is yours to probe it. A common English word is often one token; a rare or misspelled one shatters into many; an emoji or an accented café costs several byte tokens for what looks like one character. Since context windows and API bills are measured in tokens, "tokens ≠ characters ≠ words" is not pedantry — it is the unit your bill is denominated in.

This is the applying half of BPE. Its companion BPE Training lesson is where the merge table is built, from corpus frequencies. That lesson uses a deliberately tiny corpus so you can watch the table grow merge by merge; at GPT-2 scale the same procedure yields 50,257 entries — far too many to show — but every merge you see fire here is applied in exactly that learned rank order. Training molds the bricks; this lesson snaps them together.

Edge Cases

• Nothing is out of vocabulary. A never-seen word (fizzlepuff) decomposes into known subwords; an emoji (🦄) decomposes into raw byte tokens. Byte-level BPE has an atom for every possible byte.
• A space belongs to the next word. the tokenizes differently from the; the leading space is part of the token (shown as Ġ). Merges never cross a piece boundary, so fix the AI keeps fix, Ġthe, and ĠAI separate.
• Case matters. Hello and hello are different byte sequences and can tokenize differently — the tokenizer is not case-folded.
• Rank, not position, decides. When several merges are available the lowest-rank one fires, wherever it sits; the applied ranks only ever rise.
• A single byte is already a token; there is nothing to merge.

Common Mistakes

• Assuming one word = one token. tokenization is two tokens; a short common word is often one and a long rare one is many. Context limits and bills are counted in tokens, not words.
• Merging left to right. BPE is greedy by rank. Fusing the leftmost matching pair first can yield a different — wrong — tokenization.
• Confusing a token with its ID. token is the string; 30001 is its vocabulary index. The model uses the integer ID to look up a learned embedding.
• Expecting universal IDs. A token's ID depends entirely on the vocabulary its tokenizer was trained with — GPT-2's 30001 means nothing to a different tokenizer. There is no standard table.
• Forgetting spaces and case cost tokens too. Whitespace and capitalization change the byte sequence, and therefore the tokens and the bill.

A Note on Simplification

This lesson runs a real GPT-2 (r50k) byte-pair encoder: the 50,257-token vocabulary and every merge are the genuine article, and the token IDs match what the library produces. Other models use different vocabularies, special tokens, and pre-tokenization rules; their sizes and boundaries must be checked for the specific tokenizer. The mechanism you just watched — split into bytes, fuse the lowest-rank pair, map to integers — is the part they scale up.