Word Embeddings

A word embedding turns a word into a fixed-length vector: a row of numbers learned from the contexts in which that word appeared. Those numbers are useful because geometry starts to carry linguistic regularities. Words used in similar surroundings tend to point in similar directions, and some changes between words recur as offsets.

The famous example is not king − man + woman = queen as an algebraic identity. The left side produces a new vector, not a word. We then rank every candidate word by cosine similarity to that vector. In the real Stanford GloVe data used here, queen ranks first at 0.8610. That is an empirical regularity in one trained vector space, not a dictionary rule and certainly not a fact about the world.

Think of the offset as an arrow drawn on tracing paper. Draw the arrow from man to king, lift the paper, and place the arrow's tail on woman. Its head lands near queen. The arrow can be moved because subtraction kept the change while discarding the original starting point.

How Word Embeddings Work

GloVe learns one 50-dimensional vector v(w) for each word w. A dimension does not arrive with a human label such as “royalty” or “country.” Meaning is distributed across the coordinates. What matters is how the whole vectors relate.

For a source pair a → b and a new anchor c, this lesson computes:

q = normalize(v(b) − v(a) + v(c))

Then it compares q with each candidate v(w) using cosine similarity:

cos(q, v(w)) = q · v(w) / (||q|| ||v(w)||)

Cosine measures the angle between two vectors rather than their raw lengths. A score near 1 means the two directions are closely aligned. The three input words are excluded from the final ranking; otherwise a term already present in the equation can crowd out the answer we are trying to inspect.

The animation has two views that answer different questions. The scatter plot is a two-dimensional PCA projection, useful for seeing an offset move from one starting point to another. The ranking board is the actual result: its cosine scores use all 50 dimensions. A word that looks close on the page need not be close in 50D, and a true 50D neighbor can look separated after projection.

Word Embeddings Step by Step

The default trace uses { relation: "royalty", anchor: "woman" } and the Stanford GloVe 6B 50-dimensional vectors.

1. Place the anchor. Before applying any relation, the five nearest words to woman are girl (0.9065), man (0.8860), mother (0.8764), her (0.8613), and boy (0.8596). This baseline matters: it tells us what the anchor did on its own.
2. Choose a source relationship. The reference pair is man → king. In the projection, the arrow from one point to the other is the visual stand-in for a change spread over all 50 coordinates.
3. Subtract the starting word. Compute v(king) − v(man). This is the offset. Subtraction removes the original location and leaves a reusable direction.
4. Align the offset. Slide the same arrow until its tail starts at woman. The arrow does not rotate or change length; only its origin changes.
5. Translate. Add the anchor: v(king) − v(man) + v(woman), then normalize the result. The new point is a query vector. It is not itself a stored word.
6. Score candidates. Compute cosine similarity between the query and every candidate in the certified browser subset, using all 50 dimensions.
7. Rank. Exclude king, man, and woman, then sort descending. The top five are queen (0.8610), daughter (0.7685), prince (0.7641), throne (0.7635), and princess (0.7513).
8. Compare with the baseline. The nearest word changed from girl to queen. The anchor and cosine rule stayed fixed; applying the offset was the intervention.

That before/after comparison is more informative than the celebrated final word alone. Without it, queen can feel like a lookup trick. With it, you can see exactly what the offset changed.

Now remove the subtraction as a controlled counterfactual. king + woman ranks father first at 0.8827, not queen. The − man term is not decorative: it cancels the source position so the relationship can be moved.

What the Other Recipes Prove

The capital recipes reuse one source direction, paris − france, and change only the anchor:

This is the strongest evidence in the lesson. One offset transfers to three independently chosen countries, and each run recomputes the same arithmetic and ranking. The result is not guaranteed for every country or every embedding model, but it is more than a memorized one-off.

The Bias audit recipe is equally important. It computes doctor − man + woman. The same valid procedure ranks nurse first at 0.8424, followed by child, pregnant, mother, and patient. The arithmetic succeeded. The semantic conclusion is unsafe.

An embedding compresses patterns in a corpus, including its stereotypes and unequal associations. A high cosine score means “these vectors align in this trained space.” It does not mean “this association is fair,” “this person belongs in this role,” or “the world ought to be this way.” The failure case is not a broken corner of the method; it is evidence about what the method learned.

Why Offsets Can Carry Relationships

GloVe is trained from co-occurrence statistics. If two word pairs differ in similar ways across many contexts, training can place their differences in similar directions. Country names and capital cities occur in recurring constructions; gendered and family terms occur in recurring constructions too. Vector subtraction exposes those repeated changes.

But nothing in the objective names a relationship or enforces symbolic consistency. The model never stores a capitalOf function. An offset is a broad statistical tendency distributed through 50 coordinates. It works when the corpus supplied a regular pattern and the training geometry preserved it. It becomes noisy when words are rare, senses are mixed, or the proposed relation was never consistent in the data.

Edge Cases

• One vector has to carry every sense. Static GloVe gives bank one vector whether the sentence concerns a river or a loan. The two senses are averaged into one location. Contextual embedding models address this by producing a new token vector from the surrounding sentence.
• A valid equation always returns something. Cosine ranking cannot say “this relation makes no sense.” Try doctor − man + japan: the system still produces a top word, because sorting a list always has a first item. A result is not automatically an analogy.
• Source terms can dominate the list. That is why the source, target, and anchor are excluded. Exclusion is part of the evaluation protocol, not vector arithmetic.
• Projection can reverse visual intuition. PCA preserves as much variance as two axes can hold, not every local neighborhood. Trust the 50D cosine board for rank.
• Ties need a stable rule. The exporter keeps source-vocabulary order as the tie-breaker, so playback, quizzes, and tests remain deterministic.
• Out-of-vocabulary words have no vector. This lesson uses a closed certified set. Subword and contextual models can construct representations for many words a word-level GloVe vocabulary never saw.

Common Mistakes

• Calling the equation an equality between words. king − man + woman is a vector. queen is the nearest stored word after a search.
• Doing the arithmetic in the 2D picture. The picture illustrates motion; all scores and ranks use the original 50 dimensions.
• Using Euclidean distance while reading cosine scores. Cosine normalizes length. A visually longer vector is not automatically a worse or better match.
• Skipping the baseline. Without woman → girl before translation, you cannot tell what changed when queen appears.
• Treating analogy success as reasoning. The model is not applying a logical rule. It is exposing a regularity learned from word co-occurrence.
• Treating bias as a data-cleaning footnote. A biased neighbor can affect search, clustering, retrieval, and downstream decisions. The output must be interpreted and audited, not merely celebrated when the arithmetic is elegant.

A Note on Simplification

The source is Stanford GloVe 6B 50d: vectors trained on Wikipedia 2014 plus Gigaword 5, about 6 billion tokens, with a 400,000-word uncased vocabulary. The browser does not ship all 400,000 vectors. An offline exporter runs every supported baseline, analogy, and counterfactual against the complete vocabulary, retains the full-vocabulary top 64 for each query, adds background landmarks, and takes their union: 768 words in this build. Runtime ranking is recomputed from those real 50D vectors, and tests assert that every supported top five is identical to the complete source ranking. The compressed generated payload is about 135 KiB.

The 2D coordinates are PCA over that certified subset, not the original GloVe training space. They are deliberately labelled “geometry intuition.” The ranking board, quiz answers, status text, and documentation numbers all come from the 50D trace. GloVe is also a static, uncased, word-level model; modern language models use subword tokens and contextual vectors, often hundreds or thousands of dimensions wide. This smaller model is used because its arithmetic is transparent enough to inspect without changing the phenomenon.