---
title: "Positional Encoding"
description: "Add sine/cosine position vectors to repeated tokens and compare the inputs with encoding on and off."
source: https://vizlearn.app/ai/nlp/positional-encoding
---

# Positional Encoding

Positional encoding supplies a transformer with information about where tokens occur. The original sinusoidal method assigns each position a vector of sine and cosine values at different frequencies, then adds that vector to the token embedding. Two occurrences of the same token can therefore enter the network with different input vectors.

Consider the two occurrences of `the` in `the cat saw the dog`. A static embedding lookup gives them exactly the same vector. Their slot indices differ: one is at position 0, the other at position 3. Positional encoding gives the model a way to represent that distinction before it starts mixing information across tokens.

## How Positional Encoding Works

Write the token embedding as `E(the)`. A lookup depends on the token's identity, so moving that token to another slot does not change the lookup result. Let `A(p)` be the position contribution actually added at slot `p`. The input is

```text
x(p) = E(the) + A(p)
```

With sinusoidal encoding enabled, `A(p) = PE(p)`. In the controlled experiment with encoding disabled, `A(p) = 0`. The formula for `PE` stays the same in both cases; the switch decides whether its result enters the input.

The original construction uses adjacent sine/cosine pairs:

```text
PE(p)[2i]     = sin(p / 10000^(2i/d))
PE(p)[2i + 1] = cos(p / 10000^(2i/d))
```

Here `p` is the position, `d` is the embedding width, and `i` indexes a pair of components. Angles are in radians. The even component uses sine and the odd component uses cosine of the **same angle**. Both components remain between −1 and 1.

The animation uses `d = 8`, giving four pairs. Their denominators are `1`, `10`, `100`, and `1000`: each successive pair advances more slowly as position increases. Dividing position changes the frequency, not the amplitude. These are different rates of change within one position vector, not separate position numbers assigned to different tokens.

The position vector and token embedding have equal width, so they can be added component by component. Eight components plus eight corresponding components yields eight sums. Appending the position vector would instead produce sixteen components and require a different input architecture. The original Transformer also scales the token embeddings by `√d` before this addition; `E` here includes that shared scaling. [Attention Is All You Need, §§3.4–3.5](https://arxiv.org/html/1706.03762v7#S3.SS5).

## Positional Encoding Step by Step

The default run compares `the` at positions **0 and 3**, with sinusoidal encoding enabled and eight components. The animation inspects these two occurrences; it leaves the intervening tokens out of the workbench.

First, keep only the lookup vectors. Both are `E(the)`, and subtracting them gives zero. There is no need to choose numerical embedding weights to establish this: identical vectors cancel regardless of their coordinates.

For pair 0, the denominator is 1. Positions 0 and 3 become angles 0 and 3. Sampling sine gives `0.000000` and `0.141120`; sampling cosine gives `1.000000` and `−0.989992`. Notice that the position-zero pair is **[0, 1]**, not [0, 0]. Cosine contributes even when the position index is zero.

Pair 1 divides the same positions by 10. The angles are now 0 and 0.3. The sine components are `0.000000` and `0.295520`; the cosine components are `1.000000` and `0.955336`. The wave has slowed down while the position index and amplitude range stay fixed. Pairs 2 and 3 apply the same rule with denominators 100 and 1000.

| Component | Function | PE(0) | PE(3) |
|---|---|---:|---:|
| 0 | sin(p) | 0.000000 | 0.141120 |
| 1 | cos(p) | 1.000000 | −0.989992 |
| 2 | sin(p/10) | 0.000000 | 0.295520 |
| 3 | cos(p/10) | 1.000000 | 0.955336 |
| 4 | sin(p/100) | 0.000000 | 0.029996 |
| 5 | cos(p/100) | 1.000000 | 0.999550 |
| 6 | sin(p/1000) | 0.000000 | 0.003000 |
| 7 | cos(p/1000) | 1.000000 | 0.999996 |

Now add the matching position vector to each occurrence's embedding. The first coordinate is `e₀ + 0.000000` at position 0 and `e₀ + 0.141120` at position 3. Every other coordinate follows the same addition rule.

Subtract the resulting inputs:

```text
x(3) − x(0)
  = (E + PE(3)) − (E + PE(0))
  = PE(3) − PE(0)
```

The shared embedding cancels. The length of the remaining eight-component difference is approximately **2.017479**, calculated before display rounding. Every part of that difference came from position.

Here "length" means the square root of the sum of the eight squared differences. Zero means all eight coordinates match; a positive length means at least one differs. You do not need to interpret 2.017479 as a score: the experiment asks whether the two inputs are identical. It is not the three-slot gap between these occurrences.

Finally, compare the disabled case. Both inputs become `E + 0`, so their difference has length **0**. The default run leaves these two results side by side when it ends. The Encoding control and Compare position slider let you reproduce the comparison at any integer position from 1 through 32; recipes jump directly to the relevant evidence.

## Several Frequencies Preserve Distinctions One Pair Makes Small

A sine/cosine pair travels around a circle as its angle advances. After a full turn of `2π` radians, that continuous pair repeats. Integer token indices generally do not land exactly on a complete turn, but they can come close.

Set the comparison position to **6**, or choose **Fast pair nearly repeats**. The fastest pair has almost completed one turn: its distance from the position-zero pair is about **0.282240**. The next pair has advanced only 0.6 radians, and its distance is about **0.591040**. Looking at both pairs retains distinctions that one frequency makes small. The slower pairs add further scales.

This is not a guarantee that greater separation produces greater vector distance. In fact, the full difference length at position 6 is about **0.657741**, smaller than the **2.017479** at position 3. Oscillation is part of the construction. Do not read the norm as the number of intervening tokens or as a measure of semantic similarity.

For a fixed relative offset, trigonometric angle-addition identities also express each shifted sine/cosine pair as a linear transform of the original pair. This gives a model structure it could learn to use for relative positions. Availability of that structure is weaker than a guarantee about what a trained model learns.

## Time and Space Complexity

Materializing the encodings for `n` positions and width `d` takes **O(nd)** time and **O(nd)** space: one sine or cosine value per component. A vector for one requested position requires O(d) work. Sinusoidal encoding has no learned position table; it evaluates a fixed formula. These costs cover the encoding itself, not the later attention computation.

When implementing this additive method, assign `PE` by the token's slot and keep its width equal to the embedding width. A useful check is to place the same token in two slots, subtract the resulting inputs, and compare the result with `PE(q) − PE(p)`. This isolates the position path even when the embedding weights are unknown. It also exposes the mistake of broadcasting one position vector to every slot.

## Edge Cases

- **Encoding disabled.** Repeated tokens have identical input vectors in this experiment. The unused candidate `PE` can still be computed and inspected; disabling its addition does not alter the formula.
- **Position zero.** Every sine component is zero and every cosine component is one. Treating `PE(0)` as the zero vector is a different encoding.
- **Small displayed differences.** Slow cosine components can round to the same displayed value. Computation, norms, and comparisons use full precision; the focused sample shows additional decimals.
- **Longer contexts.** The formula can be evaluated beyond a training sequence's length. That does not guarantee useful behavior there. The slider's maximum of 32 is a display bound, not an encoding limit.
- **Other sources of order.** Unmasked self-attention without any position signal is permutation-equivariant: permuting its input rows permutes its output rows. A causal mask supplies ordering structure, and other architectures use rotary encodings or attention biases. Absence of this particular additive encoding does not establish that a model has no ordering information.

## Common Mistakes

- Confusing a component index with a position index. `p` selects a token slot; `2i` and `2i + 1` select two coordinates inside that slot's vector.
- Giving sine and cosine different denominators within a pair. The pair shares one frequency.
- Multiplying the wave's amplitude by the position index. The original formula changes the angle; its components remain bounded.
- Adding the same `PE(0)` to every token. A common added vector cancels when comparing identical tokens, just as a common embedding does.
- Concatenating position components while leaving the next layer's input width unchanged. Concatenation needs a compatible architecture; this lesson uses addition.
- Concluding that different input vectors prove correct understanding of word order. The calculation establishes an available position signal. What the network does with it depends on the rest of the model and its training.

## A Note on Simplification

This is an explainer of the original additive sinusoidal formula, not a substitute for formal study of transformers. It uses eight dimensions and two inspected occurrences, keeps the shared token embedding symbolic, and omits attention, training, normalization, and dropout. No semantic prediction is made. The [Word Embeddings lesson](https://vizlearn.app/ai/nlp/word-embeddings) examines learned vector geometry; [Attention Mechanism](https://vizlearn.app/ai/nlp/attention-mechanism) examines token interactions; [Transformer Block](https://vizlearn.app/ai/nlp/transformer) follows the later residual updates.