---
title: "Transformer Block"
description: "The residual stream: each layer adds, never replaces"
source: https://vizlearn.app/ai/nlp/transformer
access: premium (interactive player requires Lifetime; this doc is free)
---

# The Transformer Block

A transformer block is the repeating unit a Transformer stacks many times. It refines a sequence of token vectors with two sublayers — multi-head self-attention and a position-wise feed-forward network — and the key trick is the residual connection: each sublayer's output is *added back* to a running "residual stream" rather than replacing it, with a normalization step feeding each sublayer.

Here is the mental model worth keeping. A block is a committee editing a shared draft. Nobody retypes the page. Each member reads the current draft, scribbles a margin note — a correction — and the note gets folded back in. Attention writes the first note ("this word should lean on *river*"), the feed-forward net writes the second, and the draft moves on to the next committee unchanged except for the notes. Stack enough committees and a vague first draft becomes a precise one, and the original stream remains an explicit term in the sum, although later corrections can cancel or transform its information.

## How the Transformer Block Works

The object flowing through the network is the **residual stream**: one vector per token, carried from the bottom of the stack to the top. A block touches it twice.

First, **attention**. The block makes a normalized copy of the stream, runs self-attention on it (each token mixes in the tokens it attends to), and gets back a correction `Δ_attn` — one per token. Then the load-bearing line: `x = x + Δ_attn`. The stream is not overwritten. It has an identity route plus a learned correction; this is not a guarantee that all input information remains recoverable.

Second, the **feed-forward network**. Same shape: normalize the stream, push each token's vector through a small two-layer net (expand to a wider hidden layer, apply ReLU, contract back), and add the result: `x = x + Δ_ffn`. Attention is the only place tokens talk to each other; the feed-forward net works on each position alone, reshaping that one vector.

That is the whole block — `x = x + attention(norm(x))`, then `x = x + ffn(norm(x))`. A Transformer is this block repeated, the output of one becoming the input of the next:

```
h = embed(tokens) + positional
for _ in range(num_layers):
    h = block(h)
```

Because every sublayer adds rather than replaces, the algebra is clean: after N blocks the stream is the original embedding plus every correction ever made, `x_N = x₀ + Σ Δ`. The input is a literal term in that sum. This lesson shows it pre-norm (normalize *before* each sublayer), a common version among modern Transformers, precisely because it keeps that sum honest — the 2017 paper normalized after the addition, which muddies the running total.

## The Transformer Block Step by Step

The default run follows one word — **bank** in `by the river bank` — up through three blocks, with the residual connection on. The vectors are 4-dimensional so you can read them; think of the dimensions loosely as [geographic, financial, function-word, "bank's own identity"]. Watch dimension 0 (geographic) and dimension 3 (bank's identity).

1. **The input enters the stream.** Bank starts as `x₀ = [0.3, 0.3, 0, 1.0]` — ambiguous, with its identity loaded on dimension 3 and no committed meaning.
2. **Block 1, attention.** Attention has bank look at the sentence; its weights land mostly on *river* (the same routing the attention lesson teaches). The correction comes back as `Δ_attn ≈ [0.7, -0.3, -0.2, 0]` — a push toward the geographic dimension. Add it: the stream becomes `[1.0, 0.0, -0.2, 1.0]`. Note dimension 3 is still `1.0`. Bank is now leaning river-ward but is still itself.
3. **Block 1, feed-forward.** The FFN adds a smaller correction `Δ_ffn ≈ [0.3, -0.1, -0.2, 0.2]`, giving `[1.3, -0.1, -0.4, 1.2]`. Block 1 is done.
4. **Block 2.** The next block repeats the pattern on that stream. Attention pushes the geographic dimension again; after both sublayers the stream is about `[2.3, -0.6, -0.6, 1.4]`. The geographic reading is sharpening; the identity dimension holds near 1.
5. **Block 3.** Once more: the stream finishes at roughly `[3.4, -1.2, -0.8, 1.5]`. Across the three blocks dimension 0 climbed `0.3 → 1.3 → 2.3 → 3.4` while dimension 3 stayed around `1` — the refinement piled up on top of an identity that was never discarded.
6. **The sum.** The final vector equals `x₀` plus the six corrections added along the way (two per block). The depth map draws that sum: the slate `x₀` ribbon runs unbroken to the far end, and the green bar on each block grows with the corrections added so far (`‖stream − x₀‖`). The corrections pile up on top, and the bottom is still in there.

Now flip the **Residual** switch off and replay. Each `add` becomes an overwrite — `x = Δ` instead of `x = x + Δ`. The original `[0.3, 0.3, 0, 1.0]` is thrown away at the first block, and every block after keeps only its own small correction. Crank the depth to 6 and look at the depth map: the ribbon is cut at the entrance, each block's small red bar shows only the size of its latest output, and `x₀` is nowhere in the final vector. That is the failure the residual connection exists to prevent.

## Why the Residual Stream Works

The honest question is: why add instead of replace? Replacing seems simpler, and a single block barely cares — turn the residual off at depth 1 and the result is fine. The difference is **depth**.

Stacking is how a Transformer turns a shallow guess into a deep one. But stacking only helps if information survives the trip. With the residual connection, an explicit identity route exists by construction: `x₀` is a term in `x₁`, which is a term in `x₂`, all the way up — the input has an uninterrupted path to the top, and its sublayer Jacobian has an identity term during training. The total gradient can still shrink, grow, or cancel with the correction derivative. Without it, each block is a fresh function of the last block's output alone; after a few blocks the network is computing a function of a function of a function, and there is no explicit additive identity route. Whether input information is lost depends on those functions, rather than following from composition alone.

One subtlety the animation is careful about. You might expect "no residual" to make the signal shrink to nothing, and in a raw deep net it often does. Here it doesn't, because the normalization step rescales the stream to a stable size before every sublayer. So the magnitude stays put — around `0.4` the whole way down — but it is `0.4` of *the last correction*, not of the input. The washout is about **content**, not size: the stream is alive but amnesiac. That is the more accurate picture of why normalization and residuals are different tools that solve different problems, and why real Transformers use both.

## Edge Cases

- **One block.** Set depth to 1 and the residual barely matters — there is nothing downstream to forget the input. This selected one-block example has a small contrast, but skip connections can affect even one block.
- **The feed-forward net never mixes tokens.** It runs on each position independently. If two tokens need to exchange information, that has to happen in attention; the FFN only reshapes a single token's vector. A common misreading is to expect the FFN to "combine words."
- **Magnitude is not the washout signal.** With normalization in place, turning off the residual does not crash the stream's length — it pins it at the small per-correction scale. The thing that vanishes is the input's content, not its size.
- **Tied weights here, distinct weights in reality.** This toy reuses one set of weights for every block so the refinement reads cleanly. Many Transformers learn distinct attention and feed-forward parameters at each layer, while some architectures share them, which is what lets different blocks specialize.
- **Order still comes from somewhere else.** Attention inside the block is blind to word order; the schematic `+ positional` term represents additive position encoding; other architectures use relative terms or rotations inside attention. That is a separate lesson, and the block itself never looks at position directly.

## Common Mistakes

- **Thinking a block rewrites the token.** Each sublayer proposes a correction and the block *adds* it, so the earlier vector stays in the sum as an identity term. That is refinement, not a preservation guarantee: a correction can still cancel part of what was there.
- **Equating the block with attention.** Attention is one of two sublayers. The block is attention *plus* a feed-forward net, *plus* the residual connections, *plus* normalization — and the residual is the part that makes stacking them work.
- **Treating residuals as a minor optimization.** They are not a speed trick bolted on at the end. Removing them removes the explicit identity route and can make deep optimization harder; it does not prove that every non-residual network loses its input or gradient. They are load-bearing.
- **Confusing normalization with the residual.** Normalization keeps the stream's scale stable; the residual keeps an identity route to its *contents*. Both are in the block, and they are not interchangeable — the no-residual experiment shows a stable-sized stream that has still forgotten its input.
- **Assuming more blocks is always better.** Depth helps only as far as each block has something useful to add, and only because the residual lets the signal survive the climb. This no-skip demonstration loses the selected input identity; it is not a universal claim that all deeper non-residual models learn less.

## A Note on Simplification

The block here is deliberately tiny and curated: 4-dimensional vectors so you can read them, a single attention head (real blocks run many in parallel), a small feed-forward net, hand-set weights *tied across blocks*, and normalization shown as a labeled step rather than computed component by component. Many larger Transformers use multi-head attention and wider model dimensions, a feed-forward width several times larger, dropout, distinct learned weights at every layer, and stacks dozens deep. They also come in encoder, decoder (with a causal mask), and encoder-decoder shapes this lesson does not separate. The shared mechanism is the part you just watched: add a sublayer's output back to a residual stream, normalize, repeat. The numbers are a toy; the architecture is real.