Batch Normalization

Batch normalization, usually shortened to BatchNorm, standardizes each hidden feature channel with training-group statistics, then applies a learned scale γ and shift β. In this fixed six-layer experiment, it holds the epoch-50 activation spike to 9.15 instead of 64.35 and cuts held-out BCE from 0.6911 to 0.0985.

The useful question is not whether those two runs differ; the opening frame gives that away. It is where the difference comes from, and where it stops helping. You will first follow one channel value by value, then hold every optimizer update at 32 examples while changing only the number of values behind each BatchNorm estimate.

Batch Normalization Step by Step

One Channel, Four Transformations

Suppose a fully connected layer produces one feature channel with values z₁, …, zᴺ for a statistics group of Nstats examples. We keep the conventional subscript B for a BatchNorm statistic, while Nstats names how many values support it. BatchNorm computes the group mean

μᴮ = (1 / Nstats) Σᵢ zᵢ

and the population-form batch variance

σ²ᴮ = (1 / Nstats) Σᵢ (zᵢ − μᴮ)².

It standardizes each value with

ẑᵢ = (zᵢ − μᴮ) / √(σ²ᴮ + ε)

and applies the learned affine transform

yᵢ = γẑᵢ + β.

The lesson uses ε = 10⁻⁵. This small constant is not cosmetic. If every value in a channel is identical, the variance is zero; without ε, the forward pass would divide by zero and produce non-finite values. The same denominator also appears in the backward pass.

The animation selects layer 4, channel 1 and keeps every example in the current statistics group visible as a dot. Centering moves the whole cloud so its mean is zero. Standardization changes its spread so the variance approaches one. The affine step starts at γ = 1, β = 0, an identity transform, and both parameters then learn with the rest of the network. ReLU acts after that affine output.

How Batch Normalization Chooses an Axis

For a fully connected layer, each feature channel gets its own μᴮ, σ²ᴮ, γ, and β. The samples used to estimate those statistics are the examples in its current statistics group.

This axis is part of the definition. Normalizing every example across all its channels describes a different family of methods, such as layer normalization. Computing one global mean for every channel and every layer would also be a different operation. “Normalize the activations” is incomplete unless the axes are named.

Training and Evaluation Use Different Statistics

During training, the current statistics group supplies the mean and variance used to normalize its values. The layer also updates running estimates. To keep the boundary experiment controlled, this lesson updates that ledger once per 32-example optimizer batch from the full batch statistics. With momentum 0.9, it uses

μrun ← 0.9 μrun + 0.1 μopt(32)

and the same update form for variance. μopt(32) denotes the mean over the full optimizer batch, not one smaller training statistics group.

During evaluation, BatchNorm uses the accumulated running mean and variance. It does not compute fresh statistics from one held-out example, and it does not depend on whichever other examples happen to be evaluated beside it. This makes ordinary inference deterministic.

Switching modes incorrectly is a common source of surprising results. Evaluation with live group statistics measures a different procedure from the inference network that will actually be deployed.

You have now seen how one layer constrains a distribution. The next question is whether that constraint changes a six-layer training trajectory.

The Matched Deep-Network Experiment

Both lanes train a 2 → 12 × 6 → 1 ReLU network on 64 deterministic two-moons training points and evaluate on 400 held-out points. The output uses a sigmoid with binary cross-entropy. Training uses mini-batch SGD with exactly 32 examples per optimizer update. Initial weights are sampled from a fixed Gaussian with standard deviation 2.5 / √fan_in; this deliberately makes the six-layer run sensitive to activation scale.

The default configuration is optimizer B = 32, Nstats = 32, η = 0.1. At epoch 50, the unnormalized lane's largest raw hidden pre-activation has reached 64.35. Its held-out BCE is 0.6911, essentially a stalled classifier. The matched BatchNorm lane contains its largest raw pre-activation at 9.15 and reaches held-out BCE 0.0985.

By epoch 300, the BatchNorm lane reaches held-out BCE 0.0572 and 98% held-out accuracy. The unnormalized lane later recovers enough to reach held-out BCE 0.0901, but the epoch-50 checkpoint exposes the unstable path it took. BatchNorm did not merely shift the final score; it changed the training trajectory.

This is one controlled experiment, not a theorem that every network without BatchNorm must diverge. Initialization, residual paths, optimizer choice, activation function, data scale, and architecture all influence the result.

Why Batch Normalization Statistics Group Size Matters

BatchNorm estimates a mean and variance. With only two examples, those estimates can swing sharply even when the underlying activation distribution has not changed. The animation draws eight seeded resamples from one frozen channel pool: the Nstats = 2 means occupy a much wider range than the Nstats = 32 means.

The training comparison then fixes optimizer batch size at 32, uses the same two optimizer updates per epoch, and changes only the BatchNorm statistics group. At epoch 300, Nstats = 32 reaches held-out BCE 0.0572 and 98% accuracy. Nstats = 2 reaches 2.3795 and 37.5%. This isolates training-time estimate granularity from ordinary small-batch gradient noise and update count.

This controlled result is not a universal cutoff at 32. It shows the mechanism behind a practical warning: when BatchNorm has too few examples behind each estimate, its training transform can become erratic. Workloads that cannot supply representative groups often use layer normalization or group normalization, which estimate statistics over different axes.

Reading the Controls

The statistics-group control offers 2, 8, and 32. It does not change optimizer batch size: every update still uses 32 examples, and each epoch still contains two updates. Only the BatchNorm treatment lane partitions those 32 raw activations into smaller groups before normalization.

The learning-rate control offers 0.003, 0.03, and 0.1. Both lanes always use the selected value. The Slow steps recipe keeps Nstats = 32 but lowers η to 0.003; it stays stable yet reaches only 0.4586 held-out BCE within the fixed 300-epoch budget. Stability and speed are different claims.

Edge Cases

• A constant channel. Its variance is zero, so ε is what keeps normalization finite. The standardized values all become zero before γ and β are applied.
• A one-example statistics group. Its variance is always zero and the centered signal collapses. The lesson starts at Nstats = 2 because Nstats = 1 is mathematically computable but not a useful training regime for this BatchNorm layer.
• A tiny but non-singleton statistics group. Nstats = 2 preserves a signal, yet its mean and variance vary enough that this controlled run falls below chance-level held-out accuracy.
• Untrained running statistics. Evaluation is only meaningful after training has accumulated representative running estimates. Switching a freshly initialized layer to evaluation mode does not reveal the data distribution.

Common Mistakes

• Treating BatchNorm as “subtract the mean.” It centers, scales by variance, and then applies learned γ and β.
• Normalizing across the wrong axis. BatchNorm for a fully connected layer uses examples to estimate one set of statistics per feature channel.
• Dropping ε. A constant or nearly constant channel can make the denominator zero or numerically unsafe.
• Using current-batch statistics at inference. Ordinary evaluation uses running statistics so a prediction does not depend on neighboring examples.
• Conflating optimizer batch with statistics group. This lesson fixes the optimizer batch at 32. Nstats changes only how the BatchNorm lane partitions activations for its training-time estimates.
• Assuming smaller statistics groups are harmless. The operation is defined for Nstats = 2, but the estimates can be too noisy to optimize well.
• Calling BatchNorm a universal cure. It changes optimization and often adds useful regularization, but it does not repair every architecture, dataset, learning rate, or batch regime.
• Comparing unmatched runs. To isolate the mechanism, keep data, initialization, optimizer, learning rate, and budget fixed.

A Note on Simplification

This lesson uses a small fully connected network, synthetic two-moons data, one fixed seed, sigmoid/BCE, 32-example SGD updates, and per-channel BatchNorm after every affine hidden layer and before ReLU. Training may split one optimizer batch into virtual statistics groups; the running ledger is updated once from all 32 raw activations so inference-stat updates do not become a second changed variable. Seeded inputs, initial parameters, running statistics, and committed parameters are quantized to 11 decimal places so the long trace stays identical across browser and Node JavaScript engines. Metrics are sampled every ten epochs, with exact checkpoints at 10, 50, and 300. Real systems may use convolutional or synchronized BatchNorm, mixed precision, different momentum conventions, residual connections, or frozen statistics during fine-tuning. The animation isolates one training-time statistics boundary; it is not a prescription to place BatchNorm in every modern model.