Vanishing Gradients

Vanishing gradients occur when the chain rule multiplies enough local gains below one that early layers receive almost no learning signal, so their parameters barely update while later layers keep moving. In the default ten-layer comparison, sigmoid retains only 2.785e-7 of the output gradient; an active ReLU path with unit weights retains exactly 1.

Think of a message passed through ten people, each repeating only about one fifth of what they heard. No single handoff looks catastrophic. But after ten handoffs, almost nothing remains. Backpropagation has the same multiplicative structure: every layer contributes one local gain, and the product — not any one factor — decides whether a correction reaches the beginning of the network.

How Vanishing Gradients Work

The lesson uses a deliberately small feedforward chain. Every layer has one scalar activation, the same positive weight w, and zero bias:

z_ℓ = w · a_(ℓ−1)

a_ℓ = φ(z_ℓ)

The forward pass computes activations from left to right. The backward pass asks a different question: if the final activation a_D changed slightly, how much would that change be traced back to an earlier activation a_ℓ?

Define the relative gradient as

g_ℓ = |∂a_D / ∂a_ℓ|

and seed the output with g_D = 1. One backward hop applies the chain rule:

g_(ℓ−1) = g_ℓ · |w · φ′(z_ℓ)|

The second factor is the layer's local gain. Across D layers, the input receives the product of all local gains:

g_0 = Π[ℓ=1…D] |w · φ′(z_ℓ)|

For sigmoid, φ′(z) = a(1 − a). Its largest possible value is 0.25, and it gets smaller as the unit saturates toward 0 or 1. Even with w = 1, every local gain is at most 0.25, so a deep product collapses quickly.

For an active ReLU, φ′(z) = 1, so the local gain is simply |w|. With w = 1, the gradient is preserved. With w = 0.5, it still vanishes as 0.5ᴰ. With w = 1.5, it explodes as 1.5ᴰ. ReLU removes sigmoid's saturation factor on an active path; it does not remove multiplication from the chain rule.

The plot uses one fixed logarithmic axis from 1e-15 to 1e3 for both paths and for every recipe. It never rescales to flatter the result. Equal vertical distances mean equal orders of magnitude, so a line falling from 1 to 1e-5 has lost five decimal orders of signal.

Vanishing Gradients Step by Step

The default run sets a₀ = 1, D = 10, w = 1, and every bias to zero. Both chains receive the same input and the same normalized output gradient. Only the activation function changes.

Forward, layer 1. The sigmoid chain computes z₁ = 1 × 1 = 1, then a₁ = σ(1) = 0.7311. Its derivative is a₁(1 − a₁) = 0.1966, so this layer will keep only about 19.7% of any gradient that later crosses it. The active ReLU chain computes a₁ = 1, with derivative 1 and local gain 1.

Forward, layer 2. Sigmoid receives 0.7311, produces a₂ = 0.6750, and contributes local gain 0.2194. The ReLU activation remains 1. By layer 10, the sigmoid activations have settled near 0.6590; they look stable going forward, but every corresponding derivative is still only about 0.2247.

Pause on the contradiction. The forward state barely changes from a₂ = 0.6750 to a₁₀ = 0.6590, yet the backward sensitivity across those same layers will reach only g₀ = 2.785e-7. A calm forward trace does not prove that an early parameter will receive a useful update.

Seed the backward pass. Both output gradients start at g₁₀ = 1. This normalization removes the loss function from the comparison. A real loss would multiply both paths by its output derivative; the relative survival caused by these layers would be unchanged.

First backward hop. The sigmoid path multiplies by its layer-10 local gain and lands at g₉ = 0.2247. The active ReLU path multiplies by 1 and lands at g₉ = 1.

Keep walking left. Sigmoid's successive values are:

g₈ = 0.05049

g₇ = 0.01135

g₆ = 0.002549

g₅ = 0.0005728

g₄ = 0.0001287

g₃ = 2.888e-5

g₂ = 6.457e-6

g₁ = 1.416e-6

No hop is zero. No numerical underflow occurs. The signal disappears because ten ordinary-looking factors multiply into an extraordinary small product.

Arrive at the input. The final sigmoid hop produces g₀ = 2.785e-7. The active ReLU path remains at g₀ = 1. Under this exact setup, the ReLU path delivers about 3.59 million times as much relative gradient to the input.

The practical consequence is credit assignment. An early layer may have contributed strongly to the final error, yet receive an update millions of times smaller than a late layer. Training then concentrates on the end of the network while the beginning barely moves.

Why ReLU Helps — and When It Does Not

The default comparison makes ReLU look like a rescue because it removes the bounded sigmoid derivative from an active path. That is useful, but the weight still matters. Select Both fade and the shared weight becomes 0.5. The ReLU derivative stays 1, yet ten weights multiply to 0.5¹⁰ = 0.0009766. Sigmoid falls further, to about 7.261e-10.

Select ReLU explodes and w = 1.5. The active-ReLU product becomes 1.5¹⁰ = 57.665. This is the opposite failure: early gradients become much larger than the output gradient, so updates can become unstable. The sigmoid path still shrinks, because its activation derivatives outweigh the larger weight.

Depth controls the exponent. In Sixteen deep, w returns to 1, so active ReLU still preserves g₀ = 1. Sigmoid drops to roughly 3.585e-11. Adding six layers costs about four more orders of magnitude even though the per-layer calculation has not changed.

From Training Symptom to Gradient Diagnosis

Log a gradient norm such as ||∇W_ℓ L|| for each block, then compare early layers with late ones over the same training steps. In a real network, the scalar local gain becomes a Jacobian: its singular values and the available paths determine which directions shrink or grow. The useful question is still the same: which local gains are being multiplied along the path that must learn?

Recurrent gates and attention change the available dependency paths rather than repealing the chain rule. Gradient clipping belongs only on the explosion side of the table: it cannot restore information after a path has already reduced it to an ineffective value.

Vanishing-Gradient Edge Cases

• A nonpositive ReLU pre-activation. This lesson labels its ReLU lane “active path” because every z_ℓ is positive under the allowed inputs. For z_ℓ < 0, ReLU′(z_ℓ) = 0; at the nondifferentiable point zero, implementations commonly choose derivative zero. One zero factor makes every earlier gradient on that path exactly zero — the dying-ReLU case.
• A local gain above one. One amplifying layer does not prove that the whole gradient explodes. Other layers may shrink enough to dominate it. The complete product is the verdict.
• A local gain below one. One shrinking layer is not usually a crisis. Repeated shrinking is. Depth turns a mild factor into exponential decay.
• Negative weights. The lesson restricts w to positive values so forward activations and gradient magnitudes stay visually comparable. A negative weight can flip gradient signs; vanishing and explosion are governed by the magnitude of the product.
• Exactly zero versus effectively zero. Sigmoid derivatives are positive for finite inputs, so the mathematical gradient here never becomes exactly zero. A value like 2.785e-7 can still be ineffective once it meets finite precision, noise, regularization, or much larger gradients elsewhere.
• The output loss. The plot starts with g_D = 1. A real loss contributes another factor. This normalization isolates the effect of the hidden chain rather than claiming every training gradient begins at one.

Vanishing-Gradient Common Mistakes

• Reading stable forward activations as healthy backward flow. The default sigmoid activations settle near 0.659, but their derivatives multiply into 2.785e-7. Forward memory and backward reach are different quantities.
• Blaming one tiny derivative. The default local gains are around 0.20–0.22, not machine zero. Their repeated product is the problem.
• Saying “ReLU fixes vanishing gradients.” An active ReLU removes its own saturation derivative. Weights below one still vanish, weights above one can explode, and inactive ReLUs cut the path to zero.
• Comparing bars on changing axes. A chart that rescales each frame can make exponential decay look flat. This lesson fixes the log domain for the whole run and shares it across both lanes.
• Treating gradient clipping as a cure for vanishing. Clipping caps gradients that are too large. It cannot amplify a vanished path back into useful information.
• Assuming every parameter sees this exact scalar product. Real networks multiply Jacobian matrices and sum contributions from many paths. Singular values, branching, residual routes, and batch statistics all matter. The scalar chain isolates the multiplication mechanism, not the full geometry.

What the Scalar Chain Leaves Out — A Note on Simplification

This is an explainer, not a training system. Each layer contains one scalar rather than a vector, every weight is identical, biases are zero, and the output gradient is normalized. Real backpropagation multiplies Jacobian matrices, accumulates gradients from branches and examples, and updates parameters through an optimizer. Those details change the numbers and can create multiple routes for a signal, but they do not remove the central rule shown here: long products of local gains determine whether gradient information shrinks, survives, or grows.