Dropout

Dropout is a training-time regularizer that temporarily turns off a random subset of hidden activations on each training example. That friction makes it harder for a network to rely on fragile combinations that fit the training examples but fail on new data.

A falling training loss can hide memorization. In this lesson's fixed noisy two-moons experiment, no dropout reaches training BCE 0.0006 while held-out BCE rises to 1.5822. With p = 0.25, training BCE is higher at 0.2998, but held-out BCE falls to 0.3464. That contrast is the point: choose the dropout rate by held-out performance, not by training loss alone.

How Dropout Works

Let hⱼ be hidden unit j's activation and let p be its dropout probability. During training, each hidden unit receives an independent mask value:

mⱼ ~ Bernoulli(1 − p)

The masked activation is

h̃ⱼ = mⱼhⱼ / (1 − p)

If mⱼ = 0, that unit contributes nothing to the forward pass and receives no gradient from this example. If mⱼ = 1, its activation is divided by the keep probability. This is inverted dropout: the expected training activation stays equal to the original activation because

E[h̃ⱼ] = (1 − p)hⱼ / (1 − p) = hⱼ.

At evaluation time, the mask disappears. Every hidden unit is active, and there is no additional scaling step. That switch matters: sampling masks while measuring held-out loss would make predictions noisy and would evaluate a different procedure from the deterministic network intended for inference.

Dropout does not permanently delete neurons or weights. It changes which paths can cooperate on each training example and update. A hidden unit that is absent for one mask can return with a later mask and always returns for ordinary inference.

Step by Step

The default configuration is exactly { dropoutRate: 0.25, hidden: 16 }. Both lanes use the same 40 training examples, including eight fixed label flips, the same 400 clean-label held-out examples, the same initial weights, and full-batch Adam for 5,000 epochs at learning rate 0.01. The left lane fixes p = 0; the right lane changes only dropout.

Start from the same network. At epoch 0, both 2 → 16 → 1 networks produce the same losses. This controls the comparison: a later difference cannot be blamed on easier data, a luckier initialization, another optimizer, or a longer budget.

Sample a mask. For p = 0.25, each of the 16 hidden units has keep probability 0.75, so the expected number kept is 16 × 0.75 = 12. The actual count in one mask can be above or below 12. Expectation describes repeated samples; it does not prescribe every sample.

Run a masked update. Dropped units contribute zero and receive zero loss gradient on that example through the masked path. This does not imply that Adam’s existing momentum or other regularization terms cannot change their parameters. Kept activations are scaled by 1 / 0.75. The mask is resampled across examples and epochs, so training explores many overlapping subnetworks that share the same parameters.

Restore the full network for measurement. Both training and held-out metrics are now computed in inference mode with all hidden units enabled. This keeps the solid and dashed curves comparable.

Watch the curves separate. The animation pauses at epochs 500, 2,000, and 5,000. With no dropout, training BCE falls from 0.222 to 0.010 and finally 0.0006. Held-out BCE moves the other way: 0.495, 1.202, and finally 1.5822. The model is fitting the eight corrupted labels ever more confidently.

With p = 0.25, training is slower and never reaches the control's near-zero loss. Its final training BCE is 0.2998. But held-out BCE stays near the useful part of the curve and finishes at 0.3464, about 4.57× lower than the no-dropout control. Held-out accuracy is 86.75%, compared with 76.75% for the control.

The result is scoped to this experiment. It demonstrates the regularization mechanism; it does not establish 0.25 as a universal best rate.

Choosing the Rate and Width

The two controls are independent. p changes how often existing hidden units are unavailable during training. H changes how much hidden capacity exists in the first place.

The No dropout recipe sets p = 0. Both lanes become identical and replay the memorizing trajectory. This is the control condition, not a special implementation path.

The Sweet spot recipe uses p = 0.25, H = 16. It gives up some training fit and produces the lowest final held-out loss among the lesson's named recipes.

The Too much recipe uses p = 0.8. Only 16 × 0.2 = 3.2 hidden units are kept on average. Final training BCE rises to 0.5676, while held-out BCE is 0.4659. It still regularizes the memorizing control, but it discards so much useful capacity that the model underfits.

The Too narrow recipe keeps p = 0.25 but reduces H to 2. Its final held-out BCE is 0.4571. Dropout cannot manufacture features the narrow network does not have room to represent. Capacity and regularization solve different problems.

In practice, compare dropout with other interventions such as more data, data augmentation, weight decay, early stopping, normalization, and a model size appropriate to the task. Many modern architectures apply dropout selectively rather than placing the same rate after every layer.

Edge Cases

• p = 0. Every mask value is one, the scale factor is one, and training is exactly the no-dropout control.
• p = 1. Every hidden activation would be removed and 1 / (1 − p) would divide by zero. That rate is undefined under this formula and excluded by this lesson; library-specific p=1 conventions are outside the model.
• Very high dropout. A valid rate can still be harmful. If too few useful paths survive each update, optimization becomes noisy and the model underfits.
• A narrow hidden layer. Stronger dropout does not compensate for inadequate representational capacity. It usually makes that bottleneck harder to use.
• A small sampled keep count. One mask need not equal the expected count. The expectation emerges over many independent masks.
• Evaluation with dropout enabled. Monte Carlo dropout intentionally samples at inference for uncertainty estimates, but ordinary deterministic evaluation restores all units. These are different procedures and should be named separately.

Common Mistakes

• Calling dropout neuron deletion. It temporarily masks activations during training; it does not remove parameters from the stored model.
• Applying the scale twice. Inverted dropout scales kept activations during training, so evaluation uses the full network with no extra factor.
• Judging by training loss alone. The control's 0.0006 is the best training score and the worst final held-out score in the default comparison.
• Assuming more dropout is always safer. At p = 0.8, the network underfits. Regularization has a useful range, not a one-way quality dial.
• Confusing dropout rate and keep probability. A dropout rate of 0.25 means keep probability 0.75, not 0.25.
• Expecting exactly (1 − p)H kept units. That number is an expectation. Every mask is a random draw.
• Comparing runs with different starts or budgets. A fair causal comparison keeps the data, initialization, optimizer, and epoch count fixed.

A Note on Simplification

This lesson uses one hidden layer on synthetic two-moons data, a fixed Adam optimizer, and fixed random seeds. The training set has only 40 points and deliberately corrupts eight labels so memorization becomes visible within one short animation. The temporary subnetworks are overlapping views of one shared parameter set, not an ensemble of independently trained models. Real networks tune dropout by layer and combine it with more data, normalization, weight decay, data augmentation, and early stopping. Batch normalization, residual connections, recurrent state, attention, and structured dropout variants can all change where ordinary activation dropout is useful. The lesson isolates one causal comparison; it is not a prescription to add dropout everywhere.