Diffusion Models

Diffusion models learn to denoise data at different noise levels, then use that knowledge to generate new samples from random noise. A single denoising estimate can average several plausible origins and land between the data's groups. Here, one estimate stays near an empty center; repeated estimates produce samples near eight groups.

Imagine a map with eight towns arranged around a lake. A vague location report could refer to any town, so its average location might land in the water. That average is a reasonable estimate, but a terrible place to deliver a parcel. The distinction between an estimate and a plausible sample is the reason to watch this example.

How Diffusion Models Turn Noise into Samples

The data here consists of two-coordinate points in eight Gaussian groups. Their centers lie on a circle of radius 2, and each group has standard deviation 0.12 along both coordinates. A dot is one complete sample. For an image model, the sample would be a much longer vector of pixel or latent values.

We know this small distribution exactly, so we can calculate an ideal denoiser directly. There are no learned neural-network weights in this demonstration. That removes training error from the experiment and lets you examine what even an ideal denoising estimate can and cannot do.

Add noise with a known target

Take a clean sample x₀, sample standard Gaussian noise ε, and mix them at noise level t:

xₜ = √a(t) · x₀ + √(1 − a(t)) · ε

Here a(t) is cumulative signal variance. The coefficients are square roots: their squares add to one. They are not percentages of dots, and their ordinary sum need not equal one. At t = 0, the sample is clean. At t = 1, this example has a = 0.0001, so the signal coefficient is 0.01 and the result is nearly standard Gaussian noise.

This gives noise-prediction training its supervision. Because you sampled ε yourself, you can compare a network's prediction ε̂(xₜ, t) with the known noise and minimize the expected squared difference. The network receives both the noisy sample and its noise level. The DDPM training objective formalizes this construction. Ho, Jain, and Abbeel

The exact prediction used here is E[ε | xₜ, t]: an average over noise values consistent with the observed sample. It need not equal the particular noise used to make one training example. Several clean origins can produce similar noisy observations. Averaging those possibilities is what minimizes expected squared error.

Estimate, move, estimate again

Start generation from fresh x₁ ~ Normal(0, I). There is no original clean point to recover. Given a predicted noise vector, calculate a clean-sample estimate:

x̂₀ = (xₜ − √(1 − a(t)) · ε̂) / √a(t)

At high noise, this estimate can average over widely separated groups. In the map analogy, it can fall in the lake. Returning that average immediately would confuse a conditional mean with a sample from the distribution.

The eight origin-probability bars make that ambiguity explicit for one noisy point. Each bar measures how plausible a particular clean-origin group is, given that point and its noise level. Similar bars mean several groups remain plausible. The clean estimate averages possible origins both within and across those groups. It does not average the 48 generated points together, and the bars do not assign every point a fixed destination.

The deterministic DDIM sampler instead moves to a lower noise level s using both the clean estimate and predicted noise:

xₛ = √a(s) · x̂₀ + √(1 − a(s)) · ε̂

Now evaluate the denoiser again at the new (xₛ, s). The estimate changes because its inputs changed. Keep doing this until s = 0; there the remaining noise coefficient is zero, so the output equals the latest clean estimate. This is the η = 0 DDIM update, which introduces no fresh randomness after initialization. It differs from the stochastic DDPM sampler. Song, Meng, and Ermon

The hollow marker on the plane denotes the clean estimate. During an update, the plus marks the computed next sample and the orange point moves toward it. Early in generation, those locations can be far apart even when the actual sampling displacement is small.

Diffusion Models Step by Step

The default uses seed 42, 32 sampling steps, and 48 samples. The forward demonstration has a separate fixed random stream, so changing the generation seed leaves the training example unchanged.

First, the eight-group data is corrupted at t = 0.25, 0.5, 0.75, 1. These are samples from the correct forward marginals, with the same noise reused across levels for a readable comparison. They are not a literal stochastic DDPM Markov trajectory.

The training inset returns to one stored example at t = 0.5: clean point (-1.835, 0.278), sampled noise (0.185, -2.110), and noisy observation (-1.174, -1.287). The exact conditional noise prediction is (-0.437, -0.488). Its squared error on this pair is 3.018. A positive error for one pair is compatible with an optimal conditional-average predictor; it does not imply that the hidden noise can always be recovered.

Generation then starts with fresh noise. Follow sample 0 through its first two updates:

The first clean estimate sits close to the center, while the next sample remains noisy. Evaluation 2 consumes that new sample at its new noise level, predicts again, and takes another step. A step need not move closer to a mode center: this is a sampling trajectory, not gradient descent on distance to the nearest group.

At evaluation 1, each of the eight groups has about 12–13% of sample 0's origin probability. The lesson pauses on these weights before revealing the one-step output. It returns to the same point at evaluation 25, where t = 0.25: group 6, the lower-left diagonal group, now has about 99.8% of the origin probability. The input and noise level have changed; the denoiser has not been retrained. Another seed can favor another group.

Evaluations 3–32 repeat the same calculation; the two origin inspections only explain existing predictions. Sample 0 finishes at (-1.523356, -1.695873), near the lower-left diagonal group. Across the batch, 43 of 48 samples finish within distance 0.24 of a group center, and all eight neighborhoods contain at least one such sample. Five points lie outside those neighborhoods; Gaussian groups have tails, not hard edges.

The comparison uses exactly the same initial noise and denoiser for a one-step run. That run returns the first clean estimates directly: 0 of 48 points enter the eight neighborhoods. Its output sits around the center between the groups. The difference is produced by changing the sampling trajectory, without changing the denoiser.

More Sampling Steps Mean More Denoiser Evaluations

The sampling-step slider changes how many times the denoiser is evaluated along the fixed noise schedule. It does not change how long a network was trained. With seed 42, four steps put 33 of 48 outputs in the marked neighborhoods; 32 steps put 43 there. The One step and Coarse trajectory recipes jump directly to the comparison, so you can check the consequences without replaying the setup.

For a controlled comparison, keep seed 42 and try One step, Coarse trajectory, then Repeated denoising. Inspect both the gaps between groups and the spread within them. Then use Another noise batch to check whether your conclusion survives a different starting batch. In a learned model, increasing the sampling budget similarly spends more denoiser evaluations; it does not add training. Compare budgets with the model and starting noise held fixed before attributing a change to the sampler.

Treat those counts as geometric diagnostics. With the same seed, eight steps produce 45 nearby points, while 64 produce 42. That does not prove eight steps produce a better distribution. Collapsing every output onto one mode center would score perfectly on closeness while losing the other seven groups and all within-group variation. Occupancy supplies a second check, but neither statistic alone validates a generative model.

Change the seed independently to rerun the experiment. Seed 21 with 32 steps produces 45 nearby samples and occupies all eight neighborhoods; seed 137 produces 45 nearby samples but occupies seven. Small sample batches fluctuate. The claim is that repeated denoising resolves the groups in these experiments, not that every seed must fill every neighborhood.

For this analytical mixture, generating M points with N steps and K components costs O(NMK) time. A numerical sampler only needs O(MK) working space or less when processing points sequentially; the animation additionally stores O(NM) state snapshots for scrubbing. For a learned image model, inference cost is governed by the network and image dimensions, so this mixture-specific complexity is not its runtime formula.

Diffusion Sampling Edge Cases

• One step: the main and comparison runs are identical. Both return the initial clean estimates, and the interface reports equality.
• Exactly at the center: symmetry gives a zero clean estimate for every noise level. Deterministic sampling preserves this point. A continuous Gaussian draw hits the exact center with probability zero, but this boundary shows why an estimate is not automatically a plausible sample.
• Finite terminal noise: a(1) is small, not zero. Starting from a standard Gaussian approximates the terminal forward distribution.
• Same seed and step count: replay is deterministic, including when you change playback speed or scrub. DDIM with nonzero η and stochastic DDPM use different randomness contracts.
• Step budgets: the interface accepts 1–64 sampling steps and seeds 1–9999. These are visualization bounds, not restrictions on diffusion models.

Common Mistakes About Denoising

• Reversing a saved corruption movie. Generation begins with an independent noise batch. It does not receive the forward example's original point or sampled noise.
• Subtracting the prediction without the schedule coefficients. The sampler combines a clean estimate with a noise component at the next level. x − ε̂ alone is not the update shown here.
• Calling every clean estimate the current sample. Intermediate estimates guide the trajectory; the current sample generally retains noise until the endpoint.
• Expecting the predicted noise to equal the hidden noise of every example. Under squared error, the optimal predictor returns a conditional average.
• Claiming one-step generation is impossible. This failure concerns the high-noise conditional-mean denoiser and sampler in this experiment. Models trained specifically for one-step generation are outside its claim.

A Note on Simplification

This is a two-dimensional explainer, not a substitute for formal training in generative modeling. It uses a known Gaussian mixture and an exact analytical denoiser with deterministic DDIM sampling. It does not train a neural network or generate images. Real image systems add learned networks over image or latent tensors, training dynamics, conditioning, and often guidance; latent systems also use autoencoders, and sampling methods vary. The Training Loop lesson addresses optimization separately. Here the experiment isolates how a denoising estimate is used to generate a sample.