GAN

A generative adversarial network (GAN) learns to generate new samples by training two networks against each other. A discriminator learns to distinguish real data from generated outputs; a generator learns to fool it through its gradients. Plausible individual outputs are insufficient: the generated distribution must also preserve the data's diversity.

Think of D as a judge whose scoring rubric is still being revised. G learns to improve under that rubric, while D learns to distinguish G's latest work from real examples. The rubric can be weak, and a generator can produce the same plausible result repeatedly. You need to inspect the outputs as a distribution.

How GAN Training Works

The generator G maps a latent input z to a sample G(z). In an image GAN, that sample might contain thousands of pixel values. Here it is one number, so you can inspect every sample along a single horizontal axis. The real training examples occupy two equally populated intervals: [-1.5, -1.1] and [1.1, 1.5].

The discriminator D takes either a real example or a generated example and returns a real-source score between 0 and 1. Real training examples have target label 1; generated examples have target label 0. The curve's height is D's score at a particular value of x. It is not a probability-density curve describing how often G produces x.

The two updates take turns. D revises the judge's rubric; G then learns under that fixed rubric:

First update the discriminator

Hold G's parameters fixed and train D on both sources:

L_D = -mean(log D(real)) - mean(log(1 - D(G(z)))).

A confidently misclassified example contributes a large loss. Both means contain 64 examples in this experiment, and their losses are added. D's parameter update changes its scoring curve. It cannot change where a fixed latent input lands under the unchanged G.

Shared parameters matter here. A batch update optimizes the aggregate objective; it does not guarantee that every real example's score increases or every generated example's score decreases on that step.

Then update the generator through D

Keep D's parameters fixed and use the non-saturating generator objective:

L_G = -mean(log D(G(z))).

G is rewarded for outputs that D scores as real. Differentiation passes backward through D to the generated sample, then through G to G's parameters. Freezing D's parameters preserves this input derivative. Detaching D's output would cut off the adversarial gradient G needs.

For one generated sample, if a is D's output logit, the derivative of -log(sigmoid(a)) with respect to a is D(G(z)) - 1. D supplies the rest of the derivative back to its input. G uses that signal in the chain rule; it does not receive a real example to reconstruct. Nor are the plotted points independent trainable particles. All of them move because the same G parameters changed.

D's curve stays fixed during this update, but a generated sample can move to a different location on that curve. Its score can therefore change without any change to D's parameters.

For the first highlighted sample in the default run, dℓ/dx ≈ +0.1351: a small move left would lower that sample's current loss. This is the direction supplied by the local derivative. The subsequent batch update changes G's shared parameters, moving the highlighted output from about 0.903 to 0.734. The local arrow explains the feedback; it does not give each point its own optimizer or guarantee that a batch update follows every sample's preferred direction.

The original GAN paper, section 3 describes alternating discriminator and generator updates and motivates this non-saturating objective. The original minimax generator objective is different: it minimizes mean(log(1 - D(G(z)))), which can give weak early gradients when D confidently rejects G's samples.

GAN Step by Step

The default uses noise range 1 and three D updates per G update. Both networks start from initialization seed 42.

There are 64 fixed real training examples and 64 evenly spaced latent training inputs. The animation follows 48 different, fixed evaluation inputs spanning [-1, 1]. Keeping these probes fixed lets you attribute their movement to learning instead of a new random draw.

1. Inspect the untrained G. Its 48 outputs span approximately [-1.508, 1.508]. Ten lie near the left data band and ten near the right. The mean D score is already 0.500, although 28 outputs miss both neighborhoods. This average does not certify a learned distribution.
2. Train D three times. G's outputs remain fixed throughout all three updates. D changes its scoring curve to distinguish the real and generated training examples. Each update gets its own beat.
3. Pass feedback through fixed D. Compute G's batch gradient using the latest D. D's parameters remain unchanged; the derivative through its input remains available.
4. Update G once. The same probe inputs now produce different outputs. After this first round, ten outputs fall in the left neighborhood and seven in the right. G's first update did not improve this neighborhood count. Its objective is a learned adversarial signal, not a direct optimization of this diagnostic.
5. Repeat the full round. D judges the changed outputs and takes three more updates. G then receives the new gradient and updates once. The second round ends with eleven probes near the left band and seven near the right.
6. Continue through round 400. Later scenes group repeated rounds into labeled intervals. The coverage bars report the last completed round as the points move toward the next. By round 160, 18 outputs occupy each neighborhood. The default browser run ends with 22 near the left band and 20 near the right; six are still outside both.
7. Compare two failures of diversity. First keep the seed, data, networks, optimizer, and update ratio fixed, but set every latent input to zero. All 48 final outputs equal approximately 1.197, near the right data band. Then inspect a separate matched pair: range 1 versus 0.25, both with five D updates per G update. The narrow-range run still has distinct inputs, yet its outputs nearly coincide on the right. The distinction is explained below.
8. Generate a fresh batch. Keep the final G parameters fixed and draw new latent inputs. Each input passes through G once. D is not needed to generate these samples, and neither network takes another training update.

A sample counts as near a band when it lies within 0.25 of that band's center, -1.3 or 1.3. These diagnostic regions extend 0.05 beyond the real training intervals. Neighborhood counts and output standard deviation describe this finite batch; neither proves equality of distributions. Output standard deviation also cannot tell you whether the shape or relative frequency of the groups is correct.

Why a GAN Score of 0.5 Is Not Enough

At the idealized solution, a discriminator that cannot distinguish two matching distributions assigns a score of 0.5 on their support. The converse requires care. A weak discriminator can be uncertain about bad outputs. An average of 0.5 can also combine confident scores near zero and one, rather than reflect uncertainty at every point.

The initial state gives you a concrete counterexample: mean score 0.500, but most generated points outside the two data neighborhoods. Compare the locations and their frequencies before trusting the score. The idealized result does not guarantee that finite neural networks, alternating updates, and a fixed training budget will reach an equilibrium.

Use the two controls to isolate changes. First keep the update ratio at 3 and reduce Noise range to zero. Then restore range 1 and compare one, three, and five D updates per G update. More D updates change both the learning dynamics and the total amount of D computation. They do not promise monotonically better sample coverage.

Varied noise does not guarantee varied outputs

The Mode collapse preset sets range 0.25 and five D updates per G update. Its 48 distinct probe inputs initially produce a spread of about 0.291. After training, all 48 outputs fall near the right group and their spread is about 0.005. G learned to map different inputs to almost the same output; half of the real distribution is missing. Unlike the zero-range experiment, this failure developed despite input variation.

The comparison uses range 1 as a reference, with the same five D updates, seed, data, architecture, optimizer, and 400 rounds. That reference represents both groups. This pair demonstrates that input variation alone is insufficient; it does not establish a universal noise range or a general cure for collapse.

When checking your own generator, keep one set of latent inputs fixed across checkpoints so that changed outputs reveal changed parameters. Inspect coverage and repetition across many outputs, then also draw fresh inputs to check beyond those probes. For an image GAN, repeated plausible images can conceal missing kinds of images just as repeated right-band samples conceal the left group here. Once you have a usable generator, producing a sample requires a forward pass through G; D's role was to supply training feedback.

Edge Cases

• Identical latent inputs. A deterministic function returns the same output for the same input. More updates may move that shared output, but cannot create multiple outputs at one fixed input. Fresh random draws still become zero when multiplied by a zero noise range.
• Mode collapse with varied inputs. The range-0.25, five-D experiment produces almost one output despite distinct inputs. This differs from the guaranteed zero-range degeneracy. Other nonzero ranges and update ratios can also produce poor coverage; the two controls do not offer a general convergence guarantee.
• Changing update balance. One or five D updates per G update can produce a different trajectory. The generator still takes 400 updates, so these are not equal-compute comparisons.
• Imperfect final coverage. Some default outputs remain between or outside the bands. A fixed budget ends training; it does not declare convergence.
• Numerical replay. The same configuration replays the same browser trace. Long adversarial training can amplify small differences between numerical runtimes, so an independent NumPy computation may differ in late decimal values while reproducing the central contrast.

Common Mistakes

• Freezing D by removing its derivative. Freeze the parameters being optimized, while retaining the derivative needed to train G through D.
• Pairing each generated sample with a real target. This GAN objective uses source discrimination. It is not an autoencoder's reconstruction loss.
• Treating D's score as an image-quality percentage. It is a learned source judgment, dependent on D and the current training state.
• Moving generated points independently toward real examples. The optimizer changes G's shared weights. The displayed sample motion is the resulting change in G's outputs.
• Equating repeated plausible samples with a correct distribution. The identical-noise experiment can put every output near one real band while missing the other entirely.
• Keeping D in the generation path. Once G is trained, new noise is sufficient to generate new samples. D was the source of training feedback.

A Note on Simplification

This is a one-dimensional teaching GAN with synthetic data, small tanh networks, fixed training batches, deterministic evaluation inputs, and a bounded generator output. It is an explainer, not a substitute for formal training. Image architectures, stochastic minibatches, normalization, regularization, and advanced stabilization methods are omitted. The example trains real network parameters in the browser; it does not generate images or establish a general convergence result.

G has eight tanh hidden units and one bounded output; D has twelve tanh hidden units and a sigmoid output. Separate Adam optimizers use learning rate 0.01, beta1 0.5, beta2 0.999, and epsilon 1e-8. These settings define the reproducible experiment, rather than a recommended configuration for image synthesis.

Backpropagation explains the chain-rule bookkeeping used by both networks. Training Loop separates updates from evaluation. Diffusion Models shows a different generation procedure: repeated denoising rather than a single pass through a trained GAN generator.