Autoencoder

An autoencoder is a neural network trained to reconstruct its input through an intermediate representation called a latent code. Here, eight latent coordinates reconstruct held-out handwritten digits better on average than two. Yet the same digit-trained decoder fails on a checkerboard: a wider code does not guarantee recovery of an unfamiliar pattern.

Imagine sending a sketch through a narrow note slot. The sender chooses a short description; the receiver redraws the sketch using that description and what they already know. The receiver's prior experience matters. A description that works for handwritten digits may be useless for a checkerboard, even when you allow a longer note.

How an Autoencoder Works

The encoder maps an input x to a code z. The decoder maps z to a reconstruction x̂. Training adjusts both sets of weights to reduce a loss comparing x̂ with x. A class label is unnecessary: each training image supplies its own target. The reconstruction objective and the need to constrain copying are described in Deep Learning, Chapter 14.

If you want a compact representation of an image, reconstruction gives you a test of what that representation retains. An undercomplete autoencoder forces fewer coordinates through the code than there are input values. In the note-slot analogy, the decoder has to rebuild the sketch from the short note. A direct copying path around the slot would defeat that test. The restriction can encourage useful representations; it does not by itself guarantee one.

The decoder receives the code and its own trained weights. It cannot inspect the original image. The original pixels are used again when measuring mean squared error, MSE = Σᵢ(x̂ᵢ − xᵢ)² / 64 for the 64-pixel images here. Squaring prevents positive and negative errors from canceling; averaging makes the result a per-pixel quantity. Background pixels count too.

Each latent coordinate is a learned numerical value. A coordinate is not necessarily a pixel, a digit probability, or a named feature such as a loop. Reconstructing a recognizable 8 does not mean the network produced an 8-class prediction: its output is still 64 intensities.

The trained encoder and decoder

Here the input is an 8×8 grayscale image: 64 intensity values between 0 and 1. The architecture is 64 → 24 → k → 24 → 64, where k is 2, 4, or 8. The hidden layers use tanh; the final sigmoid gives output intensities between 0 and 1. With row-vector notation, the four transformations are:

hEnc = tanh(x W₁ + b₁)

z = tanh(hEnc W₂ + b₂)

hDec = tanh(z W₃ + b₃)

x̂ = sigmoid(hDec W₄ + b₄)

Autoencoder Step by Step

The default selects the eight-coordinate model and a held-out handwritten 8, original dataset row 249. Its pixels come from the scikit-learn 8×8 digits dataset, not MNIST. The model trained on 1,437 other images; 360 images were held out. This particular example was selected to make the reconstruction contrast visible.

1. Set the target. The network must reconstruct all 64 pixels of the selected image. Its digit label is only metadata used to select the example.
2. Encode it. The trained encoder produces the eight-coordinate code, rounded here: [0.0612, 0.4671, −0.3931, 0.0032, −0.1654, −0.0009, 0.4504, 0.2202]. The decoder must work from these numbers.
3. Decode it. The decoder produces an approximate image. The broad shape survives, but individual intensities differ from the original.
4. Inspect one pixel, then average the image. At zero-based pixel [6, 4], the original intensity is 0.4375, while the reconstruction is about 0.8077649. Its squared error is about 0.1370961, the largest of this image's 64 terms. The next beat averages all 64 terms to give MSE 0.0106606. The largest pixel error and the average error answer different questions.
5. Rewind the training record. Epoch 0 gives image MSE 0.1891698. Recorded epochs 1, 10, 50, and 200 reduce it to 0.0887845, 0.0467999, 0.0120355, and 0.0106606. The image stays held out throughout. These are real checkpoints from offline training, not training performed by playback.
6. Compare one image, then the digit set. A separately trained two-coordinate model reconstructs the same image with MSE 0.0465775. Its reconstruction loses more of the digit's detail. The following beat checks both models on the same 360 held-out digits: mean MSE is 0.0375854 for two coordinates and 0.0148576 for eight. The wider model performs better on this set as well as on the selected example.
7. Change the kind of input. Keep the eight-coordinate model's final weights, but supply a synthetic checkerboard. Its MSE is now 0.4291607: the alternating pattern is not recovered. The decoder's experience with digits did not make it a universal copier.

The training chart reports the mean MSE over 360 held-out digits, separately from the current image's MSE. At epoch 200, those values are 0.0148576 and 0.0106606, respectively. The chart's epoch spacing is literal: 0, 1, 10, 50, and 200 are measured checkpoints, not five consecutive updates.

More coordinates help these digits, but not the checkerboard

Use the bottleneck control while leaving the input unchanged. Each width selects its own trained network. The training split, 24-unit hidden layers, optimizer, epoch count, and seed policy are fixed; the parameter count and learned weights necessarily change. Selecting two coordinates does not delete six coordinates from the eight-coordinate model.

The wider networks perform better on the held-out digit set. The checkerboard reverses that ranking. This is why an aggregate reconstruction score needs an input-distribution label attached to it. None of these results guarantees that a wider network will improve every sample or that optimization will find the best possible weights.

The note-slot analogy also has a limit: eight real-valued coordinates are not eight bits. A smaller latent dimension is not a measured file-compression ratio. Coordinate precision and the cost of storing the decoder matter.

Choose a code size against the reconstruction you need

If your goal is to reconstruct digits within a code-size budget, compare candidate models on the same held-out digit set. Choose a width that meets your error tolerance, and inspect the reconstructed images for details the average score can hide. The table gives you that tradeoff for these three models; it does not choose an acceptable error tolerance for your application.

Test unfamiliar input types separately. A good digit average would have concealed the checkerboard failure. If you instead plan to reuse z for another task, evaluate it on that task: reconstruction quality alone does not establish that the representation is useful there.

For the connection to linear dimensionality reduction, continue with PCA. For how training computes gradients through the encoder and decoder, use Backpropagation.

Autoencoder Edge Cases

• Two-coordinate bottleneck: this model has the highest mean held-out digit error of the three. That is an observed result under this training setup, not proof that every possible two-dimensional encoding must fail on these images.
• Checkerboard input: all three digit-trained models have MSE above 0.4. The probe is synthetic and was not part of training. Choosing it in Inputs exposes the same failure throughout the walkthrough.
• Replay or a different speed: reconstructed values stay identical. Playback neither trains weights nor adds random noise.
• Width limits: 2, 4, and 8 are the architectures supplied here, not limits on autoencoders. Larger or overcomplete codes require other constraints if the goal is to learn something more useful than copying.

Common Mistakes

• Giving the decoder the target. A direct input-to-output copying path could achieve zero error while bypassing the code. It would no longer demonstrate that the code retained what reconstruction needed.
• Treating MSE as confidence. MSE measures pixel differences. It is not the probability that an image belongs to a digit class.
• Expecting automatic denoising. These models train on image-to-same-image pairs. A denoising autoencoder uses corrupted inputs with clean reconstruction targets, a different training task.
• Sampling arbitrary latent numbers and expecting valid images. This ordinary autoencoder learns a reconstruction map. It has no trained sampling prior; reconstruction alone does not establish a useful generative model.
• Calling reconstruction lossless compression. Low pixel error is still error, and latent dimension does not count encoded bits.

A Note on Simplification

This is an explainer, not a substitute for formal study of representation learning. It uses small dense networks and 8×8 digits, groups the hidden layers into encoder/decoder blocks, and replays five recorded training checkpoints. The models were optimized with Adam for 200 epochs; the selected images were held out. Convolutional architectures, regularization, denoising objectives, variational inference, and a latent sampling prior are omitted. Diffusion Models examines a different task: generating fresh samples from noise.