---
title: "Autoencoder"
description: "Compare learned bottlenecks, pixel reconstruction, and an unfamiliar input."
source: https://vizlearn.app/ai/generative/autoencoder
access: premium (interactive player requires Lifetime; this doc is free)
---

# Autoencoder

An autoencoder is a neural network trained to reconstruct its input through an intermediate representation called a latent code. Here, eight latent coordinates reconstruct held-out handwritten digits better on average than two. Yet the same digit-trained decoder fails on a checkerboard: a wider code does not guarantee recovery of an unfamiliar pattern.

Imagine sending a sketch through a narrow note slot. The sender chooses a short description; the receiver redraws the sketch using that description and what they already know. The receiver's prior experience matters. A description that works for handwritten digits may be useless for a checkerboard, even when you allow a longer note.

## How an Autoencoder Works

The **encoder** maps an input `x` to a code `z`. The **decoder** maps `z` to a reconstruction `x̂`. Training adjusts both sets of weights to reduce a loss comparing `x̂` with `x`. A class label is unnecessary: each training image supplies its own target. The reconstruction objective and the need to constrain copying are described in [Deep Learning, Chapter 14](https://www.deeplearningbook.org/contents/autoencoders.html).

If you want a compact representation of an image, reconstruction gives you a test of what that representation retains. An **undercomplete** autoencoder forces fewer coordinates through the code than there are input values. In the note-slot analogy, the decoder has to rebuild the sketch from the short note. A direct copying path around the slot would defeat that test. The restriction can encourage useful representations; it does not by itself guarantee one.

The decoder receives the code and its own trained weights. It cannot inspect the original image. The original pixels are used again when measuring **mean squared error**, `MSE = Σᵢ(x̂ᵢ − xᵢ)² / 64` for the 64-pixel images here. Squaring prevents positive and negative errors from canceling; averaging makes the result a per-pixel quantity. Background pixels count too.

Each latent coordinate is a learned numerical value. A coordinate is not necessarily a pixel, a digit probability, or a named feature such as a loop. Reconstructing a recognizable 8 does not mean the network produced an 8-class prediction: its output is still 64 intensities.

### The trained encoder and decoder

Here the input is an 8×8 grayscale image: 64 intensity values between 0 and 1. The architecture is `64 → 24 → k → 24 → 64`, where `k` is 2, 4, or 8. The hidden layers use `tanh`; the final sigmoid gives output intensities between 0 and 1. With row-vector notation, the four transformations are:

`hEnc = tanh(x W₁ + b₁)`

`z = tanh(hEnc W₂ + b₂)`

`hDec = tanh(z W₃ + b₃)`

`x̂ = sigmoid(hDec W₄ + b₄)`

## Autoencoder Step by Step

The default selects the eight-coordinate model and a held-out handwritten 8, original dataset row 249. Its pixels come from the [scikit-learn 8×8 digits dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html), not MNIST. The model trained on 1,437 other images; 360 images were held out. This particular example was selected to make the reconstruction contrast visible.

1. **Set the target.** The network must reconstruct all 64 pixels of the selected image. Its digit label is only metadata used to select the example.
2. **Encode it.** The trained encoder produces the eight-coordinate code, rounded here: `[0.0612, 0.4671, −0.3931, 0.0032, −0.1654, −0.0009, 0.4504, 0.2202]`. The decoder must work from these numbers.
3. **Decode it.** The decoder produces an approximate image. The broad shape survives, but individual intensities differ from the original.
4. **Inspect one pixel, then average the image.** At zero-based pixel `[6, 4]`, the original intensity is `0.4375`, while the reconstruction is about `0.8077649`. Its squared error is about `0.1370961`, the largest of this image's 64 terms. The next beat averages all 64 terms to give **MSE 0.0106606**. The largest pixel error and the average error answer different questions.
5. **Rewind the training record.** Epoch 0 gives image MSE `0.1891698`. Recorded epochs 1, 10, 50, and 200 reduce it to `0.0887845`, `0.0467999`, `0.0120355`, and `0.0106606`. The image stays held out throughout. These are real checkpoints from offline training, not training performed by playback.
6. **Compare one image, then the digit set.** A separately trained two-coordinate model reconstructs the same image with MSE `0.0465775`. Its reconstruction loses more of the digit's detail. The following beat checks both models on the same 360 held-out digits: mean MSE is `0.0375854` for two coordinates and `0.0148576` for eight. The wider model performs better on this set as well as on the selected example.
7. **Change the kind of input.** Keep the eight-coordinate model's final weights, but supply a synthetic checkerboard. Its MSE is now **0.4291607**: the alternating pattern is not recovered. The decoder's experience with digits did not make it a universal copier.

The training chart reports the **mean MSE over 360 held-out digits**, separately from the current image's MSE. At epoch 200, those values are `0.0148576` and `0.0106606`, respectively. The chart's epoch spacing is literal: 0, 1, 10, 50, and 200 are measured checkpoints, not five consecutive updates.

## More coordinates help these digits, but not the checkerboard

Use the bottleneck control while leaving the input unchanged. Each width selects its own trained network. The training split, 24-unit hidden layers, optimizer, epoch count, and seed policy are fixed; the parameter count and learned weights necessarily change. Selecting two coordinates does not delete six coordinates from the eight-coordinate model.

| Latent coordinates | Default digit MSE | Mean MSE on 360 held-out digits | Checkerboard MSE |
| --- | ---: | ---: | ---: |
| 2 | 0.0465775 | 0.0375854 | 0.4074093 |
| 4 | 0.0171610 | 0.0247478 | 0.4114591 |
| 8 | 0.0106606 | 0.0148576 | 0.4291607 |

The wider networks perform better on the held-out digit set. The checkerboard reverses that ranking. This is why an aggregate reconstruction score needs an input-distribution label attached to it. None of these results guarantees that a wider network will improve every sample or that optimization will find the best possible weights.

The note-slot analogy also has a limit: eight real-valued coordinates are not eight bits. A smaller latent dimension is not a measured file-compression ratio. Coordinate precision and the cost of storing the decoder matter.

### Choose a code size against the reconstruction you need

If your goal is to reconstruct digits within a code-size budget, compare candidate models on the **same held-out digit set**. Choose a width that meets your error tolerance, and inspect the reconstructed images for details the average score can hide. The table gives you that tradeoff for these three models; it does not choose an acceptable error tolerance for your application.

Test unfamiliar input types separately. A good digit average would have concealed the checkerboard failure. If you instead plan to reuse `z` for another task, evaluate it on that task: reconstruction quality alone does not establish that the representation is useful there.

For the connection to linear dimensionality reduction, continue with [PCA](https://vizlearn.app/ai/classical-ml/pca). For how training computes gradients through the encoder and decoder, use [Backpropagation](https://vizlearn.app/ai/neural-networks/backpropagation).

## Autoencoder Edge Cases

- **Two-coordinate bottleneck:** this model has the highest mean held-out digit error of the three. That is an observed result under this training setup, not proof that every possible two-dimensional encoding must fail on these images.
- **Checkerboard input:** all three digit-trained models have MSE above 0.4. The probe is synthetic and was not part of training. Choosing it in Inputs exposes the same failure throughout the walkthrough.
- **Replay or a different speed:** reconstructed values stay identical. Playback neither trains weights nor adds random noise.
- **Width limits:** 2, 4, and 8 are the architectures supplied here, not limits on autoencoders. Larger or overcomplete codes require other constraints if the goal is to learn something more useful than copying.

## Common Mistakes

- **Giving the decoder the target.** A direct input-to-output copying path could achieve zero error while bypassing the code. It would no longer demonstrate that the code retained what reconstruction needed.
- **Treating MSE as confidence.** MSE measures pixel differences. It is not the probability that an image belongs to a digit class.
- **Expecting automatic denoising.** These models train on image-to-same-image pairs. A denoising autoencoder uses corrupted inputs with clean reconstruction targets, a different training task.
- **Sampling arbitrary latent numbers and expecting valid images.** This ordinary autoencoder learns a reconstruction map. It has no trained sampling prior; reconstruction alone does not establish a useful generative model.
- **Calling reconstruction lossless compression.** Low pixel error is still error, and latent dimension does not count encoded bits.

## A Note on Simplification

This is an explainer, not a substitute for formal study of representation learning. It uses small dense networks and 8×8 digits, groups the hidden layers into encoder/decoder blocks, and replays five recorded training checkpoints. The models were optimized with Adam for 200 epochs; the selected images were held out. Convolutional architectures, regularization, denoising objectives, variational inference, and a latent sampling prior are omitted. [Diffusion Models](https://vizlearn.app/ai/generative/diffusion-models) examines a different task: generating fresh samples from noise.