Reinforcement Learning Visualizations
An agent takes an action, receives a reward, and uses that experience to change its next decision. A reward can arrive several decisions after the action that made it possible. These lessons show two ways to learn from that delayed feedback.
Q-learning stores an estimate for each state and action. Policy gradient changes the probabilities of actions directly. Both learn from sampled experience, but the quantity they update and the evidence used for that update differ.
What These Lessons Cover
| Lesson | What you watch |
|---|---|
| Q-Learning | A +5 terminal reward first changes one table entry. Later visits carry that estimate back toward the start, eventually making a three-step route preferable to an immediate +1. |
| Policy Gradient | REINFORCE weights each sampled action's gradient by its return. A failed episode can reduce the probability of a useful first action, even though that action is necessary to reach the goal. |
A Recommended Learning Order
Start with Q-Learning. The grid and its six legal state-action pairs make each update inspectable. Follow the observed reward, the maximum estimate at the next state, and the single entry written back to the table. Then set the exploration rate to zero: in this seeded run, the agent keeps taking the nearby reward and never discovers the larger one. Set the discount factor to zero to see a different limit: even a discovered reward cannot affect earlier decisions through the update target.
Continue with Policy Gradient. The agent now samples from learned action probabilities, with no Q table. Watch how positive and negative returns change the probability of the sampled action. The first failed route is especially useful: its update discourages both decisions, including a first turn that the successful route also needs. Later episodes show why the direction of one sampled update differs from the expected direction over many samples.
Where These Lessons Sit
Gradient Descent introduces a parameter update and learning rate. Policy gradient uses gradient ascent to increase expected return; Softmax & Cross-Entropy provides background on action probabilities and log-probabilities.
The Training Loop separates learning from evaluation. Q-learning makes that separation visible by freezing its table before following the final greedy policy. These small, deterministic environments expose individual updates. They do not establish convergence from a handful of episodes or cover the additional difficulties of learning with a neural network.
All Reinforcement Learning Visualizations
Open a lesson below to play the default example, inspect an update, or test a different parameter setting.