Tiny Transformer
Inside a tiny language model
A real transformer, trained one character at a time on Twenty Thousand Leagues under the Sea. Every weight and every number is on screen and computed live in your browser. Type to it. Take it apart.
What you're looking at
The model has 2 layers, 4 attention heads per layer and 432,384 parameters. It reads up to 128 characters and guesses the next one. Training took about four minutes on a laptop.
Overview shows the whole forward pass at once. Inside draws every weight matrix and every intermediate value, a window of the text at a time. Hover any cell to see exactly how it was computed. Click a guess to trace where it came from. The Learn panel has guided chapters and missions.
At this size there are no concepts in here, just letter statistics, heads that look a fixed number of characters back, a head that copies, and some memorized names. The chapters show each one.
The exploded layout follows Brendan Bycroft's LLM visualization. The text is Project Gutenberg eBook #164.