$ cd ..
// ~/playground/neuron-census

Neuron Census 36,864 neurons, one question

A language model we trained ourselves, opened up: every neuron, what it fires on, and whether it has one job.

We trained a GPT from scratch: 162M parameters, 3B tokens of web text, about $9 of rented GPU. It beats OpenAI’s GPT-2 small on both yardsticks we checked: validation loss 3.24 against 3.28, and the HellaSwag common-sense test 30.9% against 29.5%.

Then we asked it the question from Nick Cammarata’s essay Neural Networks Are Already Clean: if you look properly, does each neuron have one main job? He found about 90% do in a 32-billion-parameter model. Ours is 200 times smaller.

$ echo # pick a sentence, then a word in it

Loading the 3D view…

Each square is one neuron. The 12 floors are the model’s 12 layers, from the words coming in (bottom) to the guess at the next word (top). A square glows by how hard the chosen word fires it compared with that neuron’s own best: white-hot means this word is as strong as anything in 100M tokens of text. Neighbors on a floor tend to fire on the same words. Brighter idle squares are the 240 neurons we had rated.

$ ls favorites/ # neurons both raters named the same way

Some of its neurons

Two raters looked at each neuron’s strongest text without knowing which neuron, layer or side it was, and wrote down its job. These are ones where both, independently, wrote the same thing. Click one to find it in the tower.

Loading…

$ census --score # 240 random neurons, both sides each, two blind raters

The scoreboard

Loading…

$ ablate # delete firings from every neuron at once

Delete the weak, delete the strong

Loading…

$ steer # hold one neuron high and let the model write

Turn one up

Loading…

$ git log --neuron # the same neuron through training

Watch one grow up

Loading…

$ cat METHOD.md # what we did, what we got wrong

What we got wrong on the way

Three mistakes, each caught before the numbers above, each of which changed them:

  1. We only looked at half of each neuron. Our first census read each neuron after its activation function (ReLU², which turns every negative value into zero), so the negative side, half of what Cammarata rates, was invisible. The census here reads the value before it.
  2. The first few words of a passage fooled us. We feed the model text in 1,024-token windows, and at the very start of a window it has almost no context. Neurons go extreme there for no reason related to meaning: 88% of the first layer’s top examples sat in the first four positions. The first audit called layer 0 mostly unclear (10% one job). Skipping the first 64 positions of every window, its positive sides came out 68% one job.
  3. Our first “weaker examples” were noise. We first took them from a percentile band over all text, but most neurons are near zero almost everywhere, so that band sat at a few percent of the neuron’s peak. Raters rightly called it junk. The weaker examples here are 25–50% of each neuron’s own peak.

Method, and what to take with salt

  • The model: a 12-layer GPT, 162M parameters, trained from scratch on 3B tokens of FineWeb in two hours on one rented H100. A neuron is one of the 3,072 hidden units in a layer’s MLP, read before the ReLU².
  • The census: 100M tokens of held-out FineWeb text, one example per document, window positions 64 and later. Even so, top examples still lean toward earlier positions (only 10% come from the last quarter of a window, against 25% if position didn’t matter), so a position-adjusted ranking is still to do.
  • The raters are Claude Sonnet subagents, not people. Each sheet got two of them, blind to the neuron, its layer, which side it was, and each other. They agree on the strong-examples verdict 78% of the time (Cohen’s κ 0.62), but only 57% (κ 0.31) once weaker examples are included, so treat that column as soft. Two raters from the same model family also share blind spots that two people wouldn’t.
  • Negative sides. Our MLP uses ReLU², which outputs exactly zero, with zero gradient, for anything negative. So nothing downstream ever sees a neuron’s negative side, and training never pushes it to mean anything. Cammarata’s models use SwiGLU, whose negative side still carries signal. That is our best explanation for why our negative sides are much messier than his, not a test of it.
  • Size. His numbers come from Qwen3-32B, 200 times bigger. He also notes smaller models have coarser neurons.
  • The text is from FineWeb (ODC-By 1.0), raw web pages. Snippets that trip an explicit-content wordlist are swapped for the neuron’s next example, and phone numbers and email addresses are partly masked (a few snippets were swapped).

$ cd .. → more interactives