Neuron Explorer What every neuron likes
Click a photo and watch it light up the neurons of a real image model, layer by layer.
Loading the 3D view…
Each square is one neuron; layers climb from the pixels (bottom) to the answer (top). A square glows by how hard this photo fires it compared with its own favorites: orange-white means the photo would make that neuron’s top 12. Neighbors like similar photos. Towers show only the layers we studied, so a shorter tower doesn’t mean a smaller model. Threads show which neurons below pushed this one up for this photo (in ConvNeXt and our model they pass through the blocks in between).
A neuron is one number inside an image model that goes up when a photo has a certain something in it. Pick a model and a layer to see the photos each neuron fires on most, or start from a photo or a word. A few neurons have names; each one says how it was tested. Nothing here is a guess dressed up as a name.
Is each neuron doing one job? We had blind raters (Claude models working from the photos alone) look at 404 neurons picked at random: about half had an obvious job, and only 8% clearly did two. Then we investigated 30 of the unclear ones, with tests on photos we hadn't looked at, and 24 turned out to have one job too. Our estimate: about four in five neurons do one thing, if you look closely.
Watch a neuron grow up
This is our own small image model (a 9-million-parameter ConvNeXt), trained from scratch on ImageNet on a laptop and saved at 15 points through 6 epochs. For each neuron we measured how strongly it responds to each of ImageNet’s 1,000 kinds of photo, at every checkpoint, then summed the kinds it clearly prefers into familiar groups: dogs, boats, birds, food.
Two stories show up. A switch: one group fades and another takes over. A branch: the neuron keeps one group and picks up another, which the essay calls recursive generalization. We only call it one of these when the label survives at least 80% of 50 resamples of the photos.
The whole layer
Each neuron sits in the group it prefers most at each point in training (or “no clear group” when no group holds 30% of what it prefers). Columns are training epochs. Follow the streams to watch the layer find its jobs, and click any stream or block to see exactly those neurons below.
Loading…
Loading…
How we made this
The measurement. At each checkpoint we ran ImageNet’s 50,000 validation photos through the model and averaged each neuron’s response over each class’s 50 photos: a profile over 1,000 classes. We standardized it, kept the classes more than 2 standard deviations above the neuron’s own average (the ones it clearly prefers), and summed them into 32 groups taken from WordNet, the dictionary ImageNet was built from. A row’s ribbon is that group’s share. The light band reaches the top of its 5–95% range when we resample each class’s photos 50 times. Each photo is the most typical photo of one of the neuron’s top classes in that group.
The labels. We compare epoch 0.3–0.4 (when concepts start to form; before that, neurons sort by color and texture) with the last two checkpoints. A switch: the early main group held at least 35%, keeps under 10%, and another group ends with at least 35%. A branch: the early main group keeps at least 20% and a group that was under 5% early ends with at least 20%. A label counts only if it holds in at least 80% of resamples: 131 switches and 58 branches of 2,048 neurons. The median neuron changes 72% of its mix.
Is it real? We checked four ways. The labels survive resampling (above). The measurement repeats: two random halves of each class’s photos give profiles that agree at r = 0.81. The pattern of moves between groups repeats in a second training run with different random seeds, though only modestly: r = 0.59 between runs, against 0.56 for the top 5% of shuffles (p = 0.018). Fruit and vegetables → dogs and dogs → birds are the top moves in both runs. And for branches we asked whether the new concept is related to the old one: only 3 of 58 are closer than random classes by look and 9 by meaning, about what chance gives. So unlike the essay, our branches are mostly a neuron picking up an unrelated second job, not a related one.
What we got wrong first. Our first version ranked each neuron’s top 50 photos and grouped them by clustering. It failed every check: two halves of the same data agreed at only r = 0.22, the labels flipped with the clustering’s random seed, and none replicated. Measuring the whole profile instead of the top 50 fixed it.
Caveats. Dogs are 118 of ImageNet’s 1,000 classes, so moves into and out of dogs are common partly by sheer numbers. Our model is small and trained for 6 epochs; the essay’s figure follows larger models for 12, so its neurons have longer to settle.
Two groups, one trigger?
Nick Cammarata’s essay argues that a neuron which seems to do two unrelated jobs usually does one: it keys on one thing both kinds of photo share. Here we test that on InceptionV1, the model the Circuits papers mapped, reading every neuron the way the essay does: at its pole, the photos that fire it hardest.
Every photo below comes with its hot patch: the square where the neuron fires, found by covering each part of the photo and seeing which one makes it drop most.
First: can the method find what’s already known?
Distill sorted every neuron in InceptionV1’s layer 3b (and many in 4a) into families: curve detectors, eyes, fur, dog heads, stars, grids. Before trusting our method on anything new, we ran it blind on 123 of those neurons and checked its answers against theirs. The rule, written down before we looked: it must find at least 8 of the 10 published curve detectors and match the published families well above chance, or we stop and fix it.
Loading…
Then: neurons whose top photos split in two
Loading…
Decoded: one trigger behind both groups
Both raters saw one shared thing (one may call it partial), the hot patch alone still fires the neuron in both groups, and the named thing passes a test on photos it was never chosen from. Not all of these are surprises. Dogs and monkeys sharing “furry animal face”, or espresso cups and soup bowls sharing “round cup or bowl”, are unrelated only by the dictionary’s filing. The surprising ones are long flowing hair (spaniel ears and human hair), concentric rings (loudspeakers and car wheels), coils (snakes, knots and padlocks), and round tops seen from above (joysticks and water towers).
Loading…
Two jobs: the hot patches show different things
Both raters saw nothing shared. At the level of what fits in a hot patch, these look like two separate jobs. Calibration says to read that with care: the same raters missed some known families, lines and stars most of all.
Loading…
Open: maybe one thing, not proven
Loading…
The cat/car neuron
Loading…
How we tested
Written down first. The rules, thresholds and the calibration gate were committed before any rating or held-out test ran.
Discovery and test kept apart. Each ImageNet class’s 50 validation photos were split in half at random. Poles, hot patches and everything the raters saw come from the discovery half. The held-out test uses only the other half.
Which neurons. In layers 4b–4e, a neuron qualifies when the two most common of 32 WordNet groups each hold at least 10 of its top 50 discovery photos, each still holds at least 5 of its top 50 test photos, and the two aren’t the same kind of thing (vehicles; dogs, cats and wolves). We rated 30 at random per layer.
Raters. Two blind Claude Sonnet raters per neuron see only photos and hot patches, never the layer, the groups’ names or each other’s answers. For calibration, a third blind reader matched their words to Distill’s family list without seeing the right answer.
Machine checks, all from InceptionV1’s own response: the hot patch alone on gray (percentile among 50,000 real photos); edits (grayscale, swapped hues, blur, shuffled tiles, and shuffled tiles with the seams smoothed over, so seams can’t fake a layout effect); about 35 drawing families with near misses, scoring orientation-tuned ones (arcs, lines, corners) at their best angle; and a held-out test where a rater’s phrase, read by CLIP, must pick out the neuron’s top 5% of test photos outside both groups (AUC at least 0.65, and better than 95% of other raters’ phrases).
What we got wrong before. An earlier version of this section used our own small model and described each neuron by its average response per class. At the pole, half of those neurons’ top photos belonged to neither of the groups we had named them by: we were asking the wrong question. It also never checked the method against known answers. Both are fixed here.
Two kinds of photo, one job
These neurons fire on two kinds of photo that look unrelated, but on the same thing inside them. Show the hot spots to see it.
Loading…
Loading…