Why AI Neuron Explanations Often Mislead — and How Researchers Are Fixing It

Researchers have long tried to interpret individual neurons in AI models by examining the inputs that trigger their strongest activations, but this method produces unreliable and often misleading conclusions. A key problem is that top activations represent only a tiny fraction of a neuron's behavior, while moderate activations across millions of tokens can have a greater overall impact on model output. Human pattern-matching compounds the issue, as people reliably find patterns in any set of examples without a testable prediction to falsify. A 2021 study by Bolukbasi and colleagues further showed that the same neuron can yield different plausible-looking explanations depending on which dataset is used. In 2023, OpenAI researchers introduced a more rigorous pipeline that uses one language model to explain a neuron and another to score the explanation by predicting held-out activations, turning interpretations into falsifiable hypotheses.