Mechanistic interpretability
The research field that tries to read an AI's internal computation part by part and explain, as a mechanism, why it produced an answer.
Instead of probing inputs and outputs from outside, it decomposes the numbers inside a neural network into parts people can understand (features) and the connections between them (circuits). The 2020 paper "Zoom In" by Chris Olah and colleagues found parts that detect curves or wheels inside an image-recognition network, and circuits that combine them, setting out the field's approach[1].
The main subject now is large language models, where internal procedures — such as choosing a rhyme before composing the line — are becoming visible. But what has been decoded are fragments of limited tasks, and whether a whole model can be understood remains open.
Related terms
- FeatureIn interpretability research, a unit of an AI's internal representation that corresponds to a human-understandable concept, such as the Golden Gate Bridge.
- Sparse autoencoderAn auxiliary neural network used to decompose an AI's internal activity into combinations of a very large number of features, only a few of which are active at once.
- NeuronThe smallest unit of computation in a neural network. It weights and sums its inputs and produces an output depending on the result.