Source-linked AI summary
Deep Learning Models Also Recall Features
Pierre Beckmann
TL;DR
Deep learning research has emphasized feature combination, but factual recall in language models raises the question of how models generate rich information from sparse inputs. This paper defines feature recall as retrieving stored associations scaled by input activations, argues that it complements feature combination across architectures, and proposes empirical ways to distinguish the two.
Problem
Feature combination does not fully explain how models retrieve rich factual or conceptual content that is not contained in their inputs.
Method
The paper defines feature recall by rereading linear projections as input activations scaling stored weight vectors, and applies this account across deep learning architectures.
Results
Feature recall provides a complementary account of operations that feature combination struggles to explain, especially retrieval of information from learned weights.
Takeaways & Limitations
The account offers a conceptual tool for interpreting deep learning models and motivates empirical tests of whether feature recall and combination are operationally distinct.
Takeaways & Limitations
The two readings are mathematically equivalent for matrix multiplication, so establishing an operational distinction requires an appropriate mechanistic criterion and empirical investigation.
Abstract
from arXiv · showhide
Recent work in mechanistic interpretability has studied how large language models recall facts stored in their weights. This paper argues that factual recall points to something broader: a general kind of operation in deep learning models, which I call feature recall. The core observation is that a linear projection can be read as retrieving stored information scaled by input activations. I define feature recall, show it applies across architectures, and contrast it with the established paradigm of feature combination. I also consider how cases of feature recall might be mechanistically identified. The account gives philosophers a new conceptual tool for understanding deep learning, and points to empirical directions for mechanistic interpretability research.
Introduction
The paper argues that factual recall in language models reveals a broader operation called feature recall, complementing feature combination as a way to understand deep learning.
- Recent mechanistic-interpretability research on factual recall motivates a broader account of internal operations in deep learning models.
- Feature combination describes hierarchical composition from lower-level features into increasingly abstract representations.
- Feature recall explains cases where models retrieve stored information rather than assemble content from the input prompt.
- The paper presents feature recall as a complement to, rather than a replacement for, feature combination.
1 Factual recall in LLMs
The paper describes factual recall in language models as feature activations triggering associations stored in learned weights, while contrasting this operation with feature combination.
- Mechanistic interpretability analyzes transformer computation through the residual stream, where information accumulates as layers add vectors to token representations.
- Attention heads route information between token positions, while MLPs perform localized transformations that include feature combination and factual recall.
- A linear projection can encode a direct association in which activation of one feature retrieves another feature through a learned weight.
- Feature combination uses weights to detect configurations of inputs, whereas feature recall uses weights to store information for retrieval.
- Feature recall generalizes factual recall to conceptual or potentially incorrect associations without assuming the encoded connection is factual or correct.
- The paper proposes that feature recall is a general operation across deep learning architectures because each relies on learned linear projections.
2 Feature recall in deep learning models
Feature recall reinterprets learned linear projections as retrieving stored feature activations scaled by input features, complementing feature combination across deep learning architectures. This lens helps explain detailed outputs generated from sparse prompts and connects weight-based associations to possible dispositional beliefs.
- Deep learning architectures rely on learned matrix multiplications, including CNN kernels, MLP weights, attention projections, and LSTM gates.
- Feature recall treats one input feature as triggering one or many output features, with weights specifying what to retrieve rather than what to detect.Feature combination instead treats many input features as jointly producing one output feature.
- Feature recall applies across architectures because input feature activations scale stored weight vectors, retrieving associated output feature activations.In a digit-classifying CNN, a circle feature could retrieve features for digits 0, 6, 8, and 9.
- Stored associations can be retrieved and chained through successive projections, allowing a short prompt to produce a detailed biography or complex generated image or video.
- Feature recall may distinguish dispositional beliefs stored in weights from occurrent beliefs represented by features active during inference.The paper presents this mapping as a possible framework and leaves its careful exploration for future work.
3 Interpretive lens or operational kind?
Feature combination and feature recall are mathematically equivalent readings of matrix multiplication, but the paper considers whether empirical criteria could make them operationally distinct. It proposes effective connectivity and a connectivity ratio as starting points for testing that distinction.
- Any matrix multiplication supports both feature-combination and feature-recall readings, so the distinction may collapse into a difference in perspective.
- On a hermeneutic reading, projections are categorized by which interpretation is more explanatorily useful, although this blurs the boundary between the operations.
- An operational distinction would require an empirical criterion because mathematical formalism alone cannot separate the two readings.
- Feature combination has many-to-one connectivity, whereas feature recall has one-to-one or one-to-many connectivity dominated by sparse inputs.
- A connectivity ratio could measure how concentrated weight mass is on one input, with values near 1 suggesting recall and values near 0 suggesting combination.
- The ratio should be computed over interpretable features rather than raw neurons, because neurons commonly superpose multiple features.Dictionary-learning methods such as transcoders would be needed to recover those features first.
- Testing whether sparse connectivity corresponds to retrieval and whether ratio distributions are bimodal could reveal whether the criterion tracks a real boundary.The paper presents these as empirical investigations rather than established results.
- The paper’s stronger operational reading remains a proposal requiring empirical investigation, while the weaker interpretive reading is already presented as useful.
Conclusion
The paper presents feature recall as a complement to feature combination: input features can trigger retrieval of information stored in learned weights. It also proposes context-dependent analysis as a possible route toward mechanistically distinguishing the two operations.
- Feature recall describes how input features trigger retrieval of information stored in learned weights.
- A linear projection can be interpreted either as combining inputs into outputs or as retrieving stored associations scaled by input activations.
- Feature recall helps explain how deep learning models generate rich outputs from sparse inputs, where feature combination alone may be unsatisfying.
- A context-dependent ratio that scales weights by actual forward-pass activations might identify whether a projection performs feature recall or feature combination in a specific context.