PaperlayerView exact revision on arXiv

Source-linked AI summary

Friends and Grandmothers in Silico: Localizing Entity Cells in Language Models

Itay Yona, Dan Barzilay, Michael Karasik, Mor Geva

arXiv:2604.01404v2cs.CLcs.AI

TL;DR

The paper asks how language models retrieve entity-specific facts from their parameters and tests whether sparse MLP neurons provide localized access points. It ranks neurons by cross-prompt stability across seven models, then uses interventions to assess causal effects, finding the clearest evidence in Qwen2.5-7B base. These cells concentrate in early layers, support entity-specific recall, and remain robust across varied surface forms, although the strongest evidence is limited in scope and generalization remains an empirical question.

  • Problem

    The paper investigates where and how language models anchor entity-specific factual access during inference.

  • Method

    The authors rank MLP neurons by activation stability across varied prompts about each entity, then test localized candidates with causal interventions across seven models.

  • Results

    In Qwen2.5-7B base, early-layer cells show the clearest causal evidence: ablation selectively impairs matched-entity recall, while injection is mostly single-neuron sufficient on the known-answer subset.

  • Takeaways & Limitations

    The findings support a localized retrieval picture in which sparse early-layer neurons provide access to identity-level information used by downstream factual extraction.

  • Takeaways & Limitations

    The study uses one dataset and has strongest, most complete evidence in Qwen2.5-7B base, so generalization across model families remains an empirical question.

Abstract

from arXiv · show

How do language models retrieve entity-specific facts from their parameters? We investigate this question by searching for sparse, entity-selective MLP neurons - which we call entity cells, by analogy to the "grandmother cell" hypothesis in neuroscience - and testing whether they play a causal role in factual recall. We localize candidate entity cells by ranking MLP neurons for activation consistency across varied prompts about the same entity, applying this procedure across seven models on a curated subset of PopQA. In all models, localized neurons cluster predominantly in early layers, an empirical pattern not imposed by the architecture. Using Qwen2.5-7B base as a model organism, we find the clearest causal evidence: suppressing a localized cell selectively erases recall for its matched entity while leaving others intact, and activating a single cell is sufficient to recover correct knowledge for most entities - even when the entity is absent from the context. The same cells are recovered under aliases, acronyms, misspellings, and multilingual surface forms, and remain stable through instruction tuning, suggesting they encode canonical entity identity rather than surface token patterns. Causal signals vary across model families, pointing to architectural differences in how entity knowledge is organized. These findings offer concrete, interpretable access points for understanding, controlling, and correcting factual knowledge in language models, and draw a surprising empirical parallel to longstanding questions in neuroscience about sparse coding of concepts.

1 Introduction

The paper asks whether factual retrieval is anchored in sparse, entity-selective MLP neurons and develops a stability-based localization and intervention pipeline to test this. Across seven models, candidates cluster in early layers, while Qwen2.5-7B base provides the clearest causal evidence that these cells support entity-specific recall and remain robust across surface forms.

  • Motivation: The central question is whether factual access is distributed gradually or mediated by compact, localized entity representations during inference.The paper frames entity resolution and attribute retrieval as a core mechanistic-interpretability problem.
  • Approach: Entity cells are sparse, entity-selective MLP neurons whose activations remain stable across varied prompts about the same entity.The method ranks neurons by cross-prompt activation consistency at the entity mention position.
  • Findings: Across 7 models from 5 families, localized candidates consistently appear in early layers, with the strongest and most consistent effects in Qwen2.5-7B base.The tested suite includes Qwen, OLMo, Llama, Mistral, and OpenLLaMA models.
  • Causal tests: Negative ablation induces entity-specific amnesia, while controlled injection can restore matched-entity factual recall relative to mean-entity and wrong-cell controls.These interventions test causal leverage rather than correlation alone.
  • Robustness: The same localized cells remain stable across aliases, acronyms, typos, and multilingual forms, supporting access to canonical entity identity rather than one token string.This robustness links the localized neurons to entity-level representations across surface forms.

4. We characterize key properties of localized

The paper highlights robustness of localized entity cells across varied surface forms and releases the resources needed to reproduce these analyses.

  • Key properties: Localized entity cells remain robust across aliases, acronyms, misspellings, and multilingual variants.The result supports stability beyond a single lexical realization of an entity.
  • Key properties: This robustness is presented as evidence that the cells provide access to entity identity across surface forms.
  • Resources: The authors release code, prompts, and data for the study.The release is available at the repository named in the passage.

2 Related Work

The paper builds on work localizing factual behavior, forming entities in early layers, interpreting MLPs as memory, and editing model knowledge. It distinguishes its contribution by targeting entity-level retrieval with reversible causal interventions.

  • Factual recall and localization: Prior studies localized factual behavior to components and layers, often emphasizing relation-specific recall pathways.This paper instead varies attributes while keeping the entity fixed.
  • Detokenization and entity formation: The paper tests whether localized neurons persist across aliases, acronyms, misspellings, and multilingual forms as a link to canonical entity representation.
  • MLP memories and sparse features: Its findings are consistent with MLP key-value memory accounts while showing that a sparse neuron-level handle can recover entity-consistent behavior under intervention.
  • Editing and control: Unlike parameter-editing methods such as ROME and MEMIT, the paper uses reversible activation interventions to isolate retrieval-time causal effects.The approach is intended to separate entity access from persistent weight editing.
  • Editing and control: The paper's main contribution is a causal account in which sparse early-layer neurons often act as compact entity access points, though not universally or exclusively.

3 Method

The method extracts and normalizes entity-position MLP activations, ranks neurons by cross-prompt stability, and tests candidates with injection and negative ablation. Evaluation uses next-token answer probabilities and control-normalized metrics.

  • Activation extraction: The pipeline extracts each MLP neuron's down-projection activation at the entity token position.The scalar activation is the channel value immediately before the MLP down-projection.
  • Normalization: Activations are standardized using each neuron-layer pair's mean and standard deviation over generic prompts.
  • Stability ranking: For prompts sharing an entity, the stability score rewards strong mean activation and penalizes relative variability across prompts.The score is equivalent up to epsilon to |E[z]| divided by the coefficient of variation.
  • Interventions: The causal tests use controlled injection and negative ablation on localized cells.Injection overwrites a chosen activation, while negative ablation scales it by α, including sign-flipping α < 0.
  • Evaluation: Answer evaluation uses the first-token probability of the best-matching answer alias, with RelProb, pass@k, and an unknown-entity baseline supporting filtering and intervention tests.Pass@k is computed from next-token logits without sampling.

4 Experimental Setup

The study evaluates seven language models using PopQA-200, a curated 200-entity subset, and templated prompts to localize and test entity cells. Experiments identify entity positions, normalize activations, and use causal interventions and layer-localization analysis.

  • Models: Experiments cover seven models from five model families, with Qwen2.5-7B base as the main model for causal analysis.The evaluated models include Qwen, OLMo, Llama, Mistral, and OpenLLaMA variants.
  • Data: PopQA-200 contains 200 popular entities and supplies both the localization inventory and downstream causal-evaluation questions.The subset includes countries, cities, and widely known people, with at least two available questions per entity and two questions used per entity for causal checks.
  • Prompting: Localization uses templated entity prompts and records activations at the final token of each tokenized entity span.For evaluation, PopQA questions are formatted as “Question: <question>\nAnswer:”, while generic probing uses cloze-style “Fact: ...” prompts.
  • Localization and interventions: Candidate cells are selected by ranking MLP-neuron activations for stability across prompts, with interventions applied at the entity or placeholder position.The entity position is the final token in the tokenized subject span; cloze prompts use the dummy placeholder X as the analogous position.
  • Localization analysis: Figure 2 maps the layer of the top localized cell for each of the 200 PopQA-200 entities in Qwen2.5-7B base.Comparable early-layer concentration is reported for other tested models in Appendix G.

5 Results

Across the experiments, localized cells concentrate in early layers and show causal leverage over entity-specific recall. In Qwen2.5-7B base, ablation impairs matched-entity recall while injection restores it, and the same cells remain robust across varied surface forms.

  • Cross-model comparison: Localization patterns vary across model families: non-Qwen distributions are typically broader and shifted deeper than the dedicated Qwen-family plots.The comparison is reported as a localization-only analysis in Figure 21.
  • 5.1 Localizing Entity Cells Finding 1: 99.0% of entities peak in layers 0–5, while 1.0% peak in layers 22 or 27, despite ranking across all 28 layers.This depth profile is empirical rather than imposed by the architecture.
  • 5.2 Causal Necessity Finding 2: 131/200 localized cells show entity-specific recall suppression under negative ablation, with controls remaining near baseline.In the Obama case study, target retention falls from 1.0 to 0.123 at α = −3, while Trump changes from 1.0 to 0.996.
  • 5.3 Causal Sufficiency: 63.3% pass@5 follows correct-cell injection, compared with 1.8% for both mean-entity and wrong-cell controls.The evaluation uses 109 instances whose entity-present prompts were already correct under pass@5.
  • 5.3 Causal Sufficiency: Single-cell injection is largely sufficient: 41/79 entities pass with top-1 versus 42/79 with top-k, and only one entity requires multi-cell injection.The intervention replaces the entity mention with X and injects the matched cell at that placeholder position.
  • 5.4 Surface-Form Robustness Finding 4: The same localized cells are recovered across spelling variants, acronyms, and multilingual forms, consistent with access to canonical entity identity rather than one token sequence.Most spelling and phrasing variants preserve the same top cell, with similar robustness for acronym and multilingual forms.

6 Discussion

The findings support a localized retrieval account in which sparse early-layer neurons provide canonical entity access and can be causally manipulated, though the phenomenon varies across models.

  • Causal evidence: 131/200 entities passed the negative-ablation trust filter, indicating localized neurons were functionally necessary for entity-specific recall in this protocol.The strongest evidence comes from Qwen2.5-7B base.
  • Causal evidence: 41/79 known-answer instances were recovered with top-1 injection versus 42/79 with top-k, showing mostly single-cell sufficiency with marginal top-k gains.The comparison is reported for controlled injection on the known-answer subset.
  • Canonicalization: Most spelling and phrasing variants, acronyms, and multilingual forms recover the same localized cell, consistent with access to canonical identity rather than a single token sequence.The paper illustrates this robustness with Barack Obama, FBI, and Paris.
  • Cross-model scope: Qwen2.5-7B-Instruct preserves nearly the same entity-cell map as the base model, while Qwen3-8B also shows sparse early-layer cells but weaker causal evidence.Post-training robustness is the clearest extension across models.
  • Cross-model scope: Across other model families, localization and causal robustness are less consistent, making entity cells a reproducible but model-dependent phenomenon.The discussion cautions that reliable single-neuron effects are common but not universal and are clearest for popular entities.

7 Limitations and Scope

The study’s scope is constrained by its dataset, sparse localization design, limited prompt sampling, and evaluation choices that may affect generalization and measured competence.

  • Dataset and model scope: The study uses one dataset, PopQA, with the strongest and most complete evidence in Qwen2.5-7B base.Generalization to other model families is presented as an empirical question requiring like-for-like replications.
  • Localization design: The sparse individual-neuron ranking may miss distributed or multi-cell codes, despite top-k variants providing only marginal gains over top-1.The design prioritizes interpretability but does not establish that entity representations are always single-cell codes.
  • Localization design: Using K=2 prompts per entity for localization and causal checks can introduce per-entity instability.
  • Evaluation limits: First-token-based metrics can understate multi-token factual competence and reflect lexical priming effects.The evaluation therefore may not fully capture factual performance beyond first-token behavior.
  • Evaluation limits: Per-entity selection of α improves sensitivity but can introduce optimistic bias, while injection and ablation remain narrow in relation coverage.A fixed-α protocol is identified as an important next step.

8 Conclusion

The paper combines stability-based localization with controlled interventions to test whether entity-associated MLP neurons support factual recall. On the trustworthy subset, top-k injection succeeds for 75 of 131 entities, while the injection scale is selected per entity and may be optimistic.

  • Method: The injection evaluation uses a mean-entity initialization before overwriting selected neurons with entity-specific values.The intervention therefore tests directional movement from an existing mean-entity state rather than reconstructing the entity representation from scratch.
  • Caveat: The per-entity sweep selects the best-performing α from 1, 2, 4, 8, 16, 32, 64, 128, and 200, which may overstate performance relative to a fixed-α protocol.This high-sensitivity selection is explicitly identified as potentially optimistic.
  • Method: The pipeline localizes entity cells by measuring activation stability across entity-centered prompts and then applies controlled injection and latent steering interventions.Localization extracts activations at the entity position, computes a stability score, and identifies one or more top neurons for intervention.
  • Results: 75/131 entities pass the controlled-injection success criterion with top-k injection, compared with 74/131 using top-1 injection.The trustworthy set contains 131 of 200 localized cells; one entity requires top-k injection.

E Post-Training Generalization (Qwen2.5-7B-Instruct)

Qwen2.5-7B-Instruct largely preserves the base model’s entity-cell map after instruction tuning. The same cells retain early-layer localization, causal usefulness, and robustness across several surface-form variations.

  • Localization stability: 190/200 entities preserve the base model’s exact top localized cell, while 191/200 preserve the same layer after instruction tuning.Barack Obama localizes to L2-N10941 in both the base and instruction-tuned models.
  • Localization stability: The early-layer concentration pattern and 123-cell amnesia-based trust filter remain preserved in Qwen2.5-7B-Instruct.These results are described as the clearest post-training evidence in the appendix.
  • Causal interventions: Correct-cell injection again outperforms mean-entity initialization and wrong-cell controls on the trusted set.This reproduces the causal usefulness of localized cells after instruction tuning.
  • Surface-form robustness: The same top cell is recovered for most tested spelling and phrasing variants, acronym and expanded forms, and multiple scripts.The replication figures describe preservation of the localized entity handle across these surface forms.

G Lack of Generalization Across Model Families

Entity-cell localization does not transfer uniformly across model families. Although candidate cells can often be localized, strong causal validation and surface-form robustness are less reliable outside the strongest model-specific cases.

  • Within-family comparison: Qwen3-8B preserves early-layer localization but shows weaker causal separation and noisier surface-form matches than Qwen2.5.Correct-cell injection improves over controls, but the separation is weaker and less consistent; related variants and cross-script forms are also noisier.
  • Cross-family results: Cross-family transfer is limited: sparse candidate cells often localize, but causal validation and form robustness do not transfer reliably.This is the paper’s aggregate conclusion from the cross-family comparison.
  • Localization depth: Non-Qwen localization profiles are generally broader and deeper than the main Qwen result, indicating that early-layer sparsity is not uniform across families.Figure 21 compares top-neuron layer distributions across four non-Qwen models.
  • Model-dependent causality: OLMo’s negative-ablation curve affects the control entity as well as the target, unlike the clean entity-specific amnesia pattern in Qwen2.5.Its cells may therefore participate in retrieval more broadly or through a different structure.
Loading 2604.01404v2…