PaperlayerView exact revision on arXiv

Source-linked AI summary

Grammatical "grandmother neurons" are rare in LLMs

Linyang He, Nima Mesgarani

arXiv:2609.29328v1cs.CL

TL;DR

The paper asks how grammatical structure is represented at the level of individual neurons, beyond what behavioral tests or whole-vector probes reveal. It uses permutation-normalized, probe-free comparisons of minimal-pair activations across linguistic paradigms. The results show sparse, weak, narrowly tuned single-neuron selectivity and a dissociation among neuron selectivity, vector decodability, and behavior.

  • Problem

    How much grammatical competence is expressed in individual neurons, and is that selectivity broad or narrow, remains an open question not answered by behavioral tests or whole-vector probes.

    Source: §1 · pp. 1–2

  • Method

    The paper measures neuron-level selectivity directly from grammatical and ungrammatical minimal-pair activations and normalizes observed separability against a within-pair label-swapping null.

    Source: §2 · p. 2

  • Results

    Across seven checkpoints, normalized single-neuron selectivity is sparse, weak, and narrowly tuned, while whole-vector decodability, NSI, and behavioral competence are largely dissociated.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Whole-vector decodability and activation selectivity alone do not establish that the model behaviorally relies on the represented distinction.

    Source: §1 · p. 2

    The findings are bounded to seven tested models and controlled, templated English benchmarks, not universal across model types or linguistic settings.

    Source: Limitations · p. 15

Abstract

from arXiv · show

Understanding how Large Language Models (LLMs) encode linguistic structures remains a fundamental challenge in interpretability research. While diagnostic classifiers (or "probes") are widely used for this task, they face significant methodological criticism: training auxiliary classifiers introduces capacity confounds and calibration issues, often making it difficult to distinguish the model's intrinsic representations from the probe's ability to learn the task. To address these limitations, we introduce a probe-free framework for localizing linguistic selectivity at the individual neuron level. Leveraging the controlled contrasts of linguistic minimal pairs, we propose a Neuron Separability Index (NSI), a metric that directly quantifies how reliably single neurons differentiate grammatical from ungrammatical constructions without parameter updates. Applying NSI across 68 linguistic paradigms and seven checkpoints reveals three main patterns: 1) raw separability reaches near-peak levels earlier for morphological and syntactic distinctions than for syntax-semantics interface and conceptual distinctions. 2) after permutation normalization, single-unit selectivity is sparse, weak, and narrowly tuned: only a small fraction of units are sensitive to an average paradigm, and strongly selective "grandmother neurons" are rare. 3) whole-vector linear separability, single-neuron selectivity, and behavioral competence are largely dissociated, and targeted ablations further separate activation selectivity from causal reliance.

1 Introduction

The paper asks whether grammatical competence is localized in a few highly selective neurons or distributed across units, a question existing behavioral, probe-based, and phenomenon-specific studies do not settle. It introduces a probe-free minimal-pair measure and finds sparse, weak, narrowly tuned selectivity, distinct from whole-vector decodability and behavioral reliance.

1 Introduction

  • Existing behavioral tests reveal grammatical competence but not internal organization, while whole-vector probes and studies of only a few phenomena do not establish how selectivity is distributed across individual units.The paper identifies a need for a measurement defined at the unit level, comparable across linguistic phenomena, and free of auxiliary training.Source: §1 · pp. 1–2
  • The central question is how much grammatical competence is expressed in individual neurons and whether their selectivity is domain-general or domain-specific.The proposed framework uses minimal-pair contrasts and reads selectivity directly from activations without parameter updates or an auxiliary classifier.Source: §1 · p. 2
  • Across seven checkpoints, raw separability reaches near-peak levels earlier for morphological and syntactic distinctions than for syntax–semantics interface and conceptual distinctions, while normalized selectivity is sparse, weak, and narrow.Strongly selective units are rare, and even poly-selective neurons retain a dominant within-domain preference.Source: §1 · p. 2
  • Whole-vector decodability, single-neuron selectivity, and behavioral competence are largely dissociated, and targeted ablations further distinguish activation selectivity from causal reliance.Probe accuracy is essentially unrelated to behavioral accuracy across paradigms; NSI is also only weakly associated with behavior.Source: §1 · p. 2

2 Minimal pair-based Neuron Separability

The method uses controlled minimal-pair contrasts to measure neuron-level grammatical selectivity across a hierarchy of linguistic benchmarks. It computes raw separability from paired activations, then normalizes against label-swapped pairs to distinguish contrast sensitivity from lexical effects and noise.

2 Minimal pair-based Neuron Separability

  • The benchmark organizes BLiMP and COMPS into 4 domains, 13 phenomena, and 68 paradigms, from specific contrasts to broader linguistic groupings.A domain is the broadest grouping, a phenomenon groups related paradigms, and a paradigm is one specific benchmark contrast.Source: §2 · p. 2

2.1 Raw separability score.

  • For each neuron and paradigm, raw separability measures consistent within-item contrast between grammatical and ungrammatical activation vectors using their correlation.A strong, consistent difference across paired items produces a low correlation and a raw score close to 1; nearly identical activation patterns produce a score near 0.Source: §2.1 · pp. 3–4

2.2 Permuted neuron separability index.

  • NSI expresses observed separability relative to a null made by swapping grammatical labels within half of the pairs, controlling for lexical content and sampling noise.NSI>0 marks sensitive neurons and NSI>2 marks strongly sensitive neurons; the latter is a descriptive threshold, not a Gaussian significance claim.Source: §2.2 · p. 4

3 Experimental Setup

The experiments test the approach across seven decoder-only checkpoints and include robustness, probing, behavioral, and ablation comparisons. The setup also defines hierarchical tuning breadth and clusters neurons by their selectivity profiles across linguistic phenomena.

3 Experimental Setup

  • The study tests seven decoder-only checkpoints from four model families, evaluates robustness to thresholds and pair construction, and compares NSI with behavior, probing, and targeted ablations.Representations are taken at the final non-padding token; ablation checks compare candidate dimensions with bottom signed-score sets and same-layer random sets.Source: §3 · pp. 4–5

3.1 Quantification of Hierarchical Tuning Breadth

  • Tuning breadth counts the linguistic categories to which a neuron responds at paradigm, phenomenon, and domain levels, averaging the breadth across units in each layer.A phenomenon or domain counts as responsive when the neuron responds to at least one constituent paradigm; the metric is used as a proxy for polysemanticity.Source: §3.1 · p. 5

3.2 Clustering neurons by linguistic phenomenon selectivity

  • For clustering, neuron selectivity is averaged across each phenomenon, neurons with breadth of at least three are selected, and their phenomenon profiles are clustered by correlation distance and average linkage.The resulting neuron groups are displayed against phenomena ordered by domain: Concept, Syntax–Semantics Interface, Syntax, and Morphology.Source: §3.2 · pp. 5–6

4 Results

Raw separability reaches saturation earlier for morphological and syntactic distinctions, but normalized neuron selectivity is sparse, weak, and narrowly tuned. Across checkpoints, neuron selectivity, whole-vector decodability, and behavior remain distinct.

4.1 Raw Separability Patterns

  • Morphology (8.4 layers) and Syntax (9.5) reach 90% saturation before Concept (19.0), while Syntax–Semantics Interface averages 10.3 layers.Figure 2 shows layer-wise mean separability; the saturation ordering is consistent with the domain-level analysis in Appendix Figure 8.Source: §4.1 · p. 6, Figure 2 · p. 6

4.2 Neuron Separability Index Patterns

  • Only 2.50% of Syntax, 3.31% of Syntax–Semantics Interface, 3.88% of Morphology, and 8.27% of Concept units are sensitive on average.Sensitivity is highest in early-to-middle layers and declines toward deeper layers.Source: §4.2 · p. 6, Figure 3 · p. 7
  • For 65 of 68 paradigms, no neuron exceeds NSI>2; in each of the other three, only one neuron does.Among sensitive neurons, mean separability is typically around 0.5.Source: §4.2 · p. 7, Figure 4 · p. 8
  • Most neurons respond to only a few paradigms, with breadth falling sharply early, staying low through middle layers, and rising modestly near output.Breadth counts paradigms with NSI>0.Source: §4.2 · p. 7, Figure 5 · p. 8
  • Even neurons responsive to at least three phenomena retain dominant within-domain preferences rather than acting as uniform cross-domain grammar detectors.Their responses to other phenomena are weaker, consistent with spillover around a dominant preference.Source: §4.2 · p. 8, Figure 6 · p. 9

4.3 Robustness and Generalization

  • Across seven checkpoints, an average paradigm recruits only 2.52–3.94% of units, and a neuron responds to just 1.72–2.68 of 68 paradigms on average.Each checkpoint has only 2–6 paradigms with any NSI>2 unit.Source: §4.3 · p. 8, Figure 7 · p. 10
  • At NSI>1, above-threshold units average 0.094% of the network; from NSI≥1.645 through NSI>3, only 3 of 68 paradigms contain any such unit.In a clean one-prefix deletion subset, removing the critical contrast token reduces raw separability from 0.0753 to 0.0011, while deleting a matched non-critical token leaves it at 0.0746.Source: §4.3 · pp. 8–9
  • Probe accuracy and minimal-pair behavior are effectively unrelated (Pearson r=−0.032), and targeted groups are no more damaging than random ablations.Across the nine group comparisons, all empirical p≥0.109 and absolute accuracy changes are at or below 0.009.Source: §4.3 · p. 9, Table 1 · p. 10

5 Conclusion

The paper introduces NSI for probe-free neuron-level linguistic selectivity and finds that highly selective units do not show uniquely strong causal effects in targeted ablations.

5 Conclusion

  • NSI measures neuron-level linguistic selectivity from minimal pairs across layers, domains, phenomena, and paradigms.The measure is probe-free and applied consistently across those levels.Source: §5 · p. 10
  • Ablations of groups containing three rare NSI>2 candidates and their highest-scoring same-layer neighbors were indistinguishable from same-size random controls.This reinforces the distinction between activation selectivity and causal reliance.Source: §5 · p. 10
  • Future work will use patching and mediation analysis to connect neuron clusters to causal circuits and extend the analysis to multilingual minimal pairs.Source: §5 · p. 10

Limitations

The paper’s conclusions are bounded by its tested models, English benchmarks, representation choices, and intervention scope; related work motivates systematic neuron-level measurement across linguistic phenomena.

Limitations

  • The study covers seven decoder-only checkpoints and controlled, templated English benchmarks, so its conclusions are not universal.BLiMP and COMPS under-represent natural discourse, pragmatics, gradient acceptability, and typologically diverse languages.Source: Limitations · p. 15
  • Causal tests cover three paradigms in one checkpoint; held-out phenomena and circuit-level interventions would provide stronger generalization and causal tests.NSI measures activation separability, while the intervention tests selected residual coordinates.Source: Limitations · p. 15
  • Last-token layer outputs and scalar dimensions may miss earlier-position, temporally distributed, attention, or MLP signals.Neuron definitions also vary across architectures, and alternative positions, components, or readouts may change selectivity patterns.Source: Limitations · p. 15
  • The matched within-pair permutation null preserves lexical content, but remaining lexical, tokenization, template, or item dependencies could be modeled more explicitly.Robust normalization requires extensive permutation testing, although independent permutations can be parallelized.Source: Limitations · p. 15

A Related Work

  • Probes can exploit their own task-learning capacity, motivating diagnostics that measure selectivity without auxiliary training.Prior work also reports sensitivity of probe rankings to design choices.Source: A Related Work · p. 16
  • Earlier neuron-level studies targeted one or two phenomena, leaving open how typical strong single-unit selectivity is across grammar.This study applies one classifier-free index across 68 paradigms and 13 phenomena in four domains, replicated across seven checkpoints from four families.Source: A Related Work · p. 16

B Dataset

The paper combines minimal-pair resources, analyzes neuron-level response profiles with NSI, and compares raw-separability localization with whole-vector probing across linguistic domains.

B Dataset

  • COMPS adds meaning-sensitive conceptual and semantic-compositional contrasts, pooled as comps_base in the Concept domain.Its pairs preserve as much lexical overlap as possible while flipping a compositional or conceptual requirement.Source: B Dataset · p. 17
  • 116,300 English minimal pairs form a hierarchy of four domains, 13 phenomena, and 68 paradigms.The suite combines 67 BLiMP paradigms with one pooled COMPS paradigm; human agreement is 96.4% for BLiMP and 93.1% for COMPS.Source: B Dataset · p. 17

C Models

  • Generality is tested on six additional checkpoints: three larger Qwen3 models, Pythia-410M, TinyLlama-1.1B, and Llama-3.1-8B.Source: C Models · p. 17

D What does NSI measure?

  • Sraw uses Pearson correlation between paired activation vectors to measure changes in a neuron’s item-wise response profile, invariant to overall response level and scale.A uniform offset or positive gain preserves relative ordering and receives the minimum score; an order-reversing response receives the maximum.Source: D What does NSI measure? · p. 18
  • NSI tests whether profile reorganization exceeds what within-pair label swapping produces, focusing on item-specific selectivity rather than mean-level activation changes.The uncentered cosine variant retains mean-level effects as a complementary aspect of the paired contrast.Source: D What does NSI measure? · p. 18

E Raw-Score Layer Localization

  • 19.0, 10.3, 9.5, and 8.4 are the mean 90% saturation layers for Concept, Syntax–Semantics Interface, Syntax, and Morphology, respectively.Each paradigm’s saturation layer is the earliest layer reaching 90% of its own raw-separability maximum; the values average paradigm-level layers by domain.Source: Figure 8 · p. 19
  • Raw-separability and probing saturation layers show weak positive correspondence across paradigms (Pearson r = 0.15; Spearman ρ = 0.23).Saturation is measured separately for each paradigm and method.Source: Figure 9 · p. 20

F.1 Cross-model robustness

Across the reported selectivity and tuning-breadth distributions, strongly selective units and broad tuning are uncommon, including across paradigm, domain, and phenomenon levels.

F.1 Cross-model robustness

  • 5–7 paradigms is the 95th-percentile tuning breadth, and fewer than 1% of neurons respond to 10 or more paradigms in every model.This high-breadth tail remains small even when any NSI>0 counts as responsive.Source: Figure 10 · p. 21

F.2 Selectivity distributions and high-selectivity cases

  • Only three paradigms contain a unit with NSI>2, and each such case is realized by one unit across the network.The three cases are in the main 500-permutation Qwen3-0.6B analysis.Source: §F.2 · p. 21, Figure 12 · p. 23
  • ∼0.5 is the typical mean NSI among sensitive neurons across paradigms, indicating generally weak separability.Source: Figure 11 · p. 22

F.3 Tuning breadth across domains and phenomena

G Similarity Metrics and Threshold Calibration

Threshold sweeps show that strongly selective units are rare, while empirical-null calibration is too coarse for a powerful significance test. Spearman and cosine checks preserve the sparse-selectivity conclusion, but Pearson is better matched to the neuron-level estimand.

G Similarity Metrics and Threshold Calibration

  • At NSI>1, an average Qwen3-0.6B paradigm contains only 27.1 units out of 28,672; at thresholds from 1.645 to 3, only three paradigms contain any.Each of those three paradigms has one above-threshold unit.Source: G Similarity Metrics and Threshold… · p. 24, Table 3 · p. 24
  • No unit survives empirical-null correction at q<0.05 or q<0.01, but 200 permutations make the minimum attainable p-value 0.004975 and the nulls are non-Gaussian.The authors therefore treat NSI>2 as an operational strong-selectivity threshold, not a literal Gaussian significance cutoff.Source: G Similarity Metrics and Threshold… · p. 24
  • At z>2, Spearman and cosine leave more than 99.7% of units below threshold, while Pearson is retained because it centers each neuron's item-wise response pattern.Spearman discards graded response magnitudes, and cosine does not center responses across items.Source: G Similarity Metrics and Threshold… · p. 24

H.1 Random-pair controls

Pairing and deletion controls test whether NSI reflects the intended grammatical contrast rather than sentence mismatch or critical-token removal. Item alignment matters, and critical-token deletion is cleanly interpretable only for the one-prefix subset.

H.1 Random-pair controls

  • Random same-label pairs produce no NSI>2 units, whereas cross-item good–bad pairs generate sporadic outliers, supporting the need for item alignment.The original 200-permutation control run has four above-threshold paradigms; this should not be conflated with the three in the 500-permutation main analysis.Source: §H.1 · p. 25, Table 4 · p. 25

H.2 Critical-token deletion

  • In the clean 10-paradigm one-prefix subset, deleting the critical token nearly eliminates raw separability, while random deletion does not.Two-prefix paradigms are excluded from the main contrast-removal conclusion because the annotated shared target is not the contrasting word; near-identical pairs also make normalized NSI unstable.Source: §H.2 · p. 25, Table 5 · p. 25

I Probe, NSI, and Behavior

Across BLiMP paradigms, whole-vector probing, single-neuron NSI, and behavioral accuracy capture different properties. High probe accuracy can coexist with poor model behavior, while mean NSI is uncorrelated with behavior and negatively correlated with probe accuracy.

I Probe, NSI, and Behavior

  • Mean behavioral accuracy is 0.758, whereas mean probe accuracy is 0.917 across the 67 standard BLiMP paradigms.The behavioral analysis excludes COMPS; behavior is correct when the grammatical sentence receives higher mean token log-probability.Source: I Probe, NSI, and Behavior · p. 26
  • Mean NSI is uncorrelated with behavioral accuracy, while whole-vector probe accuracy and mean NSI are negatively correlated across paradigms.These comparisons distinguish distributed linear separability from average single-neuron selectivity.Source: Figure 15 · p. 26, Table 6 · p. 26
  • Near-perfect linear probing can coexist with poor model behavior in representative paradigms.The table presents representative paradigms ranked by behavioral accuracy.Source: Table 8 · p. 26

J Targeted Ablation

Targeted ablations of frozen high-NSI candidates do not show greater damage than matched random interventions. The result does not establish that the selected dimensions are causally irrelevant.

J Targeted Ablation

  • The observational analysis froze one NSI>2 coordinate in each of three Qwen3-0.6B paradigms before intervention, using the same 1,000 minimal pairs for selection and evaluation.This is a within-benchmark intervention test, not an independent replication.Source: J Targeted Ablation · p. 27
  • At each candidate's frozen layer, the study zeroes top-scoring candidate-and-neighbor groups and compares their log-probability-margin changes with same-layer random sets.The primary group interventions use k∈{5, 10, 20}; k=1 is an auxiliary necessity check.Source: J Targeted Ablation · p. 27
  • Across nine primary group comparisons, no targeted set is more damaging than same-size random controls at p<0.05.The left-branch top-20 intervention has a clear negative item-level effect, but it is not extreme relative to random interventions; its bottom control is similarly negative.Source: J Targeted Ablation · p. 27, Figure 16 · p. 28, Table 9 · p. 28

Related papers

Loading 2609.29328v1…