PaperlayerView exact revision on arXiv

Source-linked AI summary

No country for old linguists: LLM-brain alignment underdetermines neural computation

Elliot Murphy

arXiv:2609.03160v1cs.CL

TL;DR

This paper asks what LLM–brain alignment can establish about language mechanisms. It examines logical, causal, and computational underdetermination, concluding that predictive and representational success constrains explanation but does not itself identify neural operations.

  • Problem

    The paper asks what inferences LLM–brain alignment licenses, given that alignment may support claims about shared computation or mechanism.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The paper analyzes logical, causal, and computational gaps between predictive alignment, representational descriptions, and mechanistic claims.

    Source: §2 · p. 2, §5 · pp. 3, 5

  • Results

    Predictive alignment and representational geometry do not by themselves identify the operations that generate or use neural states.

    Source: §2 · p. 2, §5 · p. 3

  • Takeaways & Limitations

    Formal computational constraints can test whether proposed neural dynamics preserve relations required for linguistic capacities and rule out candidate implementations.

    Source: §5 · p. 5

    LLM–brain alignment narrows neural descriptions more than neural mechanisms, and a predictive regression relation alone does not establish shared geometry or computation.

    Source: §2 · p. 2, §5 · p. 3

Abstract

from arXiv · show

Nastase et al. (2026) argue that large language models (LLMs) may illuminate language processing because both rely on distributed, context-sensitive representations shaped by statistical learning. Their rejection of simple cortical "boxology" is persuasive, and they articulate a strong case for the value of LLM-brain alignment research. The key question is what kind of inference LLM-brain alignment licenses. My claim here will be narrow: representational alignment can in principle constrain mechanistic hypotheses, but it does not by itself identify a mechanism. Nastase et al. acknowledge that an encoding model can capture features represented in neural activity without establishing a shared architecture or algorithm. Yet the authors sometime move from alignment to "shared computational principles" and ultimately to LLMs as mechanistic models of natural language. Indeed, their methodological caveat that alignment does not establish a shared architecture or algorithm sits uneasily with their conclusion that LLMs might instantiate the same computational principles as biological brains and provide a "fully mechanistic model" of language. I discuss what I consider to be problems of logical, causal, and computational underdetermination in Nastase et al.'s (2026) proposal.

1 | INTRODUCTION

Nastase et al. argue that LLMs may illuminate language processing because LLMs and brains rely on distributed, context-sensitive representations shaped by statistical learning. The section raises what inferences LLM–brain alignment licenses.

  • LLMs may illuminate language processing because both they and brains rely on distributed, context-sensitive representations shaped by statistical learning.Source: §1 · p. 1
  • Rejecting simple cortical “boxology,” the authors make a case for LLM–brain alignment research.Source: §1 · p. 1
  • The central question is what kind of inference LLM–brain alignment licenses.Source: §1 · p. 1

2 | FROM ALIGNMENT TO MECHANISM

Predictive LLM–brain alignment does not by itself establish shared representational geometry or mechanism. The section identifies unconstrained mappings and common stimulus structure as sources of underdetermination.

  • Held-out prediction from an LLM representation to neural responses shows predictive information, but an unrestricted fitted transformation can distort representational relations.The mapping must preserve relevant relations for this analysis to support geometric claims.Source: §2 · p. 1
  • A linear map that predicts neural responses for left-out words establishes a predictive relation across the lexicon, not shared geometry.Source: §2 · p. 1
  • Under arbitrary linear maps, collinearity and parallelism may survive, while distances, angles, and neighborhood structure need not.Distance-preserving mappings or within-system similarity comparisons avoid this particular objection, but do not identify the operations that produced or use the geometry.Source: §2 · p. 2
  • Because LLM representations and neural responses both reflect the same linguistic stimulus, alignment can arise without shared internal operations.Treating prediction as proof of mechanism affirms the consequent.Source: §2 · p. 2
  • A regression coefficient alone cannot establish shared geometry, computational principles, or mechanism; a predictive linear relation is compatible with spaces whose geometric structures disagree.Source: §2 · p. 2

3 | LOST IN TRANSLATION: A LESSON FROM SURPRISAL

Surprisal and contextual embeddings can predict processing or neural responses without identifying the latent structure or biological operations involved. The section distinguishes a useful descriptor or cue from a mechanism.

  • Lexical surprisal can predict processing difficulty and neural responses, yet syntactic structure can alter surprisal without surprisal identifying that latent structure.Some surprisal variance may also reflect simpler distributional properties.Source: §3 · p. 2
  • Contextual embeddings absorb variance associated with many linguistic and contextual dimensions, making them powerful predictors while potentially obscuring the source of neural predictivity.Source: §3 · p. 2
  • An embedding can reveal stimulus dimensions represented in the brain without identifying the biological transformation that implements composition.A cue can constrain an operation without being the operation itself.Source: §3 · pp. 2–3

4 | THE MAP IS NOT THE MECHANISM

Mechanistic transformations in LLMs do not establish corresponding cortical operations. A stronger mechanistic account would connect a candidate operation to biological organization and discriminating tests.

  • BERT attention-head transformations predict fMRI activity, but the decomposed circuitry is BERT circuitry rather than an identified cortical operation.GPT-2 embeddings capture linguistic information shared between speakers and listeners, not the neuronal transformation producing or transmitting it.Source: §4 · p. 3
  • A stronger mechanistic claim would specify a candidate operation, identify biological organization capable of implementing it, derive a discriminating prediction, and ideally perturb the relevant process.Neural measurements provide evidence for mechanistic accounts but are not mechanisms themselves.Source: §4 · p. 3

5 | WITH GREAT DIMENSIONALITY COMES GREAT RESPONSIBILITY: POPULATION CODES NEED COMPUTATIONAL CONSTRAINTS

Population-level alignment can describe where linguistic information lies, but not which operations make it computationally usable. The section proposes judging candidate neural dynamics by whether they preserve the formal relations a linguistic operation requires.

5 | WITH GREAT DIMENSIONALITY COMES GREAT RESPONSIBILITY: POPULATION CODES NEED COMPUTATIONAL CONSTRAINTS

  • Population geometry and high-dimensional codes describe where linguistic information lies, but without a linking account they do not identify the causal operations generating or transforming it.Overlapping populations may multiplex variables across subspaces, time windows, trajectories, oscillatory regimes, or coupling patterns.Source: §5 · p. 3
  • Decodability establishes informational availability to an analyst, not whether the brain can selectively retrieve and reuse a constituent in linguistic computation.Long-distance dependencies, agreement, ellipsis, and structural reanalysis require such retrieval and reuse.Source: §5 · p. 4
  • Transformers’ access to thousands of prior tokens and text models’ lack of endogenous oscillatory dynamics mean their success does not reveal how brains solve those constraints.The cited passage treats these constraints as part of what an admissible implementation must satisfy, not as nuisance parameters to abstract away.Source: §5 · p. 4
  • Without outcomes that could count against it, a high-dimensional population-code account risks naming an implementation vocabulary rather than a computational theory.Vectorial computation is not necessarily opposed to algebraic theory, which can specify invariants that implementations must preserve.Source: §5 · p. 5
  • Linguistic computation requires operations that preserve structural relations; ordinary similarity, addition, or averaging does not guarantee this, and syntax can require non-associative composition.Collapsing distinct bracketings into indistinguishable states erases constituent structure; related constraints arise for entailment, scope, binding, and type-sensitive composition.Source: §5 · p. 5
  • The proposed test asks whether a candidate neural transformation preserves the relations defining a cognitive operation, rather than whether a decoder can recover the information.The relevant equivalence relation must be independently justified; language relations may include hierarchy, non-associative bracketing, recursive closure, structured access, headedness, and type compatibility. This can eliminate candidate implementations even when they explain substantial neural variance.Source: §5 · p. 5

6 | CONCLUSION

LLM–brain prediction and representational alignment offer clues about neural organization, but do not by themselves explain a causal mechanism. The conclusion calls for showing how neural processes enforce the relevant linguistic computation.

6 | CONCLUSION

  • Predictive and representational success should constrain explanation, not count as explanation itself.The LLM–brain literature has produced clues and powerful descriptions of representational organization.Source: §6 · p. 5
  • Formally explicit symbolic–algebraic models specify relations a candidate mechanism must preserve, thereby ruling out classes of implementations.The conclusion presents this as an advantage over prediction and correlation alone.Source: §6 · p. 5
  • Explaining a causal mechanism requires showing how the relevant linguistic computation undergoes neural enforcement.Source: §6 · p. 6
Loading 2609.03160v1…