PaperlayerView exact revision on arXiv

Source-linked AI summary

Which Attention Heads are like the Human Head? Not the Ones that Compute

Christopher Pinier, Gustaw Opiełka, Hannes Rosenbusch, Taylor Webb, Michael D. Nunez, Claire E. Stevenson

arXiv:2609.37991v1cs.AIq-bio.NC

TL;DR

Whether units aligned with human neural signals are causally involved in a model’s task performance is rarely tested directly. This paper compares attention-head representations with human EEG during abstract sequence completion and tests head removal. Brain alignment is a weaker guide to performance-critical heads than patching scores, while aligned heads reveal recurring attention profiles with different relationships to performance.

  • Problem

    Brain–AI alignment is often taken as evidence of shared computation, but whether units selected by alignment are causally involved in the same task has not been directly tested.

    Source: §1 · p. 1

  • Method

    The study compares LLM attention-head representations with a group-average frontal-FRP RDM during abstract sequence completion and tests the effects of removing selected heads.

    Source: §1 · p. 3, §6 · p. 8

  • Results

    Across 17 models, mean peak excess loss was 42.6 percentage points after removing 12.5% of FV-ranked heads, versus 12.8 points after removing 24.5% of brain-ranked heads.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Brain-aligned heads reveal recurring repetition- and novelty-like attention profiles, with repetition-ranked removal more disruptive and novelty-like attention more closely resembling pattern-dependent human gaze changes.

    Source: §6 · p. 9

    The main comparison used eight sequence patterns, fixed three-shot prompts, and one group-average frontal-FRP target, so brain alignment here does not represent correspondence with human brain activity in general.

    Source: §6 · p. 9

Abstract

from arXiv · show

Brain-AI alignment is often interpreted as a sign that model and brain perform similar computations. Whether the aligned units are causally involved in model computation is rarely checked. On an abstract pattern-completion task (AAABAAA $\rightarrow$ B), we compare LLM attention-head representations with human EEG and test how ablating those heads affects task performance. Alignment and causation dissociate: brain-aligned heads contribute to performance, but their removal is substantially less disruptive than removal of heads selected via attribution patching. We compare two head sets that prior interpretability work defines without reference to the brain: concept vectors (CVs), which represent abstract patterns across formats, and function vectors (FVs), selected for their contribution to correct-answer prediction. Brain alignment shows little association with FV scores, while its association with CV scores varies across models. Among brain-aligned heads, we find recurring attention profiles: one emphasizes distinctive elements (novelty heads), the other repeating elements (repetition heads). The novelty family tracks salience and attends to the same elements that humans look at, yet its removal is less damaging than random ablation on average. Repetition heads contribute modestly to performance and are associated with abstract-pattern representation (CVs). Across 17 models spanning 3B-72B parameters, FV-ranked removal is substantially more disruptive than brain-ranked removal. Brain alignment thus captures how the model reads the stimulus, and only faintly captures how it represents the pattern and solves the task.

1 INTRODUCTION

The paper tests whether attention heads aligned with human neural activity are also important to model performance on abstract sequence completion. Across its analyses, brain alignment has a weak relationship to causal importance, while aligned heads reveal recurring attention profiles.

1 INTRODUCTION

  • The study asks whether heads resembling human brain activity are also those on which task performance depends, extending prior layer-level comparisons to individual heads.The task is abstract sequence completion, such as AAABAAA → B, using text-based versions of icon sequences.Source: §1 · pp. 1–2
  • Across models, removing brain-ranked heads was far less damaging than removing function-vector (FV)-ranked heads, the most causally important set.Brain alignment was weakly associated with attribution-patching effects.Source: §1 · p. 2
  • Brain-aligned heads show recurring profiles: repetition heads attend to repeated elements, while novelty heads attend to the distinctive element.For AAABAAA, these are repeated As and the lone B, respectively.Source: §1 · p. 2
  • Novelty-ranked removal is less damaging than random removal on average, while novelty-like attention follows pattern-dependent changes in human gaze.The paper also reports that additional tasks and training-stage analyses support novelty heads’ limited task contribution.Source: §1 · pp. 2–3
  • Repetition heads correlate with format-invariant concept representation and co-occur with concept-vector heads across layers, yet their removal leaves that measured representation intact.Their removal impairs performance more than random removal.Source: §1 · p. 2
  • The human comparison uses group-average frontal fixation-related potentials from 400 unique trials, represented as an 8 × 8 RDM over abstract patterns.Human gaze data provide a separate comparison of attention profiles.Source: §1 · p. 3
  • The study began with Llama-3.1-70B and tested generality in 16 additional models spanning 3B–72B parameters, using raw prompts with three solved examples.The initial Llama-3.1-70B evaluation achieved 70.5% accuracy.Source: §1 · p. 3
  • Immediate-answer performance varied by model family and training variant, and the reported scores do not measure best achievable performance or extended reasoning.Instruction tuning improved Llama-3.1-70B performance but reduced it for Llama-3.1-8B under the same raw-prompt protocol.Source: §1 · p. 4

2 BRAIN ALIGNMENT VERSUS CAUSAL IMPORTANCE

Brain alignment was only weakly related to patching effects and varied in its association with concept scores. Brain-ranked removal was less disruptive than FV- or concept-ranked removal across the model cohort.

2 BRAIN ALIGNMENT VERSUS CAUSAL IMPORTANCE

  • The comparison scores heads by brain alignment, activation-patching contribution to correct-answer prediction, and concept representation across formats.For cohort replications, attribution patching closely agreed with activation patching in effect magnitude (Pearson r = .983) and selected 95% of the same top-20 heads in Llama-3.1-70B.Source: §2 · p. 4
  • Brain scores were only weakly related to patching scores in Llama-3.1-70B (ρ = .03) and across models (mean ρ = .017, range [−.017, .077]).Higher brain scores therefore did not identify heads with larger patching effects within a model.Source: §2 · p. 4
  • Brain–concept correlations averaged ρ = .10 across 17 models, with positive associations in Llama and base Qwen but negative associations in instruction-tuned Qwen.Brain alignment was generally more positively and variably associated with concept scores than with patching scores.Source: §2 · p. 5
  • Across 17 models, brain-ranked removal was more damaging than random in 15, versus all 17 for FV- and concept-ranked removal; peak excess losses were 12.8 pp for brain, 42.6 pp for FV, and 20.7 pp for CV.The peaks occurred at 24.5%, 12.5%, and 19.5% of heads removed, respectively.Source: §2 · p. 5

3 REPETITION- AND NOVELTY-LIKE ATTENTION PROFILES

The highest brain-scoring heads include recurring repetition-like and novelty-like attention profiles. These profiles generalize across models, but their removal has different effects on task performance.

3 REPETITION- AND NOVELTY-LIKE ATTENTION PROFILES

  • Clustering the 20 highest brain-scoring heads identified a novelty profile that attends to the distinctive symbol and a repetition profile that emphasizes repeated-symbol positions.The profile names describe attention patterns, not established computational functions; novelty-template similarity increased with attention to the distinctive symbol (r = .73).Source: §3 · p. 5
  • Across models, repetition carriers made up 22.6% of top-20 FRP heads versus 17.5% of all heads, while novelty carriers were 44.4% versus 46.7%.Template carriers were identified using a loading threshold above .50.Source: §3 · p. 5
  • Independent clustering recovered a novelty-like profile in all 17 models and a repetition-like profile in 13, despite variation in cluster number and composition.Source: §3 · p. 6
  • In Llama-3.1-70B, removing 800 novelty heads reduced accuracy from 71.5% to 63.5%, while the random-control median fell below 50%.Repetition-head removal was more disruptive than random removal and followed a trajectory similar to concept-ranked ablation.Source: §3 · p. 6
  • Across the cohort, repetition-ranked removal was more damaging than random removal, while novelty-ranked removal was generally less damaging than random.Repetition removal peaked at 11.2 pp excess loss at 15.5% of heads; novelty removal never exceeded random by more than 0.3 points.Source: §3 · p. 6

4 WHAT THE ATTENTION FAMILIES DO AND DO NOT DO

Follow-up analyses tested whether repetition and novelty profiles matter beyond the primary-task ablations. Repetition removal impaired performance without eliminating measured pattern information, while novelty-ranked removal remained minimally disruptive across tested tasks.

4 WHAT THE ATTENTION FAMILIES DO AND DO NOT DO

  • The additional-task and training-stage analyses were limited to Llama-3.1-70B and Pythia-1.4B, with Pythia used to study changes across training.Source: §4 · p. 7
  • Removing 130 strongly repetition-like heads lowered Llama-3.1-70B accuracy by 5.5 percentage points but preserved cross-format concept RSA and 100% pattern-classification accuracy in later concept heads.Repetition loading correlated with concept scores (ρ = 0.45), and the high-loading repetition and concept heads occupied overlapping layers.Source: §4 · p. 7
  • Novelty-ranked removal was less damaging than the random median on seven of eight tested tasks and within the random-control range on prose.Each task’s own attribution ranking identified heads whose removal sharply impaired performance.Source: §4 · p. 7
  • Across 19 Pythia-1.4B checkpoints, novelty-like attention became more prominent, but copying strength and attribution-based contribution remained below the analysis reference thresholds.At the final checkpoint, removing 32 heads most similar to the Llama novelty template reduced pattern-task accuracy from 42.5% to 39.5%.Source: §4 · p. 7

5 RELATIONSHIP TO HUMAN GAZE

Novelty-head attention tracks pattern-dependent human gaze more closely than repetition-head attention, but the match is concentrated where the distinctive symbol is also the answer. Because those patterns confound rarity with answer identity, the gaze comparison cannot distinguish oddity detection from answer-seeking.

Pattern dependence

  • Novelty-gaze correspondence averaged .47 on five patterns where the unrepeated symbol was the answer, versus .12 on the other three.Repetition anticorrelation was −.34 and −.20 on those respective pattern groups.Source: §5 · p. 7

Head-family correlations

  • Novelty heads correlated with gaze in every model (mean ρ = .373, range [.318, .421]); repetition heads correlated negatively (mean ρ = −.285, range [−.335, −.206]).Across models, CV and FV heads fell between these profiles on average.Source: §5 · p. 7, Figure 5 · p. 8

Figure 5

  • In Figure 5, 70% of cell-wise agreement fell in five answer-equals-unrepeated cells and 95% in their rows; the first-position case drew neither gaze nor attention.The figure compares group fixation maps with top-1% novelty-head attention across seven visible positions.Source: Figure 5 · p. 8

Interpretive boundary

  • The patterns cannot distinguish whether participants fixated the distinctive symbol because it was the answer or because it was unusual; FV heads also attended there on those patterns (.28 versus .00).Attention location alone therefore does not distinguish idle heads from working ones.Source: §5 · p. 8

6 DISCUSSION

Across the tested models, brain alignment was a weaker guide to performance-critical heads than causal patching scores, while brain-aligned heads showed recurring repetition- and novelty-like attention profiles. These profiles differed in their relationships to performance and gaze, and the paper limits the conclusions to its tested patterns, prompts, and brain target.

Brain alignment and performance

  • FV-ranked removal peaked at 42.6 percentage points of excess loss after removing 12.5% of heads, versus 12.8 points after 24.5% for brain-ranked removal; CV-ranked removal was intermediate.Brain alignment was weakly associated with patching effects, and its association with concept scores varied across model families.Source: §6 · p. 8

Recurring profiles

  • Clustering brain-aligned heads across models recovered repetition- and novelty-like attention profiles, but recurring profiles alone do not establish that those heads perform the same computations.Some clusters matched neither profile.Source: §6 · pp. 8–9

Repetition heads

  • In Llama-3.1-70B, removing 130 repetition-like heads reduced accuracy while later concept heads retained measured pattern information.Repetition loading correlated with concept scores, and the two head sets occupied overlapping layers.Source: §6 · p. 9

Novelty heads

  • Novelty-ranked removal was less damaging than random removal on average, despite novelty-like attention tracking gaze; the experiments do not establish why the profile favors distinctive elements.The authors suggest it could be a by-product of architecture and training; softmax requires every head to distribute attention even when its output has little influence on the answer.Source: §6 · p. 9

Relation to other studies

  • Earlier alignment findings need not conflict because they used different selection criteria, representational targets, interventions, and behavioral endpoints.Training models to predict brain activity poorly or selecting language-responsive units tests a different proposition than ranking attention heads by task-specific brain alignment.Source: §6 · p. 9

Scope and limitations

  • The main comparison used eight sequence patterns, fixed three-shot prompts, and one group-average frontal-FRP target; here, brain alignment refers specifically to similarity with that target.The authors also note unequal participant weighting, only 28 dependent pattern comparisons, and the need to test more tasks and prompting strategies.Source: §6 · p. 9

A SUPPORTING METHODS AND RESULTS

The section sequence moves from human and model evaluation through head scoring and ablation, then to attention-family analyses and gaze comparisons.

Supporting methods and results

A.1 HUMAN STUDY AND FRONTAL-FRP TARGET

The human study used an abstract-pattern completion task with EEG and eye tracking. Its frontal-FRP target summarizes fixation-locked responses to sequence inspection as pattern-level temporal waveforms.

A.1 HUMAN STUDY AND FRONTAL-FRP TARGET

  • The planned study comprised 400 trials over five sessions, with 50 trials per pattern; five of 25 recruited adults withdrew before completing all sessions.Their available sessions were retained in the source study; these counts differ from the participants with usable FRP RDMs in this reanalysis.Source: §A.1 · p. 13
  • Participants saw seven icons instantiating an abstract pattern and selected the missing eighth icon from four alternatives.The example pattern is ABCAABCA, completed by the star; FRP analysis retains fixations in the eight sequence-position regions, not the response options.Source: Figure 6 · p. 13
  • For each retained sequence fixation, a 600-ms EEG epoch beginning at fixation onset was averaged within the trial; answer-option fixations were excluded.The resulting FRP summarizes self-paced sequence inspection, and consecutive epochs can overlap when fixation onsets are less than 600 ms apart.Source: §A.1 · pp. 13–14
  • The target averages 17 frontal electrodes into temporal waveforms, then compares the eight pattern waveforms using Pearson correlation distance in an 8 × 8 RDM.The RDM has 28 unique pattern-pair values and compares waveform shapes rather than spatial scalp patterns.Source: §A.1 · p. 14, Figure 7 · p. 14

A.2 MODEL PROMPTS AND EVALUATION

The model evaluation uses a fixed three-shot, text-based adaptation of an abstract-sequence completion task across multiple alphabets and response formats. Performance varies across model families and prompt conditions, so scores are specific to the tested protocol.

A.2 MODEL PROMPTS AND EVALUATION

  • The task withholds the eighth element of one of eight length-eight abstract patterns and maps pattern letters to surface tokens without replacement.The patterns are evaluated using punctuation-like symbols, German words, and English words.Source: §A.2 · pp. 14–15
  • The six trial conditions cross three alphabets with open-ended and multiple-choice formats, with 25 items per pattern and 200 trials per condition.Brain scores reuse English multiple-choice trials from the head-RSA dataset, while the attention scan and ablation sweep use a separately generated English multiple-choice set.Source: Table 1 · p. 15
  • The cross-model replication uses raw three-shot prompts, including for instruction-tuned models, with three same-pattern demonstrations using fresh surface tokens before an unsolved query.No natural-language instruction or chat template is included; tokenizer-specific word pools can change literal lexical items.Source: §A.2 · p. 15

A.2.1 EVALUATION CONDITIONS AND PERFORMANCE

  • Performance is exact next-token accuracy over the full vocabulary, not accuracy restricted to the four options or an evaluation of extended reasoning.The target is the highest-probability token matching the contextually resolved correct answer.Source: §A.2.1 · p. 15
  • Performance generally improved with size within Llama and Qwen2.5-Instruct, yet Qwen2.5 base peaked at 32B (74.5%) rather than 72B (54.0%), and Phi-4 reached 82.0% with 14B parameters.Source: §A.2.1 · p. 15
  • Under this raw-prompt protocol, the reported comparisons are descriptive rather than isolated causal estimates of parameter count or post-training effects, not standalone causal conclusions about either factor.Source: §A.2.1 · p. 16
  • Prompt conditions changed accuracy substantially: chat templating raised Llama-3.3-70B-Instruct symbol multiple-choice accuracy from 33.5% to 80.5%.In standalone raw-prompt evaluations, symbol conditions yielded 18.5–33.0% exact next-token accuracy; low primary-condition performance does not establish absence of abstract-pattern competence.Source: §A.2.1 · p. 16

A.3.1 HEAD OUTPUTS, CONCEPT SCORE, AND CONTROL RSA

Head-output geometry is compared with abstract-pattern identity and with human frontal-FRP geometry. The concept score is a continuous ranking, not proof that a head causally uses the pattern representation.

A.3.1 HEAD OUTPUTS, CONCEPT SCORE, AND CONTROL RSA

  • For each head, the analysis extracts final-position attention-output-projection inputs, normalizes trial vectors, and computes pairwise cosine similarities.The extracted tensor contains the individual head outputs before they are mixed by the layer’s output projection.Source: §A.3.1 · p. 16
  • The concept score is Spearman’s ρ between head-output similarities and a design matrix marking whether item pairs instantiate the same abstract pattern.Format and alphabet controls use same-format and same-alphabet design matrices; top-k CV heads are the first k heads in descending concept-score order.Source: §A.3.1 · p. 16
  • A head’s brain score is Spearman’s ρ between its pattern-level RDM and the group-average frontal-FRP RDM, using 28 unique pattern-pair entries.Head RDMs are constructed from normalized English multiple-choice trial outputs averaged within each pattern.Source: §A.3.1 · p. 16

A.3.2 FUNCTION-VECTOR SCORE BY ATTRIBUTION PATCHING

This section defines head-selection scores and tests their behavioral and representational consequences, then characterizes attention-profile families and their relation to human gaze. Across the cohort, brain-ranked ablation is less disruptive than FV-ranked ablation, while repetition and novelty profiles show distinct associations with task representation and gaze.

A.3.2 FUNCTION-VECTOR SCORE BY ATTRIBUTION PATCHING

  • FV heads are ranked by a first-order attribution estimate of correct-answer probability change, computed from gradients and the displacement between corrupted and clean-mean activations.The corrupted prompts disrupt recoverable abstract-pattern information while retaining vocabulary; the score approximates mean activation patching, not cumulative zero ablation.Source: §A.3.2 · p. 17

A.3.4 ATTRIBUTION PATCHING VALIDATION

  • In Llama-3.1-70B, attribution estimates correlated with full activation-patching effects at Pearson r = .983, while their head rankings were not interchangeable (Spearman ρ = .60).Top-20, top-50, and top-150 selections overlapped by 95%, 90%, and 91%, respectively; validation was limited to this model.Source: §A.3.4 · p. 17

A.4.1 RANKINGS AND CUMULATIVE ZERO ABLATION

  • The cumulative ablation sweep removes progressively larger prefixes of five fixed rankings and compares accuracy against nested random-removal trajectories.The rankings include FV, brain, CV, repetition, and novelty; accuracy differences are adjusted by each model’s mean random-control accuracy and averaged equally across models.Source: §A.4.1 · p. 17, §A.4.2 · p. 18

A.5.1 ALL-HEAD ATTENTION SCAN AND FROZEN FAMILY LOADINGS

  • Template loadings correlate each head’s pattern-centered attention fingerprint with frozen repetition and novelty templates; carriers exceed the .50 cutoff.Cumulative template ablations rank all heads by loading, not only carriers; in the top-20 FRP heads, repetition carriers were modestly overrepresented, while novelty carriers were common but not overrepresented.Source: §A.5.1 · p. 19, Figure 10 · p. 20

A.5.2 INDEPENDENT ATTENTION-PROFILE CLUSTERING

  • Independent top-20 clustering yielded profiles matching novelty-like templates in 17/17 models and repetition-like templates in 13/17 at r ≥.50.Post-hoc membership overlap averaged .745 for repetition and .742 for novelty; exact set equality occurred in 4 of 17 models for each family, so these matches are not independent predictive validation.Source: §A.5.3 · p. 19, Figure 11 · p. 21

A.6.1 REPETITION-HEAD ABLATION AND THE CONCEPT REPRESENTATION

  • Removing 130 repetition heads lowered accuracy from 70.5% to 65.0%, while concept RSA rose from 0.401 to 0.427 and the pattern classifier remained at 100%.The repetition removal impaired accuracy more than random removal, yet degraded measured concept representation less than random or novelty removal.Source: §A.6.1 · p. 21, Figure 12 · p. 22

A.7.1 EXPLORATORY GAZE COMPARISON

  • After subtracting each position’s across-pattern mean, novelty-head gaze correspondence was positive (.373) and repetition correspondence negative (−.285) in all 17 models.Raw positional correspondence was weakly negative for both; these normalized comparisons do not measure total attention to the sequence versus prompt scaffolding.Source: §A.7.1 · pp. 22–23
Loading 2609.37991v1…