Source-linked AI summary
Language models and brains align due to more than next-word prediction and word-level information
Gabriele Merlin, Mariya Toneva
TL;DR
Although pretrained language models predict brain recordings and next-word prediction has been proposed as a key contributor, it remains unclear whether it is necessary or whether other information also matters. The paper uses model perturbations and controlled contrasts to separate next-word prediction and word-level information from other factors. Residual alignment remains in language-related brain regions, indicating that improvements reflect more than those two factors.
Problem
Language models predict brain recordings, but it remains unclear whether next-word prediction is necessary for that alignment or whether other shared information also contributes.
Source: §1 · p. 1Method
The authors perturb GPT-2-based models through input scrambling and stimulus-tuning, then contrast their brain alignment to control for word-level information and next-word prediction.
Source: §1 · p. 2Results
Takeaways & Limitations
The findings suggest that fine-tuning with little additional text can improve language models’ representations of multi-word information in a brain-relevant way.
Source: §6 · p. 9The study evaluates GPT-2-based models, so generalization to larger models or models trained with masked-language objectives remains to be investigated.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Pretrained language models have been shown to significantly predict brain recordings of people comprehending language. Recent work suggests that the prediction of the next word is a key mechanism that contributes to this alignment. What is not yet understood is whether prediction of the next word is necessary for this observed alignment or simply sufficient, and whether there are other shared mechanisms or information that are similarly important. In this work, we take a step towards understanding the reasons for brain alignment via two simple perturbations in popular pretrained language models. These perturbations help us design contrasts that can control for different types of information. By contrasting the brain alignment of these differently perturbed models, we show that improvements in alignment with brain recordings are due to more than improvements in next-word prediction and word-level information.
1 Introduction
The section frames whether next-word prediction is necessary for language–brain alignment and proposes contrasts to separate its contribution, word-level information, and other factors. Across GPT-2-based models, residual alignment remains in the inferior frontal and angular gyri after controlling for next-word prediction and word-level information.
1 Introduction
- Because next-word prediction correlates with other brain-relevant information, earlier approaches could not separate the contributions of word-level and multi-word information.The paper treats word-level information as a non-contextualized word representation and multi-word information as relating to multiple words; these are conceptual quantities, with word-level information assumed unaffected by word order.Source: §1 · pp. 1–2
- The study aims to disentangle next-word prediction and word-level information from other factors, such as multi-word information, in GPT-2-based models.The contribution list also reports that stimulus-tuning on validation stimulus text can improve alignment with recordings corresponding to held-out text.Source: §1 · p. 2
- The proposed contrasts compare a model with a perturbed version: controlled factors cannot explain any difference in their predicted brain activity.Input scrambling preserves word-level information while changing word order; stimulus-tuning is used to further control for next-word prediction.Source: §1 · p. 2
- The method controls next-word prediction performance at inference time, so it does not establish the importance of the training objective itself.Its conclusions concern information that emerged after training.Source: §1 · p. 2
- Residual alignment in the inferior frontal and angular gyri across three GPT-2 model types indicates that alignment reflects more than next-word prediction and word-level information.The authors speculate that multi-word information may contribute.Source: §1 · p. 2
2 Methods
The methods use GPT-2-based models and fMRI recordings from people reading a novel chapter, evaluating next-word prediction and brain alignment on the stimulus text. Input scrambling and stimulus-tuning support contrasts designed to isolate word-level information and control for next-word prediction.
2.1 Baseline models
- GPT-2-small, GPT-2-medium, and GPT-2-distilled serve as baselines, with averaged results reported across models; the study uses these models for comparison with prior brain-alignment research.The authors report that stimulus-tuning effects and residual effects decrease as model size increases, likely because the fine-tuning dataset is small.Source: §2.1 · p. 3
2.2 fMRI data
- The fMRI data come from eight participants reading chapter 9 of Harry Potter and the Sorcerer’s Stone word by word, with one image acquired every two seconds.The chapter was divided into four approximately equal runs.Source: §2.2 · p. 3
2.3 Evaluation tasks
- Both evaluation tasks use the fMRI stimulus text in overlapping 20-word windows; next-word prediction is scored by cross-entropy, while brain alignment is held-out Pearson correlation.Brain recordings are predicted with a cross-validated linear head on the final transformer block, regularized by a ridge penalty.Source: §2.3 · p. 3
2.4 Perturbations
- Input scrambling reorders words within each 20-word sequence at inference, while stimulus-tuning fine-tunes the baseline model on non-overlapping 80-word sequences from the stimulus text.Stimulus-tuning uses the next-word prediction objective; the tuned model is then paired with its scrambled version.Source: §2.4 · p. 4
2.5 Contrasts to disentangle brain alignment factors
- A baseline-versus-scrambled contrast isolates effects beyond word-level information; a second-level contrast matches the models’ scrambling-related next-word prediction drops to identify residual alignment.The second-level control assumes a linear relationship between next-word prediction and brain alignment; the study reports a 0.61 Pearson correlation in its setting.Source: §2.5 · pp. 4–5
3 Results
Across models, the results compare next-word prediction and brain alignment under stimulus-tuning and scrambling, then test whether alignment remains after controlling for next-word prediction and word-level information.
3.1 Next-word prediction
- The stimulus-tuned model predicted held-out stimulus next words better than baseline, while scrambling reduced performance relative to each model’s unscrambled counterpart.Stimulus-tuned scrambling nevertheless performed better at next-word prediction than baseline.Source: §3.1 · p. 5
3.2 Brain alignment
- Stimulus-tuning improved next-word prediction and brain alignment, with positive alignment gains over baseline in every language region.The authors could not conclude that the alignment improvement was due to improved next-word prediction, since stimulus-tuning may also change other brain-relevant representations.Source: §3.2 · p. 5, Figure 4 · p. 7
- Scrambling reduced next-word prediction and brain alignment, yet scrambled models retained alignment with fMRI recordings, particularly in language-related regions.The stimulus-tuned scrambled model showed worse brain alignment than both the stimulus-tuned model and baseline, although its next-word prediction remained better than baseline.Source: §3.2 · pp. 5–6, §3.1 · p. 5
3.3 Controlling for both word-level information and next-word prediction
- Residual alignment remained in the Inferior Frontal Gyrus (IFG) and Angular Gyrus (AG) across three GPT-2 models after controlling for next-word prediction and word-level information.The authors note that high subject variability and the number of subjects made statistical significance challenging.Source: §3.3 · pp. 6–7
4 Related Works
Prior work has linked language-model representations to brain recordings and investigated which linguistic information contributes to that alignment. This work uses perturbations to examine alignment while controlling for next-word-prediction-related information.
4 Related Works
- Goldstein et al. (2022) linked next-word prediction to alignment using ECoG evidence that electrodes can predict neural representations of upcoming words; this work uses perturbations to examine other information too.The paper describes its approach as an additional account of information relevant to brain alignment.Source: §4 · p. 7
- Prior studies disentangled supra-word from lexical meaning, syntax from semantics, and word-level semantics; the present perturbations are described as complementary, not as direct tests of semantic or syntactic significance.The cited work by Kauf et al. (2023) found syntactic perturbations had less impact on alignment than semantic perturbations.Source: §4 · pp. 7–8
- Aw and Toneva (2023) found improved brain alignment after narrative-summarization fine-tuning without improved next-word prediction; this paper’s perturbations offer additional control over representations.The paper characterizes the approaches as complementary.Source: §4 · p. 8
5 Discussion
Stimulus-tuning can improve alignment with held-out brain recordings, and the gains in IFG and AG are not explained solely by next-word prediction or word-level information. Stimulus-specific multi-word representations are one possible explanation, not an established mechanism.
5 Discussion
- Stimulus-tuning can increase alignment with held-out brain recordings, particularly in several language-processing brain areas.Possible sources include improved representations of next words, previously seen individual words, or stimulus-specific multi-word information.Source: §5 · p. 8
- Across multiple GPT-2 models, alignment gains in IFG and AG reflect more than improvements in next-word prediction and word-level information.The authors propose stimulus-specific multi-word information as one possible reason for the residual alignment.Source: §5 · p. 8
- Stimulus-specific multi-word representations are one possible explanation for improved alignment, but weak effects elsewhere do not show those regions lack multi-word processing.The discussion links IFG with syntax and AG with multi-word event structure: Friederici et al. (2003); Friederici (2012); Ramanan et al. (2018); Humphreys et al. (2021).Source: §5 · p. 8
6 Conclusion
The paper uses two perturbations to control for next-word prediction and word-level information when examining language-model alignment with brain recordings. The reported alignment gains exceed improvements in those two factors, while the findings retain a dependence on next-word prediction as a training objective.
6 Conclusion
- The perturbations together control for next-word prediction and word-level information, yet alignment improvements exceed improvements in those factors.The conclusion describes the findings as relevant to cognitive neuroscientists and NLP researchers.Source: §6 · p. 8
- The findings suggest accurate next-word prediction is not necessary for brain alignment, although it may induce other necessary properties such as multi-word information.The authors identify this possibility as a hypothesis for future work.Source: §6 · p. 8
- The inference-time controls still rely on a next-word prediction training objective; alternative objectives, including summarization, remain a direction for future work.The conclusion notes that small amounts of text can increase alignment with never-before-seen brain recordings.Source: §6 · p. 9
7 Limitations
The study’s conclusions are bounded by its model family, dataset, experimental choices, and the interpretation of next-word-prediction changes.
7 Limitations
- The analysis uses GPT-2-based models; generalization to larger models and models trained with masked language modeling remains to be tested.The authors also note that stimulus-tuning gains and residual effects decrease with model size in their comparison.Source: §7 · p. 9
- Observed effects may be specific to the single fMRI dataset, narrative text, or English, so other datasets, genres, and languages remain untested.Source: §7 · p. 9
- Results depend partly on the chosen scrambling method, and imperfect linearity between next-word prediction and brain alignment may limit the subtraction’s control.The authors cite a strong positive correlation of 0.61 Pearson correlation, but note the relationship is not perfectly linear.Source: §7 · p. 9
- Held-out next-word-prediction changes could reflect general English knowledge or Harry Potter-specific knowledge; the authors believe domain-specific learning is primary.They identify investigating the source of the increase as a next step.Source: §7 · p. 9
8 Ethics and Broader Impact
The section describes potential societal relevance for understanding artificial and biological neural networks, alongside the linear, ridge-regularized prediction head used for fMRI.
8 Ethics and Broader Impact
- The authors propose that understanding similarities and differences between human brains and neural networks may inform both neural-network power and brain-function mechanisms.They connect this understanding to the prevalence of black-box AI and responsible integration into human activities.Source: §8 · pp. 9–10
A Prediction head
- A linear prediction head maps model representations into fMRI space and uses ridge regularization, cross-validation, and held-out runs for testing.Source: A Prediction head · p. 12
- The head averages word embeddings to the fMRI repetition-time rate and concatenates the previous five averaged TR embeddings to account for hemodynamic lag.Using preceding time points allows data-driven estimation of voxel-specific hemodynamic response functions.Source: A Prediction head · p. 12
B Training hyperparameters
The appendix specifies stimulus-tuning training choices and lists the regions included in the ROI analysis.
B Training hyperparameters
- Stimulus-tuning used non-overlapping 80-word sequences, batch size 32, and two epochs, with separate models for each of four held-out runs.The selected checkpoint best matched δstim ≈δbase.Source: B Training hyperparameters · p. 12
- The stimulus text was divided into four consecutive sections for cross-validation, and models used Huggingface’s default hyperparameters.Source: B Training hyperparameters · p. 12
C List of ROI related to language processing and word semantics
- The listed regions include middle and inferior frontal, anterior and posterior temporal, angular, posterior cingulate, and dorsomedial prefrontal areas.The passage also names IFGorb and gives Figure 2B as a reference.Source: C List of ROI related to language… · p. 12
D Significance testing and Participants variability
Across three GPT-2 models, positive residual alignment was observed in IFG and AG, but the final contrast did not remain significant after correction across ROIs.
D Significance testing and Participants variability
- Experiments compared models and perturbations by percentage gain, with the final contrast testing residual brain alignment.Source: D Significance testing and… · p. 12
- Positive residual alignment in IFG and AG appeared across GPT-2-distill, GPT-2-small, and GPT-2-medium, but the final contrast’s IFG pvalue of 0.027 was not significant after ROI correction.Source: D Significance testing and… · p. 12
- The baseline-versus-stimulus-tuned contrast was not significant, and the reported significance tests used ROI-level Wilcoxon signed-rank tests with p < 0.05 and Holm-Bonferroni correction.The passage notes that high subject variability and the number of subjects make significance challenging.Source: D Significance testing and… · p. 12
E Linear relation brain alignment and cross-entropy loss
The authors test whether cross-entropy loss tracks brain alignment because matching perplexity differences across model pairs was infeasible. They also illustrate a contrast intended to control for word-level information and, when next-word prediction differences are equal, next-word prediction.
E Linear relation brain alignment and cross-entropy loss
- Brain alignment correlates with cross-entropy at 0.61, slightly below its correlation with perplexity at 0.67.The authors report that more experiments with additional models and datasets are needed to verify whether the cross-entropy relationship generalizes.Source: E Linear relation brain alignment and… · p. 13, Figure 6 · p. 13
- Because matching perplexity differences between the model pairs was infeasible, the final contrast uses cross-entropy differences.The authors note that a linear relationship between perplexity and brain alignment does not guarantee a linear relationship for cross-entropy.Source: E Linear relation brain alignment and… · pp. 12–13
F Cross perturbation contrast illustration
- The illustrated contrast compares baseline and stimulus-tuned models against their scrambled versions to control for word-level information and, if NWP differences match, next-word prediction.Residual alignment under those controls could reflect other information, such as multi-word information.Source: Figure 7 · p. 14
G Noise Ceiling estimation
Noise-ceiling estimates assess fMRI signal quality and define which voxels are included. Across GPT-2-small, GPT-2-distilled, and GPT-2-medium, stimulus-tuning improves brain alignment for nearly all participants, while residual alignment after the described controls appears in language regions.
G Noise Ceiling estimation
- Voxels with noise-ceiling estimates above 0.05 were selected to exclude noisy voxels and provide a consistent set for model contrasts.Noise-ceiling estimation assesses fMRI signal quality by estimating variance explainable by an ideal data-generating model.Source: G Noise Ceiling estimation · p. 14, Figure 8 · p. 15
H GPT-2-small
- For GPT-2-small, stimulus-tuning improves alignment for almost all participants, and several language regions favor the stimulus-tuned model after the described controls.The voxel-wise contrast compares stimulus-tuned versus scrambled against baseline versus scrambled predictions.Source: Figure 9 · p. 16, Figure 13 · p. 18
I GPT-2-Distilled
- For GPT-2-distilled, stimulus-tuning improves alignment for almost all participants, and several language regions favor the stimulus-tuned model after the described controls.The voxel-wise contrast compares stimulus-tuned versus scrambled against baseline versus scrambled predictions.Source: Figure 15 · p. 19, Figure 19 · p. 21
J GPT-2-Medium
- For GPT-2-medium, stimulus-tuning improves alignment for almost all participants, and several language regions favor the stimulus-tuned model after the described controls.All four models also exhibit significant alignment in language-processing regions despite scrambling-related reductions.Source: Figure 21 · p. 22, Figure 25 · p. 24