Source-linked AI summary
Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?
Lanni Bu, Xiulin Yang, Christian Clark, Alex Warstadt, Ethan Gotlieb Wilcox
TL;DR
Transformer attention can access the full preceding context, unlike human memory, motivating tests of constrained attention as a model of language processing. The paper systematically compares multiple constraints across models, corpora, and static or dynamic training regimes, finding that content-based constraints best align with human reading times while dynamic curricula favor grammatical competence. This dissociation suggests that modeling real-time processing and language acquisition may require different approaches.
Problem
Prior work tested few memory constraints and limited model and dataset settings, leaving the generality of cognitive-fit improvements unclear.
Method
The study compares multiple cognitively interpretable attention constraints across two model sizes, three corpora, six reading-time corpora, and static or dynamic training regimes.
Results
Content-based constraints consistently outperform distance-based constraints on human reading-time alignment, while dynamic models show stronger grammatical competence than static models.
Takeaways & Limitations
Psychometric modeling and grammatical competence may require different Transformer designs, with static constraints better suited to reading-time prediction and dynamic curricula to grammatical competence.
Takeaways & Limitations
The mechanisms are proxies rather than direct implementations of cognitive theories, and results use a single seed and English reading-time corpora.
Abstract
from arXiv · showhide
Transformer-based language models are widely used as models of human language processing, yet their attention mechanisms allow lossless access to the full preceding context, unlike the limited memory systems of humans. We hypothesize that installing memory constraints into transformers' attention mechanisms can improve their fit to human behavioral data. While previous work has explored individual constraints in isolation, we conduct a systematic comparison of multiple attention-based memory mechanisms across different model sizes and training corpora, evaluating both psychometric predictive power for human reading times and grammatical competence. We additionally compare static constraints, in which the constraint strength is fixed throughout training, to dynamic memory curricula. We find that constraints that are sensitive to the content of intervening tokens consistently achieve the highest alignment with human reading times, outperforming distance-based constraints. We observe a dissociation between psychometric fit and grammatical competence under dynamic memory curricula, suggesting that Transformers cannot serve as a one-size-fits-all cognitive model.
1 Introduction
The paper systematically tests whether attention-based memory constraints improve Transformers’ fit to human language behavior. It compares multiple constraints, model sizes, corpora, and static versus dynamic training regimes.
- Motivation: Previous studies tested only a few memory constraints and limited model and dataset settings, leaving generalizability unclear.Prior work focused mainly on n-gram-like cutoffs or distance-based decay, while alternative theories such as intervener-based interference remained untested.
- Approach: The paper tests multiple attention-based memory constraints selected from machine-learning mechanisms with plausible cognitive interpretations.The mechanisms were not originally designed as cognitive models but are mapped onto theories of memory limitation.
- Experimental design: The experiments vary two model sizes, three pretraining corpora, six reading-time corpora, and static versus dynamic constraint strengths.Static constraints remain fixed during training, whereas dynamic constraints change over training.
- Findings: Content-sensitive constraints achieve the highest alignment with human reading times across most training configurations and outperform distance-based constraints.The result identifies intervening-token content as a stronger basis for attention limitation than distance alone.
- Findings: Dynamic curricula dissociate psychometric fit from grammatical competence: static models better predict reading times, while dynamic models perform better on grammatical benchmarks.A gradually imposed constraint curriculum outperforms other curricula, challenging a straightforward application of prior cognitive theories.
2 Background
The paper frames memory constraints as computational operationalizations of theories about distance decay and content-based interference. It implements these theories through several Transformer attention mechanisms and dynamic curricula.
- Theoretical background: Cognitive constraints limit the speed, accuracy, and type of mental operations, and may promote generalization and robustness.Language research uses language models to instantiate computational-level theories of memory limitation.
- Distance-based accounts: Distance-based accounts model retrieval difficulty through activation decay or fixed context cutoffs tied to target–cue distance.Linear decay motivates lossy context surprisal, while n-gram attention implements a per-layer cutoff that remains soft across multiple layers.
- Content-based accounts: Content-based accounts attribute retrieval difficulty to feature overlap among intervening items, producing similarity-based interference and processing slowdowns.This account has been supported across several controlled psycholinguistic experiments.
- Cognitive modeling: The study applies all attention modifications to the same decoder-only vanilla Transformer and evaluates human syntactic judgments and reading times.This design supports comparison across mechanisms while connecting attention to cue-based memory retrieval.
- Static mechanisms: ALiBi adds a linear distance penalty, whereas the Forgetting Gate makes attention sensitive to intervening-token identity.ALiBi is treated as distance-based; the Forgetting Gate is treated as content-based because interveners affect attention scores.
- Static mechanisms: Stick-breaking attention down-weights attention after high-attention interveners without adding parameters that separately control the decrease.Its attention reduction is not tied directly to token positions, though distance can affect the number of interveners.
- Dynamic curricula: Dynamic curricula vary constraint strength during training, motivated by developmental theories such as Less is More.Prior work found that slowly relaxing a linear attention constraint improved grammatical competence but did not assess real-time processing fit.
- Scope: A curriculum version of n-gram attention was not continued because step changes in n caused training instability, and dynamic stick-breaking attention was left for future work.The dynamic experiments therefore focus on ALiBi and the Forgetting Transformer.
3 Methods
The study compares constrained decoder-only Transformers across model sizes, training corpora, attention mechanisms, curricula, reading-time datasets, and grammatical evaluations. Human cognitive fit is measured by the improvement in reading-time prediction from adding model surprisal to regression baselines, alongside BLiMP accuracy.
- Models and training: Models use a decoder-only OPT-based Transformer in 2-layer and 4-layer configurations, trained from scratch on three corpora.The corpora are BabyLM-10M, BabyLM-100M, and a 2-billion-token Pile subset.
- Models and training: Static constraints remain fixed during training, whereas dynamic curricula change constraint strength and are evaluated across training checkpoints.Dynamic ALiBi and Forgetting Transformer models use both Less-to-More and More-to-Less schedules; static models use the final checkpoint after 10 epochs.
- Behavioral evaluation: The evaluation uses six English reading-time corpora spanning self-paced reading and eye-tracking paradigms.The datasets are Brown, Natural Stories, UCL, Dundee, GECO, and Provo.
- Behavioral evaluation: ∆llh measures the improvement in reading-time prediction when model-derived surprisal is added to a regression model containing baseline predictors.Linear mixed-effects models include subject and item random intercepts, and ∆llh is evaluated on an exploratory partition.
- Grammatical evaluation: Grammatical competence is assessed with BLiMP accuracy across 12 linguistic phenomena using minimal grammaticality pairs.The reported score is overall accuracy averaged across all 12 categories.
4 Results
Content-sensitive attention mechanisms provide the strongest and most consistent alignment with human reading times among static models, while dynamic curricula reveal a divergence between psychometric fit and grammatical competence. Model size and training regime affect the performance of distance-based mechanisms.
- 4.1 Exp. 1: Static Memory Constraints: The Forgetting Transformer ranks first in five of six training configurations and significantly exceeds vanilla ∆llh in every setting.Stick-breaking attention ranks second or third and significantly exceeds vanilla scores in three settings.
- 4.1 Exp. 1: Static Memory Constraints: Content-based constraints provide more robust cognitive-fit gains than distance-based constraints across training conditions.The content-based mechanisms are the Forgetting Transformer and Stick-breaking attention; distance-based mechanisms show more mixed results.
- 4.1 Exp. 1: Static Memory Constraints: ALiBi performs strongly with 4-layer models but shows weaker or inconsistent gains with 2-layer models.For 2-layer variants trained on BabyLM corpora, ALiBi ∆llh is significantly higher than vanilla scores.
- 4.1 Exp. 1: Static Memory Constraints: Lower perplexity is generally associated with higher ∆llh (r = −.69, p < .001), while the Forgetting Transformer and Stick-breaking attention perform above the regression line.Their position above the line suggests that their reading-time fit is not fully explained by language-modeling quality alone.
- 4.2 Exp. 2: Dynamic Memory Constraints: Dynamic models outperform static counterparts on BLiMP, whereas static models consistently outperform dynamic models on ∆llh.No dynamic model reaches the final-epoch ∆llh of its static counterpart; More-to-Less Forgetting Transformer reaches the highest BLiMP score by epoch 10.
5 Discussion
The discussion finds that content-based memory constraints are more reliable than distance-based constraints for psychometric fit, while static and dynamic training favor different evaluation goals. These results caution against one-size-fits-all cognitive modeling with autoregressive Transformers.
- A linear attention bias can improve cognitive fit over an unbiased model in some settings, consistent with prior findings.
- Distance-based bias gains are not uniform: they depend strongly on model architecture and training data, and can reduce reading-time prediction relative to vanilla Transformers.
- For static models, content-based constraints consistently outperform distance-based constraints across architectures and training-dataset sizes.
- Intervener-based decay may better characterize human memory mechanisms, while linear-decay theories are not thereby ruled out and could be reworked.
- Static constraints may better suit psychometric modeling, whereas dynamic curricula produce stronger grammatical competence, revealing a dissociation between the two goals.
- A one-size-fits-all cognitive model may be unattainable with autoregressive Transformers, potentially requiring different architectures for different linguistic mechanisms.
- The dynamic More-to-Less Forgetting Transformer had the strongest fit to human data, but differences from human learning environments limit strong interpretation.The models lack multimodal input and social interaction, which may alter the optimal learning trajectory.
6 Conclusion
The paper compares memory constraints and training curricula to determine which best capture human sentence-processing behavior. Content-based constraints, especially the Forgetting Transformer, best align with reading times, while dynamic curricula improve grammatical competence rather than psychometric fit.
- Content-based constraints, particularly the Forgetting Transformer, most consistently align with human reading-time data and outperform distance-based constraints.
- Dynamic models do not improve psychometric fit over static models but show stronger grammatical competence.
- The results support systematic testing of cognitive theories across modeling conditions and motivate further study of intervener-based memory mechanisms.
Limitations
The study’s conclusions are bounded by proxy implementations, single-seed results, English-only reading-time data, and unequal checkpoint evaluation between static and dynamic models.
- The tested attention mechanisms are proxies for cognitive theories rather than direct implementations, and alternative mappings could produce different results.
- Results use a single random seed because of computational constraints.
- The evaluation is limited to English reading-time corpora, leaving generalization to other languages open.
- Static models are evaluated only at epoch 10, whereas dynamic models are evaluated every epoch, complicating direct BLiMP comparisons.
Use of AI Assistants
AI assistants supported code development and editing during preparation, while the authors state that they checked and validated the writing and code and retain responsibility for the scientific content and conclusions.
- AI assistants were used for code development and editing support, followed by author checking, validation, and editing.
- The authors claim responsibility for the scientific content, analysis, conclusions, and mistakes.
A Training Hyperparameters
Models use the OPT architecture and are trained from scratch in two sizes with specified optimization and context-length settings.
- The models use the OPT architecture and are trained from scratch.
- Two model sizes are evaluated: 2 layers with 256 hidden dimensions and 4 attention heads, or 4 layers with 384 hidden dimensions and 6 attention heads.
- All models train for 10 epochs with AdamW, a 0.001 learning rate, 0.1 weight decay, and betas of (0.9, 0.95).
- Context length is 512 tokens for BabyLM-10M and 1024 tokens for BabyLM-100M and Pile-2B.
B Full Dynamic Results
Experiment 2 reports per-setting ∆llh for dynamic models across training epochs, broken down by training-data and model-size conditions.
- The results are broken down by training data size and model size.
- Per-setting ∆llh is tracked across training epochs for dynamic models.
C Perplexity and Cognitive Fit over Training
Dynamic-model analyses track BLiMP scores and test-set perplexity across training epochs, using color to visualize total ∆llh and its relationship to cognitive fit.
- Models achieve lower perplexity over training, while the color coding visualizes a dissociation between training epoch and cognitive fit.
- Dynamic-model BLiMP scores are shown across training epochs.
- Training epoch is plotted against test-set perplexity for dynamic models, with color indicating total ∆llh.