Source-linked AI summary
On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior
Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, Roger Levy
TL;DR
It remains unclear which computational architecture best characterizes the expectations humans use during real-time comprehension. The paper evaluates over two dozen language models while varying architecture and training-data size, testing their surprisal against naturalistic human reading data. Better next-word prediction generally improves psychometric predictive power, but architecture matters and syntactic knowledge adds no significant predictive variance after perplexity control.
Problem
The paper addresses the open question of which computational architecture best characterizes human real-time comprehension expectations and their reading-behavior signatures.
Method
The study trains language models across architectures and four BLLIP corpus sizes, then evaluates surprisal-based predictions of three human reading datasets and syntactic generalization.
Results
Across architectures and training sizes, surprisal–reading-time relationships are near-linear; better next-word prediction generally improves psychometric power, while architecture produces substantial differences.
Takeaways & Limitations
Naturalistic reading prediction depends more on accurate next-word prediction than syntactic knowledge after perplexity control, whereas controlled syntactic tests distinguish architectures differently.
Takeaways & Limitations
Perplexity comparisons are directly valid only for models using a fixed reference vocabulary.
Abstract
from arXiv · showhide
Human reading behavior is tuned to the statistics of natural language: the time it takes human subjects to read a word can be predicted from estimates of the word's probability in context. However, it remains an open question what computational architecture best characterizes the expectations deployed in real time by humans that determine the behavioral signatures of reading. Here we test over two dozen models, independently manipulating computational architecture and training dataset size, on how well their next-word expectations predict human reading time behavior on naturalistic text corpora. We find that across model architectures and training dataset sizes the relationship between word log-probability and reading time is (near-)linear. We next evaluate how features of these models determine their psychometric predictive power, or ability to predict human reading behavior. In general, the better a model's next-word expectations, the better its psychometric predictive power. However, we find nontrivial differences across model architectures. For any given perplexity, deep Transformer models and n-gram models generally show superior psychometric predictive power over LSTM or structurally supervised neural models, especially for eye movement data. Finally, we compare models' psychometric predictive power to the depth of their syntactic knowledge, as measured by a battery of syntactic generalization tests developed using methods from controlled psycholinguistic experiments. Once perplexity is controlled for, we find no significant relationship between syntactic knowledge and predictive power. These results suggest that different approaches may be required to best model human real-time language comprehension behavior in naturalistic reading versus behavior for controlled linguistic materials designed for targeted probing of syntactic knowledge.
Introduction
The paper scales up tests of language models as predictors of human real-time reading, varying architecture and training-data scale. It examines how next-word prediction, architecture, and syntactic knowledge relate to psychometric predictive power.
- Motivation: Humans’ expectation-based comprehension is reflected in correlations between language-model next-word probabilities and online reading measures.Prior evidence includes gaze duration, self-paced reading times, and EEG N400 responses.
- Motivation: Modern neural models, including LSTMs and Transformers, achieve state-of-the-art perplexity, motivating broader tests of their reading-behavior predictions.The paper varies both model inductive bias and training-data scale.
- Research questions: Syntactic knowledge remains an open question because prior modeling results disagree about whether syntactic capacity or perplexity primarily determines fit to human reading times.Earlier studies found differing roles for hierarchical structure, inductive bias, and next-word prediction quality.
- Contribution: The study evaluates a scaled-up fleet of neural models across three online processing datasets, varying inductive bias and training-data amount.The datasets include Dundee eye-tracking, Brown self-paced reading, and Natural Stories self-paced reading.
- Main findings: Across models, better next-word prediction generally accompanies stronger prediction of human reading times, while deep Transformers show additional advantages beyond perplexity.The study also reports a near-linear surprisal–reading-time relationship and tests syntactic knowledge separately.
- Main findings: After controlling for next-word prediction accuracy, syntactic knowledge does not explain significant variance in psychometric predictive power.This contrasts naturalistic reading prediction with targeted syntactic evaluation.
Methods
The methods train language models across architectures and corpus sizes, then test whether their surprisal estimates predict human reading behavior. Psychometric models control for lexical features and are evaluated against a baseline using by-token log-likelihood differences.
- Models: Each language model predicts the next token xi from preceding context xj<i, producing Pmodel(xi | xj<i).The fleet includes LSTMs, recurrent neural network grammars, Transformers, and n-gram models.
- Training data: Four nested BLLIP training corpora range from 40K sentences and 100K tokens to 2M sentences and 42M tokens.Each architecture was trained with 1–3 random seeds per corpus.
- Representations: Sub-word Transformers use byte-pair encoding, decomposing common word substrings into independent tokens and representing sublexical co-occurrence information.For these models, word probability is defined as the joint probability of constituent subwords.
- Evaluation metric: Lower perplexity indicates more accurate next-word predictions, but perplexity is directly comparable only under a fixed reference vocabulary.The analyses therefore mostly compare models sharing the same reference vocabulary.
- Psychometric predictive power: Psychometric predictive power measures how well word-by-word surprisal explains reading measures after controlling for word length, frequency, and related lexical features.Surprisal is the negative logarithm of context-conditioned word probability.
- Psychometric data: The study uses Dundee eye-tracking, Brown self-paced reading, and Natural Stories self-paced reading data.Regression models predict subject-averaged responses while combining model-specific surprisal with model-invariant controls.
- Psychometric predictive power: ∆LogLik is the mean by-token log-likelihood difference between a model containing surprisal predictors and a controls-only baseline.Positive ∆LogLik indicates more accurate predictions of human reading behavior.
- Syntactic evaluation: Syntactic capability is assessed with 34 targeted tests whose item-level surprisal inequalities determine correctness; the mean-by-test score ranges from 0 to 1.Chance performance is approximately 0.25.
Results
Across model classes and test corpora, lower perplexity generally corresponds to greater psychometric predictive power, but the relationship varies by architecture. After controlling for perplexity, syntactic generalization does not significantly explain predictive power.
- Predictive Power vs. Perplexity: Lower perplexity predicts higher ∆LogLik across all three testing corpora, with a significant effect within each training vocabulary except BLLIP-LG.The exception was BLLIP-LG, where p = 0.07.
- Predictive Power vs. Perplexity: The perplexity–∆LogLik relationship is monotonic but more linear for n-grams and more exponential for neural models in the reported ranges.For neural models, ∆LogLik gains are stronger at lower perplexity.
- Predictive Power vs. Perplexity: N-gram models can match or exceed neural models’ predictive power despite substantially worse perplexity, especially on the Dundee corpus.For BLLIP-SM and XS on Dundee, n-gram perplexity was 2× neural perplexity while average ∆LogLik was higher.
- Psychometric Predictive Power vs. Syntactic Generalization: Syntactic generalization is measured by average accuracy across 34 targeted syntactic tests and is evaluated against psychometric predictive power after perplexity residualization.Residualization isolates the relationship unexplained by the perplexity–∆LogLik relationship.
- Psychometric Predictive Power vs. Syntactic Generalization: SG scores do not significantly improve prediction of ∆LogLik beyond perplexity, with p > 0.26 for all three testing corpora.Residual ∆LogLik varies widely among models, including models trained on the same dataset; one cited range overlaps 16/25 other models.
Discussion
Across language models, better next-word prediction generally aligned with stronger prediction of human reading times, while architecture substantially shaped psychometric performance. Syntactic generalization showed no clear relationship with reading-time predictive power after controlling for perplexity, suggesting different comprehension contexts favor different model properties.
- Word surprisal and human reading time showed a generally linear relationship across model architectures and training-data sizes.
- Within architectures, next-word prediction performance was mostly monotonically related to psychometric predictive power.
- Model architecture substantially influenced psychometric predictive power, and its influence differed from architecture effects on controlled grammatical tests.
- After perplexity was controlled, syntactic generalization did not explain significant variance in psychometric predictive power.
- The relatively strong n-gram performance reinforces the need to investigate locality as a contributor to psychometric predictive power.
- Naturalistic reading and controlled grammatical testing appear to engage different linguistic expectations, which different contemporary NLP models may capture best.