PaperlayerView exact revision on arXiv

Source-linked AI summary

STRUCTURALCOST: A controlled reading time dataset for modeling human sentence processing difficulty

Nina Nusbaumer, Iria de-Dios-Flores, Corentin Bel, Christophe Pallier, Guillaume Wisniewski, Benoît Crabbé

arXiv:2610.08208v1cs.CLcs.AI

TL;DR

Language models can fit human reading times through predictive signals, but whether they capture memory-based integration remains an open question. STRUCTURALCOST controls subject–verb dependency length in a large self-paced reading dataset; humans show structure-sensitive difficulty, while models only partly reproduce it and underestimate integration cost.

  • Problem

    Surprisal–reading-time alignment does not establish whether language models capture the memory-based cost of resolving long dependencies.

    Source: §1 · p. 1, §5 · p. 6

  • Method

    STRUCTURALCOST provides controlled self-paced reading data that varies subject–verb dependency length while holding the main subject–verb pair constant.

    Source: §1 · p. 2

  • Results

    Neural models partially mirror the syntactic ordering of human difficulty at the main verb but underestimate dependency-length processing cost across architectures and sizes.

    Source: §6 · p. 9, §7 · p. 9

  • Takeaways & Limitations

    A controlled diagnostic targeting a specific phenomenon can reveal structural-processing differences that global surprisal–RT fit misses.

    Source: §1 · p. 2, §5 · p. 6

    The English controlled constructions and online native-speaker sample limit cross-linguistic and cross-population conclusions.

    Source: Limitations · p. 10

Abstract

from arXiv · show

We introduce STRUCTURALCOST, a self-paced reading dataset of 475 participants and 40,800 observations isolating the processing cost of long-distance subject-verb dependency resolution. We replicate a low-powered psycholinguistic finding at NLP scale, namely that human reading times at the main verb increase with dependency length, driven by syntactic embedding beyond linear distance. Different language models -- spanning n-gram models, SSMs, and transformers -- partially mirror this graded difficulty profile, yet underestimate the integration cost humans incur, with a gap that persists across architectures and model sizes. This suggests these models capture the predictive component of human processing but not the full integration cost that working memory imposes. STRUCTURALCOST provides data needed to drive progress toward evaluating the cognitive plausibility of language models.

1 Introduction

The paper asks whether language models capture both prediction and memory-based integration in human sentence processing, a question with practical value for testing hypotheses at scale. STRUCTURALCOST provides controlled, large-scale evidence that human verb reading times reflect dependency length and syntactic depth, while current models capture this difficulty only partially.

1 Introduction

  • Language-model surprisal is a proxy for prediction, but its fit to reading times alone does not establish sensitivity to memory-based costs of resolving long dependencies.Dependency length is a direct test because resolving a syntactic dependency becomes more demanding as its head and dependent are farther apart.Source: §1 · p. 1
  • Existing evidence leaves a gap: naturalistic corpora are large but lexically confounded, while controlled experiments often lack the scale for comprehensive model evaluation.Separating structural effects from word frequency and length is necessary to measure graded difficulty.Source: §1 · p. 1
  • STRUCTURALCOST contains 40,800 self-paced reading observations from 475 participants and varies subject–verb distance while holding the main subject–verb pair constant.The controlled design isolates dependency-length processing cost from lexical confounds and includes individual working-memory scores.Source: §1 · p. 2
  • Human reading times at the main verb increase with dependency length, with syntactic depth contributing beyond linear distance; language models partially mirror this profile but underestimate integration cost.The gap persists across model architectures and sizes despite strong global surprisal–RT fit.Source: §1 · p. 2
  • The paper presents STRUCTURALCOST as a controlled diagnostic for a specific phenomenon, not as a general benchmark.Its contribution is a resource and controlled manipulation that scales previously low-powered psycholinguistic findings.Source: §1 · p. 2

2 Related Work

Prior work uses surprisal and architectural constraints to relate language models to human reading, but naturalistic reading-time datasets confound lexical and structural influences. The paper argues that evaluation of structural sensitivity requires controlled data alongside sufficient items and participants.

2 Related Work

  • Larger models do not always yield better psychometric fit, and transformer self-attention lacks the distance-dependent decay associated with human locality effects.Prior work links this inverse scaling trend to models becoming superhuman at next-word prediction (Oh et al., 2022; Oh and Schuler, 2023; Shain et al., 2024; Oh and Linzen, 2025).Source: §2 · p. 2
  • Surprisal from language models correlates with reading times, but constrained-model evaluations have relied mainly on naturalistic corpora where frequency, length, and syntax are confounded.Prior work reports that architectural inductive biases shape psychometric fit (De Varda and Marelli, 2024; Timkey and Linzen, 2023), and context limits can partly restore locality effects in transformers (Kuribayashi et al., 2022).Source: §2 · p. 2
  • Table 1 contrasts critical sentences varying syntax and embedding with control sentences varying linear distance through prepositional phrases.The control conditions provide a reference for distinguishing linear distance from structural complexity.Source: Table 1 · p. 3
  • The paper identifies four dataset criteria: isolation, controlled variables, sufficient items, and sufficient participants.Existing NLP datasets offer scale without experimental control, whereas psycholinguistic datasets provide control but lack the items and participants needed for model evaluation.Source: §2 · p. 3

3 Dataset Creation

STRUCTURALCOST manipulates subject–verb dependency length and syntactic structure while controlling lexical overlap, using critical and control sentences to separate linear distance from embedding. The dataset combines those stimuli with online self-paced reading from native English speakers and working-memory testing.

3 Dataset Creation

  • The dataset varies dependency length linearly through prepositional phrases and hierarchically through relative-clause embedding, which adds nested syntactic structure.The design includes critical sentences and controls so distance and structural depth can be examined separately.Source: §3 · p. 3
  • Critical items include six conditions: a 0-word baseline, a 4-word PP control, and singly or doubly embedded SRC and ORC conditions.The PP condition matches the singly embedded relatives in surface dependency length without inserting an embedded clause; controls extend linear distance with PPs only.Source: §3 · p. 4
  • The 450 sentences were drafted with language-model assistance, then manually reviewed for plausibility, semantic equivalence, and naturalness across condition-mates.There are 60 sentences in each of six critical conditions and 30 in each of three control conditions.Source: §3 · p. 4
  • Native English speakers recruited online completed self-paced reading and an operation-span working-memory test; retaining participants with comprehension scores of at least 70% yielded 475 participants.After excluding word times below 100 ms or above 3,000 ms, the dataset contains 27,103 critical and 13,663 control sentences.Source: §3 · p. 4

4 Human Processing Behavior Analysis

Human reading times at the main verb show a dependency-length effect, but condition comparisons reveal that syntactic structure matters beyond distance alone. Comparison with naturalistic datasets shows that STRUCTURALCOST isolates a stronger dependency-length effect through controlled variation.

4 Human Processing Behavior Analysis

  • Across conditions, mean reading times rise from 490 ms at 0-word dependencies to 544 ms at 4 words, then fall slightly to 528 ms at 9 words.Both contrasts with the 0-word baseline exceed the ROPE, while the 4- versus 9-word contrast is negligible.Source: §4 · pp. 4–5
  • At equal 4-word distance, main-verb slowdowns are negligible for PP, small for SRC, and largest for ORC; the same ordering holds at 9 words, while double embedding is unexpectedly faster than single embedding.The 4- versus 9-word contrast collapsed across clause types is negligible, so distance alone does not account for the condition-level pattern.Source: §4 · p. 5, Table 2 · p. 5
  • Dependency length adds a significant effect after word length and frequency are controlled, corresponding to about 3 ms per additional intervening word.The extended model improves fit by χ2(1) = 21.6, p < .001; dependency length has β = 0.0065, t = 4.73, p < .001.Source: §4 · pp. 5–6
  • At the main verb, dependency-length effects decrease in Natural Stories and UCL-SPR but increase in STRUCTURALCOST, corresponding to approximately 1–5 ms versus approximately 11 ms per SD.The controlled design varies dependency length orthogonally to lexical properties; STRUCTURALCOST complements naturalistic corpora with experimental control.Source: §4 · p. 6

5 Probing Surprisal as a Cognitive Proxy

The section tests whether language-model surprisal reflects human processing beyond broad reading-time prediction, focusing on subject–verb integration at the main verb. Models correlate with reading times globally but capture local integration difficulty only partially, and calibrated surprisal predictions substantially underestimate observed effects.

5 Probing Surprisal as a Cognitive Proxy

  • Global alignment correlates surprisal with reading times across all positions, whereas local alignment tests the main verb where subject–verb integration is resolved.Global alignment is a necessary baseline, but averaging across positions dilutes the structural signal assessed by local alignment.Source: §5 · p. 6

5.1 Models and Setup

  • The analysis spans n-grams, transformers, and state-space models, comparing surprisal and dependency-length predictors with nested mixed-effects models and likelihood-ratio tests.Model sizes vary within families; the analyses assess broad all-position correlations and verb-only correlations.Source: §5.1 · pp. 6–7

5.2 Results

  • At dependency length 4, global correlations of r = 0.51–0.53 fall to r = 0.31–0.39 for neural models and r = 0.14–0.15 for n-grams at the main verb.At dependency length 0, all models’ verb-only confidence intervals include zero; models that appear similar globally diverge at the integration site.Source: Figure 3 · p. 7, §5.2 · pp. 7–8
  • At fixed linear distance, neural models reproduce the PP < SRC < ORC surprisal ordering, but at dependency length 9 no model approaches the human 70 ms ORC penalty.Dependency length adds explanatory power beyond surprisal at the main verb, indicating that surprisal alone does not capture structural distance.Source: §5.2 · pp. 7–8
  • Scaling within model families has no consistent benefit for integration-profile fit, while MAMBA-130M reaches r = 0.41 at dependency length 9.The section suggests the gap may reflect structural factors such as inductive bias or training objective rather than capacity alone.Source: §5.2 · p. 8

5.3 Comparison of predicted and observed RTs

  • Across four models, calibrated predicted effects are one to three orders of magnitude too small; KenLM-GPT2 predicts 0.02–1.99 ms versus observed effects of 14–93 ms.No model predicts the correct effect direction in the 2×SRC condition.Source: §5.3 · p. 8, Table 4 · p. 9

6 Discussion

Global surprisal–reading-time alignment is robust but largely reflects lexical statistics and does not diagnose structural difficulty. Neural models partially track syntactic ordering yet underestimate integration cost; the authors identify double-embedding reversals and propose testing memory-sensitive architectures at the integration site.

6 Discussion

  • Neural models improve main-verb fit over n-gram baselines and partly mirror syntactic ordering, but underestimate dependency-length costs across architectures and sizes.MAMBA-130M has the highest verb-level correlations at long dependencies, though whether this reflects architectural sensitivity or training-data and tokenization confounds remains open.Source: §6 · p. 9
  • Global surprisal–reading-time correlations cluster near r ≈0.5 across model sizes, training data, and memory architectures, largely reflecting lexical frequency and word length.The authors argue that this makes global alignment a weak diagnostic of cognitive plausibility.Source: §6 · p. 9
  • At double embedding, 2×SRC and 2×ORC are read faster than singly embedded counterparts, a reversal no model predicts.Within-sentence syntactic adaptation is proposed as one possible explanation, but the mechanism remains an open question.Source: §6 · p. 9
  • The authors favor controlled evaluation at memory-sensitive positions and propose testing limited context, recency-biased attention, or capacity-constrained recurrence at the integration site.They prefer these architectural interventions over further scaling.Source: §6 · p. 9

7 Conclusions

STRUCTURALCOST is a controlled self-paced reading dataset for isolating dependency-length costs. Human main-verb reading times rise with dependency length, while language models only partly capture the graded structural difficulty profile; the dataset supplies controlled evaluation data for further work.

7 Conclusions

  • STRUCTURALCOST is a controlled self-paced reading dataset that isolates dependency-length processing costs from lexical confounds.The conclusion describes it as capturing memory-based difficulty.Source: §7 · p. 9
  • Human main-verb reading times increase with dependency length, driven by syntactic embedding beyond linear distance, while models partially mirror the profile but underestimate integration cost across architectures and sizes.The evaluation spans n-gram, SSM, and transformer architectures.Source: §7 · p. 9
  • STRUCTURALCOST provides controlled evaluation data intended to support progress on cognitive plausibility evaluation of language models..Source: §7 · p. 9

Limitations

STRUCTURALCOST prioritizes internal validity over ecological generalizability, and its findings are bounded by the study’s language, participant, and online-testing settings.

Limitations

  • The controlled English constructions and online native-speaker sample limit cross-linguistic and cross-population conclusions.The online setting also left reading environments and equipment uncontrolled.Source: Limitations · p. 10
  • A denser condition gradient could better separate linear from hierarchical contributions and clarify the double-embedding reversal.The paper leaves this extension for future work.Source: Limitations · p. 10
  • Collected working-memory scores are not yet fully analyzed, and an implementation error left only participant-level—not per-item—comprehension accuracy.Per-item results could have enabled finer comparisons between reading times and comprehension accuracy.Source: Limitations · p. 10
  • The evaluated architectures are representative but non-exhaustive, so other models may behave differently.Their evaluation on STRUCTURALCOST remains open.Source: Limitations · p. 10

Ethical considerations

The study involved human participants and received Institutional Review Board approval.

Ethical considerations

A Dataset comparison

This section presents the dataset-comparison criteria and describes STRUCTURALCOST’s controlled design, participants, and study procedures.

B.1 Design & conditions

  • Each of 60 item sets contains six sentences sharing a main subject–verb pair, with conditions varying dependency length and syntactic structure.Control sentences vary linear distance using prepositional phrases without clause embedding; critical sentences share their subject–verb pairs across conditions.Source: §B.1 · p. 12

A Dataset comparison

  • Table 5 compares reading-time datasets on sentence isolation, systematic dependency-length manipulation, item sufficiency for model evaluation, and participants per item.The table marks criteria as satisfied, not satisfied, partially satisfied, or unreported.Source: A Dataset comparison · p. 12, Table 5 · p. 13

B.2 Item counts & list structure

  • Each list contains 60 critical items and 30 controls, with one sentence per item set and condition assignments rotating across six Latin-square lists.Lists balance singular and plural items equally.Source: §B.2 · p. 13

B.3 Participants: recruitment and payment

  • Participants were recruited as native English speakers with U.S. nationality or longest U.S. residence, excluding those with diagnosed or suspected reading disorders.The experiment lasted about 30 minutes; participants were paid £5.Source: §B.3 · p. 13

B.4 Consent procedure

  • Participants received study information and consented before admission; collected data were pseudonymized and anonymized before analysis.The information letter covered study objectives, procedure, risks, and the right to withdraw.Source: §B.4 · pp. 13–14

B.5 Instructions given to participants

  • Participants read sentences one word at a time, answered some comprehension questions, and completed a separate math-and-letter memory task.The instructions set a comprehension target of at least 80%; the study described its measures as intended to constrain language models to process language more like humans.Source: §B.5 · p. 14

B.6 Data collection procedure

The analyses find that language-model fit is lowest for fast readers, while the section also details data collection, stimulus generation, preprocessing, and measurement procedures.

B.6 Data collection procedure

  • Self-paced reading recorded the time between keypresses as participants revealed words; working-memory capacity was measured with an operation-span task.The operation-span task required solving arithmetic operations while retaining letter sequences; scores ranged from 0 to 6.Source: §B.6 · p. 14, Table 7 · p. 14

B.7 Stimulus generation

  • Critical sentences were generated in six structural variants with shared subject–verb pairs and lexical overlap, while control sets used prepositional phrases at three dependency lengths.All items underwent manual review for plausibility, verb predictability, agreement, tense, and length matching.Source: §B.7 · pp. 14–15

C Preprocessing

  • The first and last sentence words were excluded from RT and surprisal analyses because sentence-boundary RTs reflect task mechanics rather than linguistic processing.Reader profiles were computed from trimmed data, and predictors were z-scored.Source: C Preprocessing · p. 15

D Dependency parsing

  • Dependency length was measured as the token distance between the main verb and its subject, using UDPipe with an English Universal Dependencies model for UCL.Natural Stories already provided parse files.Source: D Dependency parsing · p. 15

E Reader profile alignment

  • Fast readers show the lowest verb-level fit across models (r = 0.15-0.22 for neural models; near zero for KenLM), while medium and slow readers show higher fit (r = 0.36-0.40).High-working-memory readers are faster and more accurate, yet are among the readers least well fit by the models.Source: E Reader profile alignment · p. 15

F Surprisal-RT correlation

  • Spearman correlations assess model surprisal against mean per-word RT globally and locally at the main verb by dependency length.Spearman was chosen because the surprisal–RT relation is not fully linear; Table 8 reports these correlations and marks the best local score.Source: F Surprisal-RT correlation · p. 15, Table 8 · p. 15
Loading 2610.08208v1…