Source-linked AI summary
Revisiting the Estimation of Unigram Surprisal in Computational Psycholinguistics
Zachary Hopton, Francesco Ignazio Re, Samuel Kiegeland, Andreas Opedal, Eleanor Chodroff, Ryan Cotterell
TL;DR
Contextual and unigram surprisal are often estimated from different sources, despite representing conditional and marginal structure of one distribution. The paper proposes principles and a shared-language-model Monte Carlo estimator for unigram surprisal. Re-estimating it changes some predictors’ reading-time contributions, while unigram surprisal remains a stronger predictor than contextual surprisal in the reported analyses.
Problem
Method
Results
Unigram surprisal estimation significantly changes some predictors’ predictive power, and unigram surprisal is reported as a stronger reading-time predictor than contextual surprisal.
Source: §1 · p. 2Takeaways & Limitations
The findings support reconsidering unigram-surprisal estimation in computational psycholinguistics, though the empirical analyses focus on sentence processing.
Source: §1 · p. 2, Limitations · p. 9Language-model estimation is computationally expensive, and available resources limited sample size and maximum sampled-string length; analyses are limited to sentence processing.
Source: Limitations · pp. 9–10
Abstract
from arXiv · showhide
Contextual and unigram surprisal are standard predictors of reading time in computational psycholinguistics. While contextual surprisal is typically derived from neural language models, unigram surprisal is usually obtained from independent corpus counts. Although both quantities are deterministic functionals of a single underlying distribution---capturing conditional and marginal structure respectively---this practice breaks their probabilistic link and may introduce estimation error from the mismatch. We propose three principles for estimating unigram surprisal in psycholinguistic research, including the requirement that both quantities be derived from the same language model. We operationalize this principle by introducing a Monte Carlo estimator of unigram surprisal from a language model, along with a marginalized variant that has lower variance empirically. When unigram surprisal is re-estimated under this unified framework, we find evidence that predictors change in their predictive contribution to reading time. We also confirm prior findings that frequency can be a stronger predictor of processing effort than contextual surprisal under this unified framework.
1 Introduction
Contextual and unigram surprisal capture conditional and marginal structure, but are often estimated from different sources. The paper proposes principles and a shared-language-model estimator, then reports that changing the unigram estimator changes reading-time predictions.
1 Introduction
- Unigram surprisal measures a unit’s negative log marginal probability, complementing contextual surprisal’s negative log probability given preceding context.The paper frames both as functionals of an underlying distribution, capturing its marginal and conditional structure respectively.Source: §1 · p. 1
- Estimating unigram surprisal independently from contextual surprisal can break their probabilistic link and introduce estimation error.In the example, PMI is nearly identical for salt and coriander under wordfreq but diverges substantially under the shared-model estimator.Source: §1 · p. 2
- The authors propose the best estimate, cognitive plausibility, and consistent distribution principles for estimating unigram surprisal.They introduce a Monte Carlo estimator from a language model and a marginalized variant with lower variance than naive Monte Carlo at the same sample count.Source: §1 · p. 2
- Unigram surprisal is reported as a stronger reading-time predictor than contextual surprisal across analyses of reading times in eleven languages.The study compares wordfreq-derived unigram values with values from its marginalized Monte Carlo estimator.Source: §1 · p. 2
- Some predictors’ predictive power significantly changes when unigram surprisal is recalculated with the marginalized Monte Carlo estimator.The authors argue that this sentence-processing evidence motivates reconsidering unigram-surprisal estimation more broadly in computational psycholinguistics.Source: §1 · p. 2
2 Language Models
The section formalizes language models as distributions over token strings and defines word-level unigram probabilities by parsing model-generated strings. The resulting unigram distribution normalizes expected word counts.
2 Language Models
- A language model is a probability distribution over finite token strings, and a string’s prefix probability sums the probabilities of all its continuations.The model’s token probabilities can be factored autoregressively using prefix-conditioned probabilities.Source: §2 · pp. 2–3
- Word-level unigram probability is defined from how often a word occurs in strings sampled from the model, using a deterministic parser from token strings to word strings.The paper assumes parsed word positions correspond to disjoint, nonempty, contiguous token spans.Source: §2 · p. 3
- The unigram probability distribution normalizes a word’s expected count by the expected number of words in a model-generated string.The text notes that this normalization follows from the expected count of words in the parsed string.Source: §2 · p. 3
3 Which Language Model?
The section contrasts modeling language as corpus-generating probability with modeling it as a proxy for human-internalized probability. It notes that larger models may become less predictive of human behavior and that training objectives can complicate a corpus-frequency interpretation.
3 Which Language Model?
- Under the frequentist view, a language model approximates the distribution that generated a large, heterogeneous corpus.Such corpora can exceed the linguistic input any one person observes over a lifetime, and their generative process involves many writers and iterative texts.Source: §3 · p. 4
- Instruction-tuned and reasoning-oriented variants shift probabilities away from raw corpus frequency, complicating a straightforward frequentist interpretation.The paper notes this as an additional training-objective effect.Source: §3 · p. 4
- Under the cognitive view, a language model proxies the internalized distribution over linguistic strings carried by a human.As model scale grows, predictive accuracy for human data can improve only up to a point and then degrade, a pattern called the inverse-scaling law.Source: §3 · p. 4
4 Surprisal Theory
Surprisal theory relates processing difficulty to a unit’s negative log probability in context, while unigram surprisal captures its marginal probability. Reading-time studies use eye-tracking measures whose aggregation reflects the processing stage of interest.
4 Surprisal Theory
- Eye-tracking fixations are aggregated into reading-time measures, with first fixation duration associated with early lexical processing and gaze or total fixation duration with later integration.Gaze duration counts time before first leaving a unit; total fixation duration also includes regressions back to it.Source: §4 · p. 4
- Surprisal theory relates processing difficulty to a unit’s negative log probability in context, often tested against fixation duration or neural measures.Language models provide estimates of contextual probabilities.Source: §4 · pp. 4–5
- Unigram surprisal captures marginal probability, while contextual surprisal captures conditional structure; the paper’s consistent distribution principle requires estimating both from the same model.Unigram frequency is also used as a baseline predictor of reading times.Source: §4 · p. 5
5 Principles of Frequency Estimation
The section proposes three principles for estimating unigram surprisal: best estimation, cognitive plausibility, and deriving unigram and contextual surprisal from the same distribution. It notes that the statistically best estimate may not be the most cognitively plausible, and that little is known about modeling choices for unigram frequency.
Best estimate principle
- The best estimate principle treats unigram frequency as a population quantity recoverable from samples of the distribution that generated the corpus.With i.i.d. samples, normalized token counts estimate unigram probabilities and equal the multinomial maximum-likelihood estimator.Source: §5 · p. 5
- Even an unbiased unigram probability estimator yields upward-biased expected surprisal unless its variance is zero.The section attributes this to the strict convexity of −log and notes that it motivates lower-variance frequency estimators.Source: §5 · p. 5
Cognitive plausibility principle
- Cognitive plausibility targets the distribution that best reflects human baseline expectations, which need not be the statistically best estimate.The section notes that modeling choices for the cognitive plausibility of unigram frequency have received comparatively little study.Source: §5 · pp. 5–6
Consistent distribution principle
- Contextual and unigram surprisal should come from the same distribution, preserving their conditional–marginal relationship within one generative system.Estimating them from different sources makes them summaries of different probabilistic models.Source: §5 · p. 6
6 From Autoregressive Language Models to Unigram Probabilities
The section estimates the unigram probabilities induced by an autoregressive language model, since exact computation is infeasible. It introduces naive Monte Carlo sampling and a marginalized variant that uses next-token probabilities to reduce variance, while also considering a canonical-only approximation.
6 From Autoregressive Language Models to Unigram Probabilities
- Exact computation of the language model’s induced unigram probabilities is infeasible, so the section considers two estimators for words with potentially multiple tokenizations.The shared distribution is an autoregressive language model.Source: §6 · p. 6
6.1 A Naive Monte Carlo Estimator
- Naive Monte Carlo estimates a word’s unigram probability from its empirical frequency across all word occurrences in independently sampled strings.For every fixed word, the estimator is strongly consistent and asymptotically unbiased under the stated positive expected-word-count condition.Source: §6.1 · p. 6
6.2 A Marginalized Monte Carlo Estimator
- Naive Monte Carlo can have high variance over the large string space, reducing estimator efficiency and motivating a lower-variance alternative.Because surprisal is positively biased in expectation unless the probability estimator has zero variance, this variance also matters for derived surprisal.Source: §6.2 · p. 6
- MMC retains sampled prefixes but replaces each sampled next-token occurrence decision with its conditional expected contribution under the language model.This uses the model’s probabilities over next tokens and the further tokens needed to complete a word occurrence.Source: §6.2 · p. 6
- MMC splits word counts into canonical and residual occurrences, marginalizing the canonical contribution while reading residual occurrences from completed sampled strings.The per-step contribution assigns the remaining count to the terminal EOS position, making the decomposition exact for each completed string.Source: §6.2 · p. 7
- The canonical-only approximation ignores non-canonical tokenizations and underestimates expected counts whenever residual mass is nonzero.It is unbiased for the canonical share and agrees with the full estimator in expectation only when residual mass vanishes.Source: §6.2 · p. 7
7 Experiments
The experiments extend an analysis of reading times across eleven languages by replacing wordfreq unigram surprisal with estimates from the same mGPT distribution used for contextual surprisal. They evaluate predictor models with gaze duration and cross-validated likelihood comparisons.
Experimental design
- Across eleven languages, the analysis replaces wordfreq unigram surprisal with NMC and MMC estimates from the mGPT distribution while retaining contextual surprisal values.Strings are sampled from mGPT with a maximum length of 50, and estimated probabilities are converted to surprisal using the negative log.Source: §7 · pp. 7–8
Models and outcome
- The study models gaze duration with five GAMMs covering unigram surprisal, contextual surprisal, PMI, and specified predictor combinations.Each model includes a spillover term per predictor and random-effect smooths for participant and trial.Source: §7 · p. 8
Evaluation
- Model fit is measured by cross-validated ΔLL against a random-effects-only baseline, and paired permutation tests compare frequency conditions with Holm–Bonferroni correction.Positive ΔLL means the candidate model assigns higher likelihood to observed reading times than the baseline.Source: §7 · p. 8
8 Results
Marginalization yields lower-variance unigram estimates, while estimator choice also affects reading-time models. Canonical-only variants produce similar results with higher coverage and lower computational cost.
8.1 Marginalizing Reduces Variance
- Across subsample sizes and unigrams, MMC has consistently lower variance than NMC, with the variance ratio generally increasing at larger subsample sizes.Table 1 reports mean bootstrap variance for both estimators and defines the ratio as NMC / MMC.Source: §8.1 · p. 8, Table 1 · p. 8
- NMC fails to estimate every MECO type for any tested sample size from 10^2 to 10^7, and covers fewer than 50% of Finnish types at 10^7 strings.Its estimate requires the unigram to occur in the sample pool.Source: §8.1 · p. 8
- mGPT samples skew toward Hebrew, Greek, and Russian, so MMC contexts may be uninformative for other languages; priority-MMC uses language-specific strata.Unlike NMC, MMC does not have the same coverage problem.Source: §8.1 · pp. 8–9
8.2 Downstream Impact of Marginalizing
- Canonical-only samples yield similar results with higher coverage and lower computational cost.The downstream comparison concerns GAMMs of orthographic word gaze duration, with ∆LL averaged across languages.Source: §8.2 · p. 9, Figure 3 · p. 9
9 Conclusion
The paper addresses limited attention to unigram-surprisal estimation by proposing guiding principles and language-model-based Monte Carlo estimators. Its experiments find that estimator choice affects reading-time predictors and support unigram surprisal as a stronger predictor than contextual surprisal.
9 Conclusion
- Unigram-surprisal estimation has received little research attention compared with contextual-surprisal estimation.The paper proposes three principles to guide unigram-surprisal estimation.Source: §9 · p. 9
- The paper introduces Monte Carlo unigram-surprisal estimation from a language model, including marginalized variants, so both surprisal quantities can come from the same model.This permits differences in predictive power to be attributed to modeling decisions rather than mismatched estimation procedures.Source: §9 · p. 9
- Experiments show that unigram-estimator choice has empirical consequences and support unigram surprisal as a stronger reading-time predictor than contextual surprisal.The authors call for more careful consideration of frequency in computational psycholinguistics.Source: §9 · p. 9
Limitations
The analyses concern sentence processing, and the unit parser and available computational resources constrain the study's scope.
Limitations
- The empirical analysis is limited to sentence processing, so the estimator's impact in other computational psycholinguistics tasks warrants further investigation.The authors name speech segmentation during language acquisition and acceptability judgments as examples.Source: Limitations · pp. 9–10
- Language-model-based unigram estimation is more computationally expensive than lexical databases or corpus counts, and resources limited sample size and maximum sampled-string length.Lower variance in marginalized variants partly addresses the cost by allowing stable estimates from relatively few samples.Source: Limitations · p. 10
- The unit parser matches whitespace-initial token spans, adequate for the studied whitespace-delimited orthographies but not generally applicable.Whether estimator impact varies for other units, such as characters or contextually defined words, remains open.Source: Limitations · p. 10
Ethical Concerns
This block reports the use of previously published eye-tracking data and describes unigram-frequency estimation, its properties, and estimator diagnostics.
Ethical Concerns
- The study uses published human eye-tracking data and foresees no new ethical concerns from its contributions.Source: Ethical Concerns · p. 10
A Estimator description
- The estimator description counts word occurrences using canonical tokenizations and valid boundaries, then marginalizes expected per-step contributions over visited prefixes.Its EOS treatment is described as recovering the exact count.Source: A Estimator description · p. 15
B MLE of Unigram Frequency
- The corpus plug-in estimate is a word’s relative frequency; expected surprisal overestimates the true value unless the probability estimate has zero variance.Source: B MLE of Unigram Frequency · p. 18
C Implementing the Unit Parser
- The unit parser approximates word spans by matching whitespace-initial token spans, a heuristic suited to the whitespace-delimited orthographies studied but not generally linguistically adequate.Source: C Implementing the Unit Parser · p. 18
D Properties of the Naive Monte Carlo Estimator
- Under the stated positive-mass and finite-expectation conditions, the naive Monte Carlo unigram-probability estimator is strongly consistent and asymptotically unbiased.Source: D Properties of the Naive Monte Carlo… · pp. 19–20
E.1 Variance Analysis
- Figure 4 visualizes bootstrapped variance estimates for the marginalized and naive estimators, with separate lines for unigrams and their across-unigram means.Source: Figure 4 · p. 21
E.2 Naive Monte Carlo Coverage
- Figure 5 tracks the proportions of MECO types and tokens with obtainable naive Monte Carlo estimates across sample sizes; an estimate is unavailable if the item never appears in sampled strings.Samples use mGPT with maximum string length set to 50 tokens.Source: Figure 5 · p. 21
E.3 Runtime Analysis
This block reports estimator runtime behavior and documents the reading-time data, evaluation setup, and priority-sampled estimation procedure.
E.3 Runtime Analysis
- For one-token unigrams, NMC and MMC have similar costs, mainly from generating 500 strings; for multi-token units, MMC time scales with token count while NMC time remains stable.MMC’s extra forward passes are paid once per unit, while faster multi-token NMC estimates have higher variance.Source: §E.3 · p. 21
F.1 Reading Time Data
- The reading-time analyses use L1 MECO data from eleven languages supported by mGPT, with skipped words removed and first-fixation, gaze, and total-fixation durations derived.Fold sizes differ between estimators because their coverage differs.Source: §F.1 · p. 22
F.2 Model Specification and Evaluation
- The surprisal-only GAMM uses smooth effects for current and previous-unit surprisal, participant and trial random effects, and cubic regression splines limited to six basis functions.Source: §F.2 · p. 23
F.3 Estimator Configurations
- The comparison includes wordfreq, NMC, MMC, canonical-only MMC, and priority-sampled MMC variants, using estimator-specific sample pools and sizes.Source: §F.3 · p. 24
G Variation across Languages
- Language-averaged delta log-likelihood is micro-averaged across pooled word-level observations in one reporting context and macro-averaged with standard deviations for gaze-duration models in this section.Source: G Variation across Languages · p. 25
H Estimator Pseudocode
- The appendix supplies pseudocode for naive and canonical-only marginalized Monte Carlo unigram-probability estimators.Source: H Estimator Pseudocode · pp. 26–27
I Priority-Sampled Evaluation at Large Sample Sizes
- Priority sampling limits continuation evaluation to selected boundaries; the full estimator restores exact-count corrections omitted by the canonical-only approximation.Strata carrying mass must receive at least one sample, and allocation must not depend on the draws for unbiasedness; negative realized numerators leave estimates undefined and units excluded.Source: I Priority-Sampled Evaluation at Large… · pp. 27–29