Source-linked AI summary
Revisiting the Uniform Information Density Hypothesis
Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, Roger Levy
TL;DR
UID research has limited converging evidence from comprehension and lacks a settled operationalization or scope for measuring uniformity. This paper analyzes reading times and acceptability while comparing UID measures across linguistic scopes. Its findings favor super-linear surprisal effects and suggest that language-wide mean surprisal may best capture the pressure toward uniformity.
Problem
UID’s production implications are well studied, but comprehension evidence, operationalizations, and the linguistic scope of uniformity remain unclear.
Source: §1 · p. 1Method
The authors analyze reading-time and acceptability data and compare UID operationalizations across phrase, sentence, document, and language scopes.
Source: §1 · pp. 1–2Results
A super-linear function of surprisal best explains psychometric data, and global operationalizations appear to explain more than local ones.
Source: §6 · p. 9Takeaways & Limitations
Regression toward mean surprisal across the language may capture UID pressure better than phrase-, sentence-, or document-level means, consistent with a communication-channel interpretation.
Source: §1 · p. 2The experimental analysis is limited to English and Dutch, so further experiments are needed to verify the findings across languages.
Source: §5 · p. 9
Abstract
from arXiv · showhide
The uniform information density (UID) hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal. While its implications on language production have been well explored, the hypothesis potentially makes predictions about language comprehension and linguistic acceptability as well. Further, it is unclear how uniformity in a linguistic signal -- or lack thereof -- should be measured, and over which linguistic unit, e.g., the sentence or language level, this uniformity should hold. Here we investigate these facets of the UID hypothesis using reading time and acceptability data. While our reading time results are generally consistent with previous work, they are also consistent with a weakly super-linear effect of surprisal, which would be compatible with UID's predictions. For acceptability judgments, we find clearer evidence that non-uniformity in information density is predictive of lower acceptability. We then explore multiple operationalizations of UID, motivated by different interpretations of the original hypothesis, and analyze the scope over which the pressure towards uniformity is exerted. The explanatory power of a subset of the proposed operationalizations suggests that the strongest trend may be a regression towards a mean surprisal across the language, rather than the phrase, sentence, or document -- a finding that supports a typical interpretation of UID, namely that it is the byproduct of language users maximizing the use of a (hypothetical) communication channel.
1 Introduction
The paper tests whether information distribution predicts reading times and acceptability, then compares ways to measure UID and the scope over which uniformity holds. Reading-time evidence permits a slight super-linear effect, while acceptability provides clearer evidence favoring more uniform information distribution.
1 Introduction
- Prior UID evidence spans phonetics, lexical choice, syntax, and discourse, but converging comprehension evidence and justified operationalizations remain limited.Earlier studies often assessed individual linguistic units without considering information across the signal as a whole.Source: §1 · pp. 1–2
- The study relates information distribution across sentences to native speakers’ sentence-level reading times and acceptability judgments.These analyses address aspects of UID that had lacked clarity or unity.Source: §1 · p. 1
- Acceptability judgments more clearly favor super-linear surprisal effects, while reading-time results remain compatible with a slight super-linear effect alongside prior linear findings.The authors then compare operationalizations and the linguistic scope over which uniformity should be measured.Source: §1 · pp. 1–2
- The strongest trend may be regression toward mean surprisal across the language rather than the phrase, sentence, or document.This proposed global interpretation aligns with language users maximizing use of a hypothetical communication channel.Source: §1 · p. 2
2 Processing Effort in Comprehension
The section introduces information-theoretic accounts of processing effort, including surprisal theory and UID. It contrasts a linear surprisal account with UID’s claim that information distribution also matters.
2 Processing Effort in Comprehension
- Surprisal quantifies a linguistic unit’s information content, and the information content of a signal is the sum of its units’ surprisals.The section frames processing effort as varying with perceived linguistic units.Source: §2 · p. 2, §2.2 · p. 2
2.1 Surprisal Theory
- Surprisal theory links a word’s processing effort linearly to its surprisal, its unexpectedness in context.Prior work is described as offering wide empirical support, including evidence for a linear effort-surprisal relationship.Source: §2.1 · p. 2
2.2 Uniform Information Density
- With fixed bits to convey, surprisal theory predicts equal processing effort whether information is packed, spread thinly, or distributed unevenly.This counter-intuitive implication motivates UID’s claim that distribution affects overall processing effort.Source: §2.2 · pp. 2–3
- UID models processing effort as a super-linear function of surprisal plus utterance length, making a uniform surprisal distribution the least effortful for fixed length and total information.High-surprisal instances carry disproportionately high processing effort under this account.Source: §2.2 · p. 3
- Most UID studies assess individual word surprisals rather than their distribution within a sequence, and earlier verifications often tested implied tendencies instead of UID operationalizations directly.The authors identify this as a gap in direct verification of UID.Source: §2.2 · p. 3
3 Quantifying Linguistic Uniformity
The section frames UID as smoothing information density, distinguishes global regression toward a target mean from local smoothing, and develops measures linked to processing effort, reading time, and acceptability.
3 Quantifying Linguistic Uniformity
- UID treats uniformity as smoothing toward a mean information rate, which may be measured per lexical unit in text or per time unit in speech.A fixed-rate interpretation favors smaller absolute variation across a sentence; a local interpretation instead favors gradual transitions between sparse and dense regions.Source: §3 · p. 3
3.1 Regressing to Which Mean?
- Because the target mean could be phrase-, sentence-, or language-wide, literary cadence motivates context-dependent means, while prior global analyses do not test local UID.The section notes that cadence varies across sentences, and that earlier evidence against high or very low surprisal generally cannot distinguish global from local uniformity.Source: §3.1 · p. 4
3.2 Operationalizing UID
- The proposed operationalizations include super-linear surprisal cost and penalties for large deviations from a mean or for high maximum per-unit surprisal.The super-linear exponent controls the strength of the effect; the target mean can be sentence-level or language-wide, and local variance provides a separate measure.Source: §3.2 · p. 4
- A further operationalization uses Rényi entropy with k < 1, which assigns extra cost to low-probability, high-surprisal items.The distribution is normalized to sum to 1; Rényi entropy is maximized by a uniform distribution and approaches Shannon entropy as k approaches 1.Source: §3.2 · pp. 4–5
3.3 UID, Effort and Acceptability
- The paper relates UID-based processing effort to reading time and models sentence acceptability as inversely related to effort, apart from a length penalty.For the entropy-based metric, the conditional probabilities are renormalized so the measure has entropy-like properties.Source: §3.3 · p. 5
4 Experiments
The experiments estimate surprisal with several language models and test how surprisal functions predict reading times and acceptability. Results favor slightly super-linear surprisal for acceptability, while reading-time evidence remains compatible with a linear effect; among UID measures, language-wide variance is comparatively predictive.
4.1 Estimating Surprisal
- The experiments used English reading-time data from four corpora and acceptability data from CoLA and BNC, estimating surprisal with GPT-2, TransformerXL, BERT, and a 5-gram model.BERT is a cloze model, so its probabilities are pseudo-surprisal estimates; Dutch GECO data were also used in additional experiments.Source: §4 · p. 5, §4.1 · p. 5
4.2 Assessing Predictive Power
- Linear regression models compared each UID predictor with control-feature baselines using held-out data averaged over 10-fold cross-validation.Reading-time analyses used mixed-effects models, while acceptability analyses used logistic regression; sentence-level reading times summed word-level times.Source: §4.2 · p. 6
4.3 Results
- Acceptability favored k > 1 at α = 0.001 except under the n-gram model, while reading-time tests never rejected k = 1.For reading times, k = 1 was typically within the standard error of the best predictor; additional predictors and Dutch GECO experiments showed similar trends.Source: §4.3 · p. 7, Figure 3 · p. 7
- Super-Linear operationalizations for k ≥ 1 and language-wide Variance generally provided the largest increase in explanatory power, while Entropy predictors performed poorly, especially for k ≥ 1.Language-level Variance appeared stronger than sentence-level and Local Variance across psychometric data.Source: §4.3 · p. 8, Table 1 · p. 8
- Across corpora, higher-level Variance predictors appeared to explain reading times better than more local predictors.The tested means covered previous-word windows, sentences, documents, and the language as a whole; GPT-2 surprisal estimates were used.Source: §4.3 · p. 8, Figure 4 · p. 8
5 Discussion
Reading-time results remain compatible with linear surprisal effects but also allow a weakly super-linear account; acceptability provides clearer evidence consistent with UID. The discussion also identifies limits in language and modality coverage.
5 Discussion
- Reading times do not rule out linear surprisal effects, but values of k slightly above 1 often fit at least as well, especially in Provo.The analysis tested a finer range of k values than a quadratic-term test and included sentence-level times and rereading times for eye movements.Source: §5 · p. 9
- The reading-time analysis differs from prior word-level work by testing a fine-grained range of nonlinear forms and including rereading times for eye-movement data.Source: §5 · p. 9
- The experimental evidence is limited to English and Dutch, and psychometric data come only from written language; spoken-domain analyses remain needed.The authors also report no evidence about pressure toward a cross-linguistic mean information rate.Source: §5 · p. 9
- A negative-probability alternative produced poorer regression fits than surprisal in the repeated analysis.The comparison used the model-fit results in Fig. 11 versus Fig. 3.Source: §5 · p. 9
6 Conclusion
The paper evaluates UID's interpretations using comprehension and acceptability data, then compares operationalizations and their scope. It finds strongest explanatory support for super-linear surprisal and global, language-level uniformity.
6 Conclusion
- A super-linear function of surprisal best explains the psychometric data, while global operationalizations appear more explanatory than local ones.Source: §6 · p. 9
- The Levy (2005) UID formulation may better predict processing effort in language comprehension than later alternatives.Source: §6 · p. 9
- The results suggest UID may reflect regression toward a language-wide mean information rate, and the acceptability findings support a preference for UID in written language.Source: §6 · p. 9
A Theory
The theory formalizes processing effort as a function of surprisal distribution and utterance length. For convex surprisal costs, uniform distribution minimizes effort, while the optimal length is finite and may be one unit.
A Theory
- For fixed total surprisal and k > 1, processing effort is minimized when surprisal is distributed uniformly across the utterance.The theorem assumes c > 0 and N ≥ 1.Source: A Theory · p. 13
- The proof uses convexity and Jensen’s inequality to establish the uniform-distribution bound for k > 1.The appendix states that the theorem’s minimum is attained for one or two finite choices of N.Source: A Theory · p. 13
- For 0 < k < 1, concavity reverses the distribution result: uniform information density has the highest processing cost.Source: A Theory · p. 13
- The minimizing utterance length is finite, with the optimum found among floor(N⋆) and ceiling(N⋆), or at N = 1 when the first derivative is positive there.The N = 1 case occurs when total surprisal is sufficiently small and/or c is sufficiently large.Source: A Theory · p. 13
- Table 2 reports dataset statistics, defining U as unique counts and M as measured data-point counts.Source: Table 2 · p. 14
- With 0 < k < 1, processing effort is minimized at N = 1 regardless of total surprisal or c.Source: A Theory · p. 14
B Datasets and Language Models
The appendix describes the reading-time and acceptability datasets, preprocessing, language-model surprisal estimates, and the cross-validated metric used to compare predictors.
B Datasets and Language Models
- Text was tokenized and punctuation-normalized, and reading-time outliers were removed before sentence-level analysis.Source: B Datasets and Language Models · p. 14
- Reading-time data come from Natural Stories and Brown self-paced reading, and Provo and Dundee eye movements; acceptability uses CoLA and BNC judgments.Natural Stories and CoLA were designed to cover a broad range of constructions or acceptability cases.Source: B Datasets and Language Models · p. 14
- Word surprisal was estimated with pretrained GPT-2, TransformerXL, and BERT models, plus a WikiText-103 5-gram model.For neural models, per-word surprisal sums the subword surprisals.Source: B Datasets and Language Models · p. 15
- Mean ΔLogLik compares out-of-fold model log-likelihoods across 10 randomly chosen folds, with positive values favoring the comparison model.Source: B Datasets and Language Models · p. 15
C Additional Results
Additional analyses vary the reading-time data, predictors, language dataset, and surprisal estimates. Table 3 reports largely the same trends as Table 1 with different reading-time baselines and BERT estimates for acceptability.
C Additional Results
- The Dutch GECO analysis repeats Figure 3’s graph using Dutch GPT-2 surprisal estimates.Source: Figure 6 · p. 16
- Other analyses show correlations for SLOR and NormLP predictors, use linear terms for unigram log-probability and character length, or substitute probabilities for surprisal.Figure 5 adds SLOR and NormLP correlations; Figures 9 and 11 show the predictor and probability variants.Source: Figure 5 · p. 16, Figure 9 · p. 17, Figure 11 · p. 18
- Table 3 shows largely the same trends as Table 1 using different reading-time baseline predictors and BERT estimates for acceptability.The reading-time models also include unigram log-probability, total character length, and their interaction.Source: Table 3 · p. 16
- Additional figures show reading-time analyses using first-pass times, first-fixation durations, and sentence-level means across subjects.Figures 7, 8, and 10 adapt Figure 3 using these alternative reading-time measures or aggregation.Source: Figure 7 · p. 17, Figure 8 · p. 17, Figure 10 · p. 17
- For acceptability, probability-based predictors have smaller ∆LogLik than surprisal, indicating weaker predictive power.The supplied passage also contrasts this with the experimental findings of Brothers and Kuperberg (2021).Source: C Additional Results · p. 18