PaperlayerView exact revision on arXiv

Source-linked AI summary

The Cognitive Cost of LLM Writing: A Reading-Time Comparison with Human Text

Kamer Cekini, Elisabetta Biondi, Chiara Boldrini, Andrea Passarella, Marco Conti

arXiv:2610.23715v1cs.CL

TL;DR

Whether linguistic differences between human-written and LLM-generated texts are reflected in reading effort remains less explored. The paper develops an eye-tracking-trained predictor and applies it to paired articles and rewrites, finding lower total but higher per-word predicted reading times for LLM rewrites.

  • Problem

    Whether human-written texts and LLM-generated rewrites differ in predicted reading difficulty, and which linguistic differences are associated with those variations, remains underexplored.

    Source: §I · p. 1

  • Method

    The study develops and validates RoBERTa-RT on eye-tracking data, then applies it to 496 paired human-written articles and LLM rewrites of approximately equivalent informational content.

    Source: §VI · p. 9

  • Results

    LLM rewrites averaged 115.6 s total reading time versus 117.1 s (RC = −1.19%), but 233.4 ms per word versus 229.9 ms (RC = +1.57%).

    Source: A. Reading Time and Text Length · p. 7

  • Takeaways & Limitations

    Assessing LLM-generated writing should consider reader-side predicted processing demands as well as textual conciseness.

    Source: §VI · p. 9

Abstract

from arXiv · show

Although linguistic differences between human-written and Large Language Model (LLM)-generated texts have been extensively studied, their implications for the cognitive effort involved in reading remain less explored. In psycholinguistics, reading time is commonly used as an indirect measure of cognitive processing effort. Building on this perspective, we investigate whether human-written texts and their LLM-generated rewrites exhibit differences in predicted reading time that may reflect differences in processing demands for readers. To this end, we introduce RoBERTa-RT, a model trained on human eye-tracking data to predict word-level reading times. We evaluate the model on both internal and external test sets, construct a paired dataset of 496 human-written articles and their LLM-generated rewrites, and apply RoBERTa-RT to compare their predicted reading times. Our findings show that LLM-generated rewrites tend to be shorter and have slightly lower predicted total reading times, while exhibiting higher predicted reading times per word. These differences are associated with systematic linguistic changes, including longer and less frequent words, greater lexical diversity, and higher contextual predictability. From a cognitive perspective, our findings suggest that LLM-generated writing may redistribute reading demands: reducing the amount of text to process while increasing the predicted processing effort associated with individual words. This highlights the importance of considering not only the linguistic characteristics of machine-generated content, but also its potential implications for human cognitive processing.

I. INTRODUCTION

Although human and LLM texts have been compared linguistically, whether their differences affect reading effort remains underexplored. The study uses RoBERTa-RT and paired source-rewrite texts to compare predicted reading times, finding shorter rewrites with lower total but higher per-word reading times.

I. INTRODUCTION

  • Reading time provides an indirect measure of cognitive effort, and linguistic properties can affect processing even when texts convey approximately the same information.The study asks whether human texts and their LLM rewrites differ in reading difficulty and which linguistic differences are associated with that variation.Source: §I · p. 1
  • RoBERTa-RT is trained on human eye-tracking data to predict word-level reading times and is evaluated on internal and external test data.The model enables large-scale comparisons without collecting new eye-tracking data for each document collection.Source: §I · pp. 1–2
  • The paired dataset contains 496 human-written articles and LLM rewrites intended to preserve each source’s informational content while allowing linguistic reformulation.The design reduces differences attributable to topic and content while comparing predicted reading times and linguistic properties.Source: §I · p. 2
  • LLM-generated rewrites tend to be shorter and have slightly lower predicted total reading times, yet higher predicted reading times per word.They also tend to use longer, less frequent words and show greater lexical diversity and higher contextual predictability.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

Prior work links reading times to lexical and contextual properties and includes computational prediction from eye-tracking data, but does not establish whether human–LLM linguistic differences translate into reading-effort differences. The paper introduces RoBERTa-RT and describes the eye-tracking data used to develop and evaluate it.

II. BACKGROUND AND RELATED WORK

  • Earlier studies found linguistic differences between human and LLM texts but did not establish whether these translate into differences in reading effort; this study compares predicted times for paired texts of approximately equivalent content.Reading-time differences between human texts and LLM rewrites had not, to the authors’ knowledge, been systematically investigated.Source: §II · p. 3

III. READING-TIME PREDICTION MODEL: ROBERTA-RT

  • RoBERTa-RT is a RoBERTa-based model trained to predict word-level reading times from text, with evaluation on internal and external test data.It addresses the need to estimate reading times for texts without direct eye-tracking observations.Source: §III · p. 3

A. Eye-Tracking Datasets

  • Eye-tracking fixations are mapped to words, and their durations are summed to estimate time spent reading each word occurrence.The OneStop corpus supplies newspaper reading data; GECO and MECO add English-reader data, while Provo is reserved for external testing.Source: A. Eye-Tracking Datasets · p. 3
  • The evaluation uses held-out OneStop articles for internal testing and Provo for external testing.Training combines OneStop, GECO, and MECO data, with article-level train, validation, and test partitions for OneStop.Source: A. Eye-Tracking Datasets · p. 3

B. Prediction Target

The prediction target is the mean observed reading time for each word occurrence across readers. Treating occurrences separately preserves contextual distinctions, and the model produces aligned observed and predicted sequences for each text.

B. Prediction Target

  • The prediction target is the mean reading time recorded across participants for each word occurrence in a text.The resulting value represents the average observed reading time for that occurrence.Source: B. Prediction Target · pp. 3–4
  • The target is defined for word occurrences rather than word types, so the same word in different positions or contexts is treated as a distinct observation.This preserves information about the linguistic context in which each occurrence appears.Source: B. Prediction Target · p. 4
  • For each text, the model yields aligned observed and predicted reading-time sequences with one value per word occurrence.The predicted value estimates the corresponding occurrence-level target.Source: B. Prediction Target · p. 4

C. Models and Training

The study compares feature-based regressors with Transformers for word-level reading-time prediction, selecting models by text-level OneStop validation MAE. RoBERTa-base is selected as RoBERTa-RT and evaluated on held-out OneStop and Provo data.

C. Models and Training

  • Feature-based models use 15 word and context features, while Transformers encode paragraphs, mean-pool subwords by word, and predict each occurrence with an MLP regression head.Neural models minimize MSE between observed and predicted reading times; candidate models are trained on combined OneStop, GECO, and MECO data.Source: C. Models and Training · p. 4

D. Model Selection and Evaluation

  • Candidate configurations are tuned with 50 Optuna trials to minimize OneStop validation MAE, while model selection compares text-level performance.The selected configurations are the best for each model on the OneStop validation set.Source: D. Model Selection and Evaluation · p. 4
  • 15.76 ms/word: RoBERTa-base achieved the lowest OneStop validation MAE and was selected as RoBERTa-RT.Model selection used OneStop validation performance; the test sets were reserved for post-selection evaluation.Source: D. Model Selection and Evaluation · p. 5, Table IV · p. 5
  • 15.00 ms/word on OneStop Test and 16.09 ms/word on Provo: RoBERTa-RT’s held-out MAEs were small relative to observed mean reading times.The reported MAEs were approximately 7.8% and 8.0% of observed mean reading time, respectively; comparable Provo performance supports generalization beyond development data, but not precise estimates for individual texts.Source: D. Model Selection and Evaluation · p. 5

IV. HUMAN–LLM COMPARATIVE ANALYSIS

RoBERTa-RT is used to compare predicted reading effort and linguistic characteristics in paired Human–LLM texts. The paired corpus is intended to make the comparison more controlled by matching each generated text to its source article.

IV. HUMAN–LLM COMPARATIVE ANALYSIS

  • RoBERTa-RT compares predicted reading times and linguistic properties in paired source articles and LLM rewrites, reducing topic and content differences across documents.The analysis compares alternative linguistic realizations intended to retain approximately the same informational content.Source: §IV · p. 5

A. Human–LLM Paired Dataset

The paired dataset matches each human-written article with an LLM rewrite intended to preserve its substantive content while allowing changes in wording, structure, and organization. The final corpus contains 496 pairs after four rewriting requests were refused and excluded.

A. Human–LLM Paired Dataset

  • Each source article is matched with an LLM rewrite to reduce topic and content differences in within-pair comparisons.The paired design supports comparisons of reading-time estimates and linguistic properties across alternative realizations of approximately the same information.Source: A. Human–LLM Paired Dataset · p. 5
  • The source sample contains 500 CCAT50 news articles, with 10 selected from each of its 50 authors.CCAT50 contains 5,000 articles, with 100 documents per author.Source: A. Human–LLM Paired Dataset · p. 6
  • GPT-5.6 Terra rewrites each article under a prompt requiring the same facts and attributed statements without outside information or omitted substantive content.The prompt leaves lexical choices, sentence structure, and paragraphing open to change while intending to keep substantive content comparable.Source: A. Human–LLM Paired Dataset · p. 6
  • 496 Human–LLM pairs remain after four of 500 rewriting requests were refused because of content identified as sensitive and excluded.Generated texts were used as returned, without manual revision or post-processing.Source: A. Human–LLM Paired Dataset · p. 6

B. Reading-Time and Linguistic Measures

The paired corpus is used to compare predicted reading effort and linguistic properties across human-written texts and LLM rewrites. RoBERTa-RT word-level predictions are aggregated into text-level reading-time measures, alongside measures of length and linguistic characteristics.

B. Reading-Time and Linguistic Measures

  • RoBERTa-RT predicts reading times for word occurrences, which are combined into predicted total reading time and predicted mean reading time per word.Articles are processed paragraph by paragraph to fit RoBERTa’s 512-token input limit.Source: B. Reading-Time and Linguistic Measures · p. 6
  • Total reading time sums word-level predictions, while mean reading time per word averages them to compare per-word processing independently of text length.The measures capture different aspects of reading effort, especially when paired texts differ in length.Source: B. Reading-Time and Linguistic Measures · p. 6
  • Mean word length is average characters per word, and mean word frequency averages each word’s estimated relative frequency in English.Longer and less frequent words generally require more processing time.Source: B. Reading-Time and Linguistic Measures · p. 6
  • Surprisal measures how predictable each word is from preceding context; OPT-125M estimates token probabilities, which are averaged across an article.Higher mean surprisal indicates lower contextual predictability, and less predictable words generally take longer to process.Source: B. Reading-Time and Linguistic Measures · pp. 6–7
  • Word count measures article length, while MTLD measures lexical diversity and is less sensitive to text length than type–token ratio.Higher MTLD indicates greater diversity and lower MTLD greater repetition.Source: B. Reading-Time and Linguistic Measures · p. 7

C. Statistical Analysis

The analysis compares each LLM rewrite only with its corresponding human-written article. It tests paired metric differences and examines whether relative changes across metrics move together.

C. Statistical Analysis

  • Each of the 496 LLM-generated texts is compared only with its corresponding human-written source in pair-level analyses.The design preserves the link between each rewrite and its source article.Source: C. Statistical Analysis · p. 7
  • Two-sided Wilcoxon signed-rank tests assess paired differences across seven metrics, with p-values adjusted using the Bonferroni correction.The null hypothesis is that there is no systematic shift between paired observations.Source: C. Statistical Analysis · p. 7
  • Mean relative change across 496 pairs summarizes each metric’s overall magnitude and direction relative to human-written texts.Positive values indicate increases in LLM-generated texts; negative values indicate decreases.Source: C. Statistical Analysis · p. 7
  • Spearman correlations across pair-level relative changes assess whether metric changes move together or in opposite directions; nine tests receive joint Bonferroni adjustment.Positive correlation indicates changes tend to share a direction, while negative correlation indicates opposing directions.Source: C. Statistical Analysis · p. 7

V. RESULTS

Across the paired texts, LLM rewrites have slightly lower predicted total reading time but higher predicted reading time per word. Their shorter length offsets the per-word increase, and all seven examined differences remain significant after correction.

V. RESULTS

  • All seven Human–LLM differences examined remain statistically significant after multiple-comparison correction (padj < .01).The reported Spearman correlations also remain significant after correction (padj < .01).Source: §V · p. 7

A. Reading Time and Text Length

  • 115.6 s versus 117.1 s (RC = −1.19%) for predicted total reading time, while predicted mean reading time per word is 233.4 ms versus 229.9 ms (RC = +1.57%).The first value in each comparison is for LLM-generated texts and the second for human-written texts.Source: A. Reading Time and Text Length · p. 7
  • 496.4 versus 511.2 words (RC = −2.66%): shorter LLM rewrites offset their higher predicted reading time per word, leaving a modest total-time reduction.The results caution against equating textual conciseness with lower processing effort per word.Source: A. Reading Time and Text Length · pp. 7–8

B. Linguistic Differences and Predicted Reading Time

LLM-generated texts differ from human-written texts across four linguistic measures. Longer and less frequent words and greater lexical diversity are associated with higher predicted reading time per word, while contextual predictability is associated with smaller increases.

B. Linguistic Differences and Predicted Reading Time

  • Mean word length rises from 4.95 to 5.25 characters (RC = +6.06%), mean word frequency decreases by 7.61%, MTLD rises by +31.83%, and mean surprisal decreases by 6.51%.The lower surprisal indicates that LLM-generated words are more predictable from preceding context.Source: B. Linguistic Differences and… · p. 8
  • Changes in word length (ρ = .588) and lexical diversity (ρ = .292) correlate positively with per-word reading-time changes, while word frequency correlates negatively (ρ = −.405).Greater decreases in frequency tend to accompany greater increases in predicted reading time per word.Source: B. Linguistic Differences and… · p. 8
  • Surprisal changes correlate positively with per-word reading-time changes (ρ = .438), so larger reductions in surprisal tend to accompany smaller increases in predicted per-word time.Overall, the rewrites still have higher predicted mean reading time per word despite this opposing association.Source: B. Linguistic Differences and… · p. 8

C. The Compression–Reading Cost Trade-off

Across Human–LLM pairs, greater text compression tends to accompany higher predicted reading time per word and shifts in linguistic characteristics. The pattern suggests a compression–reading cost trade-off, rather than a uniform change in reading difficulty.

C. The Compression–Reading Cost Trade-off

  • ρ = −.458 links relative word-count change with change in predicted mean reading time per word, with greater shortening tending to accompany larger per-word increases.The table covers correlations across Human–LLM pairs (N = 496).Source: C. The Compression–Reading Cost… · p. 8, Table VIII · p. 9
  • ρ = −.303 for mean word length, ρ = −.269 for lexical diversity, and ρ = .364 for mean word frequency link greater compression with longer, more diverse, less frequent vocabulary.The signs reflect correlations between relative word-count change and the corresponding linguistic changes.Source: C. The Compression–Reading Cost… · p. 8
  • ρ = −.362 links relative word-count change with surprisal change; greater compression tends to accompany a smaller reduction in surprisal.LLM rewrites generally have lower surprisal than their human-written counterparts, but this difference becomes less pronounced with greater compression.Source: C. The Compression–Reading Cost… · p. 8
  • Greater reductions in text length tend to coincide with higher predicted reading time per word and shifts toward longer, less frequent, more diverse words and a smaller surprisal reduction.Together, these associations characterize the compression–reading cost trade-off.Source: C. The Compression–Reading Cost… · pp. 8–9

VI. CONCLUSION

The conclusion reports that LLM rewrites tend to be shorter and have slightly lower total predicted reading times, but higher average predicted reading times per word. The authors connect these patterns to linguistic characteristics and caution that human-participant studies are needed to establish effects on cognitive effort and comprehension.

VI. CONCLUSION

  • RoBERTa-RT predicts word-level reading times from human eye-tracking data, and the authors apply it to 496 paired articles and rewrites preserving informational content as closely as possible.The paired design supports comparison across different linguistic realizations of similar content while reducing the influence of differences in conveyed information.Source: §VI · p. 9
  • LLM rewrites tend to be more concise and have slightly lower total reading times, while exhibiting higher average reading times per word.More extensive shortening tends to accompany a larger increase in per-word reading time.Source: §VI · p. 9
  • Longer, less frequent words and greater lexical diversity are associated with higher per-word reading times, while greater contextual predictability is associated with lower times.Despite greater predictability, LLM texts still exhibit higher average predicted reading times per word.Source: §VI · p. 9
  • The findings suggest that machine-generated writing should be evaluated from the reader’s perspective as well as by content-production efficiency.The authors argue that textual conciseness alone does not capture the cognitive implications of LLM-generated writing.Source: §VI · p. 9
  • The analysis uses model-predicted rather than directly measured reading times, and human-participant studies are needed to establish whether differences translate into measurable changes in cognitive effort and comprehension.Source: §VI · pp. 9–10
Loading 2610.23715v1…