Source-linked AI summary
LLM-based Text Simplification and its Effect on User Comprehension and Cognitive Load
Theo Guidroz, Diego Ardila, Jimmy Li, Adam Mansour, Paul Jhun, Nina Gonzalez, Xiang Ji, Mike Sanchez, Sujay Kakarmath, Mathias MJ Bellaiche, Miguel Ángel Garrido, Faruk Ahmed, Divyansh Choudhary, Jay Hartford, Chenwei Xu, Henry Javier Serrano Echeverria, Yifan Wang, Jeff Shaffer, Eric, Cao, Yossi Matias, Avinatan Hassidim, Dale R Webster, Yun Liu, Sho Fujiwara, Peggy Bui, Quang Duong
TL;DR
Complex web information often exceeds general readers’ comprehension levels, so this paper develops minimally lossy LLM text simplification and evaluates it in a randomized study. Simplification improved MCQ accuracy by 3.9% overall and increased perceived ease.
Problem
Much expert information available online is written above general audiences’ reading levels, limiting its comprehensibility.
Source: Introduction · p. 2Method
The study used Gemini-based automated readability and fidelity evaluation with iterative prompt refinement, then randomized participants to read original or simplified texts across six domains.
Source: Introduction · p. 2, Autoeval system · p. 4, Statistical analysis · p. 8Results
3.9% higher MCQ accuracy was observed for simplified versus original texts, with gains robust when participants could not refer back to the text.
Source: Statistical analysis · p. 7Takeaways & Limitations
LLM text simplification can improve comprehension and make complex technical information feel easier to understand across several subject areas.
Source: Statistical analysis · p. 8The study used a broad, non-expert survey population whose incentives may have encouraged rapid responses, so effects for motivated users require further study.
Source: Statistical analysis · p. 9
Abstract
from arXiv · showhide
Information on the web, such as scientific publications and Wikipedia, often surpasses users' reading level. To help address this, we used a self-refinement approach to develop a LLM capability for minimally lossy text simplification. To validate our approach, we conducted a randomized study involving 4563 participants and 31 texts spanning 6 broad subject areas: PubMed (biomedical scientific articles), biology, law, finance, literature/philosophy, and aerospace/computer science. Participants were randomized to viewing original or simplified texts in a subject area, and answered multiple-choice questions (MCQs) that tested their comprehension of the text. The participants were also asked to provide qualitative feedback such as task difficulty. Our results indicate that participants who read the simplified text answered more MCQs correctly than their counterparts who read the original text (3.9% absolute increase, p<0.05). This gain was most striking with PubMed (14.6%), while more moderate gains were observed for finance (5.5%), aerospace/computer science (3.8%) domains, and legal (3.5%). Notably, the results were robust to whether participants could refer back to the text while answering MCQs. The absolute accuracy decreased by up to ~9% for both original and simplified setups where participants could not refer back to the text, but the ~4% overall improvement persisted. Finally, participants' self-reported perceived ease based on a simplified NASA Task Load Index was greater for those who read the simplified text (absolute change on a 5-point scale 0.33, p<0.05). This randomized study, involving an order of magnitude more participants than prior works, demonstrates the potential of LLMs to make complex information easier to understand. Our work aims to enable a broader audience to better learn and make use of expert knowledge available on the web, improving information accessibility.
Introduction
The study addresses the gap between widely available expert information and general readers’ comprehension by developing minimally lossy LLM-based text simplification and validating it in a randomized study. It evaluates readability, information preservation, comprehension, confidence, and cognitive workload across multiple expert domains.
Motivation
- Expert information often exceeds broad audiences’ reading levels, creating risks of misunderstanding, reduced engagement, and poorer informed decision-making, especially in healthcare, law, and finance.Over half of U.S. adults reportedly read below a sixth-grade level, while complex health information often surpasses that threshold.Source: Introduction · p. 2
- Traditional readability metrics are limited because they emphasize surface features while missing semantic, contextual, and domain-specific sources of difficulty.Measures such as Flesch-Kincaid and SMOG may not adequately capture jargon or technical terminology in expert discourse.Source: Introduction · p. 2
Study contribution
- LLM-based simplification was evaluated through automated readability and faithfulness measures, iterative prompt refinement, and a randomized comparison of original versus simplified texts.Gemini was used to refine prompts and few-shot examples, while participants answered comprehension questions and reported confidence and cognitive workload.Source: Introduction · p. 2
- Across PubMed, biology, law, finance, literature/philosophy, and aerospace/computer science, simplified-text readers achieved higher MCQ accuracy, greater confidence, and substantially greater reported task ease.The randomized study compared participants reading original or simplified texts and collected comprehension and subjective workload outcomes.Source: Introduction · p. 2
Related work
Prior work developed diverse text-simplification methods and evaluation metrics, but randomized evidence of comprehension benefits remained limited. This study addressed that gap with an automated minimally lossy system and a large randomized evaluation across varied web domains.
Related work
- Earlier simplification research combined syntactic or lexical rewriting with sequence-to-sequence models, external knowledge, and user-centered interfaces for accessibility.Applications included language disabilities, language learners, non-native speakers, and small-screen users; RALL and Paper Plain addressed context and navigation needs.Source: Related work · p. 3
Autoeval system
- Autoeval models separately scored readability and fidelity, using iterative prompt improvement, claim mapping, entailment, and weighted penalties for information loss, gain, and distortion.Readability was rated from 1–10, while fidelity errors included missing information, unfactual additions, and factuality or fidelity losses.Source: Autoeval system · p. 4
Prompt refinement system
- The system iteratively refined a Gemini 1.5 Flash simplification prompt by maximizing average readability minus average autoevaluated error across a dataset of 123 texts.Gemini 1.5 Pro generated prompt improvements, and the final prompt was manually selected after the iteration loop.Source: Related work · pp. 3–4, Prompt refinement system · p. 4
Study dataset
- The study used 31 excerpts across six subject areas, with two researcher-reviewed MCQs per text, comparing original, simplified, or both versions in open- and closed-book arms.Topics included PubMed, biology, law, finance, literature/philosophy, and aerospace/computer science.Source: Study dataset · p. 5
Participant recruitment and sample size
- 4,563 participants were randomly assigned across six arms and topic areas, with recruitment targeting approximate US Census age distributions and equal man-to-woman representation.The design prioritized participant responses over increasing the number of texts or questions.Source: Participant recruitment and sample size · pp. 5–6
Participant flow
- Each text was followed by two five-option comprehension MCQs, confidence ratings, and a five-point cognitive-load question; a simple attention check appeared at the end of the survey.Confidence and cognitive-load responses were mapped to integers in [-2, 2], and participants providing nonsensical attention-check answers were discarded.Source: Participant flow · p. 6
Pilot studies
- Pilot studies shortened texts, rejected unreliable free-text comprehension responses, and identified prior expertise as a potential confound, motivating the final controlled design.The pilots suggested that long open-book texts could shift the task from comprehension toward information retrieval.Source: Pilot studies · p. 6
Statistical analysis
- 3.9% (44.3% to 48.2%, p<0.05) higher MCQ accuracy followed simplified texts, with the largest gain in PubMed (14.6%); benefits persisted closed-book.Accuracy gains were more modest in finance, aerospace/computer science, and legal domains.Source: Statistical analysis · pp. 7–8
Figures & Tables
The figures and tables document the Gemini-based minimally lossy simplification pipeline, its randomized evaluation design, and comparisons of comprehension, confidence, and perceived ease between original and simplified texts.
Simplification approach
- Figure 1 depicts iterative prompt improvement using readability and information-fidelity autoevaluations to avoid adding or losing information.The evaluated models are Gemini-based, with fidelity assessed through completeness and entailment.Source: Figure 1 · p. 10
Study design
- Figure 2 shows the randomized study design across six topic areas, including original and simplified text exposure, MCQs, confidence ratings, and simplified NASA Task Load Index ratings.Participants were filtered for topic expertise and randomized to six study arms.Source: Figure 2 · p. 11
Study results
- Figure 3 compares MCQ-accuracy changes between simplified and original texts across six topics under open-book and closed-book conditions.Positive values indicate improved accuracy with simplified texts; overall and per-topic changes are shown.Source: Figure 3 · p. 12
- Figures 4 and 5 compare changes in per-question confidence and per-text perceived ease when participants viewed simplified rather than original texts.Positive values represent improvements with simplified texts.Source: Figure 4 · p. 13, Figure 5 · p. 14
- Figure 6 examines per-question accuracy across all questions and PubMed, separately for open-book and closed-book conditions, including the relationship between baseline accuracy and improvement.Accuracy is higher with simplification for points above the diagonal, and the fitted-line rotation indicates larger gains for questions with lower original-text accuracy.Source: Figure 6 · p. 15
Study materials
- Tables 1–3 characterize the text categories and participants and provide three PubMed examples with the greatest MCQ-accuracy improvements from simplification.The examples pair original and simplified excerpts with their associated questions.Source: Table 1 · p. 16, Table 2 · p. 17, Table 3 · p. 18