Source-linked AI summary
Keep it Simple: Unsupervised Simplification of Multi-Paragraph Text
Philippe Laban, Tobias Schnabel, Paul Bennett, Marti A. Hearst
TL;DR
Text simplification should improve readability while preserving relevant content, but supervised approaches rely on aligned data. KiS balances fluency, salience, and simplicity with reference-free reward training and k-SCST; on English news it outperforms strong supervised models and its simplified texts yield faster comprehension-task completion.
Problem
Text simplification aims to make complex text more readable while ideally preserving relevant content, but high-quality large public datasets for training are unavailable.
Source: §1 · p. 1Method
KiS optimizes a reference-free reward balancing fluency, salience, and simplicity, using k-SCST to compare sampled candidates against their mean reward.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The proposed comprehension task measures simplification usefulness through how efficiently readers answer questions about texts.
Source: §1 · p. 2Experiments were limited to simplifying English news articles.
Source: §6 · p. 9
Abstract
from arXiv · showhide
This work presents Keep it Simple (KiS), a new approach to unsupervised text simplification which learns to balance a reward across three properties: fluency, salience and simplicity. We train the model with a novel algorithm to optimize the reward (k-SCST), in which the model proposes several candidate simplifications, computes each candidate's reward, and encourages candidates that outperform the mean reward. Finally, we propose a realistic text comprehension task as an evaluation method for text simplification. When tested on the English news domain, the KiS model outperforms strong supervised baselines by more than 4 SARI points, and can help people complete a comprehension task an average of 18% faster while retaining accuracy, when compared to the original text. Code available: https://github.com/tingofurro/keep_it_simple
1 Introduction
KiS is an unsupervised approach to paragraph-level text simplification that balances fluency, salience, and simplicity. It introduces reference-free reward optimization, k-SCST training, and a human comprehension task; in English news, it outperforms strong supervised models on automatic metrics.
1 Introduction
- KiS trains paragraph-level simplifications to balance fluency, salience, and simplicity, ideally preserving relevant content while making text more readable.The paper contrasts simplification with summarization, which selects key content and elides the rest.Source: §1 · pp. 1–2
- A reference-free reward lets KiS optimize simplification directly without aligned data, addressing the lack of publicly available high-quality, large simplification datasets.The method is built on reinforcement-learning-based training of text generators.Source: §1 · pp. 1–2
- k-SCST extends Self-Critical Sequence Training by sampling k outputs per input and using their mean population reward as the baseline.The paper reports that k-SCST outperforms SCST in its domain.Source: §1 · p. 2
- The proposed human comprehension task tests whether readers can complete questions faster on KiS-simplified texts than on comparison texts.The evaluation is based on the assumption that simplified text should enable faster reading with better understanding.Source: §1 · p. 2
- KiS is evaluated on short documents of 3–4 paragraphs, reflecting the paper’s focus on paragraph-level simplification rather than the usual sentence-level unit.The authors note that simplification phenomena can include deleting, inserting, or reordering full sentences.Source: §1 · p. 2
- More than 4 SARI points: KiS outperforms strong supervised models on automated simplification metrics.The comparison is reported for the English news domain.Source: §1 · p. 2
2 Related Work
Prior simplification work spans aligned datasets, sequence-to-sequence and text-edit models, lexical and unsupervised methods, and automatic or human evaluation. KiS builds on Newsela data and uses reference-free reinforcement learning to avoid requiring aligned simplifications for training.
2 Related Work
- Newsela is treated as a gold standard: its public release contains 1,911 article groups, from which coarse paragraph alignment yields 40,000 paired paragraphs separated by four grade levels.The paired Newsela dataset is used for analysis and baseline training.Source: §2 · p. 2
- Sequence-to-sequence simplification includes ACCESS, which can customize compression and paraphrasing, while data availability remains a limitation for supervised systems.The paper directly compares KiS with ACCESS.Source: §2 · p. 2
- Lexical simplification methods replace words or phrases but tend not to model syntactic complexity or directly handle deletions and insertions.KiS incorporates a lexical score as part of its simplicity component.Source: §2 · p. 2
- Text-edit approaches model sequences of edits and offer explainability at the cost of added complexity; KiS can learn to copy or deviate without explicitly representing edits.KiS outputs can be post-processed into edits if desired.Source: §2 · p. 2
- Unsupervised simplification had mostly focused on lexical changes, though Unsup NTS also performs syntactic simplification with separate simple and complex decoders.The paper directly compares KiS with Unsup NTS.Source: §2 · pp. 2–3
- Earlier reinforcement-learning simplification systems required aligned reference simplifications, whereas KiS uses a reference-free reward to train without supervision.Automatic evaluation commonly uses BLEU and SARI, while the paper proposes an extrinsic comprehension task that requires resubmission after incorrect answers.Source: §2 · p. 3
3 KiS Components
KiS optimizes a combined reward for simplicity, fluency, salience, and guardrails. Its components measure different aspects of simplification and constrain generated paragraphs against common failures.
3 KiS Components
- The total reward multiplies simplicity, fluency, salience, and guardrails, so a triggered guardrail zeroes the reward.The components are normalized to [0, 1].Source: §3 · p. 3, Figure 2 · p. 3
3.1 Simplicity
- Simplicity combines syntactic SScore and lexical LScore, each ranging from 0 to 1.Syntactic scoring targets less complex sentence structure; lexical scoring favors simpler synonyms.Source: §3.1 · p. 3
3.1.1 Syntactic Simplicity: SScore
- SScore measures closeness to a target FKGL drop, which increases with the original paragraph’s FKGL rather than remaining fixed.The score is a clipped linear ramp; Figure 4 shows the positive relationship used to fit the target drop.Source: §3.1.1 · p. 4, Figure 4 · p. 4
3.1.2 Lexical Simplicity: LScore
- LScore uses word-frequency differences between inserted and deleted words; 91% of paired Newsela paragraphs have a positive shift, with median 0.4.The median shift is the target for a clipped ramp score; word frequency is log-normalized on a 0–8 scale.Source: §3.1.2 · p. 4
3.2 Fluency
- Fluency combines a language-model score comparing output and original likelihoods with a dynamically trained discriminator that scores authenticity.The static score declines linearly when output likelihood falls below the original and reaches zero at a drop of λ; the discriminator adapts during training.Source: §3.2.1 · p. 4, §3.2.2 · p. 5
3.3 Salience
- The salience score uses a Transformer coverage model that answers masked-word questions about the original; simplification masks all non-stop words.This reflects the expectation that most of the original paragraph should be recoverable; Newsela reference simplifications average 0.76 coverage.Source: §3.3 · p. 5
3.4 Guardrails
- Binary guardrails reject paragraphs outside a length ratio of 0.6–1.5 or containing an output entity absent from the original.The brevity range avoids triggering on 98% of paired Newsela data. The entity heuristic can flag accurate paraphrases that introduce new entities.Source: §3.4.2 · p. 5, §3.4.1 · p. 5, §3.4 · p. 5
4 KiS Training
KiS training uses k-SCST, which compares multiple sampled simplifications against their mean reward. In the reported setting, increasing k raised average reward and reduced variation, with k = 8 used for final models.
4 KiS Training
- SCST samples one candidate and generates another by taking the most probable word at each step, then compares their rewards in its loss.The loss increases the sampled sequence’s likelihood when its reward exceeds the greedy candidate’s and decreases it otherwise.Source: §4 · p. 6
- In SCST experiments, comparable rewards for the two candidates occurred in 30% of samples, leaving the model little to learn from those examples.When rewards are comparable, the loss nears zero and the training sample is wasted.Source: §4 · p. 6
- k-SCST samples k candidates and uses their mean reward as the baseline instead of SCST’s greedy candidate reward.The loss encourages sampled candidates whose rewards exceed the population mean.Source: §4 · p. 6
- Increasing k raised average reward and reduced variation; the authors used k = 8 for final models because larger values were impractical with GPU memory limits.The comparison used six independent runs for each setting and plotted average reward with standard error of the mean.Source: §4 · p. 6
- The authors attribute k-SCST’s advantage to better reward-distribution estimates from more samples and avoiding a separate baseline generation.They propose investigating its use in other text-generation applications.Source: §4 · p. 6
5 Experiments
The experiments compare KiS with supervised and unsupervised systems using automatic metrics, a human comprehension task, and reward-component ablations. Across these evaluations, KiS achieves the highest SARI score, speeds comprehension relative to original texts, and each reward component contributes to evaluation performance.
5 Experiments
- The experiments evaluate KiS with automatic metrics, a human comprehension task, and an ablation study against three supervised models and an unsupervised approach.The comparison includes ACCESS, ACCESS90, a GPT2-medium model finetuned on Newsela, and Unsup NTS.Source: §5 · p. 6, §5.1 · pp. 6–7
5.2 Automatic Results
- KiS achieves the highest SARI score by a margin of 0.04, despite being unsupervised.Finetune Baseline has the highest BLEU and salience scores but the lowest SARI; Newsela references outperform KiS on Lexile improvement and coverage.Source: §5.2 · p. 7, Table 1 · p. 7
- All models improve readability over original paragraphs on FKGL and Lexile, while KiS lowers Lexile grade level in 72% of outputs, close to Newsela’s 79%.The authors note that FKGL is part of the optimized simplicity score, whereas Lexile is not.Source: §5.2 · p. 7
- KiS compresses the second most, which may hurt coverage; ACCESS90’s compression rate is closest to Newsela’s but yields only a modest SARI improvement over ACCESS.The authors suggest adjusting the Brevity guardrail to encourage less compression.Source: §5.2 · p. 7, Table 1 · p. 7
5.3.1 Study Design
- The study compared five versions—the original, Newsela, KiS, Finetune Baseline, and ACCESS—of 3–4-paragraph (or 200-word) documents, each paired with five manually generated comprehension questions.Participants received feedback on incorrect answers and had to resubmit until all answers were correct; 244 submissions were used for result analysis.Source: §5.3.1 · pp. 7–8, §5.3.2 · p. 8
5.3.2 Study Results
- Participants completed the comprehension task 18% faster with KiS than with original texts, and KiS significantly outperformed originals, Newsela references, and ACCESS on completion time.Newsela achieved the best compression-adjusted speed-up (CASpeed), followed by KiS; this suggests that good simplification can involve making texts longer.Source: Table 2 · p. 8, §5.3.2 · p. 9
5.4 Ablation Study
- Removing fluency can yield incomplete sentences, removing salience can omit important information, and removing simplicity can produce more complex text.Each ablation decreased an evaluation metric, supporting the contribution of all three reward components to high-quality simplifications.Source: §5.4 · p. 9
6 Limitations and Future Work
The paper identifies limitations in factual accuracy, human-evaluation reproducibility, and experimental scope, while outlining supervised training and broader applications as future directions.
Improved Accuracy Scoring
- The current inaccuracy guardrail is rudimentary, and trained models still produce non-factual simplifications.The authors identify adapting fact-checking methods from summarization as a possible improvement.Source: §6 · p. 9
Reproducibility of Human Evaluation
- Human-evaluation results may vary because the procedure includes random elements and crowd-workers differ in literacy levels.The study releases its models, stimuli, and comprehension questions.Source: §6 · p. 9
New Settings, Domains and Languages
- Experiments are limited to English news simplification, although the authors plan to explore other languages and cite legal texts as a possible new domain.They also suggest applying the unlabeled-data approach to new settings, including reversing the effects of simplification.Source: §6 · p. 9
7 Conclusion
The conclusion presents KiS as an unsupervised simplification method that outperforms strong supervised models on automatic metrics and speeds comprehension-task completion. The authors limit demonstrated results to English news and identify new domains, languages, and factuality-aware rewards as future work.
7 Conclusion
- +0.04 in SARI: KiS outperforms strong supervised models on automatic metrics.The authors also report that KiS texts produce the best comprehension-task speed-up, averaging 18%.Source: §7 · p. 9
- 18% average speed-up: KiS texts produce the best reported comprehension-task speed-up.The authors report that simplifications tend to speed task completion measurably.Source: §7 · p. 9
- Future work should adapt the methodology to new domains and non-English languages and refine rewards to account for factuality.The authors describe these results as first steps for unsupervised text simplification.Source: §7 · p. 9
Ethical Considerations
The paper reports evaluation on English news and limits its claims to that domain because adaptation to other domains and languages was not experimentally verified. The human evaluation used reviewed protocols, avoided sensitive or political topics, and did not collect personally identifiable information.
Ethical Considerations
- The method's performance was evaluated on English news; adaptation to other domains and languages was not experimentally verified.The authors therefore limit their claims to the English news domain.Source: Ethical Considerations · p. 12
- Prior-work implementations were obtained from their authors and evaluated using recommended settings to provide strong comparison points.The authors give ACCESS as an example.Source: Ethical Considerations · p. 12
- Human-evaluation topics avoided sensitive or political subjects, and the university IRB reviewed the protocols.Protocol ID: 2018-07-11230.Source: Ethical Considerations · p. 12
- The study did not collect personally identifiable information and paid annotators above minimum wage.Amazon Mechanical Turk was used to remunerate crowd-workers.Source: Ethical Considerations · p. 12
A.1 Training Details
KiS training used a large unreleased English-news corpus and k-SCST, while the paper also reports implementation details for its fine-tuning baseline and discriminator.
A.1 Training Details
- Training used PyTorch, Hugging Face Transformers, and Apex for half-precision; KiS ran on one GPU.The GPUs were an Nvidia V-100 or Quadro RTX 8000; the authors report around 200 experiments averaging one week each.Source: §A.1 · p. 12
- KiS was trained unsupervised on an unreleased corpus of 7 million English news articles.Source: §A.1 · p. 12
- KiS used GPT2-medium initialization, Adam at 10^-6, batch size 1, and k-SCST with k = 8.Source: §A.1 · p. 12
- The fine-tuning baseline used teacher forcing on 40,000 paired Newsela samples, with 2,000 held out for validation.It used a learning rate of 10^-5 and batch size 8, and plateaued after 3 epochs.Source: §A.1 · p. 12
- The discriminator used Roberta-base and was reset whenever its training buffer reached 2,000 samples.It was trained with cross-entropy for 5 epochs per round; the validation-best checkpoint was retained.Source: §A.1 · p. 12
A.2 Human Evaluation Instructions
The appendix documents the human-evaluation instructions and stimuli, timing and participant counts, and ablation results. It also describes how omitting fluency can inflate automatic scores despite producing incomplete sentences.
A.2 Human Evaluation Instructions
- Figure A1 presents the instructions given to crowd-worker participants in the manual evaluation.Participants were recruited on Amazon Mechanical Turk, where jobs are called HITs.Source: §A.2 · p. 12, Figure A1 · p. 12
A.3 Full Example of Generated Texts
- Figure A2 shows the five stimuli for the Covid Libraries document, complementing Figure 6.Participants saw the original, Newsela, KiS, Finetune Baseline, or ACCESS version and answered five comprehension questions.Source: §A.3 · p. 12, Figure A2 · p. 13
A.4 Detailed of Human Evaluation Results
- Table A1 reports average completion time and participant counts for each document/stimulus combination, plus aggregate mean time and total participants.Source: §A.4 · p. 12, Table A1 · p. 13
A.5 Detail of Ablation Study Results
- Table A2 reports automatic results for three ablation models; SARI and BLEU are reference-based metrics.It also defines lower-FKGL and lower-Lexile percentages, average compression ratio, and average coverage score.Source: §A.5 · p. 14, Table A2 · p. 14
- Ablation examples in Figure 1 illustrate limitations associated with removing score components.Source: §A.5 · p. 14
- Without fluency, the model scores higher on almost all metrics than the full model, but generates incomplete sentences by concatenating high-scoring phrases.Source: §A.5 · p. 14
- The no-fluency result exposes a limitation of BLEU and SARI: incomplete phrase concatenations can artificially boost automatic scores.Source: §A.5 · p. 14