PaperlayerView exact revision on arXiv

Source-linked AI summary

Paragraph-level Simplification of Medical Texts

Ashwin Devaraj, Iain J. Marshall, Byron C. Wallace, Junyi Jessy Li

arXiv:2104.05767v1cs.CL

TL;DR

Dense terminology makes reliable biomedical information inaccessible to most lay readers, and manual simplification does not scale. The paper introduces paragraph-level paired data, an MLM technicality measure, and jargon-penalized generation; the measure better distinguishes technical from plain-language texts, while the penalty improves simplicity and abstractiveness by the evaluated metrics.

  • Problem

    Reliable biomedical information is often inaccessible to lay readers because it is dense with technical terminology, while manual simplification is too costly to scale.

    Source: §1 · p. 1

  • Method

    The paper builds Cochrane-derived technical–plain-language paragraph pairs, scores technicality with a scientific-text MLM, and tests BART simplification with a jargon-token penalty.

    Source: §1 · pp. 1–2, §6 · p. 9

  • Results

    The MLM measure better discriminates technical abstracts from plain-language summaries than traditional readability measures, and the jargon penalty can improve simplicity and abstractiveness by evaluated metrics.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The publicly available paired resource is intended to facilitate further research on medical text simplification.

    Source: §6 · p. 9

    Generated summaries can contain inconsistencies, including hallucinations, relative to the original review and should be manually examined before use.

    Source: §7 · p. 9

Abstract

from arXiv · show

We consider the problem of learning to simplify medical texts. This is important because most reliable, up-to-date information in biomedicine is dense with jargon and thus practically inaccessible to the lay audience. Furthermore, manual simplification does not scale to the rapidly growing body of biomedical literature, motivating the need for automated approaches. Unfortunately, there are no large-scale resources available for this task. In this work we introduce a new corpus of parallel texts in English comprising technical and lay summaries of all published evidence pertaining to different clinical topics. We then propose a new metric based on likelihood scores from a masked language model pretrained on scientific texts. We show that this automated measure better differentiates between technical and lay summaries than existing heuristics. We introduce and evaluate baseline encoder-decoder Transformer models for simplification and propose a novel augmentation to these in which we explicitly penalize the decoder for producing "jargon" terms; we find that this yields improvements over baselines in terms of readability.

1 Introduction

Medical texts are often inaccessible to lay readers, while manual simplification is costly at scale. This work introduces paragraph-level resources and methods for measuring and generating plain-language medical text.

1 Introduction

  • The dataset draws on Cochrane reviews, whose author-written plain-language summaries cover evidence on clinical topics.The summaries are written from full reviews and follow their own structure and guidelines.Source: §1 · pp. 1–2
  • The authors heuristically derive 4,459 technical–plain-English section or paragraph pairs by matching analogous abstract and plain-language-summary portions.The paired sections can involve paraphrasing, deletion, and summarization; the summaries are not direct simplifications of the abstracts.Source: §1 · pp. 1–2
  • Traditional readability differences between abstracts and plain-language summaries are small, while pretrained masked language models capture their differences better.The MLM results indicate that language differences go beyond shallow cues such as sentence and word length.Source: §1 · p. 2
  • Jargon-penalized unlikelihood training improves readability over baselines with only a minor content-quality tradeoff.Naively fine-tuned encoder-decoder models tend to delete content rather than paraphrase or explain, and often generate technical words.Source: §1 · p. 2
  • The work contributes a dataset, a masked-language-model technicality metric, analysis of plain-language style, and baselines that penalize jargon generation.The authors report that the metric better captures technical-versus-lay differences than traditional readability measures.Source: §1 · p. 2

2 Related work

Prior simplification resources and systems largely target Wikipedia, news, and sentence-level tasks; medical paragraph-level simplification remains sparsely resourced and studied.

2 Related work

  • Adduru et al. (2018) and Van den Bercken et al. (2019) introduced sentence-aligned corpora with thousands of sentence pairs, automatically derived through paraphrase mining or monolingual alignment.Source: §2 · p. 2
  • Medical text simplification lacks resources for training model-based systems, so past work focused on lexical simplification; recent sentence-aligned corpora were automatically derived and constrain approaches to sentence-operating methods.A French medical simplification corpus contains 663 sentence pairs, 112 from Cochrane, with a small subset manually sentence-aligned.Source: §2 · p. 2
  • English document simplification research is sparse, covering analysis, sentence deletion, and localized explanations; this work evaluates paragraph-level encoder-decoder simplification.Prior examples include document analysis, sentence deletion, and localized explanation generation [Srikanth and Li, 2020].Source: §2 · p. 2

3 Technical abstracts vs. plain-language summaries

The section constructs a paired Cochrane technical-abstract and plain-language-summary dataset, then examines how their language differs. SciBERT probabilities discriminate between the text types better than BERT and Flesch-Kincaid, while lexical features also separate them.

3.1 Data compilation

  • 4459 abstract–PLS pairs form the final dataset, selected by aligning results-focused sections and filtering for length ratios of 0.2–1.3.The source is the Cochrane Database of Systematic Reviews; pairs contain no more than 1024 tokens.Source: §3.1 · p. 3, §3 · p. 3

3.2 Characterizing readability differences

  • 1.5 grades with Flesch-Kincaid and 0.6 with ARI are the maximum differences in mean minimum grade between abstracts and PLS.Both differences are statistically significant (paired t-test, p < 0.01); the authors report that PLS remain above average US-adult reading levels.Source: §3.2 · p. 4, Table 3 · p. 4
  • They score technicality by averaging original-token probabilities across ten masks of 15% of sentence tokens, using BERT or SciBERT.SciBERT was pretrained on technical papers; BERT was trained on BooksCorpus and English Wikipedia.Source: §3.2 · p. 4
  • 0.70 vs 0.66 AUC: SciBERT discriminates PLS from technical abstracts better than BERT, and its AUC also exceeds Flesch-Kincaid.The authors use SciBERT masked probabilities when analyzing generated text.Source: §3.2 · p. 5, Figure 2 · p. 5

3.3 Lexical analysis

  • 92.7% cross-validation accuracy and 0.99 train-validation AUC show that bag-of-words weights distinguish abstracts from PLS; numerical and statistical terms mark abstracts, while common phrases mark PLS.The classifier uses normalized token-frequency features from the BART vocabulary.Source: §3.3 · p. 5

4 Baseline models for simplification

This section establishes pretrained BART as the simplification baseline and adds unlikelihood training to penalize selected technical tokens.

4.1 Pretrained BART

  • BART is the encoder-decoder baseline, initialized from XSum-fine-tuned weights and decoded with nucleus sampling.Nucleus sampling truncates the low-probability tail before renormalizing.Source: §4.1 · pp. 5–6

4.2 Unlikelihood training

  • The unlikelihood term discourages selected tokens by reducing their decoder probability, and is weighted by α and added to the usual log-likelihood objective.The token probability is conditioned on the technical abstract and the target prefix.Source: §4.2 · p. 6
  • The penalty set uses negative-weight tokens from simple-versus-complex classifiers, based on PLS versus abstracts, Newsela reading levels, or combined classifiers.Absolute classifier weights are softmaxed so the most technical-associated tokens contribute most to the penalty.Source: §4.2 · p. 6
  • The unlikelihood penalty is applied to a token only when the decoder distribution indicates that token should be output.Source: §4.2 · p. 6

4.3 Experimental setup

  • The 4,459 review pairs were split into 3,568 training, 411 validation, and 480 test examples; results use the full test set.Hyperparameters were selected through manual validation inspection; the reported setting was T=2 and α=100.Source: §4.3 · pp. 6–7

5 Results

The results compare model outputs on readability, style, content metrics, and manual quality. Unlikelihood (UL) training improves readability and reduces copying, but content metrics show mixed effects and manual review finds tradeoffs and artifacts.

5.1 Readability scores

  • UL training produced lower Flesch-Kincaid scores than both technical abstracts and reference PLS, while the model without UL produced a higher reading level than PLS.The differences between UL models and No-UL were statistically significant (p < 0.01, paired t-test).Source: Table 5 · p. 7, §5.1 · p. 7

5.2 Style

  • All models had lower SciBERT masked probability scores than abstracts, and their scores were not significantly different from the reference PLS.Lower scores are treated as indicating simpler, less technical language.Source: §5.2 · p. 7

5.3 Content

  • UL models had small but significantly better SARI scores, while ROUGE and BLEU scores moved in the opposite direction.The reported differences between No-UL and UL models were significant except for BLEU with UL-Newsela.Source: Table 6 · p. 8, §5.3 · p. 8
  • UL lowered n-gram overlap with abstracts, especially at larger n, but generated summaries remained more extractive than reference PLS.Cochrane weights likely reduce copying of numerical details; Newsela weights did not show the same effect, which the authors attribute to domain differences.Source: Table 7 · p. 8, §5.3 · p. 8

5.4 Manual examination and analysis

  • Manual examination found outputs shorter than abstracts and reference PLS; higher α increased readability and concision but reduced completeness and raised hallucination, spelling, repetition, and coherence problems.These drawbacks greatly impacted coherence for α ≥1000; a common hallucination claimed evidence was current to a date not stated in the abstract.Source: §5.4 · pp. 8–9

6 Conclusions

The paper introduces a publicly available resource of paired medical abstracts and plain-language summaries, along with a masked-language-model measure of technicality. Its BART-based simplification models use an explicit jargon penalty, which improved simplicity and abstractiveness on the evaluated metrics.

6 Conclusions

  • The dataset pairs technical abstracts summarizing medical evidence with plain-language versions, and is publicly available for further research.It comprises only text derived from publicly accessible abstracts.Source: §6 · p. 9
  • The proposed masked-language-model measure estimates text technicality from token likelihoods under a transformer model trained on a technical corpus, and better discriminated abstracts from PLS than traditional readability measures.Source: §6 · p. 9
  • BART models with an explicit penalty for producing jargon terms improved simplicity and abstractiveness according to the metrics considered.Source: §6 · p. 9

7 Ethical Considerations

The paper flags potential reader harm from inconsistent generated summaries and describes limits and privacy properties of the dataset.

7 Ethical Considerations

  • The dataset uses only freely available Cochrane information, involves no annotators beyond the authors in manual inspection, and contains no personally identifiable information.Source: §7 · p. 9
  • Generated summaries may contain hallucinations inconsistent with the original review and could misinform readers, so they should be manually examined before use.Source: §7 · p. 9

Appendix A: Example outputs

Appendix A provides a full example of a technical abstract, reference PLS, and model outputs.

Appendix A: Example outputs

  • Table 10 presents a full example pairing a technical abstract and reference PLS with model outputs.Source: Table 10 · p. 13

Related papers

Loading 2104.05767v1…