PaperlayerView exact revision on arXiv

Source-linked AI summary

Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature

Tomas Goldsack, Zhihao Zhang, Chenghua Lin, Carolina Scarton

arXiv:2210.09932v2cs.CL

TL;DR

Scientific articles can be difficult for non-experts, while lay-summarisation corpora remain limited in size and scope. The paper introduces PLOS and eLife datasets and evaluates their summaries and model performance. PLOS is larger, while eLife summaries are more readable and abstractive, and the datasets show differing benchmark patterns.

  • Problem

    Scientific articles’ technical language assumes specialist background, while existing lay-summarisation corpora are small and fragmented in format.

    Source: §1 · p. 1

  • Method

    The paper introduces full-article PLOS and eLife lay-summary datasets, characterizes their summaries, and benchmarks summarisation approaches with automatic metrics and expert evaluation.

    Source: §1 · p. 2

  • Results

    Extractive baselines perform better on PLOS, while BART models generally score better on eLife, whose summaries are more readable and abstractive than PLOS summaries.

    Source: §5.2 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    The datasets’ differences in readability and abstractiveness support the needs of different audiences and applications, while evaluations highlight challenges in lay summarisation.

    Source: §6 · pp. 8–9

    Both datasets cover only biomedicine, where lay summaries are more common; extending lay summarisation to other domains remains future work.

    Source: §7 · p. 9

Abstract

from arXiv · show

Lay summarisation aims to jointly summarise and simplify a given text, thus making its content more comprehensible to non-experts. Automatic approaches for lay summarisation can provide significant value in broadening access to scientific literature, enabling a greater degree of both interdisciplinary knowledge sharing and public understanding when it comes to research findings. However, current corpora for this task are limited in their size and scope, hindering the development of broadly applicable data-driven approaches. Aiming to rectify these issues, we present two novel lay summarisation datasets, PLOS (large-scale) and eLife (medium-scale), each of which contains biomedical journal articles alongside expert-written lay summaries. We provide a thorough characterisation of our lay summaries, highlighting differing levels of readability and abstractiveness between datasets that can be leveraged to support the needs of different applications. Finally, we benchmark our datasets using mainstream summarisation approaches and perform a manual evaluation with domain experts, demonstrating their utility and casting light on the key challenges of this task.

1 Introduction

Scientific articles can be difficult for non-experts to understand, and biomedical research can affect health-related decisions. This paper addresses the scarcity and limited scope of lay-summarisation resources with two datasets and evaluations of their characteristics and model performance.

1 Introduction

  • Technical language and assumed background knowledge make scientific articles difficult for lay readers, and can contribute to misinterpretation of findings.This concern is especially consequential in biomedicine, where research may affect health-related decisions.Source: §1 · p. 1
  • Lay summaries use non-specialist language to explain an article’s context and significance, often simplifying jargon and adding background for readers.The paper illustrates this contrast between abstract and lay-summary language and focus on background information.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • Existing lay-summarisation datasets are relatively small and fragmented in task framing and formats, limiting their applicability to broader biomedical literature.The cited resources range from 572 to 6,695 articles.Source: §1 · p. 2
  • The paper introduces PLOS and eLife datasets using full journal articles as sources, with different summary lengths and authorship intended to serve different audiences and applications.PLOS summaries are 150–200 words and author-written; eLife summaries are approximately twice as long and written by expert editors.Source: §1 · p. 2
  • The authors characterize the summaries, benchmark popular summarisation approaches, and evaluate outputs manually with experts to examine dataset utility and task challenges.The evaluations use automatic metrics and expert-based manual assessment.Source: §1 · p. 2

2 Related Work

Earlier lay-summarisation resources use different source-document and summary formats, limiting their fit for broader applications. The paper presents PLOS and eLife as larger or differently authored resources with distinct readability levels.

2 Related Work

  • The CL-SciSumm 2020 LaySumm task provided 572 Elsevier articles with author-generated lay summaries; organisers said the data was insufficient to train realistic summaries.The blind test set contained 37 articles, and the task drew eight submissions.Source: §2 · p. 2
  • Guo et al. (2021) paired 6,695 systematic-review abstracts with plain-language summaries, but the abstract-only source may not generalise to longer or other publication inputs.The dataset covers healthcare domains through the Cochrane Database of Systematic Reviews.Source: §2 · p. 2
  • Zaman et al. (2020) use long science-news summaries averaging over 600 words, whose length challenges training and news format limits journal applicability.Source: §2 · p. 2
  • PLOS is over 4× larger than CDSR, while eLife summaries are written by expert editors.Source: §2 · pp. 2–3
  • The paper presents two datasets with different readability levels, intended to support the needs of different audiences and applications.The authors describe their article and summary formats as more broadly applicable to wider literature.Source: §2 · pp. 2–3

3 Our Datasets

PLOS and eLife pair full biomedical research articles with manually created lay summaries. They differ in how their summaries are produced and intended for readers.

3 Our Datasets

  • The datasets pair full scientific articles from two biomedical journals with manually created lay summaries.Articles were retrieved in XML, parsed for summaries, abstracts, and article text, sectioned by headings, and split 90%/5%/5% into training, validation, and test sets.Source: §3 · p. 3
  • Several PLOS journals require authors to submit a 150–200-word non-technical summary alongside their work, intended to make paper findings accessible to a wider audience, including non-scientists.Source: §3 · p. 3
  • eLife digests are simplified summaries written by expert editors, based on the article and questions answered by its author, to explain background and significance accessibly.Source: §3 · p. 3

4 Dataset Analysis

The analysis compares abstracts and lay summaries across readability, rhetorical structure, content-word sharing, and n-gram novelty. Lay summaries are more readable, devote more space to background, share fewer abstract content words, and use more novel n-grams.

4.1 Readability

  • Lay summaries in both datasets are more readable than their abstracts across all metrics, and eLife summaries score lower than PLOS summaries on every metric.All reported differences are statistically significant (Mann–Whitney U tests, p < 0.05).Source: §4.1 · p. 4

4.2 Rhetorical Structure

  • Both datasets’ lay summaries devote more sentences to background and fewer to results, and to a lesser extent methods, than their abstracts.The authors attribute this pattern to non-experts’ need for background to understand a study’s motivation and significance.Source: §4.2 · p. 5

4.3 Content Words

  • Abstract content words are rarely shared with lay summaries, while the shared-to-unshared ratio generally rises with a word’s frequency across abstracts.More than 90% of content words other than verbs occur in 10 or fewer abstracts on average; many low-frequency nouns, proper nouns, and numbers are topic-specific or technical.Source: §4.3 · pp. 5–6

4.4 Abstractiveness

  • Lay summaries contain more novel n-grams than abstracts in both datasets, and eLife summaries appear more abstractive than PLOS summaries.eLife summaries are approximately twice as long as PLOS summaries; the authors identify readability and abstractiveness as dataset distinctions relevant to application choice.Source: §4.4 · p. 6

5 Experiments and Results

The experiments benchmark heuristic, unsupervised, and supervised summarisation approaches with automatic metrics, then assess BART outputs through expert ratings. Results differ across PLOS and eLife, and the human evaluation identifies factuality as a persistent challenge.

5 Experiments and Results

  • The experiments combine automatic benchmarking with an expert-based manual evaluation to examine model performance and challenges in lay summarisation.The benchmark reports ROUGE F1 scores alongside generated-output syntactic and lexical complexity measures.Source: §5 · p. 6, §5.2 · p. 7

5.1 Baseline Approaches

  • The benchmark includes LEAD-3 and LEAD-K, abstract and extractive-oracle baselines, four unsupervised extractors, and BART variants trained with additional data or scaffolding.The approaches are evaluated using ROUGE-1, ROUGE-2, ROUGE-L, FKGL, and DCRS.Source: §5.1 · p. 6, §5.2 · p. 7

5.2 Discussion

  • Extractive baselines perform significantly better on PLOS than eLife, while BART generally leads on eLife and exceeds ORACLE_EXT on ROUGE.On PLOS, extractive models’ readability scores match or sometimes exceed BART’s, though inconsistently; on eLife, BART generally scores higher on both ROUGE and readability metrics.Source: §5.2 · p. 7
  • Abstract-generation scaffolding lowers ROUGE on both datasets, PubMed pretraining changes performance little, and cross-dataset training performs close to standard BART.Together, these results indicate that additional out-of-domain training provides little benefit for lay summarisation; alternative approaches that use available data better may be more promising.Source: §5.2 · pp. 7–8

5.3 Human evaluation

  • Human evaluation rates model-generated summaries from a random sample of 10 test-split articles per dataset for comprehensiveness, layness, and factuality, with abstracts and reference lay summaries supplied but full articles withheld.Krippendorff’s α was 0.78 for PLOS and 0.54 for eLife.Source: §5.3 · p. 8
  • PLOS outputs are judged more comprehensive than eLife outputs, while eLife outputs sometimes introduce irrelevant or confusing information absent from the abstract and reference lay summary.Evaluators attributed PLOS’s comprehensiveness in part to extensive use of abstract content, sometimes copied directly or with minor edits.Source: §5.3 · p. 8
  • Although layness averages 3.0 on both datasets, PLOS outputs can retain difficult jargon, whereas eLife outputs can be oversimplified, misleading, or contain language errors.Evaluators also noted repeated content in eLife outputs.Source: §5.3 · p. 8
  • BART’s layness and factuality ratings average 3.0 on both datasets, and no output receives a perfect factuality score from both evaluators.Evaluators cited contradictions, unclear phrasing, and misrepresented entities as contributors to factual inconsistencies.Source: §5.3 · p. 8

6 Conclusion

The paper introduces PLOS and eLife, two biomedical article–lay-summary datasets, and benchmarks them with automatic metrics and expert evaluation. Their differing readability and abstractiveness support different audiences and applications, while the evaluations highlight challenges for lay summarisation.

6 Conclusion

  • PLOS and eLife pair full biomedical research articles with manually created lay summaries, and PLOS is significantly larger than existing resources.eLife summaries are approximately twice as long as PLOS summaries and are more readable and abstractive, making them better suited to a less technical audience.Source: §6 · p. 8
  • The paper benchmarks the datasets with popular summarisation models and automatic metrics, and evaluates model outputs with experts to identify task challenges.The authors describe the article and summary formats as more broadly applicable to wider literature than previous resources.Source: §6 · pp. 8–9

7 Limitations

The datasets remain smaller than those used for other forms of summarisation and cover only biomedicine.

7 Limitations

  • Despite being the largest dataset for lay summarisation, the resources remain smaller than datasets for other tasks, which can contain 100,000+ articles.The authors attribute this constraint to lay summaries being less common and available in relatively few journals, only some of which are open access.Source: §7 · p. 9
  • The datasets cover only biomedicine; the authors link this scope to limited lay-summary availability in other scientific domains.They hope to address other domains in future work.Source: §7 · p. 9

A Appendix

The appendix documents dataset availability and processing, supplementary comparisons, baseline details, and materials illustrating model evaluations and lay-summary examples.

A Appendix

  • PLOS and eLife data are open-access, English-only, and made available for future research; the appendix reports retrieval dates of 7/03/22 and 11/03/22, respectively.PLOS material may be mined, reused, and shared under its policy; eLife uses a CC-BY 4.0 license.Source: A Appendix · p. 11
  • eLife duplicate article versions were removed in favor of the latest, while extraction excluded tables, figures, supplementary material, acknowledgments, and DOI-ending sentences.Sentence segmentation also protected “et al.” occurrences with placeholders before restoring them afterward.Source: A Appendix · p. 11
  • Table 6 and Figure 5 compare lay-summary readability and n-gram novelty across datasets, while Table 7 gives the percentages underlying Figure 3’s content-word-sharing bars.The Table 7 percentages can be used to calculate shared-to-unshared ratios across content-word types.Source: A Appendix · p. 11, Table 7 · p. 12, Table 6 · p. 12
  • ORACLE_EXT greedily extracts article sentences that maximize mean ROUGE-1, ROUGE-2, and ROUGE-L, stopping at the dataset’s average lay-summary length.Unsupervised baselines used default configurations.Source: A Appendix · p. 11
  • BART baselines used specified Hugging Face models and AdamW training, with early stopping and validation ROUGE-2 selecting the best checkpoint.BART, BARTCross, and BARTScaffold used facebook/bart-base; BARTPubMed used mse30/bart-base-finetuned-pubmed.Source: A Appendix · p. 11
  • Figures 8 and 9 provide full PLOS and eLife lay-summary examples alongside their technical abstracts.Source: A Appendix · p. 11
  • Human-evaluation comments describe summaries as conveying core information, while also noting omitted background and instances of extra concepts that could confuse readers.Figures 6 and 7 present comments for PLOS and eLife, respectively.Source: A Appendix · pp. 11, 13, 14

Related papers

Loading 2210.09932v2…