PaperlayerView exact revision on arXiv

Source-linked AI summary

MS2: Multi-Document Summarization of Medical Studies

Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, Lucy Lu Wang

arXiv:2104.06486v3cs.CLcs.AIcs.LG

TL;DR

Biomedical literature reviews are expensive and manual, motivating systems that can summarize evidence across multiple studies. The paper releases MS^2, a large biomedical multi-document summarization dataset with free-text and structured tasks, and finds that generated summaries are fluent but agree with gold evidence direction only about half the time.

  • Problem

    Biomedical literature reviews require a time-intensive, manual process, while biomedical multi-document summarization has limited resources and modeling techniques.

  • Method

    The paper releases MS^2 with texts-to-text and structured table-to-table tasks, using study abstracts, review-derived targets, and biomedical information-extraction systems.

  • Results

    Generated summaries are fluent and on-topic but agree with gold evidence direction around half the time under both the ΔEI metric and human evaluation.

  • Takeaways & Limitations

    MS^2 supports research on biomedical literature-review automation and highlights contradiction modeling as a potential route to improving summarization.

  • Takeaways & Limitations

    The structured task is preliminary because PICO tagging, coreference resolution, and PICO relation extraction remain weak.

Abstract

from arXiv · show

To assess the effectiveness of any medical intervention, researchers must conduct a time-intensive and highly manual literature review. NLP systems can help to automate or assist in parts of this expensive process. In support of this goal, we release MS^2 (Multi-Document Summarization of Medical Studies), a dataset of over 470k documents and 20k summaries derived from the scientific literature. This dataset facilitates the development of systems that can assess and aggregate contradictory evidence across multiple studies, and is the first large-scale, publicly available multi-document summarization dataset in the biomedical domain. We experiment with a summarization system based on BART, with promising early results. We formulate our summarization inputs and targets in both free text and structured forms and modify a recently proposed metric to assess the quality of our system's generated summaries. Data and models are available at https://github.com/allenai/ms2

1 Introduction

MS^2 introduces a large biomedical multi-document summarization dataset derived from systematic reviews to support literature-review automation and research on contradictory evidence.

  • 1 Introduction: The dataset targets biomedical multi-document summarization, where existing resources are either general-domain or small.Systematic reviews synthesize findings across many studies and require expensive manual production.
  • 1 Introduction: MS^2 defines a texts-to-text task that uses study abstracts and a BACKGROUND statement to generate a TARGET summary.The formulation is a sequence-to-sequence multi-document summarization task.
  • 1 Introduction: The dataset is intended to address contradictory information, structured summary-quality assessment, and the need to distill growing biomedical literature.These goals support automated generation of literature-review summaries.

2 Background

The background frames systematic reviews as structured syntheses of evidence from diverse biomedical studies, motivated by the need to identify relevant studies and interpret their findings.

  • 2 Background: Systematic reviews extract and summarize results from relevant studies addressing one or more biomedical research questions.Reviews can concern medical or social interventions applied to groups of people.
  • 2 Background: The volume of randomized controlled trials has increased to around 120 publications per day, intensifying the challenge of reviewing biomedical evidence.The cited rate is higher than the 75-per-day estimate reported for 2010.
  • 2 Background: PICO elements identify who is studied, which intervention is used, what it is compared against, and which outcome is measured.Studies included in a review generally share the review’s PICO elements or closely related ones.
  • 2 Background: Biomedical evidence comes from varied study designs, including clinical trials, cohort studies, observational studies, and case studies.Randomized controlled trials are described as offering the highest quality of evidence among these designs.

3 Dataset

MS^2 is constructed from biomedical reviews and cited studies, with automated filtering, target extraction, structured PIO and evidence labels, and clustered dataset splits. The dataset also captures disagreement between study-level and review-level evidence directions.

  • 3.1 Identifying suitable reviews and studies: MS^2 is built by filtering Semantic Scholar papers with keyword, PubMed, study-type, and SciBERT suitability criteria, leaving 20K candidate reviews.The keyword search begins with 220K matches, the PubMed filter leaves 170K, and the final classifier produces 20K candidates.
  • 3.2 Background and target identification: Review abstracts are classified into BACKGROUND, TARGET, and OTHER sentences, using BACKGROUND as input and TARGET as the summary target while excluding review-specific or overly detailed content.Five annotators labeled 3,000 sentences, and corrected annotations achieved Cohen’s κ of 0.912.
  • 3.2 Background and target identification: The sequential classifier reaches 94.1 F1 for BACKGROUND and 77.4 F1 for TARGET, with TARGET errors often confusing review findings with individual-study results or detailed statistics.Because full manual labeling is infeasible, classifier outputs provide silver training labels, while all 4,519 TARGET sentences in the test set were manually reviewed and 1,109 corrected.
  • 3.3 Adding structured representations: The dataset augments documents with PIO spans and evidence directions for Intervention–Outcome pairs, exposing structured representations alongside textual summaries.PIO spans are marked with special tokens, while evidence sentences are paired with increases, no_change, or decreases labels.
  • 3.3.2 Adding Evidence Inference: Only 53.8% of study-level evidence classifications match review-level classifications, highlighting contradictory evidence between individual studies and synthesized reviews.Review classifications are 13.4% decreases, 57.0% no_change, and 29.6% increases; study classifications are 15.7%, 60.7%, and 23.6%, respectively.
  • 3.4 Clustering and train / test split: Reviews are clustered before train, validation, and test splitting to reduce leakage from overlapping research questions, shared studies, and common results.The temporal distribution also shows a median review year of 2016 versus 2007 for cited studies, with a peak in 2009.

4 Experiments

The experiments evaluate free-text and structured biomedical summarization with BART-based models, alongside the ∆EI metric and human assessment of evidence-direction agreement. Results show preliminary feasibility, with generated summaries matching gold evidence direction in about half of evaluated cases.

  • 4.1 Texts-to-text task: The texts-to-text task maps a BACKGROUND statement and study abstracts to a TARGET summary, while the table-to-table task predicts evidence direction from review I/O pairs.The structured task is presented as an initial area for further research.
  • 4.1 Texts-to-text task: BART encodes each study independently with the BACKGROUND, whereas LED encodes all studies together and allows cross-study interactions.LED uses global attention between studies and a 512-token sliding attention window for local context.
  • 4.2 Table-to-table task: The structured representation uses Intervention-Outcome pairs and evidence directions, generating one of increases, no_change, or decreases.The models formulate classification as text generation and evaluate with macro-averaged F-scores.
  • 4.2 Table-to-table task: Structured-task results are preliminary because PICO tagging, co-reference resolution, and PICO relation extraction remain weak.Using the full Intervention-Outcome cross-product can also create duplicated or spurious pairs.
  • 4.3 ∆EI metric: The ∆EI metric averages Jensen-Shannon distances between gold and generated evidence-direction distributions while preserving increases-versus-decreases directionality.Unlike the related metric, it aims to capture an overall picture across all outputs.
  • 4.4 Human evaluation & error analysis: Around 50% (67/133) of generated summaries have the same evidence direction as the gold summary.Most confusions occur among increases, no_change, and insufficient.

5 Related Work

Related multi-document summarization datasets largely come from general-domain settings such as news and Wikipedia. MS^2 differs by targeting biomedical literature whose input studies may contain contradictory evidence.

  • 5 Related Work: Existing multi-document summarization datasets include MultiNews, WikiSum, and Wikipedia Current Events, while biomedical resources remain distinct in domain.
  • 5 Related Work: MS^2 uniquely includes input documents with contradicting evidence, rather than applying straightforward single-document summarization methods to multiple documents.

6 Discussion

MS^2 provides biomedical multi-document summarization resources, but fluent generated summaries still struggle to represent contradictory evidence and structured findings reliably.

  • MS^2 releases biomedical summaries and documents to support literature-review automation, addressing domain-specific vocabulary and background-knowledge challenges.
  • Generated summaries are fluent and on-topic, yet agree with gold evidence direction only around half the time, as shown by both ΔEI and human evaluation.Only 54% of study evidence directions agree with their review, suggesting contradiction modeling may be important for improvement.
  • The dataset and structured representations still need better PICO labels and evidence-direction annotations because pipeline errors compound across processing stages.Future improvements include higher-quality summary targets, structured representations, and links to external clinical-trial databases.
  • The authors identify capturing nuanced claims and contradictory findings across individual studies as an unresolved modeling challenge.

7 Conclusion

MS^2 addresses accelerating biomedical publication with a dataset for multi-document summarization and systematic-review automation. Its fluent outputs nevertheless leave major challenges in structured extraction, contradiction handling, and faithful evaluation.

  • MS^2 supports biomedical literature-review automation, but substantial challenges remain in PICO tuple extraction, co-reference resolution, and evaluating summary quality and faithfulness.
  • The dataset is intended to help researchers study biomedical multi-document summarization and advance systematic-review automation as publication rates increase.

Ethical Concerns and Broader Impact

The dataset targets medical systematic reviews, but its construction and intended use raise important scope, quality, and deployment constraints. The authors view automation as valuable for research while warning that factuality guarantees remain insufficient for practice.

  • The authors aim to facilitate research rather than immediate clinical deployment, given unresolved factuality and evaluation issues in high-impact biomedical settings.
  • The authors do not consider automated review summarization ready for deployment because factuality evaluations and guarantees about output contents remain insufficient.Plausible-looking but factually incorrect summaries would be unacceptable in this high-impact domain.
  • The dataset construction filters cited studies toward clinical trials, cohort studies, and observational studies while excluding case reports and veterinary publication types.Candidate reviews are retained only when they cite at least one suitable study and no case studies.
  • MS^2 retains reviews of human medical interventions with explicit effectiveness outcomes, excluding animal, veterinary, environmental, and case-study reviews.

C Suitability Classifier

A SciBERT-based suitability classifier filters candidate reviews for the dataset. Its selected threshold achieves high precision while retaining adequate recall, but false positives remain.

  • The classifier was trained with SciBERT using standard parameters to identify suitable systematic reviews from the candidate pool.
  • A threshold of 0.75 yields precision above 80% while maintaining adequate recall in five-fold cross-validation.
  • The threshold produces many false positives, although these documents are generally reviews and later target identification helps remove some unsuitable cases.

D.1 Sentence Annotation Guidelines

The annotation guidelines classify sentences in systematic reviews by their role, including background, goals, methods, findings, evidence quality, recommendations, and further-study statements.

  • D.1 Sentence Annotation Guidelines: A systematic review is defined as an in-depth search and analysis of literature relevant to a particular topic.
  • D.1 Sentence Annotation Guidelines: Sentences can receive multiple labels, while incorrectly segmented sentences and unsuitable or accidental non-review documents should be skipped or excluded.
  • D.1 Sentence Annotation Guidelines: The guidelines define background, goal, methods, detailed findings, general findings, recommendations, and further-study categories for review sentences.General findings include effect and evidence-quality statements, while detailed findings contain study counts or numerical results.
  • D.1 Sentence Annotation Guidelines: Detailed findings report study results, often including numbers or p-values, whereas effect statements describe intervention effects without requiring those details.
  • D.1 Sentence Annotation Guidelines: Further-study statements call for additional research and may include hedging about study design, sample size, context, or bias.
  • D.1 Sentence Annotation Guidelines: Recommendations require an explicit clinical or policy recommendation, with words such as “should” or “recommend” serving as useful indicators.Passive statements that merely describe a treatment as good do not qualify.

E Dataset Contradiction Scores

This section presents contradiction-analysis artifacts for comparing review effect findings with input-study effect findings, alongside validation result tables for both task formats.

  • E Dataset Contradiction Scores: The implementation uses PyTorch, HuggingFace Transformers, PyTorch Lightning, and a BART-base checkpoint, with FP16 training on RTX 8000 GPUs.
  • E Dataset Contradiction Scores: The confusion matrix compares review effect directions with the fractions of those directions accounted for in input studies.The most frequent confusion is with no_change rather than reversal of the overall finding direction.
  • E Dataset Contradiction Scores: The paper provides generated test-set summary examples from the BART configuration.
  • E Dataset Contradiction Scores: Table 14 reports validation ROUGE, ∆EI, and macro-averaged F1-scores for the texts-to-text task.The reported validation scores are described as similar to test scores.
  • E Dataset Contradiction Scores: Table 15 reports validation precision, recall, and macro-averaged F1-scores for the table-to-table task.

H Validation Results

This section situates MS² within systematic-review automation, emphasizing interpretation of intervention evidence as a relatively underdeveloped research area.

  • H Validation Results: Systematic reviews answer scientific questions through defining questions, finding studies, assessing quality, analyzing evidence, and drawing conclusions.
  • H Validation Results: Automation efforts have focused mainly on search, extraction, screening, and quality assessment, while interpretation has received relatively little attention.
  • H Validation Results: Biomedical systematic-review automation includes active-learning search, study-type screening, PICO-based assistance, and workflow tools for manual extraction.
  • H Validation Results: Interpretation concerns overall intervention effectiveness, appropriate use, evidence strength, recommendations, and remaining research needs.
  • H Validation Results: Prior work has used structured representations and content planning to extract relations and generate summaries that highlight contradictions in relevant literature.
Loading 2104.06486v3…