PaperlayerView exact revision on arXiv

Source-linked AI summary

Extractive Summarization as Text Matching

Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, Xuanjing Huang

arXiv:2004.08795v1cs.CL

TL;DR

Sentence-level extractors can miss the semantics and coupling of complete summaries, motivating a summary-level alternative. The paper formulates extractive summarization as semantic matching between documents and candidate summaries, with gains varying by dataset and summary length.

  • Problem

    Existing sentence-level extractors overlook whole-summary semantics and sentence coupling, leaving unclear when summary-level extraction is preferable across dataset characteristics.

  • Method

    MATCHSUM matches source documents and candidate summaries in semantic space, using document and summary embeddings with cosine similarity and margin-based triplet-loss fine-tuning.

  • Results

    Performance gains vary by dataset and summary length, reaching a maximum improvement of 4.7 on CNN/DM; medium-length summaries benefit most, while short and long summaries improve relatively little.

  • Takeaways & Limitations

    The analysis links extractor suitability and summary-level gains to dataset characteristics, supporting semantic matching as an effective extractive-summarization framework.

  • Takeaways & Limitations

    The model does not perform well on samples with the largest z, which require further improvement and exploration.

Abstract

from arXiv · show

This paper creates a paradigm shift with regard to the way we build neural extractive summarization systems. Instead of following the commonly used framework of extracting sentences individually and modeling the relationship between sentences, we formulate the extractive summarization task as a semantic text matching problem, in which a source document and candidate summaries will be (extracted from the original text) matched in a semantic space. Notably, this paradigm shift to semantic matching framework is well-grounded in our comprehensive analysis of the inherent gap between sentence-level and summary-level extractors based on the property of the dataset. Besides, even instantiating the framework with a simple form of a matching model, we have driven the state-of-the-art extractive result on CNN/DailyMail to a new level (44.41 in ROUGE-1). Experiments on the other five datasets also show the effectiveness of the matching framework. We believe the power of this matching-based summarization framework has not been fully exploited. To encourage more instantiations in the future, we have released our codes, processed dataset, as well as generated summaries in https://github.com/maszhongming/MatchSum.

1 Introduction

The paper reframes extractive summarization as summary-level semantic matching rather than sentence-by-sentence extraction. Analysis of dataset-dependent gaps motivates MATCHSUM, which improves extractive performance across six datasets and reaches 44.41 ROUGE-1 on CNN/DailyMail.

  • Sentence-level extractors model sentence relationships but do not consider the semantics of the entire summary, encouraging generalized selections and overlooking sentence coupling.
  • The authors find an inherent gap between sentence-level and summary-level approaches across six benchmark datasets, motivating a summary-level method.
  • MATCHSUM formulates extractive summarization as semantic text matching between a source document and candidate summaries extracted from it.
  • Experiments with significance testing report that the matching framework outperforms strong baselines across six benchmark datasets.
  • Contrastive learning favors the gold or better candidate summary when it is more semantically similar to the source document than unqualified candidates.
  • 44.41 in ROUGE-1 is the state-of-the-art extractive result reported on CNN/DailyMail using only the base version of BERT.

2 Related Work

Prior extractive summarization research uses diverse encoder-decoder, reinforcement-learning, semantic, and multi-stage approaches. Despite these advances, many systems remain sentence-level extractors, while this work positions MATCHSUM as an extract-then-match alternative.

  • Recent neural extractive systems use RNN, Transformer, or GNN encoders with auto-regressive or non-auto-regressive decoders, but remain essentially sentence-level extractors.
  • Reinforcement learning enables summary-level scoring and improvement, yet prior efforts remain constrained by auto-regressive or non-auto-regressive architectures.
  • Earlier semantic approaches address extractive summarization through concept coverage, reconstruction, or maximizing semantic volume.
  • Two-stage systems first extract fragments and then select or modify them, including extract-then-rewrite and extract-then-compress paradigms.
  • MATCHSUM can be viewed as an extract-then-match framework that uses a sentence extractor to prune unnecessary information before matching.

3 Sentence-Level or Summary-Level? A Dataset-dependent Analysis

The analysis asks when summary-level extraction is preferable to sentence-level extraction and quantifies the dataset-dependent gap between them. Pearl-summaries expose why sentence-level ranking can miss the best candidate, while summary-level gains vary with dataset characteristics and reference-summary length.

  • Research questions: The study compares sentence-level and summary-level extractors and asks which is better for a dataset and what gap separates them.It analyzes six benchmark datasets and uses candidate summaries extracted from each document.
  • Definitions: A pearl-summary has lower sentence-level score but higher summary-level score than another candidate, making it difficult for sentence-level systems to select.Sentence-level scores average sentence overlaps with the gold summary, whereas summary-level scores evaluate the candidate as a whole.
  • Ranking of Best-Summary: z ranks the best-summary after candidates are sorted by sentence-level score; z = 1 means the best-summary comes first, while z > 1 identifies a pearl-summary.Candidates are formed approximately from the # Ext highest-scoring sentences, and the sentence-level score uses mean F1 of ROUGE-1, ROUGE-2, and ROUGE-L.
  • Ranking of Best-Summary: 18.9% of CNN/DM best-summaries are not pearl-summaries, while PubMed favors sentence-level extraction and WikiHow and Multi-News need summary-level learning more strongly.The figure normalizes z by the number of candidate summaries and reports the proportion of best-summaries at each rank.
  • Dataset-dependent analysis: The proportion of pearl-summaries among best-summaries characterizes a dataset and affects the choice of summarization extractor.The analysis treats this proportion as a dataset property relevant to extractor selection.
  • Inherent Gap: 4.7 is the maximum reported performance gain for the summary-level method on CNN/DM, while short and very long reference summaries show relatively small gains.The paper relates potential gain to reference-summary length: short summaries offer little improvement, and long summaries already contain substantial semantic overlap.

4 Summarization as Matching

The paper argues that sentence-level extractors are inherently unaware of pearl-summaries, motivating direct summary-level scoring through semantic matching. MATCHSUM matches documents with extracted candidate summaries using Siamese-BERT, margin-based objectives, and candidate pruning before selecting the best-scoring summary.

  • Sentence-level extractors are inherently unaware of pearl-summaries, making the best summary difficult to obtain.This motivates a framework that scores and extracts summaries directly at the summary level.
  • MATCHSUM formulates extractive summarization as semantic matching between a source document and candidate summaries extracted from it.The framework matches the document and candidates in a semantic space.
  • Siamese-BERT uses two tied-weight BERT encoders and cosine similarity to match document D with candidate summary C.The architecture derives embeddings for both texts and compares them during inference.
  • Margin-based objectives train the gold summary to be closest to the document and assign larger margins to candidate pairs with larger ROUGE-ranking gaps.The loss combines the gold-summary matching principle with pairwise ranking among candidate summaries.
  • At inference, the system searches the extracted candidate set and selects the summary with the highest matching score.The search is performed over candidates C extracted from document D.
  • Candidate pruning addresses combinatorial explosion by preselecting salient sentences, generating combinations from the pruned document, and restoring original sentence order.BERTEXT scores sentence salience before candidate construction.

5 Experiment

Experiments across six benchmark datasets evaluate MATCHSUM’s candidate-summary matching framework, showing strong performance and dataset-dependent gains tied to summary-level semantics. Analyses indicate improvements are largest when datasets contain more pearl-summaries, while summary length and dataset structure constrain attainable gains.

  • Results across datasets: MATCHSUM beats BERTEXT by 1.51 ROUGE-1 with BERT-base on CNN/DailyMail and remains better than a BERT-large baseline.Using RoBERTa-base improves performance further.
  • Results across datasets: 1.62 ΔR-1 on XSum and 1.04 ΔR-1 on Reddit occur when MATCHSUM compares two-sentence summaries rather than re-ranking single sentences.Single-sentence matching yields smaller gains of 0.82 ΔR-1 on XSum and 0.88 ΔR-1 on Reddit.
  • Results across datasets: 1.54 ROUGE-1 separates MATCHSUM from the state-of-the-art BERT model on WikiHow, illustrating robust performance across domains.The comparison is reported for WikiHow, where semantic-space alignment avoids strong heuristic constraints.
  • Analysis of dataset effects: XSum remains a limitation because MATCHSUM does not perform well on samples with the largest z, requiring further improvement and exploration.This differs from the generally increasing performance gap observed as z increases elsewhere.
  • Analysis of dataset effects: MATCHSUM’s performance gains concentrate in samples with more pearl-summaries, because summary-level matching can combine individually weaker sentences into a better summary.The performance gap generally increases with z, reaching 1.57 on the highest-z CNN/DailyMail subset versus 0.49 on the lowest-z subset.
  • Analysis of dataset effects: Summary length limits attainable gains: MATCHSUM reaches ψ(D)=0.64 on 23.3-word XSum summaries but remains below 0.2 on PubMed and Multi-News summaries exceeding 200 words.For similarly long summaries, Multi-News reaches ψ(D)=0.18 versus PubMed’s 0.09, associated with more pearl-summaries.

6 Conclusion

MATCHSUM formulates extractive summarization as semantic matching between a source document and candidate summaries in semantic space. It outperforms the current state-of-the-art extractive model on six benchmark datasets.

  • MATCHSUM formulates extractive summarization as semantic matching between source documents and candidate summaries in semantic space.
  • The framework replaces scoring and extracting sentences individually with a summary-level formulation.
  • MATCHSUM outperforms the current state-of-the-art extractive model on six benchmark datasets.
Loading 2004.08795v1…