PaperlayerView exact revision on arXiv

Source-linked AI summary

What Have We Achieved on Text Summarization?

Dandan Huang, Leyang Cui, Sen Yang, Guangsheng Bao, Kun Wang, Jun Xie, Yue Zhang

arXiv:2010.04529v1cs.CL

TL;DR

Automatic summaries still fall short of human-written summaries, while fine-grained evidence about systems and milestone techniques remains limited. The paper manually evaluates ten representative summarizers with MQM-inspired PolyTope metrics. Extractive systems generally outperform abstractive systems, and joint pre-training, particularly BART, shows the strongest results.

  • Problem

    Despite improved ROUGE scores, automatic summaries still fall short of human-written summaries, and existing human evaluations have limited scope and coverage.

    Source: §1 · p. 1

  • Method

    The paper manually evaluates ten representative summarizers using PolyTope, an MQM-inspired set of eight metrics for quantifying primary error sources.

    Source: §1 · p. 1, §4 · p. 4

  • Results

    Extractive summarizers generally outperform abstractive models, while joint pre-training combining text understanding and generation gives the strongest advantage, with BART achieving state-of-the-art results on automatic and human evaluations.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Fine-grained error diagnostics provide more detail about summarization systems and the effects of copy, coverage, hybrid approaches, and pre-training.

    Source: §7 · p. 9

    Applying the reported BART evaluation requires re-training BART without document rotation transformation.

    Source: §5.4 · p. 8

Abstract

from arXiv · show

Deep learning has led to significant improvement in text summarization with various methods investigated and improved ROUGE scores reported over the years. However, gaps still exist between summaries produced by automatic summarizers and human professionals. Aiming to gain more understanding of summarization systems with respect to their strengths and limits on a fine-grained syntactic and semantic level, we consult the Multidimensional Quality Metric(MQM) and quantify 8 major sources of errors on 10 representative summarization models manually. Primarily, we find that 1) under similar settings, extractive summarizers are in general better than their abstractive counterparts thanks to strength in faithfulness and factual-consistency; 2) milestone techniques such as copy, coverage and hybrid extractive/abstractive methods do bring specific improvements but also demonstrate limitations; 3) pre-training techniques, and in particular sequence-to-sequence pre-training, are highly effective for improving text summarization, with BART giving the best results.

1 Introduction

Despite improved ROUGE scores, the quality gap between automatic and human summaries remains, and fine-grained evidence about systems and milestone techniques is limited. The paper manually evaluates ten representative summarizers with PolyTope, finding advantages for extractive methods and pre-training, especially BART.

1 Introduction

  • Although ROUGE scores have improved, machine-generated summaries still fall short of human-written ones, and existing human evaluations have limited scope and coverage.The paper also releases the test set of ten system outputs with manually labeled errors and an evaluation toolkit for future research.Source: §1 · pp. 1–2
  • The study manually evaluates ten representative summarizers using eight MQM-inspired metrics designed to quantify primary error sources.It compares neural with traditional methods and extractive with abstractive systems, and examines commonly used techniques.Source: §1 · pp. 1–2
  • Under similar settings, extractive approaches generally outperform abstractive models, whose main shortcomings include omission and intrinsic hallucination.Unnecessary content is the main shortcoming identified for extractive models.Source: §1 · p. 2
  • Copy reproduces details and reduces word-level duplication but can cause redundancy; coverage reduces repetition but has limits in faithful content generation.Hybrid extractive/abstractive models reflect the relative strengths and weaknesses of both approaches.Source: §1 · p. 2
  • Joint pre-training combining text understanding and generation gives the strongest advantage, with BART achieving state-of-the-art results on automatic and human evaluations.Pre-training also improves content selection without copy or coverage mechanisms.Source: §1 · p. 2

2 Related Work

Prior summarization analysis has largely relied on automatic metrics, while human evaluation has often been coarse-grained or limited to selected model types. This work uses fine-grained human error analysis across rule-based, extractive, and abstractive systems.

2 Related Work

  • Hybrid abstractive-extractive systems use two-step content selection and text generation, and prior work reports higher ROUGE performance than end-to-end models.The cited approaches include Gehrmann et al. (2018), Hsu et al. (2018), and Celikyilmaz et al. (2018).Source: §2 · p. 2
  • Earlier analyses used automatic metrics to examine coherence, redundancy, relevance, informativeness, or model architectures, whereas this work uses fine-grained human evaluation.Automatic evaluation has been criticized for insufficiently capturing overall generated-text quality and for weak correlation with human judgments.Source: §2 · p. 2
  • Prior human evaluations often report coarse scores across two or three aspects, while this study evaluates rule-based, extractive, and abstractive systems on eight error metrics.Faithfulness and factuality are among the included metrics.Source: §2 · p. 3
  • Unlike Pyramid, ranking, best-worst scaling, and question-answering approaches that assign one score per output, the paper's MQM-motivated metrics count errors for finer-grained diagnosis.The paper contrasts these evaluation approaches empirically in Figure 3.Source: §2 · p. 3

3 Models

The study re-implements ten representative summarization methods spanning traditional and neural extractive systems and several abstractive architectures. The extractive methods select source sentences, while abstractive methods generate text, sometimes using copying, coverage, or hybrid selection.

3 Models

  • The study re-implements ten representative and influential methods, with their publicly reported ROUGE F1 scores shown in Table 1.The table covers the CNN/DM dataset.Source: §3 · p. 3

3.1 Extractive Methods

  • Lead-3 selects the first three sentences, TextRank extracts high-weight sentences from a similarity graph, and neural extractive models select sentences using learned representations.SummaRuNNer uses a hierarchical BiGRU and accumulated summary representation; BertSumExt uses pretrained BERT sentence representations and a Transformer document encoder.Source: §3.1 · p. 3

3.2 Abstractive Methods

  • Abstractive models generate summaries with attention or pointer-generator copying, while coverage supplies an additional attention input to address repetition.Bottom-Up first selects potential output words before generating; BertSumExtAbs combines a pretrained encoder with a randomly initialized Transformer decoder, and BART jointly pre-trains a bidirectional encoder and autoregressive decoder.Source: §3.2 · pp. 3–4

4 Evaluation Method

The evaluation combines ROUGE with PolyTope, a fine-grained human error analysis that classifies summary errors by issue type, syntactic location, and severity. Annotators mark incorrect segments, while predefined severity rules produce quality scores.

4 Evaluation Method

  • PolyTope complements ROUGE with eight issue types, eight syntactic labels, and severity rules for detailed, diagnostic quality scoring.Annotators compare the source and summary, mark incorrect segments, and classify them; agreement over 20 documents was 0.8621 by Pearson correlation.Source: §4 · p. 4

4.3 Severity

  • MINOR, MAJOR, and CRITICAL errors receive deduction ratios of 1:5:10, respectively, and higher-severity issues have greater impact on perceived summary quality.Source: §4 · p. 4, §4.3 · p. 5

4.1 Issue Type

  • Accuracy issues capture mismatches with the source, including additions, omissions, intrinsic or extrinsic inaccuracies, and polarity reversals.Source: §4.1 · p. 5
  • Fluency issues concern language independent of source-target relationships, including word form, word order, and unnecessary duplication.Source: §4.1 · p. 5

4.2 Syntactic Label

  • Eight syntactic labels locate errors in sentence components, including Subject, Predicate, Object, Number&Time, Place&Name, Attribute, Function Word, and Whole Sentence.Source: §4.2 · p. 5

5 Evaluating Model Performance

The evaluation compares 10 summarizers on 150 CNN/DM trials using ROUGE and PolyTope, examining model families and milestone techniques. Results show that BART leads overall, while other models reveal distinct trade-offs in faithfulness, content selection, and fluency.

5.1 Preneural vs Neural Models

  • Lead-3 remains a strong baseline, ranking fourth among all models on both ROUGE-1 and PolyTope, while TextRank ranks ninth and seventh, respectively.Both unsupervised methods have the largest Addition error counts, consistent with weaker filtering of irrelevant information than supervised methods.Source: §5.1 · p. 6

5.2 Extractive vs Abstractive Summarization

  • Under similar settings, extractive models outperform abstractive counterparts on PolyTope, while the two approaches are generally competitive on Fluency.Extractive models are stronger on intrinsic and extrinsic inaccuracy, but do not show stronger Addition or Omission performance; extractive systems make three error types, versus four to seven for abstractive systems.Source: §5.2 · p. 6

5.3 Extractive Methods

  • BertSumExt has better ROUGE-1/2 than SummaRuNNer, but no significant advantage under ROUGE-L or PolyTope and improves only on Duplication errors.The authors relate its higher ROUGE to fitting ROUGE-constructed training summaries, rather than broad gains across evaluation measures.Source: §5.3 · p. 6
  • BertSumExt and SummaRuNNer draw mostly from the source’s beginning, whereas TextRank broadens coverage slightly but incurs many Addition and Omission errors.Reference summaries use more of the article, with a high proportion from the first five sentences and a smooth tail across positions.Source: §5.3 · p. 7, Figure 2 · p. 7

5.4 Abstractive Methods

  • The copy mechanism reduces Inacc-Intrinsic errors from 304 to 14 and Duplication errors from 139 to 68, but introduces Addition errors through overly long copied sequences.Duplication reductions occur mostly at the word level, with nearly no sentence-level reduction.Source: §5.4 · p. 7
  • Coverage reduces Duplication errors from 68 to 11 and Omission errors from 286 to 256, but raises Addition and Inacc-Intrinsic errors.For outputs without Duplication errors, average PolyTope scores fall from 77.54 to 74.07; the authors suggest monotonically shifting attention can interfere with content selection.Source: §5.4 · p. 7
  • Bottom-Up achieves high ROUGE but ranks second worst on PolyTope, combining stronger recall with more Inaccuracy and weaker faithfulness.Restricting copy attention to extracted snippets leaves incomplete information sources for inference.Source: §5.4 · p. 7
  • BART ranks first among 10 models on both ROUGE and PolyTope, with joint encoder–decoder pre-training improving Addition, Omission, and Inaccuracy errors.Unlike encoder-only pre-training, BART jointly pre-trains its encoder and decoder with sequence-to-sequence denoising tasks; its source-sentence use is also closer to the reference distribution.Source: §5.4 · p. 8

6 Analysis of Evaluation Methods

The section compares ROUGE with PolyTope and other evaluation methods to assess their usefulness for system ranking and individual-summary quality. It finds that ROUGE correlates more strongly with PolyTope at the system level, while PolyTope provides finer-grained error information.

Section aim

  • The evaluation’s stated goal is to compare summarization systems, not to promote a human-evaluation metric.The authors contrast ROUGE and PolyTope quantitatively and PolyTope with other human-evaluation metrics qualitatively.Source: §6 · pp. 8–9

ROUGE and PolyTope

  • 0.78, 0.73, and 0.52 are the system-level Pearson correlations for ROUGE-1, ROUGE-2, and ROUGE-L with PolyTope, versus 0.40, 0.32, and 0.32 at the instance level.ROUGE is useful for ranking systems after aggregating samples but relatively weak for assessing individual summaries.Source: §6 · p. 9

Comparison with other evaluation methods

  • PolyTope provides finer-grained quality information than ROUGE, Pyramid, ranking, scaling, and QA in the illustrated document comparison.The comparison uses two models on one test document.Source: §6 · p. 9

Summary length

  • The passage reports corresponding PolyTope scores of 0.25 and 0.27, suggesting it is more objective and meaningful for current models that produce summaries without pre-specified length.Source: §6 · p. 9

Reference-summary evaluation

  • Reference summaries scored 96.41 overall, with 63 Accuracy errors and no Fluency errors, partly because PolyTope penalizes hallucinated content requiring knowledge beyond the document.Other cited examples include irrelevant rhetorical sentences and grammatical issues.Source: §6 · p. 9

7 Conclusion

The paper compares 10 representative summarizers with fine-grained MQM-based human-evaluation metrics to diagnose systems and milestone techniques. It concludes that extractive systems generally outperform abstractive systems in human evaluation, while copy, coverage, hybrid methods, and especially pre-training have distinct advantages.

7 Conclusion

  • Across 10 representative summarizers, extractive systems generally outperform abstractive systems in human evaluation, and the analysis identifies distinct benefits of copy, coverage, hybrid methods, and especially pre-training.The authors say their overall conclusions largely align with existing research while adding detail through error diagnostics.Source: §7 · p. 9

A Details on PolyTope

The appendix describes an Excel workbook with four interrelated sheets for annotating and scoring summarization errors. Its interface records error types and syntactic labels, while automatically calculating severity and scores.

A.1 Annotation Toolkit

  • The annotation rules are embedded in a macro-enabled Excel workbook containing Score Card, Error Log, Scores per Segment, and Severity Matrix sheets.Source: §A.1 · p. 13
  • The Score Card calculates error counts and scores to show a tested model’s overall performance, while Scores per Segment calculates word count, errors, and score for each sample.Source: §A.1 · p. 13
  • The Error Log presents source articles and summaries and lets annotators mark incorrect segments, select issue types and syntactic labels, and generate severity and scores automatically.Severity is generated in the Error Log; per-segment and overall scores are calculated in the corresponding sheets.Source: §A.1 · p. 13
  • The workbook embeds a predefined severity matrix in the Severity Matrix sheet.Source: §A.1 · p. 13

A.2 Issue Types

The appendix illustrates PolyTope issue types with examples and defines syntactic labels for locating errors in sentence components. These labels distinguish content elements such as subjects, predicates, numbers, names, and whole sentences.

A.2 Issue Types

  • Inaccuracy Intrinsic includes errors such as reporting a source distance of 10 kilometers as 3 kilometers, while Inaccuracy Extrinsic includes unsupported names absent from the source and real world.Source: §A.2 · p. 13
  • Positive-Negative Aspect captures added or omitted negative particles in actions and modifiers, such as changing “push a button” to “don’t push a button.”Source: §A.2 · p. 14

A.3 Syntactic Label

  • The syntactic labels define Subject as the event body, Predicate as verbs or related forms representing actions or state changes, and Object as the noun phrase affected by a predicate.A verb-like form acting as a modifier is labeled Attribute.Source: §A.3 · p. 14
  • Number&Time covers numeric forms and temporal expressions, while Place&Name covers geographic or administrative locations, addresses, and named entities.Source: §A.3 · p. 14
  • Attribute labels units that modify nouns; Function Word covers items such as prepositions and articles; Whole Sentence is a complete set of words.Source: §A.3 · p. 14

B Details on Human Evaluation

The evaluators were trained and kept unaware of model identities and techniques, then manually assessed summaries with cross-checking. Select models were reproduced from their papers, while outputs for others came from public sources.

B Details on Human Evaluation

  • Three linguistically qualified candidates completed training and testing, then annotated 50 instances each, with cross-checking for 1,500 examples overall.Each instance contained the source document and 10 model-generated summaries.Source: B Details on Human Evaluation · p. 14
  • Annotators were naive to model names, ROUGE scores, architectures, and techniques, and pilot-study documents were excluded from final annotation.Two pilot studies familiarized annotators with PolyTope and text summaries.Source: B Details on Human Evaluation · p. 14
  • TextRank, SummaRuNNer, and BART were implemented following their papers; other models’ summaries came from publicly available sources.The implemented models achieved their reported ROUGE scores and generated summaries on 150 test trials.Source: B Details on Human Evaluation · p. 14

C Case Study

The case study illustrates factual and content-selection problems in summarization models, including limits to coverage and hybrid approaches. The accompanying sections describe a sentence-similarity analysis of layout bias and report Pearson correlations between ROUGE-P and PolyTope.

C Case Study

  • Abstractive methods can splice source fragments together, producing factual errors in BertSumExtAbs, Pointer-Generator-with-Coverage, and Bottom-Up outputs.The examples are shown in Figures 7–9.Source: C Case Study · p. 14
  • For Pointer-Generator outputs without duplication errors, adding coverage may interfere with content selection and introduce new problems.The examples include addition errors and inaccurate intrinsic errors from incorrectly combined document information.Source: C Case Study · p. 14
  • The hybrid model has high ROUGE scores overall but does not necessarily combine the strengths of extractive and abstractive methods.Figure 12 provides an example.Source: C Case Study · p. 14

D Details on Layout Bias Calculation

  • The layout-bias analysis computes BERTSCORE similarity between each output-summary sentence and each source-document sentence, then illustrates the distribution of source sentences used.The passage notes that news summarizers typically favor leading paragraphs, where articles present salient information.Source: D Details on Layout Bias Calculation · p. 15

E Pearson Correlation

  • The Pearson-correlation analysis compares ROUGE-P with PolyTope measurements and lists its results in Table 5.The passage motivates ROUGE-P because its precision measure might correlate with the error-based PolyTope framework; the supplied table caption gives no correlation values.Source: E Pearson Correlation · p. 15, Table 5 · p. 15

F Slips in Reference

The study uses CNN/DM to assess summarization, while noting that some reference summaries contain errors or irrelevant material. Its examples also show factual-error cases and contrasting outputs from Pointer-Generator models.

F Slips in Reference

  • CNN/DM provides news articles with associated highlights, usually three sentences long, as a benchmark for comparing extractive and abstractive systems.The paper selects it because the reference length is considered closer to real-world applications and more comprehensive for analysis than single-sentence summaries.Source: F Slips in Reference · p. 15
  • Some CNN/DM references contain grammatical errors, noise such as “Click here for...”, or rhetorical sentences with little information needed to understand the document.The paper says automatic evaluation is unreliable in the noise and rhetorical-sentence cases.Source: F Slips in Reference · p. 15
  • The examples identify factual errors in BertSumExtAbs, Pointer-Generator-with-Coverage, and Bottom-Up summaries, and show different Pointer-Generator outputs for the same source.The figure captions label errors in the three models; the paired outputs show distinct content selections by Pointer-Generator and its coverage variant.Source: Figure 7 · p. 16, Figure 8 · p. 17, Figure 9 · p. 18
  • The Pointer-Generator-with-Coverage model tends to make addition errors when Pointer-Generator does not repeat content, and may combine document information incorrectly, producing intrinsic inaccuracies.These tendencies are described in Figures 10 and 11.Source: Figure 11 · p. 20, Figure 10

Related papers

Loading 2010.04529v1…