PaperlayerView exact revision on arXiv

Source-linked AI summary

News Summarization and Evaluation in the Era of GPT-3

Tanya Goyal, Junyi Jessy Li, Greg Durrett

arXiv:2209.12356v2cs.CL

TL;DR

This paper examines whether prompt-based GPT-3 can match or surpass fine-tuned models for news summarization and whether established evaluation practices remain reliable. Through systematic comparisons across generic and specialized summarization settings, it finds strong human preference for GPT-3 but ineffective automatic metrics, while identifying future use cases and scope boundaries.

  • Problem

    Summarization research has relied heavily on large-scale training data and gold test sets, limiting realistic task formulations and potentially misaligning benchmarks with user preferences.

  • Method

    The paper systematically compares prompt-based GPT-3 with fine-tuned summarization models, studies evaluation practices, and releases generated summaries and human preference judgments.

  • Results

    GPT-3 summaries outperform prior models in human evaluation despite scoring 7 ROUGE-L points lower on average, and annotators prefer GPT-3 for keyword-based summaries 70% of the time.

  • Takeaways & Limitations

    Prompt-based models can let realistic use cases drive summarization research beyond generic news summaries, including keyword-focused, update, plan-based, and aspect-based settings.

  • Takeaways & Limitations

    The experiments use English-language news datasets, and user rankings may differ in other domains, where domain-specific fine-tuning may be preferred.

Abstract

from arXiv · show

The recent success of prompting large language models like GPT-3 has led to a paradigm shift in NLP research. In this paper, we study its impact on text summarization, focusing on the classic benchmark domain of news summarization. First, we investigate how GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, prompted using only a task description, but these also do not suffer from common dataset-specific issues such as poor factuality. Next, we study what this means for evaluation, particularly the role of gold standard test sets. Our experiments show that both reference-based and reference-free automatic metrics cannot reliably evaluate GPT-3 summaries. Finally, we evaluate models on a setting beyond generic summarization, specifically keyword-based summarization, and show how dominant fine-tuning approaches compare to prompting. To support further research, we release: (a) a corpus of 10K generated summaries from fine-tuned and prompt-based models across 4 standard summarization benchmarks, (b) 1K human preference judgments comparing different systems for generic- and keyword-based summarization.

1 Introduction

This paper studies whether prompt-based GPT-3 can compete with fine-tuned summarization models and whether established evaluation metrics remain reliable for its outputs. It also examines prompting beyond generic summarization.

  • Motivation: Fine-tuning has dominated summarization, but adapting models to new domains or styles requires sizeable task-specific datasets.Prompting offers an alternative based on natural-language task instructions without updating model parameters.
  • Research Questions: GPT-3 summaries were overwhelmingly preferred by human annotators over state-of-the-art fine-tuned models across three- and single-sentence styles.The summaries also avoided limitations associated with low-quality training data in fine-tuned generic summarization models.
  • Evaluation: 7 ROUGE-L points lower on average did not prevent GPT-3 summaries from outperforming prior state-of-the-art models in human evaluation.Reference-based and reference-free metrics similarly failed to adapt to prompt-based summaries.
  • Beyond Generic Summarization: GPT-3 summaries were preferred 70% of the time over a baseline in keyword-based summarization, while aspect-based prompting produced mixed results with frequent failures.Keyword-based summaries were judged more coherent and keyword-relevant than the fine-tuned alternative.
  • Implications: The evidence suggests that prompting changes both the data requirements and the evaluation framework for summarization research.The paper argues that automatic metrics dominating the field's last decade are insufficient for evaluating these systems.

2 Models and Setup

The paper compares representative fine-tuned, zero-shot, and instruction-tuned summarization systems, using GPT3-D2 as the prompt-based case study. It adapts prompts for dataset styles and controls output length with sentence-count instructions.

  • 2.1 Current Paradigms for Summarization: Fine-tuned systems rely on large article-summary datasets, whereas zero- and few-shot models use prompts without explicit task-specific training.Instruction-tuned models fine-tune pretrained language models on multiple tasks using instruction templates aligned with inference.
  • 2.1 Current Paradigms for Summarization: The study compares GPT3-D2, BRIO, and T0 as representative prompt-based, task-specific fine-tuned, and multitask instruction-tuned approaches.BRIO is evaluated using versions fine-tuned on CNN/DM and XSum.
  • 2.1 Current Paradigms for Summarization: GPT3-D2 denotes OpenAI's Instruct-series text-davinci-002, selected to benchmark the best available prompt-based model.Its exact training details were not known, and the preceding release was trained on API and labeler prompts spanning multiple tasks.
  • 2.2 Using GPT3-D2 for summarization: GPT3-D2 prompts were adapted with sentence-count instructions to align outputs with dataset-specific summary lengths and styles.The study did not attempt to control other dataset attributes, such as CNN/DM lead bias or XSum's whole-article inference requirement.
  • 2.2 Using GPT3-D2 for summarization: 98% of GPT3-D2 summaries followed the requested length constraint in the human-study test instances.This control enabled generation of summaries with different lengths to emulate different datasets.

3 Human evaluation of GPT3-D2 summaries

Human evaluations compare GPT3-D2 with fine-tuned and prompted alternatives on CNN and BBC-style news summarization. Annotators prefer GPT3-D2 across styles, while the systems differ in abstractiveness, specificity, factuality, and adaptability.

  • 3.1 Experimental Setup: The study uses 100 recent CNN and BBC articles, with GPT3-D2 prompted for three sentences on CNN and one sentence on BBC.The comparison includes BRIO, T0, and GPT3-D2, with majority votes combining three annotators’ choices.
  • 3.2 Results: GPT3-D2 summaries are more abstractive and less specific than BRIO summaries for CNN, while covering more of the article content.Abstractiveness is measured by the percentage of novel n-grams, and specificity by named entities per 100 words.
  • 3.2 Results: For BBC, BRIO and T0 are more abstractive than GPT3-D2, whereas GPT3-D2 maintains similar abstractiveness across datasets.GPT3-D2 also tends to produce longer sentences, so equal sentence counts can yield longer summaries.
  • 3.2 Results: GPT3-D2 summaries were preferred over the next-best model by at least 20 percentage points in both CNN and BBC scenarios.The differences were statistically significant: CNN p−value = 2 × 10−3 and BBC p−value = 6 × 10−4.
  • 3.2 Results: Annotators often cited misleading or incorrect information in BRIO summaries for BBC, consistent with known problems in XSum-trained models.For CNN, T0 was most often rated worst because of shorter length, irrelevant details, missing key points, and lower coherence.
  • 3.2 Results: GPT3-D2 was unanimously preferred by all three annotators for fewer than 30% of articles, indicating disagreement even when aggregate preferences favored it.BRIO or T0 was selected as best by at least one annotator for more than 60% of articles, depending on the dataset.

4 Can current automatic metrics evaluate GPT3-D2 summaries?

The paper tests whether reference-based and reference-free automatic metrics can reliably evaluate GPT3-D2 summaries across four datasets. Both metric families fail to reproduce human preference rankings, with reference-based scores favoring fine-tuned systems and reference-free scores misjudging GPT3-D2 factuality and abstractiveness.

  • Experimental Setup: Four datasets and three summarization systems are evaluated with reference-based and reference-free automatic metrics.The evaluation samples 500 articles per dataset and includes BRIO, T0, GPT3-D2, and PEGASUS where available.
  • Reference-based metrics: Dataset-specific training improves metric scores: T0 beats GPT3-D2 on every dataset except Newsroom, the only dataset absent from T0 training.This pattern suggests that reference-based metrics reward systems trained to emulate dataset-specific references.
  • Reference-free metrics: None of the evaluated reference-free metrics reproduces human preference rankings across both CNN/DM and XSum.GPT3-D2 also receives low factuality scores except on XSum, although qualitative analysis rarely found factual errors.
  • Reference-free metrics: Reference-free metrics generally score more abstractive systems lower, revealing spurious correlations that worsen when systems have substantially different properties.GPT3-D2 is scored below BRIO by both quality metrics on all datasets except XSum, where GPT3-D2 is less abstractive.
  • Discussion: Reference-free metrics remain constrained by training and evaluation practices developed around reference-based benchmarks and fine-tuned models.The authors argue that metric components and design choices must be revisited for GPT3-D2 evaluation.

5 Beyond Generic Summarization

Beyond generic summarization, the paper compares GPT3-D2 with CTRLSum on keyword- and aspect-based tasks. GPT3-D2 produces more coherent, contextually useful keyword summaries and wins human comparisons, but both systems perform poorly or inconsistently on aspect-based summarization.

  • Task Definitions: Keyword-based summaries focus succinctly on named entities or events, whereas aspect-based summaries address high-level topics spanning related documents.The paper evaluates GPT3-D2 on both settings because users may need different information from the same article.
  • Baseline and Controls: CTRLSum is a fine-tuned BART baseline adapted to keyword- and aspect-based summarization through prompted inputs.GPT3-D2 uses named entities as keyword controls and TAC 2011 aspects for aspect-focused summarization.
  • Qualitative Analysis: GPT3-D2 keyword summaries provide article context before keyword-specific information, whereas CTRLSum summaries have poor discourse structure and read like lists of facts.Figure 6 presents qualitative examples of both systems.
  • Qualitative Analysis: GPT3-D2 produces mixed aspect-focused results: one example is factually incorrect, while another correctly maps a high-level aspect to the relevant entity.CTRLSum fails in both examples, apparently focusing on lexically similar words such as “defenders.”
  • Human Evaluation: Nearly 70% of article-keyword pairs receive majority preference for GPT3-D2 over CTRLSum.Annotators cite better contextualization of keyword-related information and better coherence as the main reasons.
  • Impact: GPT3-D2 is presented as a promising alternative for specialized summarization tasks that can be described with textual prompts, while aspect summarization is left for future work.The authors attribute the latter boundary to mixed-to-poor results for both models.

6 Discussion and Related Work

The discussion argues that GPT-3 challenges evaluation practices built around gold summaries and automatic metrics. It also suggests that prompting can let practical use cases guide summarization research, while human evaluation remains difficult to standardize and scale.

  • Evaluation: GPT-3 makes established automatic evaluation protocols ineffective and creates an urgent need for a distinct evaluation framework.The paper reports that both reference-based and reference-free metrics fail when evaluating GPT-3 summaries.
  • Manual Evaluation: Manual evaluation is considered the gold standard, but inconsistent dimensions and task designs hinder reliable, reproducible system comparisons.Recent standardization efforts do not eliminate affordability and quality-monitoring problems.
  • Research Scope: Dependence on large-scale training data has constrained summarization research to problems and domains supported by available datasets.The paper gives bullet-point news summaries as an example shaped by CNN data availability.
  • Future Directions: Prompt-based models can allow realistic use cases to drive summarization research in a more top-down way.The paper identifies update, plan-, and aspect-based summarization as future directions, alongside longer documents and structured inputs.

7 Conclusion

The paper systematically compares prompt-based GPT-3 and fine-tuned models for news summarization, examines training and evaluation practices, and releases resources for further research.

  • The study compares prompt-based GPT-3 with fine-tuned models on news summarization.
  • It analyzes how prompting affects summarization training paradigms and evaluation practices.
  • The authors release generated summaries from prompt-based and fine-tuned models, together with human preference judgments comparing these systems.

8 Limitations

The paper identifies limitations in automatic and manual evaluation practices, scope, and assumptions about GPT3-D2 training. These constraints motivate caution when generalizing the findings or evaluating new systems.

  • There is no standardized task design for comparing different system generations.
  • The chosen direct-comparison workflow may provide fewer insights into which summary-style properties distinguish GPT3 summaries than multidimensional evaluation.
  • The experiments use English-language news datasets, so user rankings may differ for other domains such as scientific summarization.
  • The exact datasets and tasks used to train GPT3-D2 are unknown, although the paper’s arguments do not depend on those specifics.

A Implementation Details

The implementation details specify prompts, factuality-metric versions, and keyword extraction constraints used to support reproducible experiments. The keyword study balances lead-biased and non-lead article content.

  • Prompts Used: GPT3-D2 prompts used N = 3 for CNN and DailyMail, N = 2 for Newsroom, and N = 1 for XSum/BBC.For XSum/BBC, the prompt was modified to request one sentence.
  • Prompts Used: T0 prompts requested 3 to 4 sentences for CNN/DM, 2 to 3 for Newsroom, and 1 sentence for XSum/BBC.
  • Factuality Metrics: The experiments used QuestEval 0.2.4 with precision-only scoring, updated document-level DAE, and SummaC-Conv with sentence-level granularity.
  • Figure 8 plots summary-length differences against annotator-score differences between GPT3-D2 and the next-best system.
  • Keyword-based data: The keyword-based study extracted two named entities per article, constraining the first to the opening three sentences and the second to the remaining article.This setup benchmarks both lead-biased and non-lead settings.

B Are annotator judgments of quality correlated with length?

The paper examines whether annotator preferences for GPT3-D2 over other systems are explained by summary length. It finds little correlation and suggests that readability and information density may also matter.

  • GPT3-D2 summaries are on average 9 words longer than BRIO summaries.
  • Pearson’s ρ between length difference and annotator-score difference is 0.17 for CNN and .02 for BBC.These correlations do not solely explain the large differences in annotator judgments.
  • GPT summaries were judged slightly less information dense but easier to read and understand despite being more verbose.
  • GPT3-D1 and GPT3-D2 produced similar high-content, high-overlap BBC single-sentence summaries, while CNN-style outputs showed more variance.
  • The study generated around 2600 GPT3-D2 summaries across the human, metric-evaluation, and keyword-based experiments.

E Long document summarization using GPT3-D2

Naive GPT3-D2 prompting produces strong summaries for individual segments of long documents, but concatenating those outputs yields incoherent, repetitive coverage. The final quality depends on segmentation, and prompting alone does not readily fix the resulting lead bias and focus errors.

  • Method: GPT3-D2 summarizes each of three disjoint article segments separately before concatenating the outputs into one final summary.This follows the segment-then-summarize procedure used in prior work.
  • Results: Although individual segment summaries are high quality, their concatenation is incoherent and repeats similar introductory content.The combined summary also fails to cover all important aspects of the article because much of its length budget goes to a high-level overview.
  • Limitations: Long-document outputs often focus on less important content because segmentation preserves GPT3-D2’s lead bias toward the beginning of each input segment.The example includes attention to opening windows for fresh air rather than the article’s most important aspects.
  • Limitations: The exact segmentation dictates final-summary quality and cannot be readily corrected by changing the prompt.This makes the document-partitioning choice a central constraint of the approach.
  • Implications: Segment-then-summarize and extract-then-abstract techniques effective for fine-tuned models are less effective when directly applied to prompting-based models.GPT3-D2 produces superior segment-level summaries but is harder to adapt to non-natural text inputs without fine-tuning.
Loading 2209.12356v2…