PaperlayerView exact revision on arXiv

Source-linked AI summary

Summarization is (Almost) Dead

Xiao Pu, Mingqi Gao, Xiaojun Wan

arXiv:2309.09558v1cs.CL

TL;DR

The paper questions whether strong LLM performance changes the value of further refinement of summarization models. Using new datasets and human evaluation across five common tasks, it finds human evaluators prefer LLM summaries and judge them more factual, while identifying dataset quality and practical evaluation as continuing priorities.

  • Problem

    The authors question the need for further refinement of summarization models with higher metric scores given LLMs’ performance, noting that about 70% of sampled recent papers proposed and validated summarization approaches on standard datasets.

    Source: §1 · p. 1

  • Method

    The study develops new evaluation datasets and conducts human experiments comparing zero-shot LLM summaries with human-written and fine-tuned summaries across five common summarization tasks.

    Source: §2.1 · p. 1, §6 · p. 4, Limitations · p. 5

  • Results

    LLM summaries are significantly preferred by human evaluators and show higher factuality; they consistently outperform human and fine-tuned summaries across all tasks.

    Source: §1 · p. 1, §3.1 · p. 2

  • Takeaways & Limitations

    The authors argue that prior summarization approaches are less meaningful given LLM advances, while high-quality datasets and more practical evaluation remain worth pursuing.

    Source: §1 · p. 1, §6 · p. 4, §4.3 · p. 4

    Because of human-experiment costs, the study covers only five common tasks, leaving less common and more challenging tasks such as slide summarization for future work.

    Source: Limitations · p. 5

Abstract

from arXiv · show

How well can large language models (LLMs) generate summaries? We develop new datasets and conduct human evaluation experiments to evaluate the zero-shot generation capability of LLMs across five distinct summarization tasks. Our findings indicate a clear preference among human evaluators for LLM-generated summaries over human-written summaries and summaries generated by fine-tuned models. Specifically, LLM-generated summaries exhibit better factual consistency and fewer instances of extrinsic hallucinations. Due to the satisfactory performance of LLMs in summarization tasks (even surpassing the benchmark of reference summaries), we believe that most conventional works in the field of text summarization are no longer necessary in the era of LLMs. However, we recognize that there are still some directions worth exploring, such as the creation of novel datasets with higher quality and more reliable evaluation methods.

1 Introduction

The paper examines whether LLMs can generate strong zero-shot summaries across varied tasks. Human evaluations find LLM summaries preferred over human-written and fine-tuned summaries, motivating a reassessment of conventional summarization research while retaining needs for better data and evaluation.

1 Introduction

  • The study evaluates zero-shot LLM summarization across single-news, multi-news, dialogue, code, and cross-lingual tasks using human-generated evaluation datasets.It compares LLM summaries with human-written summaries and summaries from fine-tuned models.Source: §1 · p. 1
  • Human evaluators significantly prefer LLM-generated summaries, which also demonstrate higher factuality than human-written and fine-tuned summaries.Source: §1 · p. 1
  • About 70% of sampled summarization papers from the previous three years primarily proposed an approach and tested it on standard datasets, prompting the authors to question further model refinement.The authors still identify high-quality reference datasets, application-oriented approaches, and improved evaluation as worthwhile directions.Source: §1 · p. 1

2 Experimental Settings

The experiments use recent datasets and human pairwise judgments to compare LLMs with task-specific fine-tuned models across five summarization tasks. The setup aims to limit training-data overlap and reports acceptable annotator agreement.

2 Experimental Settings

  • The section outlines the datasets, models, and human-evaluation procedure used in the experiments.Source: §2 · p. 1

2.1 Datasets

  • Each task dataset contains 50 recent samples selected to reduce the chance that LLMs saw the data during training.The datasets follow task-specific prior construction approaches, use current sources, and include Go programs for code summarization; cross-lingual references are translated from English into Chinese and post-edited.Source: §2.1 · pp. 1–2

2.2 Models

  • GPT-3, GPT-3.5, and GPT-4 represent LLMs, compared with one or two task-specific fine-tuned models for each task.The selected fine-tuned models include BART and T5 for single-news, Pegasus and BART for multi-news, T5 and BART for dialogue, MT5 and MBART for cross-lingual, and CodeT5 for code.Source: §2.2 · p. 2

2.3 Experimental process and details

  • For a single task, each annotator completes all 50 questions, comparing summaries pairwise after seeing each question’s source article and summaries from all systems selected for that task; overall inter-annotator agreement is acceptable (Cohen’s kappa 0.558).Source: §2.3 · p. 2

3 Experiment Results

Across tasks, human evaluators prefer LLM summaries over human-written and fine-tuned alternatives. Hallucination comparisons find GPT-4 summaries at least as factually consistent as human references, while comparative analysis identifies fluency and topic-coverage differences.

3.1 Experiment 1: Comparing the overall quality of summaries

  • LLM summaries consistently outperform human-written and fine-tuned summaries in pairwise human preferences across all tasks.WinRate represents the proportion of comparisons in which one system is preferred over another.Source: §3.1 · p. 2

3.2 Experiment 2: Comparing the factual consistency of summaries

  • Human references have equal or more sentence-level hallucinations than GPT-4 summaries, with notably worse factual consistency in multi-news and code summarization.Table 1 reports sentence-level hallucination counts for GPT-4 and human-written summaries.Source: §3.2 · p. 3, Table 1 · p. 3
  • Extrinsic hallucinations are more common in human summaries on tasks with weaker factual consistency, while their proportions are similar when factuality differs little.The authors hypothesize that extrinsic hallucinations underlie inadequate factual consistency in human-written summaries.Source: §3.2 · p. 3, Table 2 · p. 3

3.3 Comparative Analysis

  • Human-written references can be less fluent, incomplete, or hallucinated compared with LLM summaries.Source: §3.3 · p. 3
  • Fine-tuned summaries tend toward rigid lengths and lower coverage of multiple input topics, whereas LLMs adjust length and capture all topics.Source: §3.3 · p. 3

4 The Changing Landscape of Summarization: Seeking New Horizons

The authors argue that LLM progress is making many domain-specific summarization approaches less meaningful, while identifying datasets, application-oriented methods, and evaluation as directions for future work.

4.1 Summarization Datasets

  • Future datasets should shift from model training to testing, use expert-annotated references, and cover more genres, languages, and longer documents.The authors particularly call for low-resource languages and books.Source: §4.1 · p. 4

4.2 Summarization Approaches

  • Application-oriented directions include customized summaries for user preferences, real-time summarization, and interactive systems that solicit clarification or feedback.The authors also call for improving the promptness and efficiency of real-time LLM summarization.Source: §4.2 · p. 4

4.3 Summarization Evaluation

  • The authors call for moving beyond ROUGE toward LLM-based automated evaluation and greater attention to practical utility and applications.Extrinsic evaluation measures a summary by using it in another task, such as question-answering or decision-making.Source: §4.3 · p. 4

5 Related Work

Prior work found human preference for GPT-3 summaries over fine-tuned outputs and results on par with human-written summaries, while other studies considered automatic evaluation in additional settings.

  • Prior studies found that humans overwhelmingly preferred GPT-3 news summaries to fine-tuned model summaries, while another study rated LLM news summaries on par with human-written ones.Source: §5 · p. 4
  • Other studies used automatic evaluation to examine ChatGPT on aspect-based and cross-lingual summarization.Source: §5 · p. 4
  • This paper evaluates GPT-3.5 and GPT-4 with human evaluations on data their models had not been trained on.Source: §5 · p. 4

6 Conclusion

The authors report strong LLM-generated summaries across diverse tasks and outline future research directions in datasets, methods, and evaluation.

  • LLM-generated summaries showed strong performance against human-written references and fine-tuned model summaries across diverse tasks, with particular strengths in fluency, factuality, and flexibility.Source: §6 · p. 4
  • The authors argue that LLM advances are making previous summarization approaches less meaningful and identify datasets, methods, and evaluation as areas for future work.Source: §6 · p. 4
  • The study develops evaluation datasets and conducts comprehensive human evaluations across a wide range of summarization scenarios.Source: §6 · p. 4

Limitations

The limitations concern which LLMs could be evaluated and the range of tasks included in the human experiments.

  • The study excludes LLaMA and Vacuna because their training-data cutoff dates are undisclosed, making it challenging to create novel zero-shot evaluation datasets for them.The authors say more training-data details would support adding models and comparing model series.Source: Limitations · p. 5
  • High costs limited human experiments to five common summarization tasks; the authors suggest less common, more challenging tasks such as slide summarization for future study.Source: Limitations · p. 5

Ethics Statement

The dataset uses publicly available website text; annotators are paid above the local minimum wage, and the dataset is intended for research use.

Ethics Statement

  • Annotators receive 10 USD per hour, above the local minimum wage; publicly sourced texts may include biased, offensive, or violent content.The authors state that inclusion of such content does not reflect their research standpoint or endorse those views; the dataset is intended only for research.Source: Ethics Statement · p. 5

A Case Studies

The case studies contrast flaws in human-written references with topic-coverage limits in fine-tuned summaries. GPT-4 provides more complete information in the illustrated examples.

A Case Studies

  • Human-written references can omit information, lack fluency, and hallucinate facts, while GPT-4 summaries avoid these illustrated errors.One reference incorrectly specifies “three months” although the source mentions two incidents occurring within several months.Source: A Case Studies · p. 7
  • For a four-event source, fine-tuned BART covers only the first event, whereas GPT-4 summarizes all four at a similar length.The case study illustrates fine-tuned summaries' tendency to focus on only a subset of topics in long articles.Source: A Case Studies · p. 7

B Human preference score

The human preference score measures a system’s overall winning rate against other systems in annotator comparisons. Across five tasks, large-model scores exceed 50%, indicating strong preference for their summaries.

B Human preference score

  • The human preference score is a system’s overall winning rate when annotators compare it with other systems.The score indicates relative system quality within each task.Source: B Human preference score · p. 7
  • Across five tasks, large-model human preference scores exceed 50%, indicating strong evaluator preference for their summaries.Figure 4 presents preference scores for different summary systems across the five tasks.Source: B Human preference score · p. 7, Figure 4 · p. 9

Related papers

Loading 2309.09558v1…