Source-linked AI summary
Benchmarking Large Language Models for News Summarization
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, Tatsunori B. Hashimoto
TL;DR
LLM summarization succeeds for reasons that remain unclear, and existing evaluations rely on potentially poor references. The paper conducts human evaluation across ten LLMs and collects higher-quality freelance summaries, finding instruction tuning more important than model size and the best LLM comparable to human writers. It also shows that reference quality constrains comparisons among zero-shot, few-shot, and finetuned systems.
Problem
The reasons for LLM summarization success and the comparability of LLM outputs with human writing remain unclear, while benchmark references are low quality.
Method
The paper conducts human evaluation of ten LLMs and compares them with higher-quality summaries collected from freelance writers.
Results
Instruction tuning, rather than model size, is the key factor in zero-shot summarization, and the best LLM is rated comparable to freelance writers despite stylistic differences.
Takeaways & Limitations
Reference quality is crucial for evaluating and developing summarization systems, because poor references leave comparisons among zero-shot, few-shot, and finetuned performance open.
Takeaways & Limitations
As LLMs approach human-level performance, human evaluation requires more samples and less noisy measurement, while annotator preferences can vary.
Abstract
from arXiv · showhide
Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales, we make two important observations. First, we find instruction tuning, and not model size, is the key to the LLM's zero-shot summarization capability. Second, existing studies have been limited by low-quality references, leading to underestimates of human performance and lower few-shot and finetuning performance. To better evaluate LLMs, we perform human evaluation over high-quality summaries we collect from freelance writers. Despite major stylistic differences such as the amount of paraphrasing, we find that LMM summaries are judged to be on par with human written summaries.
1 Introduction
The paper investigates why LLMs succeed at news summarization and whether their outputs match human-written summaries. It identifies instruction tuning as central, exposes low-quality references, and finds the best LLM comparable to freelance writers despite stylistic differences.
- 1 Introduction: The study evaluates ten diverse LLMs with human judgments to examine summarization design choices and comparability with human writers.The evaluation focuses on news summarization.
- 1 Introduction: Instruction tuning, rather than model scale, is identified as the key factor in zero-shot summarization capability.Self-supervised learning alone cannot induce strong zero-shot summarization performance.
- 1 Introduction: Low-quality CNN/DM and XSUM reference summaries undermine human and automatic evaluation and degrade finetuning and few-shot supervision.Human annotators judge the references worse than most automatic-system outputs.
- 1 Introduction: Freelance writers re-annotate 100 CNN/DM and XSUM test articles to provide higher-quality summaries for comparison.The summaries are collected from Upwork writers.
- 1 Introduction: Instruct Davinci is more extractive and paraphrases less than freelance writers, while still combining copied segments coherently.The comparison uses manually annotated summarization operations.
- 1 Introduction: Instruct Davinci is rated comparable to freelance writers overall, although individual annotators show stable preferences for one or the other.This result holds despite major stylistic differences.
2 Background and Related Work
News summarization produces concise paragraphs capturing an article’s main points, and this paper benchmarks LLMs on CNN/DM and XSUM. The background emphasizes LLM scale, prompting, instruction tuning, and the need for human-centered evaluation.
- 2.1 News Summarization: News summarization aims to produce a concise paragraph capturing the main points of a news article.It is a core problem in automatic summarization.
- 2.1 News Summarization: The benchmark focuses on CNN/DM and XSUM, large-scale datasets created through incidental supervision.CNN/DM uses articles and website bullet-point highlights from CNN and DailyMail.
- 2.1 News Summarization: Automatic summarization metrics do not always correlate with human judgments, motivating careful evaluation of model performance.The paper compares LLMs with prior supervised-finetuning approaches.
- 2.2 Large Language Models: LLMs differ from earlier pretrained models through larger parameter and data scale and the ability to solve tasks through zero-shot or few-shot prompting.Few-shot prompting provides input-output examples for in-context learning.
- 2.2 Large Language Models: Instruction tuning reformulates diverse language tasks into prompting formats and updates model parameters through supervised finetuning or reinforcement learning.It has emerged as an effective way to improve prompting performance.
- 2.2 Large Language Models: The paper extends prior work with a ten-model benchmark examining model scale, in-context learning, and instruction tuning using human evaluation.Human evaluation is emphasized because automatic metrics may be unreliable.
3 Human Evaluation on News Summarization Benchmarks
Human evaluation across CNN/DM and XSUM benchmarks finds instruction tuning more important than model scale for zero-shot summarization, while current references distort evaluation. The study also examines automatic metrics and qualitative differences between model outputs and references.
- Experimental Setup: Ten LLMs are benchmarked on CNN/DM and XSUM using human ratings of faithfulness, coherence, and relevance.Each summary is evaluated by three annotators; coherence and relevance use 1–5 scales, while faithfulness is binary.
- Evaluation Results: Instruction-tuned models perform best overall, and scale is less important: Instruct Ada outperforms the 175B GPT-3 model on coherence and relevance.Instruct Davinci also exceeds fine-tuned models on selected CNN/DM and XSUM criteria.
- Evaluation Results: Five-shot prompting improves non-instruction-tuned LLMs, which can match instruction-tuned models on some aspects but remain worse on others.The gains occur for CNN/DM faithfulness and XSUM coherence, while instruction-tuned models retain advantages elsewhere.
- Evaluation Results: Current benchmark references are judged extremely low quality, and conditioning on them can worsen instruction-tuned summaries.On XSUM, Instruct Davinci faithfulness falls from 0.97 to 0.77 after conditioning on five reference summaries.
- Evaluation Results: Qualitative comparisons show instruction-tuned summaries are more faithful and coherent than non-instruction-tuned outputs, whereas CNN/DM references can lack coherence.The reference format originated as bullet-point highlights rather than coherent paragraphs.
- Understanding Automatic Metrics: Reference quality affects automatic-metric validity: Rouge-L reaches 0.72 Kendall’s tau with CNN/DM relevance, while correlations are very low for several XSUM aspects.Across datasets, reference-based metrics correlate better on aspects where references receive better human scores.
4 Comparing the Best LLM to Freelance Writers
The study compares Instruct Davinci with freelance-written summaries using higher-quality human references, finding major stylistic differences but comparable overall evaluations. It also shows that better references improve the relationship between Rouge-L and human ratings.
- 4.1 Experimental Setup: Six freelance writers were recruited through Upwork, selected using faithfulness, coherence, and relevance, and instructed to write approximately 50-word news summaries.The study used 50 CNN/DM articles and 50 XSUM articles, assigning each article to three writers.
- 4.1 Experimental Setup: To control length, Instruct Davinci was prompted with a 25-word limit because it otherwise exceeded the requested limit and produced summaries averaging about 50 words.The freelance writers were given the same approximately 50-word target.
- 4.1 Experimental Setup: Freelance-writer summaries received much higher quality ratings than the original CNN/DM and XSUM references, while their difference from Instruct Davinci was small.The quality-control evaluation used Mechanical Turk annotators and compared the freelance summaries with averaged Instruct Davinci and reference results.
- 4.2 Paired Comparison between LLM and Freelance Writers: Instruct Davinci summaries were more extractive than freelance-writer summaries, with coverage and density of 0.92 and 12.1 versus 0.81 and 2.07.Freelance writers used more paraphrasing, generalization/specification, and sentence reduction, whereas Instruct Davinci copied more directly.
- 4.2 Paired Comparison between LLM and Freelance Writers: Annotators equally preferred freelance-writer and Instruct Davinci summaries overall, but individual preferences varied substantially and showed low interannotator agreement.The more abstractive summaries were judged more informative only 51.1% of the time, with Krippendorff’s alpha of 0.07 for overall preference.
- 4.3 Reevaluating Reference-based Metrics: Using freelance-writer summaries instead of existing XSUM references changed Rouge-L’s relationship with human ratings from negative to much more positive.The result indicates that reference-based metric usefulness is closely linked to reference quality.
5 Discussion
The discussion identifies instruction-tuning data, learning algorithms, and multitask learning as possible contributors to summarization performance, while highlighting unresolved evaluation limitations. Low-quality references and subjective judgments complicate comparisons as LLMs approach human-level performance.
- Implication for model development: Instruction-tuning data quality may explain weak finetuned-model performance, but higher-quality data could produce better results.The authors cannot rule out substantially stronger performance from models finetuned on higher-quality data.
- Implication for model development: Learning algorithms may matter because learning from human feedback uses preference labels rather than written summaries.The authors hypothesize that this may help exploit discrepancies between how annotators write and rate summaries, but more analysis is needed.
- Implication for model development: Multitask learning is another possible contributor because Instruct Davinci is trained on diverse inputs and prior work supports multitask learning.The authors leave open how learning on other tasks benefits summarization.
- Implication for Summarization Evaluation: Reference quality limits evaluation: freelance-writer summaries may not outperform LLM summaries significantly, making ground-truth references overly restrictive.As LLMs approach or exceed average human performance, reference-based evaluation becomes harder to interpret.
- Implication for Summarization Evaluation: Subjective evaluation further limits reliability because annotators show stable individual preferences among multiple acceptable summaries.The authors argue that human evaluation requires more samples and less noisy measurement as LLMs approach human-level performance.
- Implication for Summarization Evaluation: Downstream applications may provide better evaluation contexts by defining user values more clearly and reducing annotator freedom in balancing quality dimensions.The authors suggest this as summarization systems improve and single-document news summarization evaluations approach their limits.
6 Conclusion
The paper comprehensively evaluates ten LLMs on two major news-summarization benchmarks and finds instruction tuning, rather than model scale, is central to success. Its results also expose limits of reference-based and human evaluation, including reference-quality problems and substantial annotator variation.
- 6 Conclusion: The state-of-the-art LLM performs on par with summaries written by freelance writers, with instruction tuning the key factor for success.This conclusion follows a comprehensive human evaluation of ten LLMs across two popular news-summarization benchmarks.
- 6 Conclusion: Good reference summaries are crucial for model development and evaluation, because poor references leave zero-shot, few-shot, and finetuning comparisons unresolved.The authors state that current benchmarks provide limited value when used with reference-based evaluation.
- 6 Conclusion: High-quality references do not eliminate evaluation difficulty, since the annotator pool still exhibits substantial individual variation.This variation contributes to the authors’ conclusion that single-document news-summarization evaluations may be reaching their limits.