PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation

Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev

arXiv:2212.07981v2cs.CL

TL;DR

Summarization human evaluation can be unreliable because studies may have low agreement or insufficient scale, despite serving as a basis for assessing systems and metrics. The paper introduces ACU and the RoSE benchmark, compares four protocols, and evaluates automatic metrics. It finds that RoSE supports more stable and significant comparisons, while protocol differences and annotator preferences materially affect evaluation.

  • Problem

    Existing summarization human evaluations can exhibit low inter-annotator agreement or insufficient scale, weakening their reliability as a foundation for evaluating systems and automatic metrics.

  • Method

    The paper introduces the fine-grained ACU salience protocol, curates the RoSE benchmark, compares four human-evaluation protocols, and evaluates automatic metrics using the collected annotations.

  • Results

    RoSE yields more statistically stable evaluation with tighter confidence intervals and more significant comparisons, while different protocols can produce drastically different model preferences.

  • Takeaways & Limitations

    Automatic metrics should be evaluated with human protocols aligned to the intended quality dimension, and LLM evaluation should use robust, targeted methods that limit input-agnostic preferences.

  • Takeaways & Limitations

    RoSE includes only English-language data, and dataset construction can contain noise; high annotation agreement does not necessarily imply correctness.

Abstract

from arXiv · show

Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests. However, existing human evaluation studies for summarization either exhibit a low inter-annotator agreement or have insufficient scale, and an in-depth analysis of human evaluation is lacking. Therefore, we address the shortcomings of existing summarization evaluation along the following axes: (1) We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which is based on fine-grained semantic units and allows for a high inter-annotator agreement. (2) We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of 22,000 summary-level annotations over 28 top-performing systems on three datasets. (3) We conduct a comparative study of four human evaluation protocols, underscoring potential confounding factors in evaluation setups. (4) We evaluate 50 automatic metrics and their variants using the collected human annotations across evaluation protocols and demonstrate how our benchmark leads to more statistically stable and significant results. The metrics we benchmarked include recent methods based on large language models (LLMs), GPTScore and G-Eval. Furthermore, our findings have important implications for evaluating LLMs, as we show that LLMs adjusted by human feedback (e.g., GPT-3.5) may overfit unconstrained human evaluation, which is affected by the annotators' prior, input-agnostic preferences, calling for more robust, targeted evaluation methods.

1 Introduction

The paper argues that human evaluation needs stronger agreement and scale to reliably assess summarization systems and automatic metrics. It introduces ACU and RoSE, compares evaluation protocols, and finds that protocol choice and annotator preferences can substantially affect results.

  • Motivation: Human evaluation can suffer from low inter-annotator agreement and insufficient sample sizes, limiting its reliability as an evaluation foundation.Crowd-worker and expert annotations can show near-zero correlation, while underpowered studies may miss statistically significant differences.
  • Contributions: The Atomic Content Unit protocol decomposes summaries into fine-grained semantic units to support high-agreement salience evaluation.The protocol is modified from Pyramid and LitePyramid approaches and is intended to produce more stable, reproducible system evaluations.
  • Protocol comparison: Four human evaluation protocols can yield drastically different model preferences because they target different qualities and may reflect annotators’ prior preferences.The paper emphasizes aligning the protocol with the summary quality intended for evaluation.
  • Metric evaluation: RoSE produces tighter confidence intervals and more statistically significant comparisons when evaluating automatic metrics, while recent LLM-based metrics do not outperform traditional metrics in this study.The evaluated methods include recent LLM-based approaches, and the benchmark is presented as enabling more trustworthy metric evaluation.
  • Contributions: RoSE contains 22,000 summary-level annotations across three summarization datasets and supports higher statistical power for evaluation.The benchmark includes CNN/DailyMail, XSum, and SamSum, with additional CNN/DailyMail validation annotations for metric training.
  • Implications: The authors call for more rigorous LLM evaluation because human-feedback-trained LLMs may overfit unconstrained human evaluation affected by input-agnostic preferences.The paper specifically highlights the need for robust, targeted evaluation methods.

2 Related Work

Prior work has built human-evaluation benchmarks and analyzed automatic metrics, model strengths, and evaluation protocols. It also documents annotation inconsistency, cost, and unreliability across quality dimensions and domains.

  • Human Evaluation Benchmarks: Human-evaluation benchmarks aggregate model outputs and annotate them according to specific summary-quality dimensions.The most relevant prior work uses semantic content units motivated by Pyramid and LitePyramid protocols.
  • Summarization Meta-Evaluation: Summarization meta-evaluation includes metric analyses, model-strength studies, and comparisons of human-evaluation protocols.These directions re-evaluate the current state of summarization evaluation.
  • Metric Analysis: Prior metric studies range from ROUGE analyses to broader evaluations spanning multiple domains and factual-consistency metrics.The cited work covers news, meeting, and scientific-article summarization alongside factual-consistency evaluation.
  • Human Evaluation Analysis: Previous research has examined annotation protocols for linguistic quality and factual consistency while documenting annotation inconsistency, cost, and unreliability.These issues have also been linked to effects on system rankings in prior work.

3 Atomic Content Units for Summarization Evaluation

The ACU protocol evaluates summary salience with fine-grained content units, decomposing reference-based evaluation into writing atomic facts and matching them against system outputs. The resulting annotations support large-scale evaluation across three datasets, with higher agreement than prior collections.

  • ACU Annotation Protocol: ACU evaluation decomposes reference-based salience judgments into ACU Writing and ACU Matching.Annotators extract fine-grained facts from reference summaries, then check whether those facts appear in system outputs.
  • ACU Annotation Protocol: Atomic facts are elementary information units in reference summaries, enabling more fine-grained judgments than sentence-level content units.The protocol follows prior Pyramid-based approaches while shortening the basic annotation unit.
  • ACU Annotation Protocol: The protocol controls annotation consistency by using author-written ACUs, standardized guidelines, qualification tests, and shared ACUs across candidate summaries.Using the same written ACUs for all candidates is intended to make remaining subjectivity unbiased in summary comparison.
  • Scoring Summaries with ACU: The un-normalized ACU score measures the proportion of reference ACUs matched by a candidate summary, while normalized scoring penalizes summaries longer than the reference.The normalization strength α is selected by decorrelating normalized scores from summary length.
  • RoSE Benchmark: RoSE contains approximately 22k summary-level annotations from CNNDM, XSum, and SamSum, including CNNDM validation annotations for metric training.The collection covers generated summaries from recent pretrained summarization systems and releases unified system outputs.
  • RoSE Benchmark: ACU matching achieved Krippendorff’s alpha of 0.7571 at the summary level and 0.7528 at the ACU level, exceeding prior reported crowd-worker agreement.RealSumm and SummEval reported average agreement scores of 0.66 and 0.49, respectively.

4 RoSE Benchmark Analysis

The RoSE analysis examines whether the collected human annotations provide stable, trustworthy evaluation, focusing on statistical power and system characteristics. It finds that larger samples improve power, while summary length can vary substantially across systems and merits separate consideration.

  • 4.1 Power Analysis: Statistical power measures the probability of rejecting the null hypothesis when a real effect exists, with 0.80 commonly treated as sufficient power.The analysis uses power to assess whether human evaluations can yield stable and trustworthy results.
  • 4.1 Power Analysis: High statistical power is difficult when system performance is similar, and typical 50–100-example evaluations reach 0.80 power only above a 5-point ROUGE1 recall difference.The system-pair analysis groups comparisons by ROUGE1 recall differences and assumes the systems differ significantly.
  • 4.1 Power Analysis: For 1–2-point system differences, 500 examples yield around 0.50 power, compared with around 0.20 for 100 examples.The result illustrates how increasing the number of test examples raises statistical power.
  • 4.2 Summarization System Analysis: Summary lengths vary greatly across systems and are not always captured by ROUGE F1; GSum summaries are around 40% longer than GLOBAL summaries despite similar summary characteristics.The analysis therefore treats summary length as a separate aspect of summary quality.
  • 4.2 Summarization System Analysis: The system analysis reports ACU and normalized ACU scores from 500 annotated examples, while length and ROUGE1 F1 are calculated over the entire test set.Table 3 sorts systems by average summary length and distinguishes the evaluation subsets used for each measure.

5 Evaluating Annotation Protocols

The paper compares four annotation protocols and finds that protocol choice substantially affects correlations, power, and model preferences. In particular, annotator priors and summary length can influence reference-free judgments, while ACU evaluation provides higher power than Ref-based evaluation.

  • 5.1 Annotation Collection: Prior records preferences without the input document, Ref-free judges coverage of salient input information, and Ref-based evaluates similarity to the reference.The study compares two reference-free protocols with one reference-based protocol alongside ACU evaluation.
  • 5.1 Annotation Collection: The three non-ACU protocols use 1–5 scores, with three annotations per summary collected on a 100-example CNNDM subset.The same worker pool from ACU qualification was used for these annotations.
  • 5.2 Results Analysis: The Ref-free protocol strongly correlates with Prior, both correlate strongly with summary length, and Ref-free negatively correlates with Ref-based despite targeting similar quality aspects.These system-level correlations are reported for fine-tuned models using normalized ACU scores.
  • 5.2 Results Analysis: ACU yields higher statistical power than Ref-based evaluation, while Prior and Ref-free show higher power than reference-based protocols but are not directly comparable because their targets differ.The near-zero correlation between the reference-free and reference-based protocols motivates caution in interpreting their power comparison.
  • 5.2 Results Analysis: Under Ref-free evaluation, GPT-3 receives the highest score and the reference summary the lowest, while Prior shows the same preference pattern.The results indicate that annotators’ prior preferences can affect judgments of model outputs.
  • 5.2 Results Analysis: For four annotators, the average correlation between their own Ref-free and Prior scores is 0.404, versus 0.188 between their Ref-free scores and other annotators’ Ref-free averages.The comparison suggests that an annotator’s own Prior score better predicts their Ref-free score than other annotators’ judgments do.

6 Evaluating Automatic Metrics

The metric analysis compares automatic scores with ACU annotations across datasets, system-pair differences, and human-evaluation protocols. Traditional metrics can outperform LLM-based evaluators on RoSE, while metric correlations vary by dataset, evaluation target, and sample size.

  • 6 Evaluating Automatic Metrics: The benchmark evaluates automatic metrics by correlating their scores with ACU annotations at both system and summary levels across CNNDM, XSum, and SamSum.Recall scores are used when available to align automatic metrics with the recall-based ACU scores.
  • 6 Evaluating Automatic Metrics: The evaluated metrics span lexical-overlap, pretrained-language-model, question-answering, Lite3Pyramid, and LLM-based approaches, including G-Eval variants.The analysis includes 50 automatic metric variants overall.
  • 6.1 Metric Evaluation with ACU Annotations: Metric correlations are generally stronger on SamSum and weaker on XSum, where reference summaries contain more complex structures.This dataset variation appears in both the system-level and summary-level evaluation setup.
  • 6.1 Metric Evaluation with ACU Annotations: LLM-based automatic evaluation does not outperform traditional methods such as ROUGE on RoSE and has low summary-level correlation with ACU, suggesting poor score calibration.The finding contrasts with reported successes of these methods on other human-evaluation benchmarks.
  • 6.2 Analysis of Metric Evaluation: Automatic metrics generally perform worse when distinguishing system pairs with similar ACU scores.The pairwise analysis groups system comparisons into six equal-sized buckets by ACU-score differences.
  • 6.2 Analysis of Metric Evaluation: Larger human-evaluation samples reduce correlation confidence intervals and increase the chance of statistically significant metric comparisons.Power analysis shows that significant results are difficult when metric performance is similar, but sample-size increases improve detection.
  • 6.2 Analysis of Metric Evaluation: Automatic metrics generally perform better under reference-based evaluation but can have negative correlations with reference-free protocols.This comparison reinforces the need to align metric evaluation with the human-evaluation target.

7 Conclusion and Implications

RoSE enables more robust summarization evaluation across three datasets by combining a scalable protocol with broad human and metric analyses. The authors argue that evaluation should align the human protocol with the quality dimension targeted by systems or metrics.

  • Conclusion and Implications: RoSE’s protocol and scale support more robust summarization evaluation across three datasets.The benchmark is used to reassess human evaluation and its implications for system and automatic-metric development.
  • Conclusion and Implications: Automatic metrics should be evaluated with human protocols that capture the intended quality dimension.The paper gives reference-based metrics and reference-based human evaluation as an alignment example.
  • Conclusion and Implications: Existing summarization datasets should be used for tasks matching their specific requirements and evaluation purposes.The passage highlights summary-length requirements and reference-based metric study as examples.

8 Limitations

The benchmark is limited by possible annotator and pretraining-data biases, English-only coverage, collection noise, and unresolved questions about reference quality and ACU weighting.

  • Limitations: The benchmark and analysis include only English-language data, while annotator and pretraining-data biases may affect downstream use.The authors specifically caution that language models may learn such biases.
  • Limitations: ACU writing and matching can contain noise, and high annotation agreement does not necessarily imply correctness.The authors rely on spot checks and worker filtering to curate a high-quality benchmark despite these risks.
  • Limitations: Reference-based evaluation remains sensitive to reference quality, and ACUs are not weighted during aggregation.The original Pyramid protocol supports multi-reference evaluation and semantic-unit weighting, whereas this benchmark does not weight ACUs.

A Benchmark Data Collection

The benchmark combines ACU annotation procedures, selected summarization-system outputs, multi-dataset coverage, and bootstrap power analysis. Its design emphasizes annotation consistency and the effect of sample size on statistical power.

  • Data Collection: ACUs are collected with standardized guidelines, cross-annotation proofreading, and one annotator per reference summary after consistency is established.The authors note that multiple valid formulations of the same atomic fact exist.
  • Data Collection: The benchmark covers CNNDM, XSum, and SamSum, with released data licenses specified for each dataset.CNNDM and XSum use Apache licensing, while SamSum uses CC BY-NC-ND 4.0; the benchmark is released under 3-Clause BSD.
  • Data Collection: The annotated systems include established models and variants across CNNDM, XSum, and SamSum.Examples include BART, Pegasus, MatchSum, GOLD, GSum, BRIO, CLIFF, and dialogue-summarization systems.
  • Data Collection: 500 annotated examples support ACU and normalized-ACU scores, while length and ROUGE1 F1 are computed on full test sets.This reporting setup is specified for the XSum and SamSum system-analysis tables.
  • Power Analysis: Bootstrap power analysis repeatedly samples examples and applies paired significance tests across system pairs.The analysis uses an existing human-annotated dataset, sample size n, trial count m, and statistical power p.
  • Power Analysis: Increasing sample size raises statistical power across CNNDM, XSum, and SamSum analyses.The power-analysis figures group system pairs by ACU or ROUGE1-recall performance differences.

D.2 Results Analysis

The results analysis compares protocol correlations, statistical power, system preferences, and automatic-metric evaluation. A case study shows that different protocols can favor different summaries from the same systems.

  • Protocol Results: Summary-level Pearson correlations are reported among the Prior, Ref-free, Ref-based, and ACU protocols.The analysis is conducted on fine-tuned models and includes normalized ACU scores and summary length.
  • Protocol Results: Power analysis is reported for the Prior, Ref-free, Ref-based, and ACU protocols.The comparison is presented for human-evaluation protocols on annotated CNNDM examples.
  • Case Study: BRIO and GPT-3 receive different rankings across protocols: GPT-3 scores higher on Prior and Ref-free, while BRIO scores higher on ACU and Ref-based evaluation.The reported scores are 3.33/3.33 versus 3.66/4.00 on Prior and Ref-free, 0.77 versus 0.33 on un-normalized ACU, and 3.66 versus 3.33 on Ref-based evaluation.
  • LLM-based Metrics: GPTScore uses LLM token probabilities as quality scores, whereas G-Eval asks an LLM to predict a numerical score directly.The described GPTScore setup uses the probability of “Yes” for a reference-coverage question.
  • Metric Results: The benchmark evaluates 50 automatic metrics and metric variants across three datasets using multiple correlation analyses.Reported analyses include system-level and summary-level correlations with un-normalized and normalized ACU scores.

E.4 System Pairs for Fine-grained Metric Evaluation

The evaluation organizes system pairs into performance-difference buckets and examines how automatic-metric correlations with ACU scores vary in precision and stability. Larger human-evaluation samples reduce confidence intervals and produce more stable results.

  • System-pair grouping: System pairs are grouped into six buckets according to their performance differences for fine-grained metric evaluation.The listed pairs include systems such as CLIFF, FROST, CTRLSUM, GSUM, BART, GOLD, BRIO, PEGASUS, and others.
  • Metric correlations: Confidence intervals are calculated for system-level correlations between selected automatic metrics and ACU scores.The results are presented in Figure 8, following the metric-evaluation analysis in §6.2.
  • Sample-size effects: Larger sample sizes effectively reduce confidence intervals for metric correlations with ACU scores.The study uses resampling to create synthetic samples of different sizes and averages confidence intervals across samples of the same size.
  • Metric comparison: The study uses power analysis to compare 20 automatic metrics through 190 metric pairs based on Kendall’s correlations with ACU scores.The compared metrics include BARTScore, BERTScore variants, ROUGE variants, QAEval, SummaQA, and SUPERT.
  • Evaluation protocols: Correlations between automatic metrics and different human-evaluation protocols are reported for the CNNDM dataset.These comparisons appear in Table 19 and are discussed in §6.2.

F Human Evaluation Practices in Recent Text Summarization Research

A survey of recent summarization research examines how human evaluations are conducted, reported, and released. It finds that human evaluation is common, but most reported studies use relatively small sample sizes.

  • Survey scope: The survey covers human-evaluation practices in 55 selected text-summarization papers published at NAACL 2022, ACL 2022, and EMNLP 2022.The survey follows the design of a similar study by Gehrmann et al. (2022).
  • Evaluation prevalence: 71% of the analyzed papers conducted human evaluation of some kind.The survey defines a positive result as any human evaluation being performed.
  • Reporting practices: The survey records whether studies performed significance tests and released human-evaluation data.These practices are included among the criteria summarized in Table 14.
  • Evaluation scale: Around 93% of the surveyed human evaluations had sample sizes less than or equal to 200.The reported sample-size measure may include the number of evaluators, documents, annotators per document, or demographic information when mentioned.
  • Related evaluation analyses: The appendix reports metric correlations with ACU scores across datasets, annotation normalizations, evaluation levels, and human-evaluation protocols.Tables 15–19 cover CNNDM, XSum, and SamSum, including system-level and summary-level analyses.
Loading 2212.07981v2…