PaperlayerView exact revision on arXiv

Source-linked AI summary

Finding a Balanced Degree of Automation for Summary Evaluation

Shiyue Zhang, Mohit Bansal

arXiv:2109.11503v1cs.CLcs.AI

TL;DR

Human summary evaluation is reliable but costly and difficult to reproduce, while automatic metrics may correlate poorly with human judgments. The paper proposes Pyramid-based methods that automate presence checking and vary the use of human-labeled SCUs, then reports strong correlations and reduced annotation effort with its flexible approach.

  • Problem

    Human summary evaluation is reliable but costly and not reproducible across evaluators, while automatic metrics can correlate weakly or inconsistently with human judgments.

    Source: §1 · p. 1

  • Method

    Lite2Pyramid uses NLI to judge reusable SCU presence, Lite3Pyramid substitutes SRL-extracted STUs, and Lite2.xPyramid retains SCUs for sentences its regressor predicts STUs will simulate poorly.

    Source: §1 · p. 2

  • Results

    Lite2Pyramid consistently has the best summary-level correlations and Lite3Pyramid mostly performs better or competitively; on TAC2008 and TAC2009, Lite2.xPyramid cuts SCUs by 50% for a 0.01 absolute summary-level Pearson drop and 0 system-level correlation drop.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Lite2.xPyramid offers flexible degrees of automation, and its regressor may provide useful or expense-saving guidance for future datasets.

    Source: §6 · p. 10

    On PyrXSum, summaries average 4.8 human SCUs but only 2.8 STUs, indicating that SRL decomposition can underrepresent summary content.

    Source: §5.1 · p. 7

Abstract

from arXiv · show

Human evaluation for summarization tasks is reliable but brings in issues of reproducibility and high costs. Automatic metrics are cheap and reproducible but sometimes poorly correlated with human judgment. In this work, we propose flexible semiautomatic to automatic summary evaluation metrics, following the Pyramid human evaluation method. Semi-automatic Lite2Pyramid retains the reusable human-labeled Summary Content Units (SCUs) for reference(s) but replaces the manual work of judging SCUs' presence in system summaries with a natural language inference (NLI) model. Fully automatic Lite3Pyramid further substitutes SCUs with automatically extracted Semantic Triplet Units (STUs) via a semantic role labeling (SRL) model. Finally, we propose in-between metrics, Lite2.xPyramid, where we use a simple regressor to predict how well the STUs can simulate SCUs and retain SCUs that are more difficult to simulate, which provides a smooth transition and balance between automation and manual evaluation. Comparing to 15 existing metrics, we evaluate human-metric correlations on 3 existing meta-evaluation datasets and our newly-collected PyrXSum (with 100/10 XSum examples/systems). It shows that Lite2Pyramid consistently has the best summary-level correlations; Lite3Pyramid works better than or comparable to other automatic metrics; Lite2.xPyramid trades off small correlation drops for larger manual effort reduction, which can reduce costs for future data collection. Our code and data are publicly available at: https://github.com/ZhangShiyue/Lite2-3Pyramid

1 Introduction

The paper addresses the cost and reproducibility limits of human summary evaluation and the uneven human correlations of automatic metrics. It proposes three Pyramid-based approaches that vary how much SCU annotation and presence checking are automated, and reports strong summary-level correlations alongside annotation-effort savings.

1 Introduction

  • Human Pyramid evaluation is reliable but costly and not reproducible across evaluators, while many automatic metrics correlate weakly or inconsistently with human judgments.Some metrics may reflect topic similarity more than information overlap.Source: §1 · p. 1
  • Lite2Pyramid retains reusable human-labeled SCUs and uses a finetuned NLI model instead of humans to judge their presence in system summaries.The authors frame SCU presence as an entailment judgment and use in-domain gold labels to finetune the pretrained model.Source: §1 · pp. 1–2
  • Lite3Pyramid uses SRL-extracted semantic triplets as pseudo-SCUs, enabling fully automatic evaluation but allowing units to omit or misrepresent information.The paper motivates this approach by the cost of manually extracting SCUs for large datasets.Source: §1 · p. 2
  • Lite2.xPyramid predicts which reference sentences STUs simulate poorly, retaining human SCUs for those sentences and using SRL for the rest.This regressor-based selection is presented as a flexible balance between annotation effort and reliability.Source: §1 · p. 2
  • Lite2Pyramid consistently has the best summary-level correlations; Lite3Pyramid mostly performs better or competitively, while Lite2.xPyramid reduces SCUs by 50% for a 0.01 absolute summary-level Pearson drop and 0 system-level correlation drop on TAC2008 and TAC2009.The evaluation compares the metrics on three existing meta-evaluation datasets and newly collected PyrXSum.Source: §1 · p. 2

2 Related Works & Background

This section contrasts human Pyramid evaluation with automatic metrics and prior attempts to automate or simplify evaluation. The proposed methods retain reference SCUs or replace them with SRL-derived units, while using NLI to judge their presence in system summaries.

2 Related Works & Background

  • Human evaluation includes direct ratings of responsiveness or linguistic quality, while aspect ratings may be subject to rater subjectivity.Pyramid is described as a more objective, reference-based alternative.Source: §2 · pp. 2–3
  • Pyramid evaluation extracts SCUs from references, weights them by contributor counts, and scores a system summary by the normalized weight of SCUs present.LitePyramid removes SCU merging and weighting while retaining the presence-checking procedure.Source: §2 · p. 3
  • Automatic metrics trade human evaluation's reliability for reproducibility, low cost, and speed, using approaches such as n-gram overlap, embedding-based matching, structural units, or question answering.The section notes that n-gram metrics can indicate topic similarity more than information overlap.Source: §2 · p. 3
  • Earlier automatic Pyramid methods decompose both system summaries and references, whereas Lite3Pyramid decomposes references into STUs and uses NLI to check their presence.The paper describes the latter procedure as closer to the original Pyramid process.Source: §2 · p. 3
  • Prior semi-automatic evaluation automatically decomposed both summaries and references before humans aligned their semantic units; Lite2Pyramid instead retains reusable SCUs and automates presence judgments with NLI.Source: §2 · p. 3

3 Our Method

Lite2Pyramid automates SCU-presence judgments with an NLI model, while Lite3Pyramid replaces SCUs with SRL-derived triplets. Lite2.xPyramid uses predicted simulation difficulty to balance automated scoring with human annotation.

3.1 Lite2Pyramid

  • Lite2Pyramid retains exhaustive, reference-weighted SCUs and replaces manual presence judgments with NLI entailment predictions.The NLI model is fine-tuned on human-labeled SCU-presence data and adapted to two-way classification.Source: §3.1 · pp. 4–5

3.2 Lite3Pyramid

  • Lite3Pyramid replaces SCUs with short Semantic Triplet Units generated from SRL predicate-argument frames.The procedure keeps the verb and pre-verbal arguments, then combines each post-verbal argument into a triplet; negation modifiers and Be verbs receive small fixes.Source: §3.2 · p. 5
  • STUs can poorly simulate SCUs in syntactically complex sentences, while coreference resolution is used for REALSumm but not TAC or PyrXSum.Removing coreference resolution on REALSumm causes around a 0.01 absolute correlation drop.Source: §3.2 · p. 5, §3.3 · p. 5

3.3 Lite2.xPyramid

  • Lite2.xPyramid uses STUs for sentences predicted to be easier to simulate and retains human-labeled SCUs for harder sentences.Simulation easiness is based on the average, across a sentence’s SCUs, of each SCU’s best ROUGE-1-F1 match to an STU.Source: §3.3 · pp. 5–6
  • An XGBoost regressor predicts simulation easiness from 69 sentence-complexity features, and Lite2.xPyramid replaces the top-scored sentences’ SCUs with STUs.Features include sentence length, parsing-tree length and depth, their ratio, and counts of 65 non-terminal tokens; Lite2.5Pyramid uses STUs for the top 50% of sentences.Source: §3.3 · p. 6

4 Evaluation

The evaluation compares metric-human correlations across four datasets and 15 existing metrics. On TAC, replacing Lite2Pyramid’s NLI predictions with gold labels produces scores that correlate nearly perfectly with official Pyramid scores.

4 Evaluation

  • The evaluation measures system-level and summary-level correlation with human scores, using Pearson for linear correlation and Spearman for ranking correlation.System-level correlation assesses comparison of summarization systems; summary-level correlation assesses comparisons of summaries for the same documents.Source: §4 · p. 6
  • The comparison includes 15 existing metrics, with recall reported for metrics that provide precision, recall, and F1.PyrEval is limited to multi-reference settings.Source: §4 · p. 6
  • The study evaluates TAC2008, TAC2009, REALSumm, and a newly collected PyrXSum dataset spanning long extractive and short abstractive summaries.TAC08 has 96 examples and 58 systems; TAC09 has 88 examples and 55 systems; REALSumm has 100 examples and 25 systems.Source: §4 · p. 6
  • Exhaustive-set scores using gold presence labels have close to perfect correlation with TAC’s official Pyramid scores.REALSumm likewise uses this computation as its gold score.Source: §4 · p. 6

5 Results

Across human-metric correlation tests, Lite2Pyramid leads summary-level correlations, while Lite2.xPyramid offers a regressor-guided transition toward automation. In out-of-the-box tests, Lite2Pyramid retains an advantage in most settings, while Lite3Pyramid remains competitive.

5.1 Human-Metric Correlation Results

  • The evaluation used five-fold cross-validation, splitting by examples or systems to test generalization across those units.Each fold held out examples or systems while the NLI model and regressor were trained on the remaining folds.Source: §5.1 · p. 7
  • Lite3Pyramid had the best correlations among automatic metrics except on PyrXSum, where SRL decomposition struggled with complex reference sentences.PyrXSum summaries averaged 4.8 human-labeled SCUs but only 2.8 STUs; the authors call for a better semantic-unit decomposer.Source: §5.1 · p. 7
  • Lite2.xPyramid’s 50%-STU Lite2.5Pyramid variant reduced correlation by less than half the Lite2Pyramid–Lite3Pyramid gap while saving half the human effort.Regressor-based sentence selection reduced correlation more slowly than random replacement, allowing choices to reflect available budgets.Source: §5.1 · pp. 7–8, Figure 2 · p. 8
  • 0.08 to 0.16 higher summary-level correlations than the best compared metrics were achieved by Lite2Pyramid in example-split cross-validation.It was best or close to best overall; system-split results also showed 0.06 to 0.21 higher summary-level correlations across datasets.Source: §5.1 · pp. 7–8, Table 1 · p. 8

5.2 Out-of-the-Box Generalization

  • In out-of-the-box settings, Lite2Pyramid retained its advantage in most cases, especially for summary-level correlation, while Lite3Pyramid remained competitive.Lite2.5Pyramid retained its trade-off of less than 50% correlation loss for 50% human-effort savings.Source: §5.2 · p. 9, Table 2 · p. 9
  • Learning from all three other datasets performed significantly worse on PyrXSum than learning from TAC08 alone or TAC08+TAC09.The authors conjecture that a distribution shift between REALSumm and PyrXSum explains the drop.Source: §5.2 · p. 9

6 Conclusion

The paper introduces summary-evaluation metrics that balance human effort with automation. Lite2Pyramid uses NLI for SCU-presence judgments, while Lite3Pyramid and Lite2.xPyramid extend automation through SRL and a regressor.

6 Conclusion

  • Lite2Pyramid retains human-labeled SCUs but replaces human SCU-presence judgments with an NLI model to improve reproducibility.The approach combines manual and automatic effort for summary evaluation.Source: §6 · pp. 9–10
  • Across four meta-evaluation datasets, Lite2Pyramid has the best summary-level correlations, while Lite3Pyramid performs better than or competitively with other automatic metrics.Lite2.xPyramid offers flexible automation levels, and its regressor guides which sentences merit SCU annotation.Source: §6 · p. 10
  • Lite3Pyramid decomposes references with SRL, while Lite2.xPyramid uses a regressor to choose sentences for SCU labeling.These approaches provide automated and intermediate options alongside Lite2Pyramid.Source: §6 · p. 10

A.1 Metrics for Comparison

The comparison metrics span lexical overlap, unit and graph matching, learned prediction, contextual embeddings, and question answering.

A.1 Metrics for Comparison

  • ROUGE measures n-gram overlap, AutoSummENG compares n-gram graphs, and METEOR uses text alignment with stems, synonyms, and paraphrases.BEwT-E decomposes summaries into syntactic units for similarity comparison.Source: §A.1 · p. 13
  • S3 is trained on TAC2008/2009 to predict human Pyramid or Responsiveness scores, while PyrEval simulates SCUs and returns four scores.The authors retain PyrEval's quality and comprehensive scores because they perform better; PyrEval requires multiple references.Source: §A.1 · p. 13
  • BERTScore aligns contextualized unigrams, and MoverScore minimizes transportation cost between texts.BERTScore (idf) downweights unigrams with high document frequency.Source: §A.1 · p. 13
  • QAEval compares answer similarity, using Exact Match or F1, for answers inferred from the same question over both texts.Source: §A.1 · p. 13

A.2 PyrXSum

PyrXSum adds a meta-evaluation dataset of short, abstractive XSum summaries to complement datasets with longer, extractive summaries. It includes human SCU annotations and presence labels for summaries from ten systems.

A.2 PyrXSum

  • PyrXSum samples 100 XSum test examples and adds SCU annotations for reference summaries, with a second native English speaker checking the annotations.References contain 2–11 SCUs, averaging 4.8.Source: §A.2 · pp. 13–14
  • Ten abstractive summarization systems provide generated summaries for the 100 examples; extractive systems are excluded because XSum is extremely abstractive.The systems include Fast Abs RL, PtGen, ConvS2S, T-ConvS2S, TransAbs, BertAbs, BertExtAbs, T5, BART, and PEGASUS.Source: §A.2 · p. 14
  • Crowd workers label SCU presence in system summaries, with four responses collected per summary and 104 workers involved.The study filters one noisy worker's annotations and uses majority vote, breaking ties as “not present.”Source: §A.2 · p. 14
  • PyrXSum's SCU-presence labels achieve Krippendorff’s alpha of 0.73, and REALSumm label-sampling rounds yield correlations close to majority-vote results.The sampled REALSumm rounds yield system/summary-level Pearson correlations of 0.89/0.63, 0.90/0.63, and 0.90/0.63, versus 0.89/0.64 for majority vote.Source: §A.2 · p. 14

A.3 Experimental Details

The experimental details specify pretrained models used for NLI, SRL, coreference resolution, and constituency parsing, as well as features and settings for the regressor.

A.3 Experimental Details

  • Lite2Pyramid-0 uses a pretrained RoBERTa-large NLI model, and the authors fine-tune for two epochs with learning rate 1e-5, a linear warmup schedule, and AdamW.The NLI model was trained on SNLI, MNLI, FEVER, and ANLI.Source: §A.3 · p. 14
  • The SRL model is an out-of-the-box AllenNLP model based on Shi and Lin (2019).Source: §A.3 · pp. 14–15
  • Table 3 reports ROUGE-2 and gold Pyramid scores for ten systems on 100 XSum test examples.Source: Table 3 · p. 15
  • Figure 3 depicts the Amazon Mechanical Turk interface used to collect human labels of SCU presence.Source: Figure 3 · p. 15
  • The coreference-resolution and constituency-parsing models are out-of-the-box AllenNLP models based on Lee et al. (2018) and Joshi et al. (2018), respectively.Source: §A.3 · p. 15
  • Regressor features include sentence length and parsing-tree characteristics, and XGBoost is configured with max depth 3, learning rate eta 0.1, and 40 rounds.The listed parsing features include tree length, depth, depth divided by sentence length, and non-terminal-token counts.Source: §A.3 · p. 15

A.4 Additional Results & Ablations

The appendix compares regressor-guided SCU-to-STU replacement with random replacement and tests the robustness of the NLI and regressor components.

A.4 Additional Results & Ablations

  • Lite2.xPyramid achieves higher or equal correlations than random replacement at the same human-effort reduction, including in system-level curves.Figure 4 shows the system-level comparison; Table 4 reports five-fold cross-validation split by systems.Source: §A.4 · pp. 15–16, Figure 4 · p. 16
  • Fine-tuned versus non-fine-tuned NLI accuracy was 82.34%/80.51% on REALSumm, 92.53%/87.63% on TAC08, 93.25%/88.66% on TAC09, and 92.45%/91.13% on PyrXSum.Lite2Pyramid with fine-tuned NLI had higher correlations than Lite2Pyramid-0 except on PyrXSum; the authors consider NLI accuracy positively related to results.Source: §A.4 · p. 16
  • On REALSumm, ALBERT-NLI Lite2Pyramid reaches 0.90/0.64 system/summary-level Pearson correlations, similar to RoBERTa-NLI's 0.89/0.64.Source: §A.4 · p. 16
  • Regressor MAEs on REALSumm, TAC08, TAC09, and XSum are 0.135, 0.211, 0.206, and 0.090, respectively.On REALSumm, a weaker regressor with MAE 0.167 yields Lite2.5Pyramid correlations of 0.89/0.62, close to the original regressor's 0.90/0.62.Source: §A.4 · p. 16
  • Reversing the regressor ranking to replace SCUs for more complex sentences lowers REALSumm Lite2.5Pyramid correlations to 0.88/0.60.Source: §A.4 · p. 16
  • On REALSumm, a RoBERTa-based regressor yields Lite2.5Pyramid correlations of 0.89/0.62, similar to XGBoost's 0.90/0.62.Source: §A.4 · p. 17
Loading 2109.11503v1…