PaperlayerView exact revision on arXiv

Source-linked AI summary

Perturbation CheckLists for Evaluating NLG Evaluation Metrics

Ananya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, Mitesh M. Khapra

arXiv:2109.05771v1cs.CL

TL;DR

Human evaluation of NLG outputs covers multiple criteria, but metric evaluation often focuses on overall-quality scores. The paper proposes criterion-targeted perturbation templates and finds that metrics often disagree with human judgments on those perturbations.

  • Problem

    Automatic metrics are commonly evaluated against human overall-quality scores, leaving their ability to assess multiple criteria insufficiently examined.

    Source: §1 · p. 1

  • Method

    The paper proposes CheckList-style templates that perturb outputs to target specific evaluation criteria and test whether metrics detect those changes.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across six tasks, human scores for different criteria are often weakly correlated, and existing metrics have low correlations with most criteria; their scores also disagree with human scores on perturbations.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The templates provide a finer-grained way to assess metric strengths and limitations than correlations with overall-quality scores alone.

    Source: §7 · p. 9

    For automatic summarization, the authors use Fabbri et al. (2020) because they could not find a dataset containing the criteria set from Sai et al. (2020b).

    Source: §2.1 · p. 2

Abstract

from arXiv · show

Natural Language Generation (NLG) evaluation is a multifaceted task requiring assessment of multiple desirable criteria, e.g., fluency, coherency, coverage, relevance, adequacy, overall quality, etc. Across existing datasets for 6 NLG tasks, we observe that the human evaluation scores on these multiple criteria are often not correlated. For example, there is a very low correlation between human scores on fluency and data coverage for the task of structured data to text generation. This suggests that the current recipe of proposing new automatic evaluation metrics for NLG by showing that they correlate well with scores assigned by humans for a single criteria (overall quality) alone is inadequate. Indeed, our extensive study involving 25 automatic evaluation metrics across 6 different tasks and 18 different evaluation criteria shows that there is no single metric which correlates well with human scores on all desirable criteria, for most NLG tasks. Given this situation, we propose CheckLists for better design and evaluation of automatic metrics. We design templates which target a specific criteria (e.g., coverage) and perturb the output such that the quality gets affected only along this specific criteria (e.g., the coverage drops). We show that existing evaluation metrics are not robust against even such simple perturbations and disagree with scores assigned by humans to the perturbed output. The proposed templates thus allow for a fine-grained assessment of automatic evaluation metrics exposing their limitations and will facilitate better design, analysis and evaluation of such metrics.

1 Introduction

The introduction argues that NLG metric evaluation should assess multiple human criteria rather than only correlations with overall-quality scores. It proposes criterion-targeted perturbation templates and reports that automatic metrics disagree with human judgments on perturbed outputs.

  • The study examines multiple criteria because a single overall metric score does not reveal which aspect of generated output needs improvement.Source: §1 · p. 1
  • Across six tasks, human scores for different criteria often correlate weakly, and no metric has high correlation with human scores for any desired criterion on a task.Source: §1 · p. 2
  • The proposed CheckList-style templates perturb outputs along specific criteria, such as changing named entities to affect factual correctness.Source: §1 · p. 2
  • The authors create perturbation templates for six NLG tasks, instantiate them as test cases, and collect human judgments for the perturbed outputs.Source: §1 · p. 2
  • Automatic metrics disagree with human scores on perturbed outputs, indicating that they are not robust to these perturbations.Source: §1 · p. 2

2 Criteria used in Human Evaluations

The section describes the human-evaluation criteria and datasets used to examine whether criteria should be combined into a single score. It reports that most criterion pairs have low-to-moderate correlations across tasks.

2 Criteria used in Human Evaluations

  • The study considers popular human-evaluation criteria for six NLG tasks and computes their pairwise correlations to test whether one overall score is sufficient.Source: §2 · p. 2

2.1 Correlations between different criteria

  • The authors compute pairwise Kendall tau correlations from existing multi-criteria human ratings; for automatic summarization, they use Fabbri et al. (2020) because the specified criteria set was unavailable.Source: §2.1 · p. 2
  • Some datasets do not report Inter Annotator Agreement scores, while others report values ranging from 0.63–0.71.Source: §2.1 · p. 2
  • Across tasks, most criterion pairs correlate moderately (0.3–0.5) or weakly (< 0.3), while dialogue interestingness–enjoyability reaches 0.76.Source: Figure 1 · p. 3, §2.1 · p. 3

3 Perturbation Checklists

Perturbation Checklists test whether evaluation metrics detect output changes targeted at particular criteria. The framework also includes invariant perturbations, which should not lower a metric’s score.

3 Perturbation Checklists

  • Table 4 presents examples of templates, with blue highlighting affected portions of the original and red highlighting changes in the modified output.Source: Table 4 · p. 4
  • The framework tests whether a metric detects criterion-specific changes by comparing its score change with humans’ perceived quality change.A metric’s score is normalized to [0, 1], and the defined score captures the deviation between the human-perceived and metric-perceived quality drops.Source: §3 · p. 4
  • The authors create criterion-targeted templates for each task, including invariant templates whose perturbations should not reduce metric scores.Fluency templates are also included; the invariant and fluency templates are common across tasks.Source: §3 · p. 4
  • Perturbations are generated using Checklist operations such as dropping or replacing words and entities, jumbling words, reordering sentences, and adding spurious text or redundancy.Source: §3 · p. 5

4 Experimental setup

The experimental setup combines a coarse-grained correlation study with a fine-grained test of metric robustness to criterion-specific perturbations. It uses task-specific datasets, 25 metrics, and human judgments to assess perturbation effects.

4 Experimental setup

  • The study combines correlations between metrics and human scores on multiple criteria with robustness tests using criterion-specific perturbations.The authors describe the latter as augmenting the coarse-grained evaluation.Source: §4 · p. 5
  • For fine-grained evaluation, the authors use reference datasets for MT, QG, AS, DG, IC, and D2T, including WebNLG 2020 data for D2T.Source: §4 · p. 5
  • Perturbations are applied to reference sentences after preprocessing; targeted text is modified with string operations or words and phrases generated using RoBERTa predictions.Source: §4 · p. 5
  • The study examines 25 automatic evaluation metrics, with implementations drawn from prior work or official metric code repositories.Source: §4 · pp. 5–6
  • Fifteen NLP-proficient computer science graduate annotators rated perturbation penalties from 0 to 10, which the authors normalized to [0, 1].The perturbed-output human score is computed by subtracting the mean normalized penalty from 1.Source: §4 · p. 6
  • Inter-annotator agreement was 0.79, measured as Kendall tau between averages from two randomly formed annotator groups across five runs.The standard deviation of normalized annotator scores for each perturbation ranged from 0.03 to 0.2.Source: §4 · p. 6

5 Results and Discussion

Across tasks and criteria, many metrics correlate poorly with human judgments, and fine-grained perturbation tests reveal failures that coarse correlations miss. Metrics also differ in which perturbations they handle well.

5.1 Insights from coarse-grained evaluation

  • DEB, BLANC, and MaUde correlate poorly with human judgments on criteria beyond overall quality despite task-specific design and BERT-based components.The passage notes that these metrics are fine-tuned on human judgments for overall quality.Source: §5.1 · p. 7
  • Task-agnostic BERTScore, MoverScore, and BLEURT often rank among the top metrics alongside task-specific metrics, despite lacking task-specific human-judgment fine-tuning.The passage also identifies scope for improving task-specific metrics on criteria peculiar to each task.Source: §5.1 · p. 8

5.2 Insights from fine-grained evaluation

  • BERTScore, BLEURT, and MoverScore can miss meaning-changing MT edits, while similar perturbation failures appear across tasks.Examples include adding negation, changing names or numbers, and replacing words with antonyms.Source: §5.2 · p. 8
  • Task-specific grievances remain: metrics miss generic dialogue responses, question-answerability changes, image-caption attributes, data-to-text coverage, and summary coherence or referential clarity.For generic dialogue responses, expert annotators assigned a high penalty; task-specific metrics generally use context as input, unlike task-agnostic metrics.Source: §5.2 · p. 8
  • BLEURT and MoverScore handle word jumbling better than BERTScore, while several metrics handle number-to-word transformations and BERTScore respects alternative references.For image captioning, SPICE and ViLBERTScore show complementary behavior across the tested perturbation criteria.Source: §5.2 · p. 9

6 Related Work

Prior work documents weak human-score correlations and gaps in what automatic metrics capture across NLG tasks. Other work calls for more robust evaluation and multiple evaluation dimensions.

6 Related Work

7 Conclusion

The paper studies automatic NLG metrics across tasks and criteria, finding that criteria often diverge and metrics correlate poorly with many of them. It proposes perturbation templates to assess metrics more finely and expose their limitations.

7 Conclusion

  • The study spans 6 NLG tasks, 25 automatic evaluation metrics, and 18 human evaluation criteria.Source: §7 · p. 9
  • Human scores on different criteria often do not correlate, and existing metrics have low correlation with most criteria across tasks.Source: §7 · p. 9
  • The proposed perturbation templates target specific criteria to assess metric effectiveness more finely and expose limitations.The authors hope future metric-design work will use the checklist to evaluate effectiveness across relevant criteria.Source: §7 · p. 9

A Criteria correlations

The appendix compares correlations among human evaluation criteria and reports that annotator expertise affects those relationships. Expert annotations show less correlation among criteria than crowdsourced annotations.

A Criteria correlations

B Detailed examples to illustrate the implementation of the perturbation templates

The implementation uses NLP tools and lexical resources to construct targeted perturbations, excludes samples where a perturbation cannot be applied, and makes the complete implementation details publicly available.

B Detailed examples to illustrate the implementation of the perturbation templates

C Perturbation Templates for various criteria

Table 5 is identified as the comprehensive inventory of the perturbation templates used in the study.

C Perturbation Templates for various criteria

D Correlations of Metrics with various Criteria (complete plots)

Figure 7 presents correlations between the study’s metrics and evaluation criteria across tasks, expanding on Figure 3.

D Correlations of Metrics with various Criteria (complete plots)

Loading 2109.05771v1…