PaperlayerView exact revision on arXiv

Source-linked AI summary

Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration

Daniel Deutsch, George Foster, Markus Freitag

arXiv:2305.14324v2cs.CL

TL;DR

Modern MT meta-evaluation has to handle ties in human judgments and metric scores, but existing Kendall variants have weaknesses and can be gamed. The paper proposes tie-aware pairwise accuracy together with tie calibration, and reports that these changes produce fairer ranking-based assessments.

  • Problem

    Existing Kendall variants handle ties inadequately as ties arise in human judgments and metric scores, leaving tie-sensitive MT meta-evaluation vulnerable to gaming.

    Source: §1 · p. 1

  • Method

    The paper combines pairwise accuracy that credits correctly predicted ties with tie calibration that automatically introduces ties into metric scores.

    Source: §1 · p. 1

  • Results

    Existing Kendall variants fail to handle some simple tie-related phenomena and can be gamed for large improvements in certain evaluation settings.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors argue and report experimentally that pairwise accuracy and tie calibration yield fairer assessments of MT metric performance.

    Source: §1 · p. 1

    The authors are unaware of a way to prove the proposed meta-evaluation is fair, relying instead on experiments and its resistance to known methodological issues.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Kendall's tau is frequently used to meta-evaluate how well machine translation (MT) evaluation metrics score individual translations. Its focus on pairwise score comparisons is intuitive but raises the question of how ties should be handled, a gray area that has motivated different variants in the literature. We demonstrate that, in settings like modern MT meta-evaluation, existing variants have weaknesses arising from their handling of ties, and in some situations can even be gamed. We propose instead to meta-evaluate metrics with a version of pairwise accuracy that gives metrics credit for correctly predicting ties, in combination with a tie calibration procedure that automatically introduces ties into metric scores, enabling fair comparison between metrics that do and do not predict ties. We argue and provide experimental evidence that these modifications lead to fairer ranking-based assessments of metric performance.

1 Introduction

Modern MT creates many ties in both human judgments and metric scores, exposing weaknesses in Kendall’s τ variants used for meta-evaluation. The paper proposes tie-aware pairwise accuracy and tie calibration as a fairer alternative.

1 Introduction

  • As MT systems improve, human scores increasingly tie for perfect outputs, while error counts and discrete metric scores also produce ties.These ties arise on both the human and metric sides.Source: §1 · p. 1
  • Existing Kendall variants mishandle simple tie-related phenomena and can be gamed to produce large improvements in certain evaluation settings.The paper examines WMT metrics-task data.Source: §1 · p. 1
  • Tie-aware pairwise accuracy gives credit for correctly predicting ties and is presented as an interpretable proportion of correctly ranked pairs.The paper argues this formulation is easier to interpret than a closely related modification to τ.Source: §1 · p. 1
  • Tie calibration automatically introduces ties into metric scores to address pairwise accuracy’s disadvantage to metrics that rarely predict ties.The authors argue and report experimentally that the two modifications yield fairer MT metric assessments; experiments are limited to MT, though the approach may apply to other generative-AI metrics with similar characteristics.Source: §1 · pp. 1–2

2 Background & Related Work

The background contrasts ranking-based evaluation with other correlations, describes WMT’s system- and segment-level meta-evaluation, and reviews how Kendall’s τ variants handle ties.

2 Background & Related Work

  • The background frames the work as an update to ranking-based MT meta-evaluation and notes that WMT has used different τ variants over time.Table 1 presents variants and identifies acceq and τeq as proposals in this work.Source: §2 · p. 2, Table 1 · p. 3

2.1 Why not Pearson or Spearman?

  • Pearson captures linear correspondence, Kendall captures pairwise ranking, and Spearman behaves as a compromise across noisy-linear and perfectly ordered nonlinear examples.For applications where linear correspondence and ranking decisions both matter, the text advises measuring Pearson and Kendall.Source: §2.1 · p. 2

2.2 Metric Meta-Evaluation

  • WMT meta-evaluates metrics against human-annotated scores at system and segment levels, with segment-level approaches including no-grouping, group-by-item, and group-by-system.Segment-level correlations are described as better at discriminating metrics and more closely related to generation-improvement applications.Source: §2.2 · p. 2

2.3 The Landscape of Kendall’s τ

  • Kendall’s τ compares concordant and discordant translation-pair rankings, while its variants mainly differ in tie handling.WMT variants do not include human-score ties and penalize metric-score ties; no τ variant directly rewards correctly predicting human-score ties.Source: §2.3 · p. 3

3 Analysis Setup

The analysis uses WMT’22 MQM ratings across three language pairs to compare meta-evaluation statistics and metric submissions.

3 Analysis Setup

  • The study uses WMT’22 MQM ratings as human ground truth for en→de, zh→en, and en→ru, with 13–15 systems and about 1,300–1,900 rated segments per system.The dataset description gives the system and segment ranges across the language pairs.Source: §3 · p. 3
  • The analysis compares meta-evaluation statistics for WMT’22 primary submissions and GEMBA metrics, with closer examination of Metric-X, COMET-22, BLEURT-20, MaTESe, and GEMBA.Metric-X, COMET-22, and BLEURT-20 are regression-based; MaTESe predicts weighted span-level errors, while GEMBA predicts scores using zero-shot GPT-3.5 and GPT-4 prompting.Source: §3 · pp. 3–4
  • Table 3 reports segment and pair counts, including tied MQM pairs and pairs tied at MQM=0, across WMT’22 language pairs.Its scope is group-by-item correlations; statistics for other segment-level correlations are in Appendix C.Source: Table 3 · p. 4

4 Why Ties are Important

Ties in human scores are increasingly meaningful and common in modern MT evaluation, while some newer metrics also predict many ties. Existing Kendall variants do not directly reward metrics for correctly predicting human ties.

4 Why Ties are Important

  • MQM scores from expert translators are more reliable than crowdsourced direct-assessment scores, making ties or small differences more likely to reflect translation quality.Up to 53% of possible en-de pairs have tied MQM scores; most have no translation errors.Source: §4 · p. 4
  • As MT quality improves, tied human scores are likely to become more common, and removing them can discard a useful signal for distinguishing metrics.For example, up to 53% of possible en-de pairs have tied MQM scores, most with no errors.Source: §4 · p. 4
  • MaTESe and GEMBA predict many ties, unlike regression metrics such as Metric-X and COMET, but existing Kendall variants do not reward correct predictions of human ties.The percent of tied pairs differs drastically between these metric types when grouping by source segment.Source: §4 · p. 4, Table 4 · p. 4

5 Shortcomings of Kendall’s Variants

Existing Kendall variants can fail to distinguish metrics that differ in their treatment of ties, and NaN correlations can let metrics improve apparent performance by excluding difficult groups. These issues can make comparisons unfair or misleading.

5 Shortcomings of Kendall’s Variants

  • Kendall variants’ handling of ties creates blind spots and opportunities to exploit τ-specific properties in metric meta-evaluation.The section demonstrates these shortcomings with an example and experimental analysis.Source: §5 · p. 4

5.1 A Motivating Example

  • With ties considered, m1 incorrectly ranks 1 of 15 pairs and m2 ranks 6, yet existing τ coefficients can treat m1 as approximately equal to or worse than m2.Existing τ variants do not credit metrics for correctly predicting ties, making them inadequate for this example.Source: §5.1 · pp. 4–5

5.2 The NaN Problem

  • When a metric or human score vector is constant, many τ values are undefined; removing NaN group correlations can let a metric exclude difficult groups and raise its score.Bucketing Metric-X scores can reduce non-NaN groups while producing much higher correlations, so comparisons use different subsets and can lead to erroneous conclusions.Source: §5.2 · p. 5
  • A metric could exploit the NaN problem in WMT’22 because the meta-evaluation does not account for the number of non-NaN segments.Defining correlations for constant vectors would close this loophole; assigning zero instead could penalize metrics when human scores are tied or nearly tied.Source: §5.2 · p. 5

6 Evaluating with Pairwise Accuracy

The paper proposes pairwise accuracy that credits correct rankings and correctly predicted ties, avoiding key weaknesses of existing τ variants. It also describes class-specific statistics for examining tie and non-tie performance.

6 Evaluating with Pairwise Accuracy

  • acceq measures the proportion of all pairs that a metric ranks correctly or correctly predicts as tied, incorporating ties in both human and metric scores.Its value is between 0 and 1 and can be read as the proportion of correctly handled pairs, unlike τ’s range of -1 to 1.Source: §6 · p. 5
  • Pairwise accuracy avoids the presented τ problems: acceq prefers m1, which has fewer incorrectly ranked pairs, and its value is never NaN.Thus all examples remain in the evaluation, unlike cases affected by the NaN problem.Source: §6 · p. 6

6.1 Evaluating Ties and Non-Ties

  • Pairwise accuracy treats each pair as a three-way decision—tie or either ranking—enabling class-specific precision, recall, and F1 analysis.These statistics can reveal class imbalance between tied and non-tied pairs that overall accuracy may hide.Source: §6.1 · p. 6, Table 5 · p. 6

7 Tie Calibration

Tie calibration introduces ties for metrics that rarely predict them, helping make comparisons fair under pairwise accuracy. It selects a score-difference threshold to optimize a ranking statistic, though the chosen statistic must handle ties appropriately.

7 Tie Calibration

  • Regression metrics such as BLEURT and COMET rarely assign tied scores, disadvantaging them when human scores are tied and motivating tie calibration.Tie calibration is intended to make fair comparisons between metrics that do and do not predict ties.Source: §7 · p. 6
  • Tie calibration searches for an ϵ that maximizes a rank-based statistic, treating score differences below ϵ as ties.The implementation considers possible score differences as candidate thresholds and runs in O(n^2 log n), where n is the number of translations.Source: §7 · p. 6
  • The optimal tie threshold prevents under-prediction of ties from penalizing metrics and can be interpreted as a dataset-specific threshold for treating score differences as significant.This interpretability is scoped to a specific dataset.Source: §7 · p. 6
  • Tie calibration can produce unexpected results with statistics that mishandle ties; with τ13, it could improve the statistic by turning a discordant pair into a tie even when human scores are not tied.The authors caution that tie calibration requires a statistic that handles ties desirably.Source: §7 · pp. 6–7

8 Analysis

The analysis examines how ranking statistics affect metric comparisons, whether tie calibration generalizes across datasets, where it introduces ties, and how metrics perform on tied versus non-tied pairs.

8.1 Comparing Metric Rankings

  • Under τb, GEMBA-GPT-4 and MaTESe rank highest, but MaTESe uses 773 non-NaN segments versus Metric-X’s 1,133.On the same 773 segments, Metric-X’s correlation is 0.296 versus 0.281 for MaTESe, illustrating the problem of comparing correlations computed on different segments.Source: §8.1 · p. 7
  • τ10 pushes tie-heavy MaTESe and GEMBA to the bottom, while randomly breaking their ties raises correlations by about 0.5–0.6 points.MaTESe changes from −0.459 to ≈0.15 and GEMBA from −0.344 to ≈0.15; COMET-22 improves by only ≈0.005.Source: §8.1 · p. 7
  • With acc*_eq and tie calibration, MaTESe and GEMBA rank 4th, 6th, and 15th, and the authors argue this is fairer and more reliable than τ variants.The passage says τ_eq and acc*_eq are never NaN and do not systematically favor or penalize ties.Source: §8.1 · p. 7

8.2 Generalization of Epsilon

  • For en-de, held-out ϵ changes by 0.03 and acc_eq changes by a relative 2%, but ϵ does not generalize well between zh-en datasets.The authors hypothesize this relates to WMT’21 zh-en having 23% ties versus 41% in WMT’22; they argue calibrating ϵ on the current test set still permits fair metric comparisons.Source: §8.2 · p. 8

8.3 Where are Ties Introduced?

  • For Metric-X on WMT’22 zh-en, ties introduced by ϵ* are concentrated among higher-scoring pairs, suggesting they mostly predict perfect translations.The authors suggest first identifying a perfect-translation threshold, then applying tie calibration to the remaining pairs.Source: §8.3 · p. 8, Figure 5 · p. 8

8.4 Class-Specific Statistics

  • For COMET-22 on en-de, ties-F1 exceeds correct-rank-F1 for almost every ϵ, indicating more reliable prediction of tied pairs than non-tied pair rankings.The authors suggest this may reflect many perfect translations and ϵ values biased toward introducing ties for them.Source: §8.4 · p. 8, Figure 6 · p. 8
  • Tie calibration can select an ϵ to balance performance on class-specific statistics when those statistics better match metric use in practice.Source: §8.4 · p. 8

9 Conclusion

The paper argues that rank-based meta-evaluation must account for ties and advocates pairwise accuracy with tie calibration for comparing MT metrics.

9 Conclusion

  • Pairwise accuracy rewards correct rankings and correctly predicted ties, while tie calibration enables comparison of metrics that do and do not predict ties.Although the experiments focus on MT, the authors say the methods generally apply to NLP metric meta-evaluation.Source: §9 · p. 9

Limitations

The paper notes limitations of tie calibration and pair-level decisions, then provides definitions of segment-level correlations and WMT tabular notation.

Limitations

  • Tie calibration assumes equal absolute score differences represent equal quality changes across the metric range; an untested metric may work better with relative ϵ.Experiments with relative differences did not show a significant difference in results.Source: Limitations · p. 9
  • Because tie decisions are pairwise, ϵ does not create a global ordering; with scores 1, 2, and 3 at ϵ = 1, adjacent pairs tie but the endpoints do not.The authors note a similar limitation in pairwise statistical significance testing.Source: Limitations · p. 9
  • The authors argue their proposal is fairer but say they cannot prove this, relying instead on experiments and its avoidance of known issues in existing methods.Source: Limitations · p. 9

A Correlation Definitions

  • Segment-level correlation compares human and metric scores over translations, using a correlation coefficient such as Pearson’s r, Spearman’s ρ, Kendall’s τ, or pairwise accuracy.The appendix identifies three ways to compute segment-level correlations.Source: A Correlation Definitions · p. 10

B WMT Tabular Notation

  • The appendix presents WMT tabular notation for computing τ and defines C_h,m as a bucket coefficient and S_h,m as the number of pairs in that bucket.It maps the paper’s notation to WMT’14 notation and includes tabular versions of τ variants, τ_eq, and acc_eq.Source: B WMT Tabular Notation · p. 11, Table 9 · p. 11

C Additional Results

Appendix analyses find that downsampling preserves the accuracy of tie calibration’s epsilon search, while Unbabel normalization leaves the reported tie-related metric-ranking biases largely intact.

C Additional Results

  • Table 10 summarizes tied-pair counts across WMT’22 language pairs and segment-level correlations, while Table 11 points to the full correlation and metric-ranking results.The tie-count statistics include pairs tied at MQM=0, corresponding to error-free translations.Source: C Additional Results · p. 11, Table 10 · p. 12, Table 11 · p. 12

D Tie Calibration Psuedocode

  • Tie calibration runs in O(n^2 log n), with O(1) extra bookkeeping when applied to grouped human and metric score matrices.The pseudocode describes calibration on two score vectors; grouped use tracks each pair’s originating group.Source: D Tie Calibration Psuedocode · p. 11

F Unbabel Normalization

  • Under Unbabel normalization, τb still favors tie-predicting GEMBA-GPT-4 and τ10 remains biased against them; fewer ties did not significantly change those biases.Non-perfect ties decreased from 44% to 37%. The authors expect larger changes under Google normalization; Table 21 says group-by-item results largely follow the main-body results.Source: F Unbabel Normalization · pp. 12–13, Table 21 · p. 16

E Epsilon Search Approximation

  • With 10% of possible pairs, epsilon-search approximations were precise; the largest observed differences over 30 iterations were 2.5e-3 for epsilon and 4.3e-5 for acceq.The figure plots the best epsilon found on downsampled pairs and the resulting acceq.Source: E Epsilon Search Approximation · p. 12

Related papers

Loading 2305.14324v2…