The price of debiasing automatic metrics in natural language evaluation

• Automatic metrics including BLEU, ROUGE, METEOR, and CiDER have been found to correlate poorly with human metrics across datasets and systems.Human evaluation is often too expensive, motivating automatic evaluation for generation tasks with diverse correct responses.Source: §1 · p. 1
• The control variates estimator averages differences between human judgments and automatic metrics, reducing variance and cost when the two are correlated.Source: §1 · p. 1
• The estimator is optimal among unbiased estimators using the same automatic metric, and efficiency depends on annotator variance and metric–human correlation.Annotator variance depends on the human evaluation prompt.Source: §1 · p. 1
• The evaluation studies state-of-the-art summarization and open-response question answering systems, using 10,000 precollected human judgments.Source: §1 · p. 1
• ROUGE-L and human judgments correlate well at system level, but instance-level correlation is low (ρ = 0.31).The figure describes local improvements in answer quality that do not significantly improve ROUGE scores, and vice versa.Source: Figure 1 · p. 2
• About 10% cost reduction was achieved in practice, while perfect metric correlation would yield data efficiency around 3 and noiseless human judgments about 1.5.Compared with a Likert survey, measuring post-editing needed to fix a generated sentence reduced annotator variance three-fold.Source: §1 · p. 2

• Prior studies report instance-level metric correlations below 0.3 for generation methods and below 0.35 for dialogue metrics, dropping below 0.16 across domains.The reported findings are from Novikova et al. (2017) and Liu et al. (2016b), respectively.Source: §2 · p. 2
• The paper examines open-response question answering on MS MARCO, where systems answer human-generated questions using supplied context.Source: §2 · p. 2
• Instance-level ROUGE correlation is only ρ = 0.31; it scores bad examples low more reliably than it judges good examples.This pattern agrees with Novikova et al. (2017), who found metrics correlate better on bad examples than average or good ones.Source: §2 · p. 2
• On MS MARCO, a majority of system responses were correct but poorly scored by ROUGE-L; CNN/Daily Mail examples show VecSim and human ratings disagreeing in both directions.Source: Table 1 · p. 3
• Automatic metrics can miss genuine quality improvements: ROUGE optimization has produced systems indistinguishable from human summaries on ROUGE but hardly improved on human evaluation.Hillclimbing on ROUGE can also produce worse human scores, while genuine quality improvements may not raise ROUGE.Source: §2 · pp. 2, 4

3.1 Sample mean

• The sample-mean baseline independently samples system outputs and human judgments, although annotators are not fully independent in practice.The baseline estimates the average human judgment using sampled outputs.Source: §3.1 · p. 4

3.2 Control variates estimator

• The estimator averages noisy human judgments and adjusts them with standardized g(z), reducing variance when it correlates with f(z).Figure 2 illustrates the same-mean, lower-variance comparison between f(z) and f(z) − g(z).Source: §3.2 · p. 4, Figure 2 · p. 5
• Among unbiased estimators using the same human judgments and automatic metric, the control-variates estimator has the least worst-case variance.The optimality result is stated for distributions with fixed variance and correlation parameters.Source: §3.2 · p. 4
• 2x data efficiency requires |ρ| ≥ 0.707, and even perfect correlation is capped unless γ approaches 0.Data efficiency is the multiplicative reduction in samples needed versus the sample-mean estimator.Source: §3.2 · p. 5

3.3 Using the control variates estimator

• Implementation samples human judgments but evaluates the metric on far more outputs; estimating its covariance coefficient adds O(1/n) bias.The paper notes this bias quickly becomes small relative to the standard deviation, which decays as Θ(1/√n).Source: §3.3 · p. 5

3.4 Discussion of assumptions

• Efficiency depends on metric-human correlation and prompt-linked annotator variance; γ is high for most prompts, with Edit a notable exception.The discussion also suggests richer annotator models or metric confidence information as possible routes to further gains.Source: Table 2 · p. 6, §3.4 · p. 6

Evaluating language quality in automatic summarization

• For summaries, workers rated fluency, redundancy, and overall quality on a 1–3 scale or post-edited outputs; post-editing cost about $0.40 per summary.The study used CNN/Daily Mail articles paired with reference summaries and left content selection for future work.Source: §4 · p. 6

Evaluating answer correctness

• For MS MARCO answers, annotators judged plausibility and correctness against each context paragraph, while AvgCorrect and AnyCorrect captured different paragraph-level outcomes.Some answers could be correct under one context and incorrect under another.Source: §4 · pp. 6–7
• 62% of incorrectly annotated reference answers were answerable from some paragraph, indicating a real task ceiling around 90%.The authors manually inspected 30 reference answers marked incorrect; about 95% were indeed incorrect.Source: §4 · p. 7

5 Experimental results

• Correlations between automatic metrics and human judgments vary across systems, with token-based metrics stronger for more extractive systems.The evaluated metrics include BLEU, ROUGE, METEOR, and VecSim.Source: §5 · p. 7, Figure 5 · p. 8
• 1.08–1.15 data efficiency (7–13% cost reduction) is achieved across tasks, prompts, and systems using current automatic metrics.The reported efficiency varies with task, prompt, and system.Source: §5 · p. 8
• Improving the evaluation prompt or automatic metric reduces variance and can increase data efficiency.Post-editing decreases variance relative to a Likert-scale Overall prompt, while ROUGE-L reduces variance relative to VecSim.Source: §5 · p. 8
• With zero annotator variance, data efficiency ranges from 1.42 to 1.69; with perfectly correlated automatic metrics, it ranges from 2.38 to 7.25.These estimates indicate potential gains from improving either human evaluation prompts or automatic metrics.Source: §5 · p. 8

6 Related work

• Learned automatic metrics can struggle to generalize across systems, while the framework incorporates them without introducing bias when their correlation is low.It can exploit such metrics when their correlation is high.Source: §6 · p. 8
• Control variates reduce variance in Monte Carlo estimates and, to the authors’ knowledge, are first applied here to language evaluation.Prior applications cited include machine learning and reinforcement learning.Source: §6 · p. 8
• A related bias-correction approach for knowledge-base population relies on shared structured relation triples and does not apply to natural-language generation.Chang et al. (2017) instead dynamically collect human feedback to learn dialog policies.Source: §6 · pp. 8–9
• Figure 6 compares confidence-interval lengths across prompts and metrics; its caption reports data efficiency of 1.06 for VecSim with Overall and 1.15 with Edits.The caption also reports that replacing VecSim with ROUGE-1 changes data efficiency from 1.03 to 1.16.Source: Figure 6 · p. 9

7 Discussion

• Current automatic metrics and prompts yield data efficiencies of 1.08–1.15, equivalent to 7–13% cost reduction.Post-edits of summaries decreased normalized annotator variance by a factor of three relative to a Likert-scale survey.Source: §7 · p. 9
• Further gains require improving automatic-metric correlation and reducing annotator variance through the evaluation prompt.The authors caution that changing the prompt also changes the underlying ground truth f(z).Source: §7 · p. 9
• Improving automatic metrics may be as difficult as solving the task, while alternative ways of soliciting evaluation have been less explored.The authors also raise stable ranking of methods as an alternative to measuring absolute scores.Source: §7 · p. 9
• The authors frame the estimator’s account of unbiased-estimation limits as applying without stronger assumptions.They suggest that human evaluation will become more important as NLP tasks become more difficult.Source: §7 · p. 9

A Crowdsourcing data collection

• The section describes annotation-interface design and quality-control measures.Source: A Crowdsourcing data collection · p. 12

A.1 Language quality evaluation.

• Five-level Likert ratings increased annotator variance relative to three-level ratings, while requiring post-edits significantly decreased it.Annotators rated fluency, redundancy, and overall quality, then post-edited summaries rated as lacking on any facet.Source: §A.1 · p. 12
• Annotators completed an interactive tutorial with guidelines and examples for rating fluency, redundancy, and overall quality.The tutorial also tested whether workers could identify and correct language errors in the post-editing interface.Source: §A.1 · p. 12
• Annotators could not reliably identify focus, coherency, and referential-clarity errors in short summaries.They also could not reliably localize language errors when asked to highlight them.Source: §A.1 · p. 12
• Requiring post-edits reduced spam; other rejection criteria included task duration, Likert agreement, and edits consistently shorter than five characters.Attention-check ratings were too subjective to reliably identify poor work.Source: §A.1 · p. 12

A.2 Answer correctness evaluation.

• Annotators judged whether each question made sense and its answer was plausibly correct, then identified paragraphs justifying the answer.Figure 8 shows the crowdworker interface and instructions for this task.Source: §A.2 · p. 12, Figure 8 · p. 14
• Asking whether questions made sense and answers were plausible addressed confusion from ambiguous questions and implausible system responses.Annotators were also asked to double-check their work after pilots found they sometimes marked an unrelated paragraph as correct.Source: §A.2 · p. 12

B Proofs

• The B Proofs section provides proofs for the theorems stated in the main paper.Source: B Proofs · p. 15

B.1 Main Theorem

• For the stated Gaussian distribution, the control-variate estimator is the minimal variance unbiased estimate.The proof uses sufficiency and the Lehmann–Scheffé theorem.Source: §B.1 · p. 16
• The proof establishes sufficiency by decomposing the expression into data-only terms and terms depending on the estimator and μ.It then applies the Lehmann–Scheffé theorem to obtain the MVUE result.Source: §B.1 · pp. 16–17
• The control-variate estimator has minimax variance among the stated unbiased estimators; no other estimator has lower worst-case variance.The section identifies the estimator’s minimax variance with its variance.Source: §B.1 · p. 17

B.2 Added Bias

• Proposition 3.1 states that Algorithm 1 has O(1/n) bias.Source: §B.2 · p. 17