On the Limitations of Reference-Free Evaluations of Generated Text

• Reference texts can be expensive to collect or unavailable in real-time applications, motivating interest in metrics that evaluate generated text without references.Source: §1 · p. 1
• Because they use the same input as generation models, reference-free metrics can be viewed as underlying generation models evaluating other models.Source: §1 · p. 2
• Their underlying models define the best possible metric score, enabling approximate inference procedures to find outputs that score best under the metric.Source: §1 · p. 2
• Metrics can favor outputs from similar models and disfavor higher-quality outputs, including human-written text, encouraging systems to become more like the metric’s model and less human-like.Source: §1 · p. 2
• The paper recommends using reference-free metrics as diagnostic statistics for model behavior rather than measures of progress on tasks such as machine translation.Source: §1 · p. 2

• A conditional text generation model scores an output y for an input x and uses an inference procedure to find an output at test time.Source: §2 · p. 2
• Reference-free metrics score a candidate without a gold reference, unlike reference-based metrics that can use the reference text.Source: §2 · p. 2
• Because both the model and reference-free metric are functions of the input and output, the metric can itself be viewed as a conditional generation model.Source: §2 · p. 2
• An inference procedure for a reference-free metric can in principle return its highest-scoring output, though computing it may be expensive.Source: §2 · p. 2

• Machine translation experiments use WMT’19 metrics-task data spanning 18 language pairs and 10 to 20 systems; summarization uses SummEval and REALSumm outputs from 16 and 25 models.Source: §3 · p. 3
• Prism-src scores translations by their log-probability conditioned on the source text under a learned multilingual sequence-to-sequence model.Source: §3 · p. 3
• COMET-QE uses COMET’s architecture to score a candidate translation from the candidate and source, omitting the reference used by COMET.Source: §3 · p. 3
• QuestEval scores summaries using question-answer pairs generated from the source document and summary, optionally weighting source questions with a learned query model.Source: §3 · p. 3
• The analysis uses BLEU, ROUGE, BERTScore, QAEval, and BLEURT as reference-based metrics with established correlations to human judgments.Source: §3 · p. 3

4 Metric Optimization

• Metric optimization uses approximate inference procedures that can run at test time without a reference text.Source: §4 · p. 3

4.1 Direct Optimization

• For Prism-src, beam search with its learned translation model produces an approximate best translation under the metric.Source: §4.1 · pp. 3–4

4.2 Greedy Optimization for Extractive Summarization

• Greedy extractive optimization repeatedly selects the sentence that most increases the metric score until the target of k sentences is reached.Source: §4.2 · p. 4

4.3 Reranking

• Exact inference by enumerating all outputs is computationally intractable for practical text-generation applications, motivating approximate procedures.Source: §4.3 · p. 4
• Reranking restricts the search space to candidates from a pretrained task model, then scores and orders those candidates with the reference-free metric.Source: §4.3 · p. 4

5.1 Approximate Inference Effectiveness

• Across all MT language pairs and both summarization datasets, inference algorithms achieved the highest reference-free scores; on de→en, COMET-QE rose 38% relative to the best-scoring WMT’19 submission (0.347 → 0.478).The procedures worked even when the metric did not directly support efficient inference.Source: §5.1 · pp. 4–5
• The optimized outputs were often average or worse by reference-based metrics, including QuestEval-optimized REALSumm summaries, which were among the dataset’s lowest by ROUGE-2.The authors used reference-based metrics as quality indicators because human ratings would require rejudging other systems’ outputs.Source: §5.1 · p. 5

5.2 Undesirable Metric Biases

• Prism-src and QuestEval often scored learned-model outputs above human references, while this bias was potentially less severe for COMET-QE, trained on human judgments that included human-written translations.The authors suspect that task-trained internal models and generation models learn similar signals, which may help explain the bias.Source: §5.2 · p. 6

5.3 Reference-Free Metrics as Pseudo-References

• Metric-optimized outputs used as pseudo-references yielded strongly correlated system scores: BERTScore correlations averaged 0.95 with Prism-src and 0.92 with COMET-QE, while QAEval correlations averaged 0.88 with QuestEval.The analysis compares system-level scores from reference-based metrics using pseudo-references with scores from reference-free metrics.Source: §5.3 · pp. 6–7, Figure 5 · p. 7, Figure 6 · p. 7
• Because a metric’s pseudo-reference is its own underlying model’s output, a higher-quality output can be penalized for differing from it, making the metric misleading for systems better than that model.The analysis frames this as a limit on evaluating other systems with the reference-free metric.Source: §5.3 · pp. 6–7

6.1 Reference-Free Evaluation

• Reference-free scores should not rank task progress: the metric’s underlying model is already the known best scorer, and inference procedures can optimize its score.Under this evaluation approach, gains shift toward better inference-time metric optimization rather than better task-trained models; resulting system quality is limited by the metric.Source: §6.1 · p. 8
• When references are unavailable, the authors recommend collecting human-written references for evaluation because reference-based metrics encourage human-like outputs.They acknowledge that individual reference-based metrics have flaws.Source: §6.1 · p. 8

6.2 What about Inherently Reference-Free Evaluations?

• For inherently reference-free aspects such as fluency or faithfulness, the authors recommend using reference-free metrics as diagnostics rather than measures of task progress.A summary’s perplexity under a large language model can approximate fluency, but measures how likely the text is under that model and is biased toward it.Source: §6.2 · p. 8
• Reference-free scores can be useful statistics to report, but maximizing them should not become the goal, including for metrics applied to aspects that ought to use references.Source: §6.2 · p. 8

Related work

• Reference-free metrics have been proposed across MT, summarization, dialogue, image captioning, and simplification, and some settings report that they outperform reference-based metrics.Source: §7 · p. 8
• The authors warn that stronger human-judgment correlations may lead readers to treat metric types as interchangeable, and recommend presenting reference-free metrics as complements rather than replacements.They recommend authors state the metrics’ limitations clearly.Source: §7 · p. 8
• Quality-estimation metrics can flag translations for post-editing or catastrophic errors, but their intended purpose is not to rank and compare MT systems.Source: §7 · pp. 8–9
• The authors recommend calling proposed no-reference evaluation methods “QE metrics” rather than “reference-free metrics” to make their intended distinction clearer.Source: §7 · p. 9
• Agrawal et al. (2021) found no evidence that Prism-src’s pseudo-reference similarity bias harmed system rankings, while the authors argue their own results indicate that the bias exists.The authors distinguish evidence of bias from evidence that it negatively affects rankings.Source: §7 · p. 9
• The reranking experiment illustrates that reference-free scores can be optimized during inference; the authors say its aim was to expose this possibility, not propose a new technique.Reference-free reward functions can be optimized directly, unlike reference-based approaches that approximate their reward function.Source: §7 · p. 9

Conclusion

• Reference-free metrics can be optimized at test time, favor their underlying or similar models, and be biased against higher-quality text, including human-written outputs.Source: §8 · p. 9

Limitations

• Experiments cover one translation dataset and two summarization datasets, so the extent of the experimentally demonstrated bias on other tasks is unknown.Source: Limitations · p. 10
• The paper provides no clear alternative for automatically evaluating inherently reference-free aspects without the identified problems, and it is unclear whether such an alternative is possible.Source: Limitations · p. 10
• The §5.1 analysis uses reference-based metrics as surrogates for human judgments of output quality rather than direct human judgments.The authors state that human evaluation of inference-algorithm outputs was outside the paper’s scope.Source: Limitations · p. 10

A Implementation Details

• Experiments ran on one Titan RTX GPU with 24 GB of memory, and most scoring runs finished in several hours.The experiments did not require training models and were described as relatively inexpensive.Source: A Implementation Details · p. 13
• ROUGE, BLEU, and QAEval used SacreROUGE or SacreBLEU implementations, while BLEURT used the BLEURT-base-128 version.ROUGE and QAEval used SacreROUGE; BLEU used SacreBLEU.Source: A Implementation Details · p. 13
• BERTScore used RoBERTa-Large embeddings for English and default contextual embeddings for other languages; QuestEval used v0.1.1.Prism-src and COMET-QE were based on the authors’ original code, and the implementations were available in Repro.Source: A Implementation Details · p. 13

B Additional Results

• The section provides additional results to supplement those in the main body of the paper.Source: B Additional Results · p. 13
• In every setup, inference-generated outputs performed best compared with all other available systems in the datasets.Table 1 maps each setup to a figure showing reference-based and reference-free scores for optimized outputs and the reference text.Source: B Additional Results · p. 13, Table 1 · p. 14
• Tables 2–4 report correlations between systems’ reference-free scores and reference-based scores calculated against metric-specific pseudo-references.Prism-src uses direct optimization or reranking; COMET-QE uses reranking; QuestEval uses greedy extractive summarization or reranking.Source: B Additional Results · p. 13, Table 2 · p. 14, Table 3 · p. 14