Source-linked AI summary
Source-Free MT Evaluation Is Not MT Evaluation
Baban Gain, Ramakrishna Appicharla, Asif Ekbal
TL;DR
MT adequacy is source-relative, but evaluation commonly relies on references that represent only one possible rendering. The paper proposes source-grounded evaluation and a counterfactual diagnostic, finding that prominent hybrid metrics can be more sensitive to reference corruption than source corruption.
Problem
Reference-based evaluation can conflate adequacy with similarity to one reference, while source-aware metrics may still let references dominate the judgment.
Method
The paper defines adequacy through the source–hypothesis relationship and uses counterfactual source and reference replacements to test which input governs hybrid metrics.
Results
For COMET, reference replacement causes an average score change more than ten times as large as source replacement, with reference sensitivity exceeding source sensitivity for 99.54% of analyzed examples.
Takeaways & Limitations
Source access alone is insufficient for adequacy evaluation; hybrid metrics should prioritize source–hypothesis faithfulness and use references as auxiliary evidence.
Takeaways & Limitations
Current LLM judges may exploit target-side reference comparison more effectively than cross-lingual source reasoning, so higher human-score agreement does not establish source faithfulness.
Abstract
from arXiv · showhide
Reference-based metrics remain the standard choice in machine translation evaluation, partly because quality estimation methods often correlate less well with human judgments. As a result, source-free, reference-based evaluation has become the practical norm, even though it is unfaithful to the definition of translation adequacy and unfair to systems whose outputs preserve the source meaning while differing from the reference. This paper argues that adequacy must be judged with respect to the source. A reference is only one possible rendering of the source and may introduce bias, under-specification, or errors. We further argue that source-reference-hypothesis evaluation is fair only when the judge treats the reference as auxiliary evidence rather than as the primary standard. Otherwise, even source-aware evaluation can reduce adequacy to preference towards reference. We show the existing hybrid metrics are highly reliant on reference compared to source. Our argument is not that all automatic MT metrics fail to use the source. Rather, we argue that any evaluation protocol that removes the source, or allows the reference to dominate the source, is structurally incomplete for adequacy evaluation. However, existing MT papers generally prefer reference-based metrics and use QE metrics only when reference is unavailable. We therefore call for QE to be reframed as a primary approach to source-grounded adequacy evaluation, rather than as a fallback motivated by missing references. We further call for hybrid metrics whose designs explicitly prioritize source--hypothesis faithfulness while using references only as complementary evidence.
1 Introduction
MT evaluation is difficult because adequacy concerns source meaning while automatic metrics have largely relied on references. The paper argues for source-grounded evaluation and tests whether hybrid metrics actually prioritize the source.
- Automatic metrics became the primary means of MT evaluation because human assessment is costly, time-consuming, and difficult to reproduce at scale.
- Reference-based metrics can conflate translation quality with similarity to one human rendering, especially when references are ambiguous, underspecified, noisy, or erroneous.
- QE evaluates quality from the source and hypothesis but has usually been framed as a fallback for settings where references are unavailable.
- The paper argues that adequacy should be judged primarily from the source–hypothesis relationship, with references serving only as auxiliary evidence.
- The proposed counterfactual diagnostic separately corrupts the source and reference to measure which input governs a hybrid metric’s score.
- For COMET, reference replacement changes scores more than ten times as much as source replacement, with reference sensitivity exceeding source sensitivity for 99.54% of examples.
2 Related Work
Prior work shows that metrics may use references heavily even when the source is available. This paper introduces a paired intervention that measures whether source or reference evidence governs hybrid-metric scores.
- Reference-based evaluation can favor hypotheses resembling one human rendering, although multiple realizations may be acceptable.
- Learned metrics can accept source, hypothesis, and reference inputs without materially using the source in their scores.
- ACES evaluates metric accuracy across 68 phenomena and 146 language pairs, including cases where ambiguous references make source information necessary.
- Unlike ACES, this paper holds a good hypothesis fixed and separately replaces the source and reference with incompatible examples.
- Comparing the resulting score changes measures which input governs a hybrid metric’s score rather than whether it detects a particular translation error.
3 Adequacy and Fluency
Adequacy is a source-relative property, whereas fluency is primarily a target-language property. References provide useful evidence but cannot fully specify source meaning or establish source faithfulness.
- Adequacy concerns preserving source meaning, while fluency concerns grammaticality, naturalness, and acceptability in the target language.
- A fluent translation can omit content, reverse semantic relations, alter named entities, add unsupported information, or resolve ambiguity incorrectly.
- A reference demonstrates one acceptable realization but does not completely specify the source meaning.
- In the sentence “Ravi told Mohan that his proposal had been rejected,” the source leaves the antecedent of “his” unresolved.
- A hypothesis that preserves genuine source ambiguity should not automatically be penalized when a reference selects one interpretation.
- Target-language constraints may force choices absent from the source, so multiple realizations can remain equally compatible with it.
- Agreement between a hypothesis and reference cannot establish adequacy when their shared interpretation is unsupported or inconsistent with the source.
- The paper defines adequacy strictly as preserving meaning licensed by the source, including ambiguity and underspecification where reasonably possible.
4 Hybrid Metrics are Biased Towards References
Across COMET, XCOMET-XXL, and MetricX-24-Hybrid-XL, replacing references generally affects scores more than replacing sources, though the strength and consistency vary by metric and direction. These results show that source access alone does not ensure source-grounded adequacy evaluation.
- COMET: COMET’s reference sensitivity was 10.43× its source sensitivity, with reference effects larger in 99.54% of examples.Mean source and reference sensitivities were 0.0417 and 0.4354, respectively; the median sensitivities were similarly separated.
- COMET: COMET showed reference dominance across directions, with Ref/Src ratios of 11.19× into English and 9.47× out of English.Reference sensitivity was larger for 99.92% of into-English examples and 99.01% of out-of-English examples.
- COMET: COMET’s reference dominance was not language-pair-specific: every examined direction had a ratio of at least 8× and dominance of at least 96%.The en-es ratio was 8.04×, while en-ru reached 15.43× with a 96.0% dominance rate.
- Overall observations: Overall, hybrid metrics can use the source while remaining more dependent on the reference, so input access alone is insufficient evidence of source-grounded adequacy evaluation.The pattern is systematic for COMET and XCOMET-XXL but direction-dependent for MetricX.
- XCOMET-XXL: XCOMET-XXL responded strongly to both inputs, but reference sensitivity still exceeded source sensitivity in 96.18% of examples.Mean source and reference sensitivities were 0.4075 and 0.6100, respectively, indicating a smaller asymmetry than COMET.
- XCOMET-XXL: XCOMET-XXL remained reference-dominated across directions, with Ref/Src ratios from 1.42× to 1.87× and dominance rates from 79.0% to 100.0%.Reference sensitivity exceeded source sensitivity in all 14 directions, although example-level consistency was lower for some pairs.
5 LLMs as Judges and the Source-Free Evaluation Problem
LLM judges can evaluate MT with or without references, but controlled studies show that reference-based comparison often contributes more to human-score agreement than source-based reasoning. This higher agreement does not establish source-faithful adequacy, and source-dependent tests reveal failures on phenomena where the source is decisive.
- Evaluation settings: LLM evaluation protocols differ in whether they provide the source, reference, or both alongside the hypothesis.GEMBA-MQM and AutoMQM can operate without references by prompting judges to identify source-grounded translation errors.
- Empirical evidence: Across multiple models, reference-and-translation inputs generally contribute more to human-score agreement than source-and-translation inputs.For GPT-3.5-Turbo, the reference-and-translation configuration reaches system-level accuracy 0.891.
- Interpretation: Higher agreement from reference comparison does not show that the source is unnecessary or that the resulting judgment faithfully assesses adequacy.The results may instead reflect stronger monolingual target-side comparison than reliable cross-lingual reasoning.
- Source-dependent failures: Judges can achieve high average correlation while failing on omissions, additions, ambiguity, negation, named entities, and other source-decisive phenomena.Source-dependent challenge sets report unreliable LLM-based evaluation when the better hypothesis can only be identified from the source.
- Source grounding: Providing a source does not guarantee source grounding because judges may rely on fluency, plausibility, memorized patterns, or reference similarity.Hybrid judges should therefore be treated as source-grounded only when their judgments respond empirically to source-side meaning.
6 Recommendations
The paper recommends treating quality estimation as a primary form of source-grounded adequacy evaluation and redesigning hybrid metrics so references remain complementary evidence. It also calls for evaluation protocols that measure source grounding directly alongside human-score correlation.
- Quality estimation: Quality estimation should be treated as a primary approach to adequacy evaluation rather than as a fallback for unavailable references.The proposed focus is on whether hypotheses preserve source meaning, using source-grounded data and challenge sets.
- Hybrid metrics: Hybrid metrics should separately model source–hypothesis adequacy and hypothesis–reference compatibility so references cannot easily override the source.Suggested strategies include reference dropout, multiple valid references, counterfactual replacements, and source-first LLM judging.
- Evaluation protocols: Metric evaluation should report source-dependent challenge-set performance, source and reference ablations, and results on incomplete or misleading references.Results should also be separated by translation direction and include non-English-to-non-English pairs.
- Reporting standards: Source-free scores should not be presented as complete measures of translation quality.Reference-based metrics remain useful for target-side agreement, but adequacy claims require a source-grounded evaluator whose source use is empirically demonstrated.
7 Conclusion
The conclusion defines MT adequacy by preservation of source meaning and argues that source grounding, not reference availability or reference agreement, should govern adequacy evaluation. References remain useful as auxiliary evidence but should not override a valid source–hypothesis relationship.
- Conclusion: MT adequacy is defined by whether a hypothesis preserves the meaning of its source.Source-free evaluation can assess fluency or reference agreement but cannot independently establish source–hypothesis adequacy.
- Conclusion: The paper reports that COMET and XCOMET-XXL are more sensitive to reference corruption than source corruption, while MetricX varies substantially by direction.These findings support treating source grounding as an empirical criterion rather than assuming it from the presence of source and reference fields.
- Conclusion: The paper concludes that source grounding should be the primary criterion for measuring adequacy, with references retained as auxiliary evidence.References should not override a valid source–hypothesis relationship.