Source-linked AI summary
High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics
Markus Freitag, David Grangier, Qijun Tan, Bowen Liang
TL;DR
The paper questions whether the most probable NMT output is also the highest-quality translation, given the weak correspondence between model estimates and human quality. It applies sampled MBR with neural metrics, and reports significantly higher expert human ratings for BLEURT-based MBR than for beam search, despite lower-probability, different translations.
Problem
The paper questions the assumption that the highest-probability NMT output is also highest in human-rated quality, noting that model probability and translation quality do not always correlate.
Source: §1 · p. 1Method
The paper applies sampling-based MBR with neural reference-based metrics, selecting candidates by expected utility over unbiased samples rather than by model probability.
Source: §3.2 · p. 4Results
BLEURT-based MBR significantly outperforms beam search in expert human evaluation, producing translations that are less likely and differ from beam and overlap-based MBR outputs.
Source: §1 · p. 2, §8 · p. 10Takeaways & Limitations
In these experiments, higher-quality human-rated translations can differ from beam output and have lower model probability.
Source: §1 · p. 2, §8 · p. 10MBR uses model-distribution expectations as a proxy for human-translator expectations, and metric validity across reference and hypothesis spaces is not guaranteed.
Source: §1 · p. 2
Abstract
from arXiv · showhide
In Neural Machine Translation, it is typically assumed that the sentence with the highest estimated probability should also be the translation with the highest quality as measured by humans. In this work, we question this assumption and show that model estimates and translation quality only vaguely correlate. We apply Minimum Bayes Risk (MBR) decoding on unbiased samples to optimize diverse automated metrics of translation quality as an alternative inference strategy to beam search. Instead of targeting the hypotheses with the highest model probability, MBR decoding extracts the hypotheses with the highest estimated quality. Our experiments show that the combination of a neural translation model with a neural reference-based metric, BLEURT, results in significant improvement in human evaluations. This improvement is obtained with translations different from classical beam-search output: these translations have much lower model likelihood and are less favored by surface metrics like BLEU.
1 Introduction
The introduction questions whether the most probable translation is also the highest-quality one, and proposes sampled MBR decoding with automated quality metrics as an alternative to beam search. It reports that BLEURT-based MBR earns higher human ratings than beam search, while producing less likely, different translations.
1 Introduction
- Beam search approximates the highest-probability translation, but estimated probability and human-rated quality do not always correlate.The example contrasts frequent-word beam output that is inaccurate with correct human translations containing infrequent, low-probability words.Source: §1 · p. 1, Table 1 · p. 2
- The paper applies sampling-based MBR, selecting the candidate with the highest average utility when sampled hypotheses serve as pseudo-references.The samples are drawn from the NMT model without bias.Source: §1 · p. 2
- MBR relies on model-distribution expectations as proxies for human-translator expectations, and metric validity may be limited across reference and hypothesis spaces.Even high utility against human references does not guarantee high human judgments for hypotheses outside metric design conditions.Source: §1 · p. 2
- BLEURT-based MBR translations receive significantly higher human quality ratings than beam search and other MBR hypotheses.The BLEU-based MBR improvement over beam search is slight, and those translations differ only slightly.Source: §1 · p. 2
- The authors introduce neural metrics YISI and BLEURT as MBR utility functions and assess different utilities through professional-translator evaluation.The motivation is that learned metrics estimate human judgments and can handle choices beyond lexical overlap.Source: §1 · pp. 1–2
- BLEURT-based MBR produces less likely translations that differ from beam output and MBR outputs using overlap-based utilities.These translations are less favored by surface metrics such as BLEU.Source: §1 · p. 2
2 Related Work
Related work traces MBR from statistical decision theory through earlier machine-translation applications, where overlap metrics and beam-derived candidate lists were common. This paper builds on unbiased sampling for MBR and applies neural evaluation metrics as utility functions.
2 Related Work
- Earlier MBR work applied the decision-theoretic approach to parsing, speech recognition, word alignment, and machine translation.For statistical MT, prior work used MBR to maximize overlap metrics such as BLEU.Source: §2 · p. 3
- Neural MT commonly used beam search to approximate MAP decoding, while other efforts optimized metrics such as BLEU by changing training procedures.Those approaches included structured risk minimization and reinforcement learning.Source: §2 · p. 3
- Sampling-based MBR uses unbiased model samples rather than beam-derived k-best lists, which prior work found more faithful to training-data statistics.The paper adopts this approximation in all experiments.Source: §2 · p. 3
- The paper extends MBR beyond overlap-based utilities by using neural metrics that score hypothesis-reference pairs or source-hypothesis-reference triplets.Prior neural metrics include representation-based YISI and BERTSCORE, and fine-tuned metrics such as BLEURT and COMET.Source: §2 · p. 3
3 Method
MBR selects a candidate by estimated expected utility against reference hypotheses, using the translation model to approximate an unknown human-reference distribution. The method applies lexical, embedding-based, and fine-tuned neural metrics as utility functions.
3.1 Minimum Bayes Risk Decoding
- MBR selects the candidate with highest expected utility over human references, substituting the model distribution because the human-reference distribution is unknown and intractable to integrate directly.This substitution assumes the model approximates the true human-translation distribution; a finite sample of model-generated pseudo-references estimates the expectation.Source: §3.1 · p. 3
- Using the same sample as both candidates and pseudo-references makes a larger set improve the expectation estimate and expand the candidate pool, but increases computation.Sampling cost is linear in set size, while cross-utility computation is generally quadratic; independent, undeduplicated samples retain model-probability frequency without reweighting.Source: §3.1 · pp. 3–4
3.2 Utility Metrics
- BLEU uses smoothed sentence-level n-gram overlap for MBR, while CHRF compares character n-grams; corpus BLEU is reported separately.BLEU measures corpus-level n-gram precision with a brevity penalty, and sentence-level BLEU is used as an approximation during MBR.Source: §3.2 · p. 4
- YiSi-1-BERT represents embedding-based overlap by aligning contextual word-embedding n-grams bidirectionally and computing an F-score.The experiments use base-cased BERT for English evaluation and multilingual BERT for other languages.Source: §3.2 · p. 4
- BLEURT represents fine-tuned neural utilities: it scores hypothesis-reference pairs using a learned embedding and is not focused on surface overlap.BLEURT v0.1 is based on RemBERT and fine-tuned on WMT human evaluations; the v0.2 model is joint across language pairs. A BLEURT variant also uses synthetic pairs and 0–1-scaled, clipped scores.Source: §3.2 · pp. 4–5
4 Experimental Setup
The experiments compare English→German and German→English systems trained on WMT data, using beam search and sampled MBR decoding. Professional translators evaluate the main results with document-context MQM and ablations with pSQM.
4.1 Data and Model
- Experiments use English→German and German→English models trained on about 57 million filtered examples, with newstest2019 for checkpoint selection and newstest2021 for testing.The test set has two reference translations for each language direction.Source: §4.1 · p. 5
4.2 Model
- The Transformer models have six encoder and six decoder layers, 1,024 model dimensions, 8,192 hidden dimensions, and 16 attention heads; beam search uses size 4.Models are trained without label smoothing, which lowers BLEU by 0.5 points on both language pairs relative to label smoothing; MBR uses 1,000 unbiased samples per source sentence.Source: §4.2 · p. 5
4.3 Human Evaluation
- Main results use professional-translator MQM evaluation, while ablation studies use a cheaper, less time-consuming scalar evaluation with professional translators.The MQM setup is described as more reliable than typical scalar evaluation with crowd-workers.Source: §4.3 · p. 5
4.3.1 MQM
- Document-context MQM scores each segment by averaging annotator error scores, weighting major errors 5, minor errors 1, and minor punctuation errors 0.1.Nine professional translators evaluate the two directions: four for English→German and five for German→English.Source: §4.3.1 · p. 5
4.3.2 pSQM
- In English→German ablations, pSQM asks three professional translators to rate each source-and-translation segment from 0 to 6 in document context.System scores average the segment-level ratings.Source: §4.3.2 · p. 5
5 Experimental Results
MBR with BLEURT produced better human-evaluation results than beam search and other tested MBR metrics, while choosing translations that differed from high-likelihood outputs. Automatic evaluation also found that MBR selections achieved higher actual utility than beam-search output across the tested utilities.
5 Experimental Results
- The experiments assess whether MBR’s automatic utility gains transfer to human judgments, alongside comparisons of automatic scores.The study reports both automatic metrics and MQM-based evaluations by professional translators.Source: §5 · pp. 5–6, §5.2 · p. 6
5.2 Human Evaluation
- MBR decoding with BLEURT clearly outperforms beam search and MBR with sBLEU, CHRF, and YISI in human evaluation, significantly for En→De.MQM scores are lower when translation quality is better.Source: Table 2 · p. 6, §5.2 · p. 7
5.1 Automatic Evaluation
- For all five tested utilities, MBR hypotheses achieve higher actual utility than beam-search output in experiments using 1,000 unbiased samples.The tested utilities are sBLEU, CHRF, YISI, BLEURT v0.1, and BLEURT v0.2; the authors describe model-distribution expected utility as a good proxy for utility against a human translation.Source: §5.1 · p. 6, Table 2 · p. 6, Table 3 · p. 6
- BLEURT selects alternative translations with lower model likelihood that overlap-based metrics such as BLEU do not score highly.By contrast, MBR with sBLEU, CHRF, and YISI favors high-likelihood hypotheses with logP similar to MAP decodes.Source: §5.1 · p. 6
5.2 Human Evaluation
- The De→En human translation is ranked behind machine-generated translations because it contains critical errors, while the En→De human translation outperforms all machine outputs.The authors note the De→En result aligns with the official WMT21 human evaluation.Source: §5.2 · p. 7
6 Ablation
The ablations focus mainly on English→German because of space and cost constraints, testing the effects of model size, candidate-list pruning, reference choice, and quality-estimation reranking.
6.1 Smaller Model
- MBR beats beam by 0.8 BLEURT points on transformer-base, versus 7.4 on the bigger model, yet human evaluation finds lower quality with the smaller setup.The results indicate that MBR needs a good-quality candidate list to outperform beam search.Source: Table 4 · p. 7, §6.1 · p. 7
6.2 Candidate List Size
- Randomly pruning both MBR lists to 8 (En→De) or 13 (De→En) candidates already beats beam on BLEURT.Random subsampling outperforms selecting candidates by logP, and pruning the expectation list is more effective than pruning the maximization pool.Source: §6.2 · p. 8, Figure 1 · p. 8
- Pruning the En→De expectation list to 5 candidates yields major improvements over beam and requires 5,000 BLEURT computations per source sentence.The results indicate that candidate-pool quality and size on the maximization step matter more than expectation-list size.Source: §6.2 · p. 8
6.3 Oracle Experiments
- With Ref-C selection and Ref-D evaluation, cross-human oracle selection scores 0.774 BLEURT versus MBR’s 0.789, illustrating single-reference selection bias.The passage attributes greater damage to inter-translator variability in single-reference evaluation than to replacing human references with model estimates.Source: §6.3 · p. 8, Table 5 · p. 9
- MBR’s median Ref-C ranking is 8 out of 1,000 candidates, versus 181 for MAP, and it ranks above the Ref-D oracle candidate.The authors interpret these comparisons as evidence that model-estimated MBR provides more reliable quality estimates than selection with a single human reference.Source: §6.3 · p. 8, Table 6 · p. 9
6.4 Comparison to QE Metrics
- QE-based reranking beats beam and BLEURT-MBR on QE metrics, but human evaluation finds lower-quality translations, sometimes with crucial errors or source-unrelated content.The authors suggest that bad or source-unrelated unbiased samples may contribute: unlike MBR, QE reranking does not compare samples with one another to penalize outliers.Source: §6.4 · p. 9
7 How Different are Beam and MBR Hypotheses?
The section compares how closely beam and MBR translations resemble one another and reports differences in human-evaluated error categories. Cross-BLEU measures surface proximity between translations, not their quality.
7.1 Cross BLEU
- BLEURT-MBR translations have low cross-BLEU with beam outputs, unlike BLEU-, CHRF-, and YISI-based MBR outputs; human references are similarly distinct from beam.The two human translations are as different from each other as they are from BLEURT-MBR output.Source: §7.1 · p. 9, Table 8 · p. 10
7.2 MQM Error Categories
- For En→De, BLEURT-MBR reduces major Terminology/Inappropriate for context errors by one third; the table reports advantages across four error categories.The reported categories exclude those with similar counts for beam and MBR.Source: §7.2 · p. 10
8 Conclusion
The paper applies sampling-based MBR to optimize BLEU, CHRF, YISI, and BLEURT. BLEURT-based MBR significantly outperforms beam search in expert human evaluation while producing lower-probability, markedly different translations.
8 Conclusion
- Sampling-based MBR is used to optimize BLEU, CHRF, YISI, and BLEURT as an alternative to beam search.Source: §8 · p. 10
- BLEURT-based MBR significantly outperforms beam search in expert human evaluation, producing translations that differ from beam and overlap-based MBR outputs and have lower model probability.Source: §8 · p. 10