Source-linked AI summary
Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation
Bryan Eikema, Wilker Aziz
TL;DR
The paper asks whether NMT pathologies reflect the model and MLE or instead the MAP decoding rule used to inspect them. It analyzes learned distributions with sampling and distribution-wide decision rules, finding that they reproduce data statistics better than beam outputs and that approximate MBR decoding performs competitively.
Problem
Prior NMT pathologies are mainly interpreted through MAP decoding, leaving it unclear whether they reflect NMT and MLE or the mode-based decision rule.
Method
The paper evaluates MLE-trained NMT distributions across settings using ancestral samples, comparisons with beam outputs, and an approximate minimum Bayes risk decision rule.
Results
NMT distributions reproduce many data statistics well, while beam search outputs often stray from them; approximate MBR decoding achieves competitive results.
Takeaways & Limitations
NMT should be criticized as a probability distribution and predictions should use decision rules that account for the distribution holistically.
Takeaways & Limitations
For English–German and German–English, the test domain is not considered out-of-domain because both training and test data concern newswire.
Abstract
from arXiv · showhide
Recent studies have revealed a number of pathologies of neural machine translation (NMT) systems. Hypotheses explaining these mostly suggest there is something fundamentally wrong with NMT as a model or its training algorithm, maximum likelihood estimation (MLE). Most of this evidence was gathered using maximum a posteriori (MAP) decoding, a decision rule aimed at identifying the highest-scoring translation, i.e. the mode. We argue that the evidence corroborates the inadequacy of MAP decoding more than casts doubt on the model and its training algorithm. In this work, we show that translation distributions do reproduce various statistics of the data well, but that beam search strays from such statistics. We show that some of the known pathologies and biases of NMT are due to MAP decoding and not to NMT's statistical assumptions nor MLE. In particular, we show that the most likely translations under the model accumulate so little probability mass that the mode can be considered essentially arbitrary. We therefore advocate for the use of decision rules that take into account the translation distribution holistically. We show that an approximation to minimum Bayes risk decoding gives competitive results confirming that NMT models do capture important aspects of translation well in expectation.
1 Introduction
The paper argues that apparent NMT flaws may reflect MAP decoding rather than fundamental defects in NMT or MLE. It evaluates the distributions learned by MLE and advocates holistic decision rules such as approximate MBR decoding.
- Motivation: Observed NMT pathologies include underestimated output length, declining quality with larger beams, empty modes, and negative likelihood–quality correlation.These observations motivate what the paper calls the inadequacy of the mode problem.
- Problem: Existing explanations largely attribute these observations to NMT's model structure or MLE while assuming that selecting the distribution's mode is the obvious prediction rule.The paper identifies MAP decoding as the decision rule underlying that assumption.
- Problem: Mode-related pathologies cannot be unambiguously assigned to NMT or MLE because the mode may be an inappropriate summary of the learned distribution.Thus, these pathologies do not rule out models capturing important translation properties in expectation.
- Approach and findings: Across language pairs, training-data amounts, and test domains, learned distributions reproduce data statistics well while approximate MAP decoding introduces length and lexical biases.Beam outputs are especially rare events away from the training domain, whereas model samples are generally reasonable quality.
- Approach and findings: A straightforward approximation to minimum Bayes risk decoding performs well, especially when models are more uncertain.This supports decision rules that use the translation distribution holistically.
2 Observed Pathologies in NMT
Prior NMT research reports several decoding-related pathologies, especially length underestimation and worsening quality with larger beam sizes. These findings have often been linked to model normalization or search behavior.
- Length bias: NMT systems often underestimate output length, which can reduce translation adequacy, particularly for longer sequences.This length bias has been reported across multiple studies and tasks.
- Beam search curse: Studies commonly associate length bias with the beam search curse, where larger beam sizes worsen NMT performance.Larger beams have been reported to exacerbate length bias.
- Explanations: Prior explanations attribute these problems to local normalization and the resulting label bias in NMT.This frames the pathologies as consequences of model specification rather than only decoding.
- Beam search behavior: Other accounts link larger-beam degradation to excessive probability on copies of the input and to unlikely prefixes followed by nearly deterministic suffixes.These outputs are associated with lower translation quality.
3 NMT and its Many Biases
NMT uses locally normalized conditional distributions trained with MLE, and predictions are typically approximated by beam-search MAP decoding. The literature attributes several resulting biases to training–test mismatch, search heuristics, and local normalization.
- Model and training: NMT models generate target words sequentially from locally normalized Categorical distributions conditioned on the source and previous target prefix.The neural network maps each source–prefix pair to vocabulary probabilities.
- Model and training: Models are typically trained with regularized maximum likelihood estimation, or MLE.The supplied passage introduces MLE as the standard training procedure.
- Prediction: At test time, NMT predicts the sequence maximizing model log probability, known as MAP decoding, with beam search used because exact decoding is intractable.Beam search is therefore an approximation to mode seeking.
- Biases: Exposure bias arises because MLE conditions on training observations that are unavailable during test-time search through the learned distribution.This mismatch has motivated training modifications intended to expose models to their own predictions.
- Biases: Beam search can prune good translations early because partial hypotheses are ranked without adequately accounting for their future scores.This is characterized as non-admissible heuristic search bias.
- Biases: Label bias occurs when local normalization prevents a conditional model from revising decisions about variables it does not generate.The paper describes this as a model-specification limitation on representable distributions.
4 Biased Statistics and the Inadequacy of the Mode
The paper argues that mode-based decoding produces biased statistics because NMT distributions spread probability across many translations. It therefore uses unbiased ancestral samples to assess model fit and distinguish distributional behavior from search artifacts.
- Biased statistics: Mode-based criticism can misrepresent an NMT distribution because a single or few decoded outcomes may be rare and need not reflect distribution-wide statistics.A short mode, for example, does not imply underestimated average sequence length.
- Inadequacy of the mode: In NMT's high-dimensional sample space, the mode may contain only a tiny fraction of total probability mass and be extremely unlikely under the learned distribution.This makes the mode an inadequate decision statistic when probability mass is broadly dispersed.
- Ancestral sampling: Ancestral sampling generates unbiased model samples by sequentially drawing from the conditional Categorical distributions until the end-of-sequence symbol appears.These samples imitate the model's generative process and can diagnose fit or approximate distribution-aware decision rules.
5 Data & System
The study trains NMT systems in multiple language directions and evaluates model statistics against references and beam-search outputs. Figure 1 compares these groups using hierarchical Bayesian analyses of length and training-data agreement.
- Data and systems: Systems cover German-English, Sinhala-English, and Nepali-English in both translation directions.German-English uses about 5.9 million sentence pairs and a Transformer base model; Sinhala and Nepali use a low-resource setup.
- Decoding: The paper relates replacing ancestral sampling with approximate MAP decoding to exposure bias.
- Analysis: The analysis compares beam-search outputs, model samples, and gold-standard references using posterior distributions of extracted statistics.Figure 1 displays posterior density against mean Poisson rate for length and agreement with training data for unigrams, bigrams, and skip-bigrams.
6 Assessing the Fit of MLE-Trained NMT
The authors assess whether MLE-trained NMT distributions fit observed translation statistics by comparing unbiased samples with references and beam-search outputs. Samples generally match reference statistics better than beam search, whose outputs diverge from data patterns.
- Evaluation design: The held-out training-data evaluation removes domain-shift confounding and tests whether sampled statistics match observed reference statistics.
- Evaluation design: Hierarchical Bayesian models jointly analyze statistics from references, ancestral samples, and beam-search outputs, incorporating training-data statistics.
- Length statistics: NMT samples capture sequence-length statistics reasonably well and overlap substantially with gold-standard references.The comparison uses expected posterior Poisson rates, interpreted as average sequence lengths.
- Length statistics: Beam-search outputs stray from length statistics and usually produce shorter sequences than the data.
- Probability mass: Figure 2 plots cumulative probability for unique translations among 1,000 ancestral samples across held-out and test-domain sets.The figure shows an average curve, a one-standard-deviation band, and final per-sentence values.
- Lexical statistics: For unigram, bigram, and skip-bigram agreement, references generally align most with training data, followed by model samples and beam-search outputs.Higher concentration values indicate closer resemblance to training-data statistics.
7 Examining the Translation Distribution
The translation distribution spreads probability across many candidates, often becoming flatter in the test domain, while beam-search outputs can be rare events. Samples are reasonably consistent and, when selected with an oracle or MBR, can match or outperform beam search in several settings.
- 7.1 Number of Likely Translations: After 1,000 samples, held-out data covered 16.4%–57.8% of probability mass, with substantially more uncertainty in the test domain.Coverage varied widely across individual inputs, showing that sampling explored only part of the translation space.
- 7.2 Sampling the Mode: Beam-search outputs appeared in 54.3%–92.2% of held-out inputs but only 4.8%–49.3% of test-domain inputs among 1,000 unbiased samples.Thus, beam search often selected a rare outcome under the model, particularly in low-resource test-domain settings.
- 7.2 Sampling the Mode: The empty sequence was sampled at least once for 7.2%–29.1% of held-out inputs and 2.8%–33.3% of test-domain inputs, but occurred only 1.2 ± 0.5 times on average.Even when it may be the true mode, the empty string remained an unlikely sampled event.
- 7.3 Sample Quality: Average ancestral samples always scored below beam search, with gaps of 0–14.4 METEOR and standard deviations below 0.2.Sampling is used to explore the distribution rather than as a complete decision rule, while beam search uses adjustments such as beam size and length penalty.
- 7.3 Sample Quality: Oracle-selected samples improved steadily with sample size; 5–10 samples beat beam search in low-resource pairs and training-domain English-German, while 15–25 were needed for test-domain English-German.With fewer than 30 random samples, oracle selection matched or exceeded beam search in most cases, motivating sampling-based decision rules.
- 7.4 Minimum Bayes Risk Decoding: Minimum Bayes risk decoding using 30 ancestral samples considerably outperformed average single-sample quality, often matched beam search, and consistently beat it for low-resource pairs.It used METEOR as the utility function and estimated expected utility from sampled translations; more samples may be needed for test-domain English-German.
8 Related Work
Prior work motivates sampling-based decision rules by showing that beam search can distort translation statistics and that MBR offers a distribution-aware alternative. Existing NMT sampling results also suggest strong gains over beam search.
- Beam search and MBR: Random samples reproduce unigram statistics better than beam-search outputs, which stray from the data’s statistics.
- Distributional structure: NMT distributions can be flat: 10,000 samples from a high-resource English-French system explored less than 25% of the probability space.Despite this, likely translations share statistics with references, motivating MBR.
- Decision-rule motivation: MAP decoding is equivalent to MBR with an exact-match utility, which is inadequate because translation admits multiple valid solutions.
- Beam search and MBR: Beam search can emphasize rare, inadequate translations because it defines the support and expected utility using high-scoring hypotheses.This may explain why prior MBR approximations required very large beams.
- Sampling-based decision rules: Up to 3 BLEU points of improvement over beam search was reported by reranking 100 sampled translations according to oracle BLEU.The result supports sampling-based decision rules in NMT.
9 Conclusion
The conclusion argues that MAP decoding is poorly suited to MLE-trained NMT because modes represent little probability mass and can induce misleading pathologies. It therefore advocates holistic, sampling-based decision rules, with MBR achieving competitive results.
- 9 Conclusion: MAP decoding is not well-suited to MLE-trained NMT because the mode often represents little probability mass in a distribution spread across many translations.
- 9 Conclusion: Many observed NMT pathologies and biases are at least partially due to approximate MAP decoding rather than inherent properties of the model or training objective.
- 9 Conclusion: Samples can outperform beam-search outputs even at small sample sizes, and an approximate MBR decoder achieves competitive results.Likely translations share statistics that correlate well with the reference.
- 9 Conclusion: The paper advocates evaluating MLE-trained NMT models as probability distributions and using decision rules that account for those distributions holistically.
A.1 Length Analysis
The length analysis models sequence lengths with hierarchical Gamma-Poisson distributions, assigning individual Poisson rates while sharing population- or group-level priors. Test groups extend the training model through group-specific parameterization.
- A.1 Length Analysis: Training sequence lengths are modeled with a hierarchical Gamma-Poisson model, where each sequence has its own Poisson rate.The rates share a population-level Gamma prior with Exponential hyperpriors.
- A.1 Length Analysis: The model is flexible because each data point receives an individual Poisson rate, while test groups use a slightly different Gamma-Poisson parameterization.
- A.1 Length Analysis: Test-group Poisson rates share group-level Gamma priors, with sg scaling the expected posterior training rate for each group.Posterior approximations are inferred with SVI, then predictive samples are compared with observed first- through fourth-order moments.
A.2 Lexical & Word Order Analyses
The lexical and word-order analysis uses hierarchical Dirichlet-Multinomial models for unigram and bigram counts. Test-group concentration parameters encode agreement with the training lexical distribution, and posterior predictive checks assess fit.
- A.2 Lexical & Word Order Analyses: Unigram and skip-bigram counts are modeled with hierarchical Dirichlet-Multinomial distributions.
- A.2 Lexical & Word Order Analyses: The model uses V + 1 Dirichlet-Multinomial models: one unigram model and V bigram models sharing a common Gamma prior.Here V is the BPE vocabulary size.
- A.2 Lexical & Word Order Analyses: Test-group lexical distributions are formed by scaling normalized posterior concentration from the training group.The scalar sg for unigrams or mg for bigrams measures agreement with the training distribution.
- A.2 Lexical & Word Order Analyses: Posterior predictive samples are checked against observed unigram and bigram data using absolute frequency errors and ranking correlation.