Source-linked AI summary
Minimum Risk Training for Neural Machine Translation
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu
TL;DR
Neural machine translation is commonly trained with MLE objectives that do not directly optimize translation-quality metrics and expose models to different inputs during training and generation. The paper introduces MRT, which minimizes expected metric-based loss using arbitrary sentence-level losses, and finds significant improvements over MLE across several language pairs. MRT is presented as architecture-transparent, but its expectations are difficult to compute over the full search space and performance is constrained by a 50-word training-length limit in one system.
Problem
MLE trains NMT on the training distribution with word-level losses rather than directly optimizing sentence- and corpus-level translation-quality metrics.
Method
MRT minimizes expected loss on training data, using evaluation metrics as arbitrary sentence-level loss functions that need not be differentiable.
Results
MRT achieves significant improvements over MLE on end-to-end NMT across Chinese-English, English-French, and English-German translation tasks.
Takeaways & Limitations
MRT can be applied to any end-to-end NMT architecture and can optimize parameters directly with respect to evaluation metrics.
Takeaways & Limitations
MRT expectations are usually intractable because of the exponential search space, non-decomposable losses, and NMT context sensitivity.
Abstract
from arXiv · showhide
We propose minimum risk training for end-to-end neural machine translation. Unlike conventional maximum likelihood estimation, minimum risk training is capable of optimizing model parameters directly with respect to arbitrary evaluation metrics, which are not necessarily differentiable. Experiments show that our approach achieves significant improvements over maximum likelihood estimation on a state-of-the-art neural machine translation system across various languages pairs. Transparent to architectures, our approach can be applied to more neural networks and potentially benefit more NLP tasks.
1 Introduction
End-to-end NMT replaces explicit SMT structures with a single neural network, but conventional MLE optimizes training-data likelihood rather than translation-quality metrics. The paper introduces MRT to optimize expected evaluation loss directly, with arbitrary sentence-level losses and no architecture restriction, and reports improvements over MLE across language pairs.
- End-to-end NMT directly transforms source sentences into target sentences without explicitly modeling latent structures used in conventional SMT.These structures include word alignment, phrase segmentation, phrase reordering, and SCFG derivation.
- MLE exposes NMT models to the training distribution and uses word-level losses, although translation quality is evaluated with sentence- or corpus-level metrics.The stated drawbacks include exposure bias and weak correspondence between word-level cross-entropy and metrics such as BLEU and TER.
- MRT minimizes expected loss on training data instead of maximizing training-data likelihood.The objective is framed as minimizing risk over the training examples.
- MRT directly optimizes evaluation metrics by using them as loss functions, including arbitrary sentence-level losses that need not be differentiable.This allows the training objective to target metrics that quantify translation quality directly.
- MRT does not assume a specific NMT architecture and can be applied to any end-to-end NMT system.The approach is presented as architecture-transparent rather than tied to one model design.
- Experiments on Chinese-English, English-French, and English-German show significant improvements over MLE on a state-of-the-art NMT system.The paper positions this as the first effort, to the authors’ knowledge, to introduce MRT into end-to-end NMT.
2 Background
The background describes end-to-end NMT as modeling target-word generation with recurrent states and source context, while standard MLE maximizes training-data log-likelihood. Its main limitations are exposure bias and a mismatch between word-level training loss and sentence- or corpus-level evaluation metrics.
- End-to-end NMT directly models the translation probability of a target sentence given a source sentence.The target sentence is represented as a sequence, and the model conditions generation on the source sentence and prior target words.
- A recurrent neural network predicts the n-th target word using a target-side hidden state, a context vector, and a nonlinear function.NMT approaches differ in how they calculate the hidden state and context and define the nonlinear function.
- Standard MLE training maximizes the log-likelihood of the training data.The background contrasts this objective with later training objectives designed around evaluation loss.
- MLE suffers from exposure bias because models train on the data distribution but generate using previous model predictions at test time.Previous predictions can be erroneous, creating a mismatch between training inputs and generation-time inputs.
- MLE’s word-level cross-entropy may correlate poorly with sentence-level and corpus-level metrics such as BLEU and TER.This motivates training algorithms that include model predictions and optimize parameters directly with respect to evaluation metrics.
3 Minimum Risk Training for Neural Machine Translation
Minimum risk training minimizes expected translation loss rather than maximizing training-data likelihood, using evaluation metrics to discriminate among candidate translations. Because the full candidate space is intractable, it approximates expectations over sampled subsets while retaining differentiability with respect to model parameters.
- MRT defines risk as expected loss over all possible candidate translations and trains parameters by minimizing that risk.
- Unlike MLE, MRT discriminates among candidate translations according to their losses against the gold-standard translation.The intended model distribution should rank candidates in agreement with the gold-standard ordering.
- MRT can use nondifferentiable evaluation metrics as loss functions because its gradient does not require differentiating the loss itself.The approach is also described as applicable to arbitrary end-to-end NMT architectures.
- To make risk tractable, MRT samples a subset of the exponential translation space and defines an approximate posterior distribution over that subset.The subset includes the gold-standard translation and is built by sampling target words until complete candidates are formed.
- The hyper-parameter α controls the sharpness of the approximate posterior distribution Q.
- The approximate expectation can be computed by enumerating sampled candidates, with 100 samples generally sufficient because larger sample sizes add memory cost without significant gains.Sampling is preferred to top-k translation lists for efficiency and GPU parallelism.
4 Experiments
The experiments evaluate MRT on Chinese-English, English-French, and English-German translation tasks using BLEU and compare it with phrase-based SMT and attention-based NMT baselines. The setup replaces MLE with MRT on top of RNNSEARCH and uses negative smoothed sentence-level BLEU by default.
- MRT is evaluated on Chinese-English, English-French, and English-German translation tasks with BLEU as the evaluation metric.
- The comparison includes MOSES, a phrase-based SMT system, and RNNSEARCH, an attention-based NMT system trained with maximum likelihood estimation.
- The MRT system replaces MLE on top of RNNSEARCH, with beam size 10 and vocabulary sizes of 30K or 50K depending on the language pair.
4.2 Effect of α
The smoothing parameter α strongly affects BLEU on the Chinese-English validation set, with an intermediate setting improving translation quality and smaller or larger values performing worse.
- α has a critical effect on BLEU scores on the Chinese-English validation set.
- α = 1 × 10−1 decreases BLEU scores dramatically.
- α = 5 × 10−3 improves translation quality significantly and consistently.
- Reducing α to 1 × 10−4 results in lower BLEU scores.
4.3 Effect of Sample Size
MRT approximates the full search space by sampling candidate translations, and sample sizes above 100 provide little additional improvement while increasing memory demands.
- MRT samples k candidate translations from the full search space to build an approximate posterior distribution Q.
- BLEU scores consistently rise as sample size k increases.
- k = 200 usually does not lead to significant improvements over k = 100 and increases GPU memory requirements.
- Negative smoothed sentence-level BLEU leads to statistically significant improvements over MLE (p < 0.01).
- MRT processes 13,000 mini-batches per hour, compared with 20,000 for MLE on the same cluster.
- After initialization with the best MLE model, MRT increases BLEU scores dramatically within about 30 hours.
4.6 Results on Chinese-English Translation
On Chinese-English translation, MRT substantially improves BLEU and TER over MLE, while sentence-length analysis identifies weaker performance for inputs longer than 60 words.
- Overall results: Up to +8.61 BLEU points over MOSES and +7.20 over RNNSEARCH-MLE were achieved across all Chinese-English test sets, with statistically significant improvements.These gains were reported for RNNSEARCH trained with MRT.
- Overall results: Up to -10.27 TER points over MOSES and -8.32 over RNNSEARCH-MLE were obtained across all test sets, with statistically significant improvements.MRT used −sBLEU as its loss function for these experiments.
- Sentence-length analysis: MRT consistently improves BLEU over MLE across input lengths but performs worse for sentences longer than 60 words.Figure 4 compares MOSES, RNNSEARCH-MLE, and RNNSEARCH-MRT by input sentence length.
- Sentence-length analysis: RNNSEARCH produces shorter translations for long sentences because training sentences are limited to 50 words, which harms test sentences usually exceeding that length.Figure 5 compares output sentence lengths for MLE and MRE against MOSES.
- Human evaluation: Human evaluation found approximately 54% of MLE translations worse than MRE, with 23% judged equal and 23% better.Two evaluators compared translations for 100 randomly sampled source sentences without system identification.
- Example translations: Example translations suggest MRT improves sentence-level consistency, while MLE can produce duplicated words and omit content despite handling long-distance reordering.The example contrasts MOSES, RNNSEARCH-MLE, and RNNSEARCH-MRT.
4.7 Results on English-French Translation
On English-French translation, RNNSEARCH-MRT achieves the highest BLEU score among the compared end-to-end NMT systems, despite using a smaller vocabulary than some systems.
- English-French results: RNNSEARCH-MRT achieves the highest BLEU score in the English-French comparison, while RNNSEARCH-MLE is comparable to Jean et al. (2015).The compared systems use the same WMT 2014 training subset but differ in architectures and vocabulary sizes.
4.8 Results on English-German Translation
On English-German translation, MRT significantly outperforms MLE and reaches results comparable to state-of-the-art systems, although gains are smaller than on Chinese-English.
- English-German results: RNNSEARCH-MRT significantly outperforms MLE and achieves results comparable to state-of-the-art systems despite Luong et al. (2015a) using a much deeper neural network.The authors believe their approach can be applied easily to that architecture.
- Cross-language comparison: Margins on English-German and English-French datasets are much smaller than on Chinese-English.The authors conjecture that reference-count differences and language relatedness may explain the smaller margins.
5 Related Work
The work builds on minimum risk training in statistical machine translation and differs from reinforcement-learning approaches by estimating expected risk over multiple sampled candidates.
- Minimum risk training: The authors adapt minimum risk training from conventional statistical machine translation and use a smoothing parameter α for the neural objective.Because neural networks are nonlinear, the approach minimizes expected sentence-level loss rather than loss on 1-best translations.
- Reinforcement-learning comparison: MIXER is the closest related approach, but it uses reinforcement learning and hybrid losses whereas MRT minimizes expected loss directly.MIXER samples one candidate for reward calculation; MRT generates multiple samples to calculate expected risk.
- Broader applicability: Experiments support the use of evaluation metrics during optimization for improving sentence-level text generation.The paper also notes successful application of the approach to neural summarization with ROUGE.
6 Conclusion
The paper presents a minimum risk training framework for end-to-end neural machine translation that minimizes expected evaluation loss and improves over maximum likelihood estimation. The authors identify broader testing across language pairs and systems, plus extensions to other neural NLP architectures, as future directions.
- MRT minimizes expected loss in terms of evaluation metrics on NMT training data.
- MRT significantly improves over maximum likelihood estimation for neural machine translation, especially for distantly related Chinese-English language pairs.
- The framework samples the full search space to approximate the posterior distribution and improve efficiency.
- Future work will test MRT on more language pairs and additional end-to-end neural machine translation systems.
- The authors propose extending MRT to minimum risk annealing and other end-to-end neural architectures for NLP tasks.