Source-linked AI summary
Mutual Information and Diverse Decoding Improve Neural Machine Translation
Jiwei Li, Dan Jurafsky
TL;DR
Standard sequence-to-sequence translation models optimize p(y|x) while ignoring p(x|y), a dependency useful in phrase-based translation. The paper incorporates bidirectional dependency by modeling maximum mutual information between source and target, equivalent to linearly combining p(x|y) and p(y|x). The proposed models perform better than corresponding standard models on English→French and English→German, with both mutual information and diversity methods contributing to the gains.
Problem
Standard sequence-to-sequence translation models optimize p(y|x) while ignoring p(x|y), a dependency useful in phrase-based translation.
Method
The paper incorporates bidirectional dependency by modeling maximum mutual information between source and target, equivalent to linearly combining p(x|y) and p(y|x).
Results
The proposed models perform better than corresponding standard models on English→French and English→German, with both mutual information and diversity methods contributing to the gains.
Takeaways & Limitations
Bidirectional source-target modeling and diverse decoding improve neural machine translation across standard LSTM and attention-based settings tested in the paper.
Takeaways & Limitations
Reported results are not comparable to ensemble mechanisms, and unknown-word replacement requires a pretrained attention model for vanilla sequence-to-sequence outputs.
Abstract
from arXiv · showhide
Sequence-to-sequence neural translation models learn semantic and syntactic relations between sentence pairs by optimizing the likelihood of the target given the source, i.e., $p(y|x)$, an objective that ignores other potentially useful sources of information. We introduce an alternative objective function for neural MT that maximizes the mutual information between the source and target sentences, modeling the bi-directional dependency of sources and targets. We implement the model with a simple re-ranking method, and also introduce a decoding algorithm that increases diversity in the N-best list produced by the first pass. Applied to the WMT German/English and French/English tasks, the proposed models offers a consistent performance boost on both standard LSTM and attention-based neural MT architectures.
1 Introduction
Standard neural translation models optimize p(y|x), capturing only source-to-target dependency, whereas this paper adds the reverse dependency through mutual information and approximates it with reranking and diverse decoding.
- 1 Introduction: Sequence-to-sequence models encode a source sentence into a continuous representation and decode a target by maximizing its conditional likelihood.This approach supports end-to-end learning with limited domain knowledge and smaller memory requirements than phrase-based systems.
- 1 Introduction: Mutual information augments p(y|x) with p(x|y), modeling bidirectional source-target dependency in sequence-to-sequence translation.The objective combines the two conditional probabilities with a tuning weight λ.
- 1 Introduction: Reranking uses p(y|x) to generate an N-best list, then scores candidates with a separately trained p(x|y) model.This approximates mutual-information decoding because direct computation of p(x|y) is infeasible before completing the target.
- 1 Introduction: A diversity-promoting decoder expands the usefulness of reranking by producing a more diverse first-pass N-best list.The paper evaluates mutual-information and diversity methods on English→French, English→German, and German→English with standard LSTM and attention models.
2 Related Work
The paper connects neural translation to prior work on sequence-to-sequence modeling, mutual information, and diverse decoding, adapting phrase-based translation features to neural MT.
- 2 Related Work: Prior sequence-to-sequence systems use encoder-decoder architectures to model conditional target probabilities across several NLP generation tasks.The surveyed architectures include convolutional and recurrent encoders and stacked or bidirectional recurrent networks.
- 2 Related Work: Earlier mutual-information work addressed generic conversational responses by modeling message-response dependence to promote more diverse outputs.This paper transfers that objective to translation rather than conversational response generation.
- 2 Related Work: The paper’s novelty is testing whether bidirectional mutual-information modeling improves translation by incorporating source-target dependencies into sequence-to-sequence models.It specifically targets reverse translation probability and sentence length as features previously useful in phrase-based MT.
- 2 Related Work: Diverse translation methods include compact search structures, length features, ensembles, and dissimilarity-based N-best generation, but many were designed for phrase-based decoding.The paper motivates a diversity method tailored to neural MT’s batched computation.
3 Background: Neural Machine Translation
The background describes encoder-decoder LSTM translation, attention-based decoding, and unknown-word replacement used to improve neural MT outputs.
- 3 Background: Neural Machine Translation: Neural MT maps source tokens to continuous representations and sequentially generates target tokens with an encoder-decoder model.The target distribution is factorized over decoding time steps and computed with a softmax.
- 3.1 LSTM Models: LSTM models use input, memory, and output gates to maintain recurrent hidden and cell-state representations during decoding.Generation stops when the model predicts an end-of-sentence symbol.
- 3.2 Attention Models: Attention models link each decoding step to encoded source representations using relevance scores and averaged attention weights.The implementation adopts the general scoring strategy and input-feeding approach.
- 3.3 Unknown Word Replacements: Neural MT’s full-vocabulary softmax is computationally expensive, so systems use shortlists and an unknown-word token for infrequent words.Unknown-word replacement extracts alignments from attention models and substitutes UNK tokens using a bilingual dictionary.
- 3.3 Unknown Word Replacements: Unknown-word replacement is not directly available in vanilla sequence-to-sequence models because they do not explicitly model automatic word alignments.The paper instead postprocesses their generated translations with a pretrained attention model.
4 Mutual Information via Reranking
The method approximates mutual-information reranking with separately trained directional models and improves candidate diversity before reranking. It combines standard beam search, diversity promotion, and additional reranking features to produce broader N-best lists and revised rankings.
- 4 Mutual Information via Reranking: Separate p(y|x) and p(x|y) models generate N-best lists and rerank them by adding the reverse probability.This approximates mutual information because direct decoding requires completing the target before computing p(x|y).
- 4.1 Standard Beam Search for N-best lists: Standard beam search expands K hypotheses into K × K candidates but retains only the top K unfinished hypotheses at each step.Completed hypotheses are added to the N-best list, while the active beam remains fixed; the final list can exceed the beam size.
- 4.2 Generating a Diverse N-best List: The diversity-promoting decoder penalizes lower-ranked sibling expansions and favors high-ranked hypotheses from different parents, producing more diverse N-best lists.The adjustment changes beam-search scores using the sibling rank γk′ while leaving the implementation close to standard beam search.
- 4.2 Generating a Diverse N-best List: distinct-1 rises from 0.54% to 0.95% and distinct-2 from 1.55% to 2.84% for English–German translation with diversity-promoting decoding.The metrics count distinct unigrams and bigrams in each source sentence’s N-best list, scaled by total tokens; similar English–French trends are reported without details.
- 4.3 Reranking: The reranking score linearly combines log p(y|x), reverse probability log p(x|y), average language-model probability log p(y), and a target-length term.
5 Experiments
Experiments evaluate mutual-information reranking and diversity-promoting decoding across WMT English–German and English–French tasks, using standard and attention-based neural MT models.
- 5 Experiments: The experiments use WMT’14 training data for English–German and English–French translation and report translation performance in BLEU on newstest2014 and news-test-2014, respectively.The training data contain 4.5 million English–German pairs and 12 million English–French pairs; evaluation uses 2,737 newstest2014 sentences and 3,003 news-test-2014 pairs.
- 5.1 Training Details for p(x|y) and p(y|x): The two directional models are trained separately, with p(y|x) generating N-best lists and p(x|y) used during reranking.The reverse-direction model is trained by swapping sources and targets, following standard training protocols.
- 5.3 English-German Results: +1.4, +1.1, and +0.9 gains come from reverse probability p(x|y) for standard models without unknown-word replacement, standard models with replacement, and attention models.Reverse probability is described as the most significant reranking feature among those tested.
- 5.3 English-German Results: The authors observe a consistent performance boost from language models, while target-length effects are smaller because decoding strictly considers candidates 0.75–1.5 times the source length.
- 5.3 English-German Results: +2.6, +2.6, and +2.1 total gains are reported for English–German models, with diversity decoding adding +0.4, +0.3, and +0.3 beyond reranking improvements.The comparison attributes additional gains to the diverse N-best lists.
- 5.3 English-German Results: The English–German comparisons are not directly comparable to cited ensemble systems because this implementation did not use ensembling.The reported single-model baselines nevertheless achieve similar performances to corresponding models from Luong et al. (2015a).
- 5.4 French-English Results: +3.2, +2.6, and +2.7 BLEU are reported for English–French models versus standard neural models without and with unknown-word replacement, and attention models, respectively.The English–French results show a larger gap between the proposed and vanilla models than English–German.
6 Discussion
The discussion presents mutual-information reranking and diversity-promoting decoding as broadly applicable additions to neural generation systems. It reports improvements over standard models while identifying direct mutual-information decoding without reranking as future work.
- 6 Discussion: The mutual-information objective models mutual dependency between source and target sentences and is implemented approximately through reranking.The objective is inspired by Li et al. (2015).
- 6 Discussion: On English→French and English→German, the proposed models outperform corresponding standard neural MT models, with both mutual information and diversity decoding contributing to the gain.The conclusion treats the two components as separate contributors to the reported performance improvement.
- 6 Discussion: The diversity-enhancing decoder produces more diverse first-pass N-best lists and can be applied to other NLP reranking tasks.The paper describes the implementation as requiring only minor adjustments for integration into neural generation models.
- 6 Discussion: Directly incorporating mutual-information decoding into a first-pass decoder without reranking remains future work.