Source-linked AI summary
On Using Very Large Target Vocabulary for Neural Machine Translation
Sébastien Jean, Kyunghyun Cho, Roland Memisevic, Yoshua Bengio
TL;DR
Neural machine translation is limited by the computational cost of large target vocabularies and the unknown-word errors caused by shortlists. The paper applies biased importance sampling for training and subset-based candidate lists for decoding, achieving comparable or better translation performance, including 21.59 BLEU on English→German.
Problem
NMT training and decoding become more complex as the target vocabulary grows, motivating shortlists that exclude words and can degrade translation quality.
Method
The paper uses biased importance sampling to train very-large-vocabulary NMT models and candidate lists to restrict decoding to selected target-word subsets.
Results
21.59 BLEU on English→German exceeded the previous state of the art at 20.67, while English→French surpassed a prior best single NMT model by approximately 1 BLEU point.
Takeaways & Limitations
Very-large-vocabulary NMT models can match or outperform limited-vocabulary models while retaining practical decoding through selected target-word subsets.
Takeaways & Limitations
The importance-sampling estimator is biased, and rebuilding candidate lists for each source sentence can be time-consuming.
Abstract
from arXiv · showhide
Neural machine translation, a recently proposed approach to machine translation based purely on neural networks, has shown promising results compared to the existing approaches such as phrase-based statistical machine translation. Despite its recent success, neural machine translation has its limitation in handling a larger vocabulary, as training complexity as well as decoding complexity increase proportionally to the number of target words. In this paper, we propose a method that allows us to use a very large target vocabulary without increasing training complexity, based on importance sampling. We show that decoding can be efficiently done even with the model having a very large target vocabulary by selecting only a small subset of the whole target vocabulary. The models trained by the proposed approach are empirically found to outperform the baseline models with a small vocabulary as well as the LSTM-based neural machine translation models. Furthermore, when we use the ensemble of a few models with very large target vocabularies, we achieve the state-of-the-art translation performance (measured by BLEU) on the English->German translation and almost as high performance as state-of-the-art English->French translation system.
1 Introduction
Neural machine translation offers a neural-network approach with promising performance, but large target vocabularies make training and use more complex. The paper proposes importance-sampling training to support much larger vocabularies and reports stronger translation results.
- Neural machine translation: Neural machine translation jointly trains a single neural network to generate translations from source sentences.The model is trained on bilingual data to maximize the conditional probability of a correct translation.
- Limited vocabulary problem: The number of target words must be limited because training and model-use complexity increases with target-vocabulary size.This limitation distinguishes NMT from existing phrase-based approaches.
- Limited vocabulary problem: Shortlists of 30,000–80,000 frequent words map excluded words to [UNK], while translation quality degrades as unknown words increase.The shortlist strategy works best when target sentences contain few unknown words.
- Proposed approach: Importance-sampling training enables NMT models with much larger target vocabularies without increasing training complexity.The proposed algorithm keeps training computation at the level of a small vocabulary subset.
- Results: The proposed approach achieved better translation performance with larger vocabularies while avoiding substantial speed losses in English→French and English→German tasks.It also produced the best performance among single NMT models on WMT’14 English→French.
2 Neural Machine Translation and Limited Vocabulary Problem
NMT encodes source sentences and generates translations with an attention-based encoder–decoder, but computing target-word probabilities over large vocabularies is costly. Shortlists reduce computation but introduce unknown-word errors, especially for richly inflected languages.
- Neural machine translation: NMT uses a single jointly trained neural network, commonly implemented as an encoder–decoder that maps source sentences to translations.The encoder produces hidden states, and the decoder generates target symbols from them.
- Neural machine translation: The attention-based decoder computes context vectors as convex combinations of encoder hidden states and updates its recurrent state from prior outputs and context.The implementation uses bidirectional recurrent encoding and gated recurrent units.
- Limited vocabulary issue: Target-word probabilities require dot products for every vocabulary word, making training expensive and memory use linear in vocabulary size.The computation is repeated for roughly 20–30 target words per sentence, which can become prohibitive.
- Conventional solutions: Shortlists of 30,000–80,000 frequent words make training feasible but degrade performance when translations require many excluded words.The problem is especially relevant for German and other highly inflected languages.
- Conventional solutions: Model-specific alternatives approximate target probabilities or factorize them through word classes, while translation-specific methods replace rare words using alignment-based tokens.These approaches are often complementary and can be combined.
3 Approximate Learning Approach to Very Large Target Vocabulary
The approach approximates the expensive target-vocabulary normalization during training with a small sampled subset, while enabling large-vocabulary decoding through candidate lists. It reduces training cost but introduces estimator bias and practical memory constraints.
- Computational properties: Training complexity becomes constant with respect to target-vocabulary size, supporting much larger vocabularies on limited-memory devices.The approach is designed to exploit matrix-matrix operations and GPU memory efficiently.
- Approximate gradient computation: Importance sampling approximates the negative expected-gradient term using a small set of target-word samples.The approximation targets the expectation responsible for the expensive negative gradient component.
- Approximate gradient computation: The normalization constant is computed from only a small target-vocabulary subset at each update, lowering per-update complexity.Only vectors for the correct word and sampled words are updated.
- Practical training constraints: The method does not inherently bound the number of parameters updated per sentence pair, creating a memory concern during training.The paper addresses this operationally by partitioning the corpus and fixing subset thresholds before training.
- Proposal distributions: Corpus partitions use predefined vocabulary subsets, with each proposal distribution assigning equal probability to words included in its partition.This construction cancels the importance-weight correction term, but the resulting estimator is biased.
- Decoding: At decoding time, candidate lists combine frequent target words with dictionary-linked translations for source words, rather than using the entire vocabulary.The full vocabulary remains available, but subset decoding is used to reduce computational expense.
4 Experiments
Experiments compare shortlist-based RNNsearch models with 500,000-word RNNsearch-LV models trained using the proposed approach on English→French and English→German. Larger-vocabulary models generally improve translation performance, while candidate lists substantially reduce decoding cost.
- Experimental setup: The experiments use WMT’14 English→French and English→German data, with BLEU computed on cased tokenized translations.Evaluation uses news-test 2014, with news-test-2012 and news-test-2013 for model selection.
- Experimental setup: Baseline RNNsearch models use 30,000 target words for English→French and 50,000 for English→German, whereas RNNsearch-LV models use 500,000 source and target words.RNNsearch-LV training varies the shortlist size τ and reports the best development-set performance.
- Translation performance: Without translation-specific strategies, RNNsearch-LV outperforms baseline RNNsearch on the reported translation experiments.The paper compares the models alongside previously reported systems in Table 2.
- Translation performance: On English→French, RNNsearch-LV exceeds the previous best single NMT model when translation-specific techniques are added and reaches performance comparable to the state of the art with an eight-model ensemble.It also outperforms a standard phrase-based translation system.
- Translation performance: On English→German, RNNsearch-LV beats the baseline before unknown-word replacement, while an eight-model ensemble surpasses the previously reported best translation result.After unknown-word replacement, the large-vocabulary and baseline systems perform similarly.
- Decoding speed: Candidate lists substantially improve RNNsearch-LV decoding speed, and a common list for multiple source sentences can match baseline RNNsearch speed.Full-vocabulary decoding is slowest, while candidate-list decoding approaches baseline speed.
- Decoding target vocabulary: The tested decoding vocabulary should resemble the training vocabulary because the best K depends on the shortlist size τ.For τ = 15,000, increasing K from 15,000 to 50,000 reduced English→French validation BLEU; for τ = 30,000, scores increased moderately.
5 Conclusion
The proposed approach extends neural machine translation to much larger target vocabularies without substantial additional training complexity. Models match or exceed limited-vocabulary systems, while selected-vocabulary decoding improves practicality and yields strong BLEU results.
- 5 Conclusion: Importance sampling enables training neural machine translation models with much larger target vocabularies without substantial increases in computational complexity.The approach derives from importance sampling methods for reducing normalization-cost complexity in neural language models.
- 5 Conclusion: On English→French and English→German, models trained with the proposed method performed as well as or better than limited-target-vocabulary models, including when replacing unknown words.
- 5 Conclusion: Selected subsets of the target vocabulary improved RNNsearch-LV decoding performance, making the learning algorithm more practical.
- 5 Conclusion: 21.59 BLEU on English→German exceeded the previous state of the art of 20.67, while English→French performance was approximately 1 BLEU point above the best single NMT model.