Source-linked AI summary
Addressing the Rare Word Problem in Neural Machine Translation
Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, Wojciech Zaremba
TL;DR
Conventional NMT systems struggle with rare words because limited vocabularies represent many OOV words with one unk symbol. The paper adds alignment-derived source positions to NMT outputs and replaces OOV tokens during post-processing, improving WMT’14 English–French performance by up to 2.8 BLEU points and reaching 37.5 BLEU points. The approach remains subject to dictionary errors, alignment errors, and implementation trade-offs.
Problem
Conventional NMT systems struggle to translate rare words because limited vocabularies represent many out-of-vocabulary words with a single unk symbol.
Method
The method augments NMT training data with alignment information so the model emits relative source positions for OOV words, which post-processing translates using a dictionary or identity translation.
Results
37.5 BLEU points established the first NMT system to outperform the best MT system on a WMT’14 contest dataset, while gains reached 2.8 BLEU points over various NMT baselines.
Takeaways & Limitations
The alignment-based technique mitigates and can overcome NMT’s inability to translate words outside its vocabulary and applies across NMT architectures.
Takeaways & Limitations
The PosAll variant doubles target-sentence length and slows parameter updates by a factor of two, while examples also show dictionary and alignment errors.
Abstract
from arXiv · showhide
Neural Machine Translation (NMT) is a new approach to machine translation that has shown promising results that are comparable to traditional approaches. A significant weakness in conventional NMT systems is their inability to correctly translate very rare words: end-to-end NMTs tend to have relatively small vocabularies with a single unk symbol that represents every possible out-of-vocabulary (OOV) word. In this paper, we propose and implement an effective technique to address this problem. We train an NMT system on data that is augmented by the output of a word alignment algorithm, allowing the NMT system to emit, for each OOV word in the target sentence, the position of its corresponding word in the source sentence. This information is later utilized in a post-processing step that translates every OOV word using a dictionary. Our experiments on the WMT14 English to French translation task show that this method provides a substantial improvement of up to 2.8 BLEU points over an equivalent NMT system that does not use this technique. With 37.5 BLEU points, our NMT system is the first to surpass the best result achieved on a WMT14 contest task.
1 Introduction
NMT offers promising, general-purpose translation but struggles with rare words because modest vocabularies collapse OOV words into a single unk symbol. The paper addresses this weakness by adding alignment-based positional information and post-processing, yielding substantial gains on WMT’14 English–French.
- Problem: NMT systems use modest-sized vocabularies, forcing many rare out-of-vocabulary words to share a single unk symbol.Sentences containing many rare words are translated more poorly than sentences containing mainly frequent words.
- Problem: Standard phrase-based systems mitigate rare-word problems more effectively through larger vocabularies, explicit alignments, and phrase tables.These mechanisms allow them to memorize translations of extremely rare words.
- Approach: The proposed method augments training data with alignment information so the NMT system emits a source-position pointer for each target-side OOV word.A post-processing step uses the pointer to translate OOV words with a dictionary or identity translation when no dictionary entry is found.
- Results: Up to 2.8 BLEU points are gained on the WMT’14 English-to-French task over an equivalent NMT system without the technique.The paper reports that its system was the first NMT system to outperform the winner of a WMT’14 task.
2 Neural Machine Translation
NMT models map source sentences to target sentences by parameterizing conditional word distributions. This section describes common architectures and the deep-LSTM encoder–decoder used in the paper, while noting that rare-word handling remained unresolved.
- NMT formulation: An NMT system maps a source sentence to a target sentence, with both sentences ending in a special <eos> token.The model parameterizes conditional distributions for target words given the source and preceding target words.
- NMT formulation: Training maximizes the log probability of the target sentence conditioned on the source sentence.The conditional word probabilities are combined across target positions.
- Architectures: NMT conditional distributions can be parameterized with convolutional, recurrent, LSTM, or attention-based neural architectures.The cited examples include deep LSTM models and architectures using attention over the input sequence.
- Paper model: The paper uses a deep-LSTM encoder–decoder whose encoder represents the source sentence with a vector and whose decoder generates the translation until <eos>.The encoder and decoder are separate deep LSTMs.
- Rare-word context: Prior NMT work had not addressed rare words adequately, while very large vocabularies remained complementary to alignment-based techniques.Even a 500K-word vocabulary did not eliminate rare-word problems such as names and numbers.
3 Rare Word Models
The paper develops three alignment-based annotation strategies that let NMT systems track source positions for unknown target words and replace them during post-processing. The models progressively move from copying unknown tokens to encoding relative positions for all words or only unknown words.
- Overview: The approach augments NMT training data with alignment annotations so unknown target words can be linked to source words and translated during post-processing.A dictionary supplies the translation, with identity translation used when no dictionary entry exists.
- 3.1 Copyable Model: The copyable model uses distinct unknown tokens to match unknown target words aligned to unknown source words, while null tokens mark unaligned or source-known cases.This enables translation of every non-null unknown token but cannot handle unknown target words aligned to known source words.
- 3.2 Positional All Model: The PosAll model appends relative positional tokens to every target word, encoding alignment to source position i = j − d or a null token when alignment is unavailable.Relative positions range from d = −7 to 7, with distant alignments treated as unaligned.
- 3.2 Positional All Model: PosAll doubles target-sentence length and slows parameter updates by a factor of two, motivating annotation only for unknown target words.The paper reports that the anticipated benefit from learning more alignment examples does not occur.
- 3.3 Positional Unknown Model (PosUnk): The PosUnk model uses unkpos_d tokens that jointly indicate an unknown target word and its relative source position, with unkpos_∅ for unaligned unknowns.Unlike PosAll, it annotates only aligned unknown words while retaining a universal unknown token on the source side.
4 Experiments
The experiments evaluate the rare-word models on WMT’14 English-to-French translation using BLEU on newstest2014. PosUnk improves both individual and ensemble LSTM systems, with gains depending on vocabulary size and model accuracy.
- 4 Experiments: The evaluation uses BLEU on the 3003-sentence newstest2014 English-to-French test set, with models trained on 12M parallel sentences.Target vocabularies contain either the 40K or 80K most frequent French words, while the source vocabulary contains 200K English words.
- 4.1 Experimental Setup: The experiments compare the proposed systems with state-of-the-art, neural, and phrase-based baselines using tokenized and detokenized BLEU scores.Table 1 varies architecture, vocabulary size, and training corpus; Table 2 compares the existing state-of-the-art system with the best proposed system.
- 4.4 Main Results: 2.3–2.8 BLEU points are gained with 40K-word vocabularies, while 80K-word vocabularies yield 1.6–1.9 BLEU-point gains from PosUnk.The technique improves both individual and ensemble LSTM models, though gains diminish with larger vocabularies.
- 4.4 Main Results: More accurate models obtain larger post-processing gains because they locate the source origin of unknown words more accurately.This pattern is especially visible for ensemble models compared with individual models at the same vocabulary size.
- 4.4 Main Results: 37.5 BLEU is achieved by the best system, surpassing other NMT systems and establishing a new WMT’14 English-to-French record.The paper reports that this result outperforms the existing WMT’14 state-of-the-art system, which achieved 35.8 detokenized BLEU.
5 Analysis
The analysis evaluates how rare-word translation affects BLEU, compares alignment strategies, and examines model depth, perplexity, and translation examples. Positional alignment models yield the strongest gains, especially for more accurate models.
- 5.1 Rare Word Analysis: The analysis measures rare-word effects by grouping newstest2014 sentences by average inverse frequency and evaluating BLEU before and after OOV post-processing.The groups contain sentences with comparable prevalence of rare words.
- 5.1 Rare Word Analysis: 4.8 BLEU points: unknown-word translation improves the last group of 500 sentences, which contains the greatest proportion of OOV words.The rare-word model approaches the winning system on sentences with many OOV words while outperforming it on predominantly frequent-word sentences.
- 5.2 Rare Word Models: 0.8 BLEU points: the NoAlign monotone-alignment baseline improves modestly, while Copyable provides a 1.0-BLEU-point gain.The comparison is made on English-French models, whose similar word order may favor monotone alignment.
- 5.2 Rare Word Models: More than 2 BLEU points: PosAll and PosUnk outperform the alignment-free alternatives, while PosUnk performs better than PosAll.Unlike Copyable, positional models can align unknown target words with any source word; PosUnk also predicts positions only for unknown target words.
- 5.3 Other Effects: Roughly 1.0 BLEU point per added LSTM layer: PosUnk gains increase with overall model accuracy across 3-, 4-, and 6-layer models.The analysis links this pattern to more accurate source-word pointing by larger models.
- 5.3 Other Effects: A 0.5-point perplexity reduction corresponds on average to roughly 1.0 BLEU point improvement during 4-layer LSTM training.The correlation is measured at multiple training stages.
6 Conclusion
The paper concludes that alignment-based post-processing can mitigate NMT’s rare-word weakness and applies across NMT systems. On WMT’14 English-French, it improves BLEU by up to 2.8 points and reaches 37.5 BLEU.
- 6 Conclusion: Alignment-based rare-word translation can mitigate or overcome NMT’s inability to translate words outside its vocabulary.The authors identify this inability as a major weakness of current NMT systems.
- 6 Conclusion: The technique applies to any NMT system, rather than only the deep LSTM architecture used in the experiments.The authors characterize this general applicability as a key advantage.
- 6 Conclusion: Up to 2.8 BLEU points: the technique consistently improves various NMT architectures on WMT’14 English-French.The reported gains are described as substantial and consistent.
- 6 Conclusion: 37.5 BLEU points: the system became the first NMT system to outperform the best MT system on a WMT’14 contest dataset.This is the paper’s principal benchmark achievement.