Source-linked AI summary
Neural Machine Translation of Rare Words with Subword Units
Rico Sennrich, Barry Haddow, Alexandra Birch
TL;DR
Rare-word translation remains difficult for fixed-vocabulary NMT, especially when languages require subword-level mechanisms. This paper encodes rare and unseen words as subword sequences, achieving performance gains over back-off baselines across segmentation methods.
Problem
Fixed-vocabulary NMT leaves rare-word and open-vocabulary translation unresolved, especially for languages with productive compounding or agglutination.
Method
The paper represents rare and unseen words as subword sequences and adapts byte pair encoding to create compact variable-length segmentations.
Results
Performance gains over the baseline are shown with both byte pair encoding and simple character bigram segmentation.
Takeaways & Limitations
Subword representations enable NMT systems to perform open-vocabulary translation and productively generate words unseen during training.
Takeaways & Limitations
The chosen subword vocabulary size is somewhat arbitrary and may depend on the language pair and amount of training data.
Abstract
from arXiv · showhide
Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character n-gram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English-German and English-Russian by 1.1 and 1.3 BLEU, respectively.
1 Introduction
Word-level NMT struggles with rare words because translation is open-vocabulary, especially for languages with productive word formation. The paper addresses this by encoding rare words as subword units within the NMT model and adapting byte pair encoding for segmentation.
- Problem and motivation: Word-level NMT typically uses limited vocabularies, but translation remains an open-vocabulary problem, particularly for languages with agglutination and compounding.Such languages require mechanisms below the word level.
- Problem and motivation: Fixed word representations are poorly suited to compounds such as German Abwasser|behandlungs|anlange, for which segmented variable-length representations are more intuitive.The example illustrates why translation models need subword-level mechanisms.
- Problem and motivation: Back-off dictionary methods for out-of-vocabulary words can fail when source and target languages differ in morphological synthesis and lack one-to-one word correspondence.The introductory compounding example demonstrates this limitation.
- Approach and contributions: The paper models open-vocabulary translation directly in the NMT network by operating on subword units instead of requiring a separate rare-word back-off model.The authors report simpler processing, improved rare-word accuracy, and productive generation of unseen words.
- Approach and contributions: The authors adapt byte pair encoding, a compression algorithm, for word segmentation, representing an open vocabulary with fixed-size variable-length character sequences.This representation is presented as suitable for neural network models.
2 Neural Machine Translation
The system follows Bahdanau et al.’s neural machine translation architecture, implemented as a recurrent encoder-decoder network. Its bidirectional encoder produces annotation vectors, while an attention-based recurrent decoder predicts each target word from prior state, output, and context.
- Architecture: The system is an encoder-decoder network implemented with recurrent neural networks.The approach follows Bahdanau et al. (2015), though it is not specific to this architecture.
- Encoder: The bidirectional encoder uses gated recurrent units to read the input and concatenate forward and backward hidden states into annotation vectors.For each input position x_j, the forward and backward states are concatenated to obtain h_j.
- Decoder: The recurrent decoder predicts each target word from its hidden state, previously predicted word, and an attention context vector.The context vector is a weighted sum of encoder annotations, with weights computed by a jointly learned alignment model α_ij.
3 Subword Translation
Subword translation targets rare and unseen words by decomposing them into smaller units whose translations can be learned compositionally. The approach is motivated by transparent patterns in names, cognates and loanwords, and morphologically complex words.
- Motivation: Rare words can be translated through smaller units when their structure is transparent to a translator.Relevant units include morphemes and phonemes.
- Transparent word categories: Named entities may be copied across languages sharing an alphabet or transcribed and transliterated when alphabets differ.The examples include Barack Obama across English, German, Russian, and Japanese.
- Transparent word categories: Cognates and loanwords can follow regular character-level transformations, while compounds and other morphologically complex words can be translated morpheme by morpheme.The passage gives claustrophobia→Klaustrophobie and solar system→Sonnensystem/Naprendszer as examples.
- Empirical motivation: 56 compounds, 21 names, 6 loanwords, 5 transparent affixations, 1 number, and 1 computer language identifier occurred among 100 rare German-training tokens.The majority were potentially translatable from English through smaller units.
- Hypothesis and limitation: The hypothesis is that suitable rare-word segmentation enables NMT to learn transparent translations and generalize them to produce unseen words.The models are expected to remain robust even when some segmentations are opaque or oversplit.
4 Evaluation
The evaluation tests subword segmentation for open-vocabulary NMT on WMT 2015 English→German and English→Russian, using translation-quality and rare-word metrics. Subword systems operate without a back-off dictionary and improve rare-word translation, with BPE-J90k particularly effective for English→Russian.
- Evaluation setup: Experiments use WMT 2015 English→German and English→Russian data, with newstest2013 for development and newstest2014/2015 for testing.The training sets contain approximately 100 million and 50 million tokens, respectively.
- Segmentation methods: BPE provides open-vocabulary segmentation with no unknown symbols at test time, while its compact units yield shorter sequences than character-level models.The evaluation compares BPE with character n-grams, compound splitting, rule-based hyphenation, and Morfessor.
- Rare and unseen words: 36.8%→41.8% for EN→DE and 26.5%→29.7% for EN→RU: all subword systems improve unigram F1 over the baseline, especially for rare words.All subword systems operate without a back-off dictionary; gains are especially pronounced for English→Russian OOVs because the alphabets differ.
- Segmentation methods: BPE-J90k outperforms BPE-60k and C2-50k on unigram F1, although all reported subword segmentations outperform the back-off dictionary baseline.BPE-J90k learns symbols on the vocabulary union, whereas BPE-60k learns them separately.
- Overall translation quality: 0.3–1.3 BLEU and 0.6–2 CHRF3: subword ensembles outperform the WDict baseline, despite rare and unseen words comprising only 9–11% of test sets.The study also reports improvements in total unigram F1, BLEU, and CHRF3.
5 Analysis
The analysis shows that subword models improve rare- and unknown-word translation over word-level and back-off-dictionary approaches, with gains arising from productive composition and transliteration. Their advantages vary by frequency range, language pair, and segmentation strategy.
- Frequency effects: Subword C2-3/500k outperforms the back-off dictionary for words ranked above 500,000, while subword systems can productively form compounds.The back-off dictionary improves over producing UNK, but copied OOVs are usually names; subword systems can form new words.
- Frequency effects: For the 50,000 most frequent words, all neural networks use the same representation and achieve comparable unigram F1.Between ranks 50,000 and 500,000, C2-3/500k represents words as single units whereas C2-50k uses subword units.
- English→German: 26.5% recall and 60.6% precision characterize WDict on English→German OOVs, while C2-50k reaches 33.0% recall but 29.1% precision.WDict produces few OOVs with high precision; subword systems achieve higher recall but lower precision, and C2-50k produces the most OOV words.
- English→Russian: 21.9% precision and 15.6% recall for BPE-J90k improve on WDict’s 9.2% precision and 5.2% recall for English→Russian OOVs.Unknown Russian names usually require transliteration, so they can rarely be copied successfully.
- Qualitative analysis: Subword systems learn translations and transliterations that the baseline deletes or copies incorrectly, but inconsistent segmentation and ambiguous mappings cause transliteration errors.The joint BPE segmentation is more consistent in the cited example, while BPE-60k encodes the same name inconsistently across language pairs.
6 Conclusion
The paper shows that subword units enable open-vocabulary neural machine translation more simply and effectively than back-off models. It also identifies vocabulary-size optimization and bilingual segmentation as promising directions for improving subword-based translation.
- Main contribution: Subword sequences enable NMT systems to translate rare and unseen words with open vocabularies.The approach uses variable-length subword units and a byte pair encoding variant for compact word segmentation.
- Analysis: The baseline NMT system translates both out-of-vocabulary and rare in-vocabulary words poorly.The conclusion also reports that reducing subword-model vocabulary size can improve performance.
- Future work: The optimal vocabulary size remains an open research question that may depend on the language pair and amount of training data.The paper proposes learning vocabulary size automatically for each translation task.
- Implications: Subword segmentations are considered suitable for most language pairs and could eliminate the need for large NMT vocabularies or back-off models.Their relative effectiveness is expected to depend on language-specific factors such as vocabulary size.