Source-linked AI summary
Six Challenges for Neural Machine Translation
Philipp Koehn, Rebecca Knowles
TL;DR
Neural machine translation faces unresolved challenges involving domain mismatch, training-data size, rare words, sentence length, alignment, and decoding. The paper empirically compares NMT with phrase-based SMT and finds weaknesses alongside improvements, especially across resource levels and rare-word conditions. It concludes that out-of-domain and low-resource performance remain the most notable challenges, while interpretability is an additional unexamined limitation.
Problem
The paper asks how well NMT holds up against traditional SMT across six challenges, given reported poor performance in low-resource conditions and domain-specific translation needs.
Method
The authors empirically compare attention-based NMT and phrase-based SMT across domain mismatch, training-data size, rare words, long sentences, word alignment, and beam search.
Results
NMT is weaker out of domain and in low-resource settings, but better in high-resource settings and on extremely low-frequency words; it remains weaker on very long sentences, alignment, and large beam spaces.
Takeaways & Limitations
Despite recent successes, NMT still needs improvement most notably for out-of-domain and low-resource performance.
Takeaways & Limitations
NMT is much less interpretable, and this challenge is not examined empirically.
Abstract
from arXiv · showhide
We explore six challenges for neural machine translation: domain mismatch, amount of training data, rare words, long sentences, word alignment, and beam search. We show both deficiencies and improvements over the quality of phrase-based statistical machine translation.
1 Introduction
The paper evaluates six challenges for neural machine translation against phrase-based statistical machine translation, finding both important weaknesses and areas of improvement. It also identifies interpretability as an additional challenge not examined empirically.
- NMT is evaluated empirically across domain mismatch, training-data size, rare words, long sentences, word alignment, and beam search.The comparison is with traditional phrase-based SMT.
- Out-of-domain NMT can sacrifice adequacy for fluency, while its steeper data-learning curve yields worse low-resource but better high-resource performance.
- Sub-word NMT outperforms SMT on extremely low-frequency words but remains weak on low-frequency words in highly inflected categories such as verbs.
- NMT quality is lower on very long sentences, although it performs comparably better than SMT up to about 60 words.
- NMT attention does not always act as word alignment and can diverge dramatically, while larger beam-search spaces can deteriorate translation quality.
- NMT is much less interpretable because word-choice decisions are buried in large matrices of real-numbered values.
2 Experimental Setup
The experiments compare attention-based NMT and phrase-based SMT using common toolkits and data across English–Spanish and German–English. Evaluation uses WMT and OPUS resources, with news test sets for most experiments.
- Both systems are trained and evaluated with common datasets drawn from WMT and OPUS.
- The study compares attention-based NMT using Nematus with phrase-based SMT using Moses.NMT uses default settings including beam search and single-model decoding.
- Phrase-based SMT uses standard features common in recent WMT system submissions.
- Experiments cover English–Spanish and German–English, with large training datasets available for both language pairs.
- Domain experiments use OPUS, while other experiments use WMT news test sets averaging about 30 words per sentence.
3 Challenges
Across six challenges, NMT shows serious weaknesses under domain mismatch, limited data, long inputs, and some alignment conditions, while improving over SMT for rare words and high-resource training. The results also expose fluent but inadequate translations and non-monotonic benefits from NMT's mechanisms.
- Domain mismatch: Out-of-domain NMT quality is often dramatically worse than SMT, sometimes sacrificing adequacy for fluent but unrelated output.On the Law test set, NMT scores 3.9 BLEU versus 10.2 for SMT; an example produces fluent content unrelated to the input.
- Amount of training data: NMT has a steeper data-learning curve: it performs worse with few million words but surpasses SMT as training data grows.NMT scores 1.6 versus SMT's 16.4 with 1/1024 of the data, then reaches 25.7 versus 24.7 with 1/16 of the data.
- Rare words: NMT with byte-pair encoding outperforms SMT on very infrequent words, but highly inflected adjectives and verbs remain difficult.Unknown words are translated correctly 60.1% of the time by NMT versus 53.2% for SMT, while inflected categories have worse accuracy and higher deletion rates.
- Rare words: NMT's overall word precision is similar to SMT's but its deletion rate is higher across frequency ranges.Overall precision is 70.3% for NMT versus 70.1% for SMT; NMT deletes 3769 of 64379 words compared with 2274 for SMT.
- Long sentences and word alignment: Attention can diverge from traditional word alignments despite high translation quality, while long-sentence quality declines beyond about 60 words.NMT is lower quality from length 60 onward and produces shorter translations, with length ratio 0.859 versus SMT's 1.024.
- Word alignment: For German–English, attention alignments can be off by one position even when both systems retain high translation quality.The paper reports this divergent behavior without an intuitive explanation and notes that larger divergence has appeared under other data conditions.
4 Conclusions
Neural machine translation still faces important challenges, especially when tested outside its training conditions or with limited resources. These problems point toward the need for more robust training approaches.
- NMT remains challenged by out-of-domain and low-resource conditions despite its recent successes.
- Many problems share a lack of robust behavior when test conditions differ substantially from training conditions.The paper identifies limited training exposure, unusual out-of-domain inputs, and unlikely initial beam-search choices as examples.
- The authors suggest that addressing these problems may require training beyond optimizing single-word predictions from perfectly matching prior sequences.