A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation

• Multilingual NMT enables one system to handle multiple translation directions and zero-shot inference through a target-language flag.The approach uses shared encoder, decoder, and attention modules, with the flag specifying the requested target language.Source: §1 · pp. 1–2
• The study compares bilingual, multilingual, and zero-shot systems on translation quality and examines whether each models particular translation aspects better.It also compares Recurrent and Transformer architectures and studies how related-language data affects zero-shot translation.Source: §1 · p. 2
• The experiments use IWSLT 2017’s five-language shared task, covering twenty directions and professional post-edits for selected related and unrelated language pairs.French and Spanish are additionally used to examine related-language training effects for zero-shot translation into Italian.Source: §1 · p. 2
• The analysis reports BLEU and TER alongside multiple-reference evaluation and examines lexical, morphological, and word-order errors.The paper’s sections cover NMT architectures, multilingual and zero-shot translation, experimental design, overall quality, and fine-grained error analysis.Source: §1 · p. 2

• The error analysis follows a multiple-post-edit approach that quantifies lexical, morphological, and word-order errors.The classification is simpler than one earlier scheme, which additionally distinguished missing and extra words.Source: §2 · p. 2
• The IWSLT 2017 evaluation compared bilingual, multilingual, and zero-shot systems using direct assessment, post-editing, and fine-grained error analysis.It found a single multilingual system could replace multiple bilingual systems and that zero-shot translation was viable in low-resource settings.Source: §2 · pp. 2–3
• This work investigates whether adding training data from related languages benefits zero-shot translation and compares Transformer with Recurrent NMT models.Source: §2 · p. 3

3 Neural Machine Translation

• Standard NMT combines an encoder, decoder, and attention mechanism trained end-to-end; this paper focuses on LSTM-based Recurrent and Transformer variants.Source: §3 · p. 3

3.1 Recurrent NMT

• Recurrent NMT encodes source words sequentially, applies attention over encoder states during decoding, and feeds each prediction back into the decoder.Source: §3.1 · p. 3, §3 · p. 3

3.2 Transformer NMT

• Transformer NMT replaces recurrence with self-attention, adds positional encoding, and stacks encoder-decoder layers containing multi-head attention and feed-forward sublayers.Source: §3.2 · p. 3

3.3 Multilingual NMT

• Multilingual NMT trains one encoder-decoder across multiple language directions by prepending a target-language flag, which can direct zero-shot translation.Its reported advantages include reduced system complexity, low-resource performance gains, and translation between language pairs absent from training.Source: §3.3 · pp. 3–4
• The paper emphasizes that multilingual and zero-shot outputs require detailed investigation because language flags may be weaker in low-resource settings.Source: §3.3 · p. 4

4.1 Datasets and preprocessing

• The study uses approximately 200,000 parallel sentences per language pair, shared subword preprocessing, and language flags for multilingual and zero-shot systems.The data cover seven languages and use 8,000 shared BPE segmentation rules in small-data conditions.Source: §4.1 · p. 4

4.2 Evaluation data

• Professional post-edits cover 603 segments for Dutch→German and Romanian→Italian, with matched subsets of 444 and 478 segments reused for the other directions.Source: §4.2 · p. 4
• The post-edits are generally external references rather than edits of outputs generated by these experiments.This limits their interpretation as direct post-edit evaluations of the tested systems.Source: §4.2 · p. 5

4.3 Training setting

• Both Recurrent and Transformer models are tuned for the low-resource setting, including distinct dropout and learning-rate schedules.Recurrent models use OpenNMT-py, whereas Transformer models use Tensor2Tensor.Source: §4.3 · p. 5

4.4 Models

• The systems include four bilingual models, one many-to-many multilingual model, and zero-shot models for related and unrelated language directions.Additional zero-shot systems incorporate English–French or English–Spanish data related to Romanian and Italian.Source: §4.4 · p. 5

4.5 Evaluation methods

• Evaluation combines BLEU and TER on official references with multiple-reference TER and lemma-based TER on professional post-edits.Lexical, morphological, and word-order errors are derived from lemmatized, POS-tagged outputs and TER operations.Source: §4.5 · pp. 5–6

5.1 Related languages

• Transformer systems outperform Recurrent systems on BLEU and TER across bilingual, multilingual, and zero-shot variants, although post-edit metrics favor Recurrent systems because their outputs contributed to the post-edits.The authors caution that the post-edit comparison is unfair because nine post-edits corrected Recurrent-system outputs.Source: §5.1 · p. 6
• Bilingual systems are best in three of four related-language cases, while multilingual systems show strong mTER and lmmTER performance in both architectures.The authors attribute this robustness to multilingual training exposing models to more linguistic phenomena.Source: §5.1 · p. 6

5.2 Unrelated languages

• For unrelated directions, the experiments evaluate how source-language relatedness affects zero-shot translation against bilingual references.The study focuses on zero-shot systems because they must translate directions without direct training data.Source: §5.2 · pp. 6–7
• Multilingual systems support effective zero-shot and non-zero-shot translation, with related target-language data producing limited gains and Transformer systems performing best, especially in zero-shot directions.In one Recurrent Ro→De condition, zero-shot training even outperforms the bilingual system; other cases show no significant difference.Source: §5.2 · p. 7

6.1 Related languages

• Transformer models reduce total errors by 14.88 and 9.40 points for multilingual and zero-shot systems, respectively, whereas Recurrent models show a 9.69-point decrease and a 9.84-point increase.Lexical errors dominate at 76–77%, followed by morphology at 15–16% and reordering at 3–6%.Source: §6.1 · p. 8, Table 5 · p. 8
• Adding two related-language directions improves Recurrent error variation from +33.81 to +26.00, while Transformer zero-shot performance improves from +1.53 to −5.04 and outperforms bilingual models.Transformer zero-shot models are less sensitive to removing or adding language directions than Recurrent models.Source: §6.1 · p. 9
• For Ro→It, zero-shot translation is competitive with bilingual NMT only with Transformers, while multilingual models outperform their bilingual counterparts, especially with Recurrent models.The Transformer zero-shot model is slightly better than bilingual NMT on reordering errors, with a −0.54 difference.Source: Table 6 · p. 8, §6.1 · p. 9

6.2 Unrelated languages

• In unrelated directions, Recurrent Ro→De zero-shot models reduce errors by 5.60 and 6.21 points, while Transformers show no significant differences from bilingual references.The Recurrent ZST_B gain is mainly lexical (−4.76 points) and reordering (−1.17 points), likely reflecting its weaker bilingual counterpart.Source: §6.2 · p. 9, Table 8 · p. 10
• For unrelated Nl→It, Transformer and Recurrent zero-shot models reduce errors by −2.29 and −1.11 points in ZST_B, and both benefit from additional Italian-related training data.The observed zero-shot error changes are associated with the relatedness and variety of languages in training data.Source: §6.2 · p. 9, §6.1 · p. 9, Table 9

• Multilingual models consistently outperform bilingual models across lexical, morphological, and reordering errors.Source: Summary and Conclusions · p. 10
• Transformer models deliver the best-performing multilingual systems, with larger gains over bilingual baselines than RNNs.Source: Summary and Conclusions · p. 10
• Multilingual models trained on related languages achieve the best performance scores and relative gains over corresponding bilingual models.Source: Summary and Conclusions · p. 10
• Language relatedness does not play a crucial role when comparing zero-shot systems with bilingual models.Source: Summary and Conclusions · p. 10
• Transformers deliver the best quality across all considered zero-shot conditions and translation directions.Source: Summary and Conclusions · p. 10
• Error distributions differ significantly across translation directions, including between source languages from the same family, while architectures show no particular distributional differences.The authors identify deeper analysis of these differences as future work.Source: Summary and Conclusions · p. 10