Universal Neural Machine Translation for Extremely Low Resource Languages

1 Introduction

• The approach targets low-resource languages by leveraging multilingual NMT capabilities through transfer learning.It focuses on sharing representations from multiple source languages into one target language.Source: §1 · p. 1
• A universal word-level representation enables lexical sharing across source languages.The representation uses monolingual embeddings and seed parallel data from all languages.Source: §1 · p. 1
• Sentence-level sharing uses source-language representations to support translation for languages with extremely limited or zero parallel data.Source: §1 · p. 1
• The evaluation covers three languages with tiny or zero parallel data and reports consistent gains over a strong multilingual NMT baseline.Source: §1 · p. 1

2 Motivation

• A standard NMT encoder transforms input sequences into continuous representations, while an attention-based decoder predicts target words.The model is trained by maximizing likelihood over parallel sentences.Source: §2 · pp. 1–2
• NMT performance is highly correlated with parallel-data quantity and becomes unreasonable with extremely small or unavailable corpora.The motivation identifies approximately 13k sentences as extremely small and N = 0 as no parallel data.Source: §2 · p. 2, Figure 1 · p. 2
• Multilingual NMT trains on multiple language pairs to regularize models and transfer translation knowledge to low-resource languages.Source: §2 · p. 2

2.1 Challenges

• Conventional multilingual vocabularies give each source word its own embedding, leaving low-resource vocabulary items poorly trained.Source: §2.1 · p. 2, §2 · p. 2
• Sharing subwords or characters cannot guarantee semantic overlap because low-resource languages may differ in surface forms or character sets.Source: §2.1 · pp. 2–3
• Low-resource languages also need shared sentence representations, including representations learned from monolingual data.Syntactic similarity and monolingual resources are identified as useful sources for sentence-level sharing.Source: §2.1 · p. 3

• ULR and MoLE extend multilingual NMT to enable word-level and sentence-level sharing when minimal parallel sentences are available.Source: §3 · p. 3

Universal Lexical Representation

• ULR represents each source word as a probabilistic mixture of universal-token embeddings learned during NMT training.The universal token set is discrete, while each word's representation is determined through a distribution based on semantic similarity.Source: §3.1 · p. 3
• The ULR key-value mechanism uses monolingual embeddings as queries, universal-token embeddings as keys, and distributions over universal NMT embeddings as values.A shared transformation matrix is optimized during NMT training to tune similarity scores toward the translation task.Source: §3.1 · p. 4
• Language-specific monolingual embeddings are projected into the universal-token space using learned language-specific transformations and seed word alignments.The paper also notes that seed-free transformation methods could support purely zero-parallel-resource cases.Source: §3.1 · p. 4

Mixture of Language Experts

• MoLE adds expert networks and a gating network after the encoder, producing a weighted sum of expert outputs as the encoder representation.One expert is created per auxiliary language, while low-resource-language training leaves MoLE unchanged and represents its tokens through mixtures of auxiliary experts.Source: §3.2 · p. 5

Settings

• The experiments compare the proposed system with vanilla single-source NMT and multilingual NMT baselines.Source: §4 · p. 5
• The study evaluates Universal NMT on Romanian, Latvian, and Korean to English in near-zero-resource settings.Auxiliary languages are jointly trained from Czech, German, Greek, Spanish, Finnish, French, Italian, Portuguese, and Russian.Source: §4.1 · p. 5
• The targeted-language parallel corpora are down-sampled to simulate zero-resource conditions, while monolingual data come from Wikipedia dumps.The monolingual corpora are typically much larger than the parallel corpora.Source: §4.1 · p. 5

Implementation

• All systems use tokenized and BPE-segmented data, with sentences limited to 50 subword symbols and source vocabularies concatenated in multilingual settings.Source: §4.1 · p. 6
• The shared experimental architecture is a one-layer bidirectional RNN encoder with a two-layer attention-based RNN decoder, using 512 LSTM units.Models are trained with Adam for 1 million steps on mixed data using batches of 128 and dropout rate 0.4.Source: §4.1 · p. 6

Back-translation

• Back-translation is used to encourage use of zero-resource-language information, although applying it directly is difficult without a reasonably good initial translation system.The authors initialize this process from a trained multilingual NMT model.Source: §4.2 · p. 6

Multilingual results

• Applying universal tokens and mixture-of-experts modules improves translation quality across Romanian-English, Latvian-English, and Korean-English settings, with additive gains.Experiments use 6k sentences for Romanian and Latvian and 10k for Korean.Source: §5 · p. 6
• Romanian performs best with Romance-language auxiliaries, while Latvian improves when Russian is added as an auxiliary language.The paper attributes these gains to greater linguistic relatedness and shared knowledge.Source: §5 · p. 7
• The proposed methods consistently help less-related Korean-English translation, although its BLEU score remains relatively low.The experiments use Romance-family languages as Korean auxiliary languages and conclude that similar auxiliaries are more reasonable.Source: §5 · p. 7

Ablation study

• +7.98 BLEU is obtained over the vanilla low-resource setup when ULR is combined with MoLE and back-translation.ULR alone gives +5.07 BLEU over Multi-NMT, while a trainable transformation matrix improves ULR further; the best result remains 5–6 BLEU below the full-data model.Source: §5.1 · p. 7
• ULR remains strongest when monolingual Romanian data and back-translation are added, indicating additive gains.Back-translation improves both systems, but ULR still achieves the best score.Source: §5.1 · p. 8

Corpus size and unknown tokens

• ULR works especially well with small training corpora and supports pure zero-resource translation through shared multilingual representations.Performance is evaluated at 0k, 6k, 60k, and 600k Romanian-English sentence pairs; larger corpora improve all systems.Source: §5.1 · p. 8

• Multilingual NMT has been extensively studied by Lee et al., Johnson et al., Zoph et al., and Firat et al.Source: §6 · p. 9
• Earlier multilingual approaches have significant limitations in zero-resource cases.Source: §6 · p. 9
• The proposed work extends related multilingual NMT by enabling lexical and sentence representation sharing across multiple languages for very low-resource settings.Source: §6 · p. 9

• The approach enables resource sharing between high-resource languages and extremely low-resource languages.Source: §7 · p. 9
• 23 BLEU is achieved on Romanian-English WMT2016 using a tiny parallel corpus of 6k sentences.Source: §7 · p. 9
• 18 BLEU is reported for the strong multilingual baseline system.Source: §7 · p. 9