Source-linked AI summary
Universal Neural Machine Translation for Extremely Low Resource Languages
Jiatao Gu, Hany Hassan, Jacob Devlin, Victor O. K. Li
TL;DR
Extremely low-resource translation lacks enough parallel data for standard NMT and remains difficult even with multilingual systems. The paper introduces universal lexical and sentence-level sharing, achieving 23 BLEU on Romanian-English with 6k sentences versus 18 BLEU for a strong baseline.
Problem
Extremely low-resource languages lack sufficient parallel data, while conventional multilingual vocabularies provide unreliable representations for their words.
Method
The approach shares lexical representations through Universal Lexical Representation and sentence representations through multilingual source encoders and experts.
Results
23 BLEU is achieved on Romanian-English WMT2016 using 6k parallel sentences, compared with 18 BLEU for a strong multilingual baseline.
Takeaways & Limitations
The approach enables sharing resources between high-resource and extremely low-resource languages, including settings with tiny or zero parallel data.
Abstract
from arXiv · showhide
In this paper, we propose a new universal machine translation approach focusing on languages with a limited amount of parallel data. Our proposed approach utilizes a transfer-learning approach to share lexical and sentence level representations across multiple source languages into one target language. The lexical part is shared through a Universal Lexical Representation to support multilingual word-level sharing. The sentence-level sharing is represented by a model of experts from all source languages that share the source encoders with all other languages. This enables the low-resource language to utilize the lexical and sentence representations of the higher resource languages. Our approach is able to achieve 23 BLEU on Romanian-English WMT2016 using a tiny parallel corpus of 6k sentences, compared to the 18 BLEU of strong baseline system which uses multilingual training and back-translation. Furthermore, we show that the proposed approach can achieve almost 20 BLEU on the same dataset through fine-tuning a pre-trained multi-lingual system in a zero-shot setting.
1 Introduction
The paper proposes a universal multilingual NMT approach for extremely low-resource languages, sharing lexical and sentence-level representations across source languages. It reports improved Romanian-English translation with tiny parallel data and zero-resource settings.
- 1 Introduction: The approach targets low-resource languages by leveraging multilingual NMT capabilities through transfer learning.It focuses on sharing representations from multiple source languages into one target language.
- 1 Introduction: A universal word-level representation enables lexical sharing across source languages.The representation uses monolingual embeddings and seed parallel data from all languages.
- 1 Introduction: Sentence-level sharing uses source-language representations to support translation for languages with extremely limited or zero parallel data.
- 1 Introduction: The evaluation covers three languages with tiny or zero parallel data and reports consistent gains over a strong multilingual NMT baseline.
2 Motivation
NMT quality depends strongly on parallel-data volume, making extremely low-resource translation difficult. Multilingual systems offer transfer benefits but still face lexical and sentence-level sharing challenges when languages have limited overlap or data.
- 2 Motivation: A standard NMT encoder transforms input sequences into continuous representations, while an attention-based decoder predicts target words.The model is trained by maximizing likelihood over parallel sentences.
- 2 Motivation: NMT performance is highly correlated with parallel-data quantity and becomes unreasonable with extremely small or unavailable corpora.The motivation identifies approximately 13k sentences as extremely small and N = 0 as no parallel data.
- 2 Motivation: Multilingual NMT trains on multiple language pairs to regularize models and transfer translation knowledge to low-resource languages.
- 2.1 Challenges: Conventional multilingual vocabularies give each source word its own embedding, leaving low-resource vocabulary items poorly trained.
- 2.1 Challenges: Sharing subwords or characters cannot guarantee semantic overlap because low-resource languages may differ in surface forms or character sets.
- 2.1 Challenges: Low-resource languages also need shared sentence representations, including representations learned from monolingual data.Syntactic similarity and monolingual resources are identified as useful sources for sentence-level sharing.
3 Universal Neural Machine Translation
The proposed Universal NMT system addresses minimal parallel-data settings with Universal Lexical Representation for word-level sharing and Mixture of Language Experts for sentence-level sharing. ULR maps multilingual words into a shared probabilistic universal-token space, while MoLE combines language-specific experts for encoder representations.
- ULR and MoLE extend multilingual NMT to enable word-level and sentence-level sharing when minimal parallel sentences are available.
- Universal Lexical Representation: ULR represents each source word as a probabilistic mixture of universal-token embeddings learned during NMT training.The universal token set is discrete, while each word's representation is determined through a distribution based on semantic similarity.
- Universal Lexical Representation: The ULR key-value mechanism uses monolingual embeddings as queries, universal-token embeddings as keys, and distributions over universal NMT embeddings as values.A shared transformation matrix is optimized during NMT training to tune similarity scores toward the translation task.
- Universal Lexical Representation: Language-specific monolingual embeddings are projected into the universal-token space using learned language-specific transformations and seed word alignments.The paper also notes that seed-free transformation methods could support purely zero-parallel-resource cases.
- Mixture of Language Experts: MoLE adds expert networks and a gating network after the encoder, producing a weighted sum of expert outputs as the encoder representation.One expert is created per auxiliary language, while low-resource-language training leaves MoLE unchanged and represents its tokens through mixtures of auxiliary experts.
4 Experiments
The experiments evaluate Universal NMT on Romanian, Latvian, and Korean translating to English under near-zero-resource conditions, using auxiliary languages and vanilla or multilingual NMT baselines. The setup combines tiny down-sampled parallel corpora with larger monolingual resources and standard validation and test sets.
- Settings: The experiments compare the proposed system with vanilla single-source NMT and multilingual NMT baselines.
- Settings: The study evaluates Universal NMT on Romanian, Latvian, and Korean to English in near-zero-resource settings.Auxiliary languages are jointly trained from Czech, German, Greek, Spanish, Finnish, French, Italian, Portuguese, and Russian.
- Settings: The targeted-language parallel corpora are down-sampled to simulate zero-resource conditions, while monolingual data come from Wikipedia dumps.The monolingual corpora are typically much larger than the parallel corpora.
- Implementation: All systems use tokenized and BPE-segmented data, with sentences limited to 50 subword symbols and source vocabularies concatenated in multilingual settings.
- Implementation: The shared experimental architecture is a one-layer bidirectional RNN encoder with a two-layer attention-based RNN decoder, using 512 LSTM units.Models are trained with Adam for 1 million steps on mixed data using batches of 128 and dropout rate 0.4.
- Back-translation: Back-translation is used to encourage use of zero-resource-language information, although applying it directly is difficult without a reasonably good initial translation system.The authors initialize this process from a trained multilingual NMT model.
5 Results
Across low-resource translation experiments, universal lexical and sentence-level sharing improve performance, with gains depending on corpus size, auxiliary-language relatedness, and model configuration.
- Multilingual results: Applying universal tokens and mixture-of-experts modules improves translation quality across Romanian-English, Latvian-English, and Korean-English settings, with additive gains.Experiments use 6k sentences for Romanian and Latvian and 10k for Korean.
- Multilingual results: Romanian performs best with Romance-language auxiliaries, while Latvian improves when Russian is added as an auxiliary language.The paper attributes these gains to greater linguistic relatedness and shared knowledge.
- Multilingual results: The proposed methods consistently help less-related Korean-English translation, although its BLEU score remains relatively low.The experiments use Romance-family languages as Korean auxiliary languages and conclude that similar auxiliaries are more reasonable.
- Ablation study: +7.98 BLEU is obtained over the vanilla low-resource setup when ULR is combined with MoLE and back-translation.ULR alone gives +5.07 BLEU over Multi-NMT, while a trainable transformation matrix improves ULR further; the best result remains 5–6 BLEU below the full-data model.
- Ablation study: ULR remains strongest when monolingual Romanian data and back-translation are added, indicating additive gains.Back-translation improves both systems, but ULR still achieves the best score.
- Corpus size and unknown tokens: ULR works especially well with small training corpora and supports pure zero-resource translation through shared multilingual representations.Performance is evaluated at 0k, 6k, 60k, and 600k Romanian-English sentence pairs; larger corpora improve all systems.
6 Related Work
Prior multilingual NMT work established strong multilingual translation, but related approaches have limitations in zero-resource cases. This work extends that line toward very low-resource settings by sharing lexical and sentence representations across languages.
- Multilingual NMT has been extensively studied by Lee et al., Johnson et al., Zoph et al., and Firat et al.
- Earlier multilingual approaches have significant limitations in zero-resource cases.
- The proposed work extends related multilingual NMT by enabling lexical and sentence representation sharing across multiple languages for very low-resource settings.
7 Conclusion
The paper proposes universal machine translation to share resources between high-resource and extremely low-resource languages. On Romanian-English WMT2016, it achieves 23 BLEU with only 6k sentences, versus 18 BLEU for a strong multilingual baseline.
- The approach enables resource sharing between high-resource languages and extremely low-resource languages.
- 23 BLEU is achieved on Romanian-English WMT2016 using a tiny parallel corpus of 6k sentences.
- 18 BLEU is reported for the strong multilingual baseline system.