Source-linked AI summary
Zero-Resource Translation with Multi-Lingual Neural Machine Translation
Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman Vural, Kyunghyun Cho
TL;DR
Zero-resource neural translation lacks direct parallel examples between the target languages. The paper introduces a pseudo-parallel-pair finetuning strategy for multilingual models, achieving performance comparable to single-pair models trained with up to 1M direct parallel sentences and surpassing pivot-based translation.
Problem
The paper investigates whether multilingual neural machine translation can directly translate language pairs without any direct parallel examples.
Method
The authors use many-to-one translation strategies and finetune an additional attention-parameter copy using pseudo-parallel sentence pairs without direct parallel data.
Results
The proposed strategy performs comparably to or better than single-pair models trained with 1M true parallel sentences and outperforms pivot-based translation.
Takeaways & Limitations
Zero-resource neural translation can achieve strong quality and data efficiency without direct parallel corpora or deep linguistic knowledge between the languages.
Takeaways & Limitations
Experiments cover only Spanish, French, and English, so broader language diversity and the effects of varying parallel-corpus sizes remain to be studied.
Abstract
from arXiv · showhide
In this paper, we propose a novel finetuning algorithm for the recently introduced multi-way, mulitlingual neural machine translate that enables zero-resource machine translation. When used together with novel many-to-one translation strategies, we empirically show that this finetuning algorithm allows the multi-way, multilingual model to translate a zero-resource language pair (1) as well as a single-pair neural translation model trained with up to 1M direct parallel sentences of the same language pair and (2) better than pivot-based translation strategy, while keeping only one additional copy of attention-related parameters.
1 Introduction
The paper investigates zero-resource translation with a multi-way, multilingual neural machine translation model, finding that many-to-one strategies outperform one-to-one translation and motivate a parameter-efficient finetuning method. The proposed method enables zero-resource translation as well as a single-pair model trained with up to 1M true parallel sentences.
- Research focus: The study investigates whether a multi-way, multilingual neural machine translation model can translate language pairs without any direct parallel examples.Zero-resource translation is contrasted with traditional pivot-based translation.
- Translation strategies: Many-to-one multi-source translation strategies significantly outperform the usual one-to-one strategy.These strategies are studied before designing the zero-resource finetuning method.
- Translation strategies: The vanilla multi-way, multilingual model cannot perform zero-resource translation.This evaluation motivates a separate finetuning strategy.
- Finetuning method: The proposed finetuning strategy requires no direct parallel corpus and adds only an additional copy of the attention mechanism, finetuning this small parameter set.It uses the idea of generating a pseudo-parallel corpus.
- Results: Up to 1M true parallel sentences suffice for the proposed model to perform zero-resource translation as well as a single-pair neural translation model.Large-scale experiments use Spanish, French, and English.
2 Multi-Way, Multilingual Neural Machine Translation
The multi-way, multilingual model translates among N source and M target languages using language-specific encoders and decoders with a single attention mechanism shared across all M × N language pairs. It can be trained from bilingual corpora rather than requiring multi-way parallel data.
- Architecture: The model translates a source sentence in one of N languages into one of M target languages using N encoders and M decoders.Each encoder and decoder is language-specific.
- Architecture: A single attention mechanism is shared across all M × N language pairs, unlike the language-specific encoders and decoders.The shared mechanism computes time-dependent context from encoder context vectors.
- Encoder: Each source-language encoder reads a sentence as linguistic symbols and returns context vectors, typically using a bidirectional recurrent network.Context vectors concatenate the forward and reverse recurrent networks’ hidden states.
- Decoder and Attention Mechanism: Each target-language decoder is a conditional recurrent language model whose hidden state updates at every time step using a gated recurrent unit.The decoder computes probabilities over the next target symbol from its updated hidden state.
- Training: Training requires only a set of bilingual corpora, optimizing each pair’s conditional log-probability by gradient-based adjustment of relevant parameters.Multi-way parallel corpora are not required.
3 Translation Strategies
The multi-way multilingual model supports many-to-one translation from multiple source languages, despite not being trained on multi-way parallel corpora. The section presents early, late, and combined averaging strategies for multi-source translation.
- Translation strategies: The original one-to-one strategy combines a source-language encoder, target-language decoder, and shared attention mechanism as a single-pair translation model.This was the only strategy evaluated by Firat et al. (2016).
- Translation strategies: Many-to-one translation exploits multiple encoders and decoders with shared attention to translate a sentence provided in multiple source languages.The approach extends multi-source translation to the multi-way multilingual model.
- Translation strategies: The model is not trained with multi-way parallel corpora, but the section proposes two alternatives for multi-source translation.The approaches are introduced for a sentence available in two languages, X1 and X2, and extend trivially to more source languages.
- Early Average: Early averaging averages the source-specific time-dependent context vectors at each decoder time step and averages the encoders’ initializers for decoder-state initialization.For Es+Fr to En, this combines the Es→En and Fr→En translation paths before output generation.
- Late Average: Late averaging averages the target-vocabulary distributions produced independently by the two translation paths at each decoder time step.Unlike early averaging, it can combine paths from separate single-pair models and is equivalent to an ensemble when both paths use the same language pair and direction.
- Early+Late Average: Early+late averaging combines the early-averaged model’s output distribution with the late-averaged model’s output distribution.The paper also empirically evaluates this combined strategy.
4 Experiments: Translation Strategies and Multi-Source Translation
The experiments evaluate multi-source translation from Spanish and French to English using only En-Es and En-Fr parallel corpora, then show that the multilingual model effectively combines multiple sources without multi-way parallel training data. Averaging strategies improve quality by up to +3.5 BLEU, with output-level combination performing best.
- Experimental setup: The multi-source evaluation uses English, Spanish, and French with only En-Es and En-Fr parallel corpora available.newstest-2012 serves as development data and newstest-2013 as the test set; no additional monolingual corpus is used.
- One-to-one translation: The multilingual model performs comparably to four separate single-pair models on all four trained translation directions while using fewer parameters through shared attention.The four directions are Es↔En and Fr↔En.
- Many-to-one translation: Late averaging marginally outperforms early averaging for both the multilingual model and an ensemble of two single-pair models.The ensemble of two separate single-pair models is reported in the second column of Table 3.
- Many-to-one translation: 3 BLEU points: early averaging improves test-set quality for Es+Fr→En despite no multi-way parallel training corpus.The comparison is between Table 2 (a–b) and Table 3 (a).
- Many-to-one translation: The multilingual model exploits multiple sources effectively without requiring any Spanish–French–English multi-way parallel corpus.This property motivates using the model with the proposed many-to-one strategies for later zero-resource translation experiments.
5 Zero-Resource Translation Strategies
The section develops zero-resource translation between language pairs lacking direct parallel data, focusing on Spanish–French when only Spanish–English and French–English corpora are available. It contrasts ineffective direct decoding with pivot-based and finetuning strategies that use no additional bilingual corpus.
- Motivation: Zero-resource paths arise because the multilingual model can connect languages indirectly, such as Spanish–French through the trained Spanish–English and French–English pairs.The considered zero-resource direction is Es↔Fr, while training uses Es↔En and Fr↔En parallel corpora.
- Baseline: The naive one-to-one strategy treats the zero-resource path like a trained path, but experiments show that it fails completely.This approach corresponds to the one-to-one strategy described in Section 3.1.
- Target-Specific Finetuning: The proposed finetuning method creates pseudo Spanish–French pairs and updates only a copied target-specific attention mechanism, leaving the encoder and decoder parameters unchanged.After finetuning, the model can use the earlier translation strategies, including many-to-one translation, without requiring additional bilingual data.
- Pivot-Based Translation: Pivot-based translation decomposes Es→Fr into Es→En followed by En→Fr, with independent one-to-one decoding adding a multiplicative computational factor of two.Maintaining k-best translations from the first stage further increases computational complexity by a factor of k and is impractical.
- Many-to-One Translation: Many-to-one pivot translation uses both the original Spanish sentence and its English translation as paired source languages for the final English-to-French stage.The strategy first generates English from Spanish, then translates using Es+En to produce French.
6 Experiments: Zero-Resource Translation
Experiments evaluate Spanish-to-French zero-resource translation using a multilingual model trained on Spanish-English and French-English data without Spanish-French parallel data. Finetuning with pseudo-parallel pairs improves over pivot translation and can approach or exceed a single-pair model trained with 1M true parallel sentences.
- Experimental setup: The model was trained only on Es-En and Fr-En bilingual corpora, with Es-Fr selected as the zero-resource translation path.No Es-Fr parallel corpus was used during the model’s original training.
- Baseline strategies: Direct translation failed when the language path was absent from training, while pivot-based translation was decent and many-to-one strategies performed worse.The many-to-one degradation was attributed to including a useless translation component.
- Pseudo-parallel finetuning: 1,000 pseudo-parallel pairs were sufficient for finetuning to outperform pivot-based translation, while larger pseudo corpora produced clear further improvements.Pseudo corpora contained 1k, 10k, 100k, or 1m pairs.
- True versus pseudo supervision: 1M true parallel pairs were beneficial for finetuning, but with fewer true pairs, pseudo pairs could be more beneficial, especially for direct one-to-one translation.The comparison used true multi-way parallel data assembled from UN, Europarl-v7, OpenSubtitles-2013, news-commentary-v7, LDC2011T07, and news-crawl.
- Overall findings: The results show that zero-resource translation quality can improve without direct parallel data and may improve further when a small number of direct parallel pairs is available.Finetuned multilingual models performed comparably to or better than the single-pair model trained with 1M true parallel sentences.
- Decoding strategies: Further averaging early- and late-averaged decoding did not improve results because pseudo-source context vectors already contained the pseudo source’s information.The early averaging scheme was therefore sufficient to use those context vectors.
7 Conclusion: Implications and Limitations
The paper shows that multilingual translation can exploit shared structures and that zero-resource translation does not require direct parallel data or deep linguistic knowledge. Its conclusions are limited by the narrow language and source-language evaluations and by parameter growth with target language pairs.
- Implications: The multilingual model exploits common underlying structures across languages, demonstrating positive language transfer in machine translation.This result concerns improved translation when a source sentence is given in multiple languages.
- Implications: Zero-resource translation requires neither a direct parallel corpus nor deep linguistic knowledge between the two languages.The proposed approach was also better than pivot-based translation in translation quality and data efficiency.
- Limitations: The experiments covered only Spanish, French, and English, so broader language diversity and varying parallel-corpus sizes require further investigation.The authors identify these as limitations for drawing more solid conclusions.
- Limitations: Although many-to-one translation is generally applicable to any number of source languages, the study tested source sentences in only two languages.The authors expect greater improvement with more languages, but state that this requires thorough future testing.
- Limitations: The finetuning strategy requires additional attention-related parameters for each target zero-resource pair, potentially causing parameter counts to grow linearly with target language pairs.Future work could mix high-resource language-pair parallel corpora into finetuning to address this issue.