Source-linked AI summary
Unsupervised Neural Machine Translation
Mikel Artetxe, Gorka Labaka, Eneko Agirre, Kyunghyun Cho
TL;DR
Large parallel corpora limit NMT for many language pairs, motivating a method that trains translation models from monolingual data alone. The paper combines cross-lingual embeddings, a shared attentional encoder-decoder, denoising, and backtranslation, reaching 15.56 and 10.21 BLEU points on WMT 2014 French-to-English and German-to-English. The method also improves with 100,000 parallel sentences, but translation quality often lags supervised NMT and concrete source details remain difficult to preserve.
Problem
NMT requires large parallel corpora to be effective, but such corpora are unavailable for many language pairs and existing alternatives still require a strong cross-lingual signal.
Method
The method uses fixed unsupervised cross-lingual embeddings in a shared attentional encoder-decoder and trains on monolingual corpora with denoising and backtranslation.
Results
15.56 BLEU points for French →English and 10.21 BLEU points for German →English were obtained on WMT 2014 using only monolingual training data.
Takeaways & Limitations
The approach learns non-trivial cross-lingual translation relations beyond word-by-word substitution and can improve further when combined with a small parallel corpus.
Takeaways & Limitations
Translation quality often lags supervised NMT, and the model has difficulty preserving concrete source details.
Abstract
from arXiv · showhide
In spite of the recent success of neural machine translation (NMT) in standard benchmarks, the lack of large parallel corpora poses a major practical problem for many language pairs. There have been several proposals to alleviate this issue with, for instance, triangulation and semi-supervised learning techniques, but they still require a strong cross-lingual signal. In this work, we completely remove the need of parallel data and propose a novel method to train an NMT system in a completely unsupervised manner, relying on nothing but monolingual corpora. Our model builds upon the recent work on unsupervised embedding mappings, and consists of a slightly modified attentional encoder-decoder model that can be trained on monolingual corpora alone using a combination of denoising and backtranslation. Despite the simplicity of the approach, our system obtains 15.56 and 10.21 BLEU points in WMT 2014 French-to-English and German-to-English translation. The model can also profit from small parallel corpora, and attains 21.81 and 15.24 points when combined with 100,000 parallel sentences, respectively. Our implementation is released as an open source project.
1 INTRODUCTION
NMT performs well but typically depends on large parallel corpora, which are unavailable for many language pairs. The proposed system removes this requirement by training an NMT model from monolingual data with denoising and backtranslation, achieving strong WMT results.
- Large parallel corpora are required for effective NMT, creating a practical obstacle for many language pairs.
- The proposed method trains NMT completely unsupervised using only monolingual corpora and unsupervised cross-lingual embeddings.The system uses a shared encoder and combines denoising with backtranslation.
- The architecture alternates denoising and on-the-fly backtranslation for sentences from both languages.Denoising reconstructs noised inputs, while backtranslation translates through the other decoder before reconstructing the original sentence.
- 15.56 BLEU points for French →English and 10.21 BLEU points for German →English were obtained on WMT 2014 using only monolingual training data.
2 RELATED WORK
Prior work uses cross-lingual signals, decipherment, pivoting, or scarce parallel data to address low-resource translation. This paper targets the more ambitious setting of training NMT from monolingual corpora alone.
- Cross-lingual embedding methods commonly rely on bilingual signals, while unsupervised alternatives learn mappings from independently trained monolingual embeddings.
- Statistical decipherment induces translation models from monolingual data using noisy-channel models, language models, and estimated channel parameters.
- Pivoting and backtranslation exploit indirect or monolingual resources, but these approaches generally remain connected to scarce parallel data.
- Prior dual-agent NMT required 1.2 million parallel sentences for a warm start, whereas this work uses no parallel data.
3 PROPOSED METHOD
The proposed system modifies attentional encoder-decoder NMT for unsupervised training by jointly handling both translation directions with a shared encoder and fixed cross-lingual embeddings. Denoising and on-the-fly backtranslation provide training objectives from monolingual sentences.
- Architecture: The model uses a dual architecture that handles both translation directions together rather than targeting a single direction.
- Architecture: A shared encoder produces language-independent representations for both languages, which separate decoders transform into their respective languages.
- Architecture: Fixed pre-trained cross-lingual embeddings provide language-independent word representations that the encoder composes into larger phrase representations.
- Unsupervised training: Denoising trains each language pathway to reconstruct its input after local token perturbations.
- Unsupervised training: On-the-fly backtranslation translates a monolingual sentence into the other language and trains the system to reconstruct the original sentence.
- Unsupervised training: Training alternates denoising and backtranslation mini-batches across both language directions, with optional direct prediction when parallel data is available.
4 EXPERIMENTAL SETTINGS
Experiments evaluate French-English and German-English translation on WMT 2014 under unsupervised, semi-supervised, and supervised settings. The unsupervised condition uses only monolingual News Crawl data, with standard preprocessing and BLEU evaluation.
- Evaluation: French-English and German-English systems are evaluated on WMT 2014 newstest2014 using tokenized BLEU scores.
- Training settings: The unsupervised setting uses nothing but monolingual News Crawl articles from 2007 to 2013.
- Training settings: The semi-supervised setting combines the monolingual data with either 10,000 or 100,000 random in-domain parallel sentence pairs.
- Training settings: The supervised setting uses all relevant WMT 2014 parallel corpora and serves as an approximate upper bound for the proposed system.
- Experimental controls: No parallel data from the evaluated language pairs is used for development or tuning.
- Implementation: The systems use independently learned 50,000-operation BPE models, cross-lingual embeddings, cross-entropy training, Adam optimization, and dropout.
- Implementation: Greedy decoding is used during training backtranslation, while test-time inference uses beam search with beam size 12.
5 RESULTS AND DISCUSSION
The proposed unsupervised system achieves substantial translation quality using monolingual data, while backtranslation is essential and small parallel corpora provide further gains. Qualitative analysis shows non-trivial translation ability but persistent detail, fluency, and adequacy limitations.
- Quantitative results: 14-15 BLEU points in French-English and 6-10 BLEU points in German-English were achieved using only monolingual corpora, exceeding word-by-word substitution by 40-140%.The results indicate that the system uses context and models internal linguistic structure beyond literal substitution.
- Quantitative results: Backtranslation is essential: denoising alone falls below the baseline, whereas introducing backtranslation produces large improvements.French→English per-word perplexity decreases from 634.79 with denoising alone to 44.74 with backtranslation.
- Quantitative results: Both denoising and backtranslation play essential training roles, with denoising capturing broad word-level equivalences and backtranslation encouraging subtler relations.The procedure would not work with backtranslation alone because initial random translations could encourage learning only a target-language model.
- Quantitative results: BPE is slightly beneficial for German targets, detrimental for French targets, and practically equivalent for English targets, while rare words and named entities remain difficult.The authors suggest character-level information and better handling of rare words, named entities, and numerals as future improvements.
- Quantitative results: 100,000 parallel sentences improve results by 4-7 BLEU points, and the resulting semi-supervised system surpasses comparable full-corpus NMT in all but one case.The semi-supervised systems alternate denoising, backtranslation, and parallel-sentence training; the authors attribute the comparison partly to domain matching.
- Quantitative results: Additional constraints introduced for unsupervised learning may limit performance, and limited hyperparameter exploration leaves considerable room for improvement.The authors propose progressively relaxing fixed embeddings, shared encoders, and noise during training.
- Qualitative analysis: Qualitative examples show non-trivial phrase and structural translation, including idiomatic expressions, reordered noun phrases, and longer complex sentences.The analysis translates “a eu lieu” as “occurred” and “l’aéroport international de Los Angeles” as “Los Angeles International Airport.”
- Qualitative analysis: The system often lags behind supervised NMT and can mistranslate concrete details, while some outputs contain fluency and adequacy problems that hinder understanding.Examples include confusing October with May and 1 073 with 1 064; the authors identify character-level information as a possible mitigation.
6 CONCLUSIONS AND FUTURE WORK
The paper presents an unsupervised NMT method trained from monolingual corpora and reports improvements on French-English and German-English benchmarks, while identifying substantial room for improvement.
- The method trains a modified attentional encoder-decoder using fixed cross-lingual embeddings, a shared encoder, denoising, and backtranslation from monolingual corpora alone.
- The experiments show significant BLEU improvements over word-by-word substitution on standard WMT 2014 French-English and German-English benchmarks.
- Combining the unsupervised method with a small parallel corpus further improves results, indicating potential beyond the strictly unsupervised setting.
- The authors identify considerable room for improvement, including relaxing architectural constraints and adding character-level information to address adequacy issues.