Source-linked AI summary
Unsupervised Machine Translation Using Monolingual Corpora Only
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, Marc'Aurelio Ranzato
TL;DR
The paper asks whether machine translation can be learned without parallel data, addressing the scarcity and cost of aligned corpora. It maps two languages into a shared latent space and learns reconstruction-based translation from monolingual corpora, achieving effective translation without supervision. A remaining limitation is that shared latent representations do not guarantee accurate translations.
Problem
Machine translation requires large parallel corpora, which are costly and often unavailable for low-resource languages.
Method
The model uses shared encoder-decoder representations and reconstructs noisy monolingual inputs and noisy cross-language translations in both directions.
Results
The approach learns effective translation models without any supervision and reaches BLEU 32.76 on Multi30k English-French.
Takeaways & Limitations
Monolingual datasets alone can support effective translation models without sentence or document alignment.
Takeaways & Limitations
Aligning latent distributions across languages does not ensure that the decoder produces a good translation rather than merely a valid target-language sentence.
Abstract
from arXiv · showhide
Machine translation has recently achieved impressive performance thanks to recent advances in deep learning and the availability of large-scale parallel corpora. There have been numerous attempts to extend these successes to low-resource language pairs, yet requiring tens of thousands of parallel sentences. In this work, we take this research direction to the extreme and investigate whether it is possible to learn to translate even without any parallel data. We propose a model that takes sentences from monolingual corpora in two different languages and maps them into the same latent space. By learning to reconstruct in both languages from this shared feature space, the model effectively learns to translate without using any labeled data. We demonstrate our model on two widely used datasets and two language pairs, reporting BLEU scores of 32.8 and 15.1 on the Multi30k and WMT English-French datasets, without using even a single parallel sentence at training time.
1 INTRODUCTION
Machine translation traditionally depends on massive parallel corpora, which are costly and often unavailable for low-resource languages. This work asks whether translation can instead be learned from monolingual corpora alone by sharing a latent space and reconstructing across languages.
- Millions of parallel sentences enable strong translation systems, but parallel corpora are costly to build and often nonexistent for low-resource languages.
- Earlier methods used monolingual data through back-translation, language models, or auto-encoding, but still required tens of thousands of parallel sentences.
- Prior zero-resource approaches generally relied on labeled information from related language pairs or other modalities, while deciphering methods handled only restricted settings.
- The paper investigates fully unsupervised translation assuming only one monolingual corpus per language, applicable to unannotated language pairs and providing a lower bound for semi-supervised systems.
- The model maps both languages into a common latent space and learns reconstruction from noisy monolingual inputs and noisy translations in both directions.
- BLEU 32.76 on Multi30K English-French and parity with a fully supervised system trained on 100,000 WMT sentence pairs demonstrate strong performance without parallel training data.
2 UNSUPERVISED NEURAL MACHINE TRANSLATION
The proposed translation system uses shared encoder–decoder components and trains them on monolingual data through denoising, cross-domain reconstruction, and adversarial alignment. The approach iteratively improves translations while addressing the risk that shared latent representations may not enforce sentence-level correspondence.
- 2.1 NEURAL MACHINE TRANSLATION MODEL: The model uses one encoder and one decoder for both languages, differing only in their language-specific lookup tables.The encoder maps source and target sentences into a latent space, while the decoder generates outputs in either language.
- 2.2 OVERVIEW OF THE METHOD: Training reconstructs monolingual sentences from noisy inputs within the same language or from noisy translations in the other language.Word dropout and local shuffling create denoising inputs; the reported settings use word-dropout probability 0.1 and shuffle parameter k = 3.
- 2.2 OVERVIEW OF THE METHOD: The method initializes translation with unsupervised word-by-word mappings, then iteratively trains reconstruction and translation objectives while regenerating translations.At test time, the encoder and decoder compose into a standard machine translation system despite having no parallel training data.
- 2.5 ADVERSARIAL TRAINING: The encoder is encouraged to produce language-independent features by adversarially training a discriminator to classify whether encoded sentences come from the source or target domain.The encoder is trained to fool the discriminator, promoting alignment between the two latent distributions.
- 2.5 ADVERSARIAL TRAINING: Shared latent representations alone may yield fluent target-language sentences without preserving sentence correspondence, so cross-domain reconstruction is used to mitigate this problem.The paper connects this strategy to evidence that aligned latent representations can be effective at the word level when feature-space structure is strong.
3 TRAINING
Training alternates between translating monolingual data with the current model and retraining an encoder-decoder, while an unsupervised reconstruction criterion selects models without parallel validation data.
- Iterative training: At each iteration, the current model translates both monolingual datasets, and a new encoder-decoder is trained by minimizing the objective.The resulting components are composed into the next translation model.
- Iterative training: Training begins with a word-by-word translation model initialized from a bilingual dictionary learned using monolingual data.This initial model supplies the first translations needed by the iterative procedure.
- Architecture and objectives: The architecture uses shared sequence-to-sequence components across two languages, with language identifiers selecting the corresponding lookup tables.Its objectives combine denoising auto-encoding with cross-language reconstruction using translations produced at the previous iteration.
- Unsupervised model selection: Model selection translates each sentence to the other domain and back, then averages BLEU over both directions against the original inputs.The model with the highest average score is selected.
- Unsupervised model selection: The proposed criterion correlates with test BLEU at an average Spearman coefficient of 0.95, although selected models can be 1 or 2 BLEU points below models chosen with 500 parallel validation sentences.It is used both to stop training and to select hyper-parameters.
4 EXPERIMENTS
Experiments evaluate the unsupervised system on English-French and English-German across WMT and Multi30k-Task1, comparing it with translation baselines and supervised models. The model achieves strong BLEU, converges quickly, and depends especially on unsupervised word alignment, adversarial training, and denoising.
- 4.1 DATASETS: The experiments cover English-French and English-German on WMT and Multi30k-Task1, using monolingual training and validation corpora derived from the datasets.WMT English-French uses about 30 million filtered parallel sentences split into non-overlapping English and French monolingual sets; WMT English-German uses 1.8 million monolingual sentences per language, while Multi30k provides 14,500 training sentences per language.
- 4.5 EXPERIMENTAL RESULTS: The model outperforms word-by-word and word-reordering baselines, reaching BLEU 32.76 on Multi30k-Task1 and 15.05 on WMT for English-to-French translation.After one iteration, English-to-French BLEU is 27.48 on Multi30k-Task1 and 12.10 on WMT; after a few iterations, it reaches the reported final scores.
- 4.5 EXPERIMENTAL RESULTS: The unsupervised approach matches a supervised NMT model trained on about 100,000 parallel sentences on WMT, although additional parallel data lets supervised models outperform it.The comparison uses the same NMT architecture across varying amounts of parallel data.
- 4.5 EXPERIMENTAL RESULTS: Performance is already high after the first iteration, improves with subsequent iterations, and shows marginal gains by iteration 3, indicating rapid convergence.Translation examples also improve at every iteration from the word-by-word initialization.
- 4.5 EXPERIMENTAL RESULTS: Ablations identify unsupervised word alignment as the most critical component, while the adversarial and auto-encoding losses and input corruption also materially affect performance.Removing both alignment components reduces BLEU to 8.78 and 9.15; removing auto-encoding yields 20.02, 8.05 BLEU below the full method, and removing corruption causes substantial degradation.
5 RELATED WORK
The paper situates its approach among unsupervised representation-learning and non-parallel translation methods, emphasizing shared latent spaces, adversarial alignment, and differentiable iterative training.
- Adversarial representation learning: Adversarial training is used to make latent representations invariant to domain-specific attributes or labels in related non-parallel learning approaches.
- Related machine translation methods: He et al. optimize a related model-selection metric but require reinforcement learning to back-propagate through discrete predictions.
- This work: This work instead uses a symmetric architecture, alternately freezes one directional translator, and retains a fully differentiable training process.
- Unsupervised domain translation: Prior non-parallel methods learn shared representations across domains, including shared convolutional layers in CoGAN and adversarially disentangled latent states in Fader Networks.
6 CONCLUSION
The paper presents neural machine translation learned from monolingual datasets without sentence or document alignment. It iteratively improves an initial word-by-word model through reconstruction and adversarial latent-distribution alignment, and reports effective translation without supervision.
- The approach learns neural machine translation from monolingual datasets only, without alignment between sentences or documents.
- The method starts with unsupervised word-by-word translation and iteratively improves it using reconstruction loss and a discriminator aligning source and target latent distributions.
- Experiments demonstrate effective translation models without supervision of any sort.