Source-linked AI summary
An Effective Approach to Unsupervised Machine Translation
Mikel Artetxe, Gorka Labaka, Eneko Agirre
TL;DR
Machine translation systems typically depend on large parallel corpora, limiting coverage across language pairs. The paper improves unsupervised SMT with subword information, principled tuning, and joint refinement, then uses it to initialize dual NMT through on-the-fly back-translation. The resulting approach improves previous unsupervised state-of-the-art results by 5-7 BLEU points across French-English and German-English WMT 2014 and 2016.
Problem
Existing machine translation systems rely on large parallel corpora, while unsupervised SMT systems have several deficiencies, including limited use of character-level information.
Method
The paper improves unsupervised SMT with subword information, theoretically well founded unsupervised tuning, and joint refinement, then uses it to initialize dual NMT with on-the-fly back-translation.
Results
5-7 BLEU points: the approach improves previous unsupervised state-of-the-art results across French-English and German-English WMT 2014 and 2016.
Takeaways & Limitations
The best results come from initializing NMT with SMT, while pure SMT systems outperform pure NMT systems.
Takeaways & Limitations
The paper identifies ungrammatical back-translated source phrases as a limitation because they affect backward translation probability estimation.
Abstract
from arXiv · showhide
While machine translation has traditionally relied on large amounts of parallel corpora, a recent research line has managed to train both Neural Machine Translation (NMT) and Statistical Machine Translation (SMT) systems using monolingual corpora only. In this paper, we identify and address several deficiencies of existing unsupervised SMT approaches by exploiting subword information, developing a theoretically well founded unsupervised tuning method, and incorporating a joint refinement procedure. Moreover, we use our improved SMT system to initialize a dual NMT model, which is further fine-tuned through on-the-fly back-translation. Together, we obtain large improvements over the previous state-of-the-art in unsupervised machine translation. For instance, we get 22.5 BLEU points in English-to-German WMT 2014, 5.5 points more than the previous best unsupervised system, and 0.5 points more than the (supervised) shared task winner back in 2014.
1 Introduction
Unsupervised machine translation removes dependence on parallel corpora but earlier approaches had distinct NMT and SMT limitations. This paper develops a principled SMT approach, combines it with NMT, and reports substantial gains over prior systems.
- Parallel corpora support recent translation progress but are available for only a few major-language combinations.
- Unsupervised machine translation trains systems using monolingual corpora only, initially through NMT methods combining denoising autoencoding and back-translation.
- Earlier SMT approaches induced phrase tables from cross-lingual embeddings, combined them with n-gram language models, and refined them through iterative back-translation.
- This paper addresses prior SMT deficiencies with subword information, theoretically well founded unsupervised tuning, joint refinement, and NMT initialization.
- 5-7 BLEU points: the proposed system outperforms previous unsupervised state-of-the-art systems across WMT 2014/2016 French-English and German-English datasets and directions.It also outperforms the supervised WMT 2014 shared-task winner in English-to-German and is around 2 BLEU points behind it in the remaining directions.
2 Related work
Unsupervised translation progressed from statistical decipherment and cross-lingual embedding methods to NMT and then SMT-based systems. Recent hybrid approaches combine SMT-generated synthetic data with NMT training.
- Statistical decipherment modeled the source language as ciphertext generated through a noisy channel and estimated parameters with expectation maximization or Bayesian inference.
- Unsupervised NMT first achieved promising standard-benchmark results using monolingual corpora and cross-lingual embedding mappings.
- Unsupervised SMT adapted these principles by learning cross-lingual n-gram embeddings, inducing phrase tables, adding language and distortion models, and refining through back-translation.
- Hybrid systems used synthetic parallel data from unsupervised SMT to assist or train NMT systems.
3 Principled unsupervised SMT
The paper develops principled unsupervised SMT by addressing phrase-table limitations, tuning model weights without parallel data, and jointly refining opposite-direction systems. It combines subword-aware phrase modeling with cycle- and language-model-based optimization and iterative refinement.
- Phrase-table construction is the central unsupervised SMT challenge because other log-linear components can be learned from monolingual data or remain parameterless.
- The method extends an embedding-induced initial phrase-table with character-level weights to exploit subword information and improve named-entity translation.The character-level features complement word- or phrase-level alignment using normalized Levenshtein similarity.
- The character-similarity design remains limited because more elaborate and learnable similarity functions are left for future work.
- Unsupervised tuning combines cyclic consistency and language-model losses, then adapts MERT to jointly optimize both translation directions toward a local optimum.The objective favors translations that reconstruct monolingual inputs and remain fluent in the target language.
- Joint refinement uses back-translated synthetic parallel data to address unnatural phrase-table probabilities and the absence of a lexical reordering model.The authors note that back-translation can introduce ungrammatical n-grams and artifacts into the induced phrase-table.
4 NMT hybridization
The hybrid approach uses unsupervised SMT to warm up dual NMT systems, addressing the performance ceiling imposed by SMT while retaining monolingual-data training. The NMT models are then improved through iterative back-translation.
- SMT provides a useful unsupervised framework, but its locality and sparsity limitations impose a ceiling compared with supervised NMT.
- The hybrid method trains opposite-direction SMT systems first and uses them to initialize two opposite-direction NMT systems.
- The NMT systems continue training through alternating updates with iterative back-translation rather than restarting from scratch each iteration.
- Table 1 compares the proposed method with prior systems using BLEU, distinguishing overall best results from the best results within each group.The table caption also distinguishes detokenized BLEU from tokenized BLEU implementations.
5 Experiments and results
Experiments show that the proposed unsupervised translation system substantially improves over prior unsupervised methods across WMT directions and approaches supervised performance in selected settings. The strongest gains come from combining principled SMT with NMT hybridization, while translations are generally fluent and meaning-preserving.
- Main results: 5-7 BLEU points separate the full system from the previous state-of-the-art across all reported datasets and translation directions.The experiments use French-English and German-English WMT data, with test sets from WMT 2014 and, for German-English, WMT 2016.
- SMT and NMT hybridization: Around 2 BLEU points distinguish the principled SMT approach from previous SMT-based systems.The initial SMT system provides a stronger starting point before NMT hybridization.
- SMT and NMT hybridization: 5-9 BLEU points are gained by NMT hybridization over the initial SMT systems.The hybridization gains exceed those reported for previous hybridization methods despite starting from stronger SMT systems.
- SMT and NMT hybridization: Pure SMT systems outperform pure NMT systems, but the best results come from initializing NMT with SMT.The authors suggest SMT is useful for initial cross-lingual alignment, while NMT produces the strongest final system.
- Comparison with supervised systems: The unsupervised system beats the WMT 2014 shared task winner in English-to-German and trails it by around 2 BLEU points in the other directions.The comparison uses supervised systems evaluated on the same WMT 2014 test set.
- Qualitative results: The proposed translations are generally more fluent and avoid the named-entity and numeral adequacy errors illustrated for the earlier SMT system.The examples are randomly selected from French-English newstest2014 and compare against Artetxe et al. (2018b).
6 Conclusions and future work
The paper presents a more principled unsupervised machine translation approach and reports substantial gains over prior systems. Future work targets learnable character-level similarity functions and broader multilingual or limited-parallel-data settings.
- The approach improves previous unsupervised machine translation state of the art by 5–7 BLEU points in French-English and German-English WMT 2014 and 2016.
- The method combines subword information, theoretically well founded unsupervised tuning, and joint refinement to address deficiencies in prior unsupervised SMT systems.
- The improved SMT system initializes a dual NMT model that is further improved through on-the-fly back-translation.
- Future work: Future work will explore learnable similarity functions for computing character-level scores in the initial phrase-table.
- Future work: The authors also plan to adapt the approach to settings involving multiple languages and/or small parallel corpora.