Source-linked AI summary
Unsupervised Statistical Machine Translation
Mikel Artetxe, Gorka Labaka, Eneko Agirre
TL;DR
Existing unsupervised NMT systems trained from monolingual data remain far behind supervised systems. This paper instead builds phrase-based SMT from cross-lingual phrase embeddings, tunes it with unsupervised MERT, and refines it through iterative backtranslation, reducing the supervised SMT gap to 2–5 points.
Problem
Existing unsupervised NMT systems trained only on monolingual corpora remain far behind supervised counterparts, limiting their practical usability.
Method
The paper induces a phrase table from cross-lingual n-gram embeddings, combines it with an n-gram language model and distortion model, and tunes weights through unsupervised MERT and iterative backtranslation.
Results
The system improves over previous NMT-based approaches by more than 7 BLEU points for German-English and above 10 BLEU points for French-English, narrowing the supervised SMT gap to 2–5 points.
Takeaways & Limitations
Phrase-based SMT trained from monolingual corpora can substantially close the gap between unsupervised and supervised translation systems.
Takeaways & Limitations
The system omits lexical reordering, limits its phrase table to the embedding vocabulary, and estimates translation probabilities from cross-lingual embeddings.
Abstract
from arXiv · showhide
While modern machine translation has relied on large parallel corpora, a recent line of work has managed to train Neural Machine Translation (NMT) systems from monolingual corpora only (Artetxe et al., 2018c; Lample et al., 2018). Despite the potential of this approach for low-resource settings, existing systems are far behind their supervised counterparts, limiting their practical interest. In this paper, we propose an alternative approach based on phrase-based Statistical Machine Translation (SMT) that significantly closes the gap with supervised systems. Our method profits from the modular architecture of SMT: we first induce a phrase table from monolingual corpora through cross-lingual embedding mappings, combine it with an n-gram language model, and fine-tune hyperparameters through an unsupervised MERT variant. In addition, iterative backtranslation improves results further, yielding, for instance, 14.08 and 26.22 BLEU points in WMT 2014 English-German and English-French, respectively, an improvement of more than 7-10 BLEU points over previous unsupervised systems, and closing the gap with supervised SMT (Moses trained on Europarl) down to 2-5 BLEU points. Our implementation is available at https://github.com/artetxem/monoses
1 Introduction
Unsupervised NMT uses monolingual corpora but remains far behind supervised translation, motivating an alternative based on modular phrase-based SMT. The proposed system combines cross-lingual phrase embeddings, standard SMT components, unsupervised tuning, and iterative backtranslation.
- The paper investigates whether SMT’s rigid, modular architecture is better suited to unsupervised translation than end-to-end NMT.
- The system induces a phrase table from cross-lingual phrase embeddings, combines it with a language model and distortion model, and tunes weights unsupervised.
- Iterative backtranslation further improves the system’s results.
- Experiments report improvements above 7–10 BLEU points over previous unsupervised approaches and reduce the supervised SMT gap to 2–5 points.
2 Background: phrase-based SMT
Phrase-based SMT scores translation candidates with a modular log-linear combination of statistical models. Its components cover phrase translations, target-language fluency, reordering, output length, and tuned model weights.
- Phrase-based SMT uses a log-linear model combining independently estimated scoring functions whose weights are tuned for translation quality.
- The phrase table lists source n-grams, possible target translations, and multiple scores derived from aligned parallel data.
- The language model assigns probabilities to target-language word sequences using n-gram counts from monolingual data.
- The reordering model scores word-order deviations through distance-based distortion and lexical reordering components.
- Word and phrase penalties control output length and preferences for shorter or longer phrases.
- MERT commonly tunes model weights by maximizing BLEU on a separate parallel validation corpus.
3 Cross-lingual n-gram embeddings
The paper extends skip-gram to learn word and phrase embeddings simultaneously, then maps them across languages for phrase-based translation. This avoids problematic stochastic segmentation while retaining unigram representations.
- Negative-sampling skip-gram learns embeddings from word-context pairs by predicting true corpus pairs against noise samples.
- Standard phrase preprocessing merges selected non-compositional phrases into single tokens using co-occurrence scores and a rarity discount.
- Compositional n-grams also require phrase-level representations because translation can map an entire source phrase to one target word.
- Merging all overlapping n-grams is difficult, and stochastic segmentation performed poorly by changing co-occurrence distributions and worsening sparsity.
- The proposed skip-gram generalization learns n-gram embeddings on-the-fly while preserving the original skip-gram embeddings for unigrams.
- Cross-lingual mapping uses self-learning to create a shared space, learning on the 20,000 most frequent unigrams before applying the mapping to longer n-grams.
4 Unsupervised SMT
The unsupervised SMT system replaces parallel-data-dependent components with cross-lingual embedding-based phrase induction while retaining monolingual language modeling and distortion scoring. Synthetic parallel data enables iterative unsupervised weight tuning and backtranslation refinement.
- SMT’s language model, distortion model, and penalties can be integrated from monolingual data, while lexical reordering is omitted.
- Without parallel data, each target n-gram is a candidate translation, but the phrase table keeps only the 100 nearest target neighbors for each source phrase.
- Phrase translation probabilities are computed by applying softmax to cosine similarities between cross-lingual phrase embeddings.
- The temperature parameter controls prediction confidence and is tuned using maximum likelihood on a dictionary induced in the opposite retrieval direction.
- Lexical weightings align target words with their most likely source generators and multiply the corresponding translation probabilities.
- Unsupervised MERT creates synthetic parallel data through backtranslation and iteratively tunes weights in both translation directions.
5 Iterative refinement
The initial unsupervised SMT system has important performance-limiting simplifications. Iterative backtranslation addresses these restrictions by creating synthetic parallel data for standard SMT training, though the procedure is computationally expensive.
- Limitations: The initial system omits lexical reordering, limits phrases to trigrams, and estimates phrase translation probabilities and lexical weightings from cross-lingual embeddings.These simplifications may compromise performance.
- Backtranslation: Iterative refinement generates synthetic parallel data by translating monolingual text with the initial system, then trains and tunes a standard SMT system in the opposite direction.The refined system uses standard word alignment and is not subject to the initial restrictions.
- Experimental setup: The procedure is computationally expensive, so experiments use 2 million training sentences per monolingual corpus, 10,000 validation sentences, and three iterations.The original language model is reused from the full corpus.
- Results: Table 1 compares the proposed method with existing unsupervised NMT systems using BLEU.The table reports results for the proposed method and comparison systems.
6 Experiments and results
Experiments on WMT 2014 French-English and German-English show that the unsupervised phrase-based SMT system substantially outperforms previous unsupervised systems and approaches supervised SMT.
- 749 million French, 1,606 million German, and 2,109 million English tokens from News Crawl monolingual corpora were used for training.
- Main results: Around 10 BLEU points were gained over previous unsupervised NMT systems for French-English, and more than 7 BLEU points for German-English.
- Main results: Improvements exceeded 50% in all cases, suggesting phrase-based SMT may suit unsupervised translation better than previous NMT-based approaches.
- Ablation analysis: Unsupervised MERT improved French-English results by over 1 BLEU point, but helped German-English less and did not affect English-German.
- Ablation analysis: Iterative refinement improved results by over 2.5 BLEU points in all cases and up to 5 BLEU points, with most gains in the first iteration.
- Comparison with supervised systems: The unsupervised system stayed below 5 BLEU points from comparable Moses+Europarl systems, with differences as small as 2.5 points.
- Comparison with supervised systems: Removing lexical reordering and restricting phrase-table vocabulary cost less than 1 BLEU point, while unsupervised tuning lost under 2.5 points in the worst case.
- Translation examples: Randomly selected French-to-English examples were generally accurate and fluent, despite some adequacy and grammaticality issues.
7 Related work
The paper relates unsupervised SMT to statistical decipherment, unsupervised NMT, and methods that use monolingual data to enhance supervised systems. It differs by learning a fully featured phrase-based SMT system from monolingual corpora alone.
- Statistical decipherment models translation as a noisy-channel process, but has mainly been demonstrated in limited, often word-level settings.
- Unsupervised NMT trains attentional encoder-decoder systems from monolingual corpora using shared or separate encoders, denoising, backtranslation, and sometimes adversarial training.
- Backtranslation and cross-lingual embeddings have previously enhanced systems trained on parallel data, whereas this method learns a fully featured phrase-based SMT system from monolingual data alone.
8 Conclusions and future work
The paper presents an unsupervised phrase-based SMT system trained from monolingual corpora, combining cross-lingual phrase embeddings with language modeling, tuning, and iterative backtranslation. Experiments show substantial gains over previous unsupervised approaches and a reduced gap to supervised SMT.
- Cross-lingual phrase embeddings induce a phrase table, which is coupled with an n-gram language model and distance-based distortion.
- Unsupervised MERT tuning improves the system, while iterative backtranslation provides additional improvements.
- Above 10 and 7 BLEU points are obtained over previous NMT-based approaches on WMT French-English and German-English, respectively.
- The gap with supervised SMT, using Moses trained on Europarl, is reduced to 2-5 points.
- Future work targets semi-supervised settings with small parallel corpora and hybrid systems combining unsupervised SMT with NMT.