PaperlayerView exact revision on arXiv

Source-linked AI summary

Phrase-Based & Neural Unsupervised Machine Translation

Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, Marc'Aurelio Ranzato

arXiv:1804.07755v2cs.CL

TL;DR

Large parallel corpora remain a major obstacle for applying machine translation broadly, motivating methods that learn from monolingual data. This paper develops unsupervised neural and phrase-based systems using shared initialization principles and iterative training, achieving state-of-the-art results against prior unsupervised approaches.

  • Problem

    Current machine-translation learning algorithms rely on large parallel corpora, although most language pairs have little or no parallel data.

  • Method

    The paper applies shared initialization principles to neural and phrase-based systems, including joint BPE processing, inferred bilingual dictionaries, and iterative back-translation.

  • Results

    The systems largely outperform previous unsupervised baselines across language pairs and directions, with combined PBSMT and NMT reaching 25.2 BLEU on de → en.

  • Takeaways & Limitations

    PBSMT often outperforms NMT in fully unsupervised translation, while combining both systems can greatly outperform previous approaches.

  • Takeaways & Limitations

    It remains open whether more effective principles exist and under what conditions the iterative process is guaranteed to converge.

Abstract

from arXiv · show

Machine translation systems achieve near human-level performance on some languages, yet their effectiveness strongly relies on the availability of large amounts of parallel sentences, which hinders their applicability to the majority of language pairs. This work investigates how to learn to translate when having access to only large monolingual corpora in each language. We propose two model variants, a neural and a phrase-based model. Both versions leverage a careful initialization of the parameters, the denoising effect of language models and automatic generation of parallel data by iterative back-translation. These models are significantly better than methods from the literature, while being simpler and having fewer hyper-parameters. On the widely used WMT'14 English-French and WMT'16 German-English benchmarks, our models respectively obtain 28.1 and 25.2 BLEU points without using a single parallel sentence, outperforming the state of the art by more than 11 BLEU points. On low-resource languages like English-Urdu and English-Romanian, our methods achieve even better results than semi-supervised and supervised approaches leveraging the paucity of available bitexts. Our code for NMT and PBSMT is publicly available.

1 Introduction

This work addresses the reliance of machine translation on large parallel corpora by learning translation from monolingual data alone. It combines neural and phrase-based models with shared principles and reports strong benchmark and low-resource results.

  • Large parallel corpora limit the applicability of machine translation because most language pairs have little or no parallel data.
  • Fully unsupervised translation relies on monolingual corpora in each language and combines initialization, language modeling, and iterative back-translation.
  • The proposed neural model simplifies prior architectures and loss functions, while the phrase-based model is simple, interpretable, and fast to train.
  • +10 BLEU points are reported for the NMT model and +12 points for the PBSMT model on widely used benchmarks.
  • The methods are also evaluated on distant and low-resource language pairs, including English-Russian, English-Romanian, and English-Urdu.

2 Principles of Unsupervised MT

Unsupervised MT is treated as an ill-posed problem addressed through three connected principles: initialization, language modeling, and iterative back-translation. Together, these principles provide priors, denoising, and training signal using monolingual data.

  • Learning translation from monolingual data alone is ill-posed because multiple associations between source and target sentences are possible.
  • The approach combines suitable initialization, language modeling, and iterative back-translation to construct and improve translation models.
  • Initialization: Initialization provides a prior by starting from approximate translations of words, phrases, or sub-word units.
  • Language Modeling: Language models trained on source and target monolingual data encode how sentences should read and help denoise translation outputs.
  • Iterative Back-translation: Back-translation generates synthetic source sentences for target sentences, turning the unsupervised problem into supervised training.

3 Unsupervised MT systems

The paper develops unsupervised neural and phrase-based MT systems that combine initialization, language modeling, and iterative back-translation, with shared representations as an additional NMT property. The systems use these principles to construct translation models and improve them through repeated synthetic-data training.

  • 3 Unsupervised MT systems: The proposed systems combine careful initialization, language modeling, and iterative back-translation for unsupervised MT.These principles are applied to both NMT and PBSMT, with shared internal representations added specifically for NMT.
  • 3.1 Unsupervised NMT: NMT initialization jointly applies BPE to both monolingual corpora and uses shared token embeddings, avoiding an explicit bilingual dictionary for related languages.For unrelated languages, the method still requires an inferred dictionary to initialize embeddings.
  • 3.1 Unsupervised NMT: NMT language modeling uses denoising autoencoding, with words dropped or swapped before reconstruction on the source and target sides.The source- and target-side objectives compose encoders and decoders operating in their respective languages.
  • 3.1 Unsupervised NMT: Back-translation generates source or target sentences for monolingual data, producing synthetic parallel pairs used to train both translation directions.The reverse model is not differentiated through during optimization; each iteration minimizes language-modeling and back-translation losses.
  • 3.1 Unsupervised NMT: Shared encoder representations act as an interlingua so denoising-language-model benefits transfer to translation from noisy sources.This constraint is intended to prevent language-modeling and translation tasks from using different latent subspaces.
  • 3.2 Unsupervised PBSMT: PBSMT initializes phrase tables with an inferred bilingual dictionary, scores translations with embedding similarities, and keeps smoothed source- and target-side language models fixed.The scoring uses a rotation matrix between embedding spaces and a temperature-like hyper-parameter controlling distribution peakiness.
  • 3.2 Unsupervised PBSMT: PBSMT iteratively translates monolingual data, retrains in the reverse direction, and repeats the process so language-model corrections can improve phrase tables.The method can expand correct phrase coverage and enable phrase swaps over longer spans across rounds.

4 Experiments

The experiments evaluate the proposed NMT and PBSMT systems across five language pairs using monolingual corpora and selected benchmark test sets. They compare unsupervised methods with prior work, varying labeled-data conditions, and phrase-table configurations.

  • 4 Experiments: The study evaluates English-French, English-German, English-Romanian, English-Russian, and English-Urdu.The first two pairs support comparisons with recent unsupervised MT work, while the last three test low-resource or unrelated-language settings.
  • 4.1 Datasets and Methodology: The datasets use WMT News Crawl monolingual data for several languages, augmented Romanian data, and approximately 5.5 million Urdu monolingual sentences.Results are reported on newstest 2014 for English-French and newstest 2016 for English-German, English-Romanian, and English-Russian.
  • 4.1 Datasets and Methodology: NMT uses 60,000 BPE codes, while PBSMT uses true-casing and removes Romanian source-side diacritics.The preprocessing choices address tokenization and inconsistent diacritic use in the Romanian monolingual data.
  • 4.1 Datasets and Methodology: PBSMT constructs a 60-million-pair phrase table from the 300,000 most frequent source phrases and their 200 nearest target neighbors.The phrase pairs are scored using the embedding-based formula described for initialization.
  • 4.1 Datasets and Methodology: A bigram phrase table improves performance by about 1 BLEU point over unigrams, while longer phrases provide no observed improvement.The reported phrase-table examples also show correct unigram-to-bigram and bigram-to-unigram alignments.

4.3 Training

The models use shared architectures and iterative PBSMT training, with default hyper-parameters preferred because small validation sets overfit. Across benchmarks and resource settings, unsupervised systems substantially outperform prior methods and can match supervised PBSMT trained on over 100,000 parallel sentences.

  • Training: The NMT models use LSTM or Transformer cells, while PBSMT uses Moses with default smoothed n-gram language modeling and iterative training.PBSMT translates 5 million randomly sampled source-language sentences per iteration and uses phrases up to length 4 after initialization.
  • Training: 100 labeled validation sentences cause PBSMT to overfit, so the paper uses default hyper-parameters without parallel resources.On en →fr, tuning on 100 sentences yields 26.42 BLEU, versus 27.09 with defaults and 28.02 when tuning on 3,000 sentences.
  • Results: 28.1 BLEU on en →fr lets unsupervised PBSMT outperform the previous best result by more than 11 BLEU points.The systems report large gains across language pairs and directions; on en →de, both PBSMT and NMT exceed the baseline by more than 10 BLEU points.
  • Results: 17.5 to 27.2 BLEU measures the fr →en PBSMT improvement from iteration 0 to iteration 4.Iteration 0 follows unsupervised phrase-table construction, while later iterations use back-translation.
  • Results: 20.2 and 25.2 BLEU are achieved on en →de and de →en by combining PBSMT with NMT.Adding PBSMT-generated data to NMT back-translated data improves the combined system, whereas the reverse bootstrap does not improve PBSMT alone.
  • Results: Unsupervised PBSMT matches supervised PBSMT trained on more than 100,000 parallel sentences.On low-resource languages, it reaches 23.9 BLEU on ro →en and 12.3 BLEU on ur →en under the reported settings.

4.6 Ablation Study

The ablations show that initialization, language modeling, and back-translation data each materially affect iterative PBSMT, while NMT depends critically on back-translation, auto-encoding, and embedding initialization.

  • 4.6 Ablation Study: More iterations only partially compensate for weaker phrase-table initialization, smaller language models, or less monolingual back-translation data.The PBSMT ablation varies initialization quality, language-model training data, and back-translation data across iterations.
  • 4.6 Ablation Study: Language-model quality becomes more influential as PBSMT iterations proceed.The ablation suggests that better initialization and more powerful language models could further improve results.
  • 4.6 Ablation Study: Back-translation and auto-encoding are critical for NMT, without which the system fails to learn.Embedding initialization is also very important; concatenated-corpus BPE embeddings yield a 7-BLEU gain over prior work.

5 Related Work

The work extends fully unsupervised MT by simplifying initialization and transferring its core principles to PBSMT, while distinguishing itself from earlier dictionary-, parallel-data-, and pretraining-dependent approaches.

  • 5 Related Work: Earlier unsupervised MT used linguistic priors or bilingual dictionaries, often on short sentences, limited vocabularies, or supervised phrase tables.These approaches also relied on target-side language models and, in later work, small parallel corpora or dictionary resources.
  • 5 Related Work: Back-translation commonly augments target-side monolingual data, whereas this approach applies it in a fully unsupervised setting.The comparison with dual learning highlights that earlier systems used gradients through the reverse model and relatively large labeled pretraining data.
  • 5 Related Work: The paper contributes a simpler, more effective initialization for related languages and applies three unsupervised principles to PBSMT.It reports that PBSMT outperforms the original NMT and that combining PBSMT with NMT performs even better.

6 Conclusions and Future Work

The paper identifies three principles for fully unsupervised MT, applies them to PBSMT and NMT, and reports strong benchmark and low-resource results. It leaves the effectiveness of alternative principles and convergence conditions open.

  • 6 Conclusions and Future Work: The three principles are applied to PBSMT and NMT, with PBSMT often outperforming NMT and their combination greatly outperforming prior approaches.The approach is evaluated on benchmark, low-resource, and under-explored language pairs, obtaining state-of-the-art results.
  • 6 Conclusions and Future Work: The effectiveness of other principle instantiations and the conditions guaranteeing convergence of the iterative process remain open questions.The authors also identify extension to the semi-supervised setting as future work.

A Supplemental Material

The supplemental material specifies the unsupervised PBSMT algorithm, beginning with bilingual dictionary induction and phrase-table construction alongside language-model training.

  • Unsupervised PBSMT is presented as a distinct algorithm in the supplemental material.
  • The algorithm learns a bilingual dictionary before constructing phrase tables and a language model.

A.1 NMT Ablation study

The NMT ablation study finds that embedding initialization, auto-encoding, and back-translation are central to unsupervised translation performance, while the approach remains robust to architecture choices.

  • Initialization: Random BPE-embedding initialization reaches a BLEU score of 10.5 and converges more slowly to lower accuracy.
  • Loss components: Auto-encoding and back-translation are critical loss components, without which the system fails to learn.
  • Architecture: The approach is robust to the choice of architectures.

A.2 Qualitative study

Qualitative and tabulated evidence shows translation quality improving across unsupervised training iterations, with model combinations producing stronger examples than individual systems.

  • NMT ablation: The reported initialization variants range from 25.1 for concatenated fastText BPE embeddings to 10.5 for random initialization.
  • NMT ablation: Removing decoder sharing or replacing the Transformer with an LSTM yields scores of 24.6 and 24.5, respectively.
  • NMT ablation: The ablation table reports BLEU scores for unsupervised NMT on newstest 2014.
  • Training progression: The PBSMT model improves continuously across iterations, moving from word-by-word translations toward better syntax and quality.
  • Model comparison: German-English examples compare PBSMT, NMT, and PBSMT+NMT, with the combined system performing better than either individual model.
  • Training progression: PBSMT BLEU rises from 15.4 at iteration 0 to 24.7 at iteration 4, while NMT rises from 12.3 after epoch 1 to 24.2 after epoch 42.Table 5 reports intermediate PBSMT and NMT scores during training on the French-English newstest 2014 pair.
Loading 1804.07755v2…