Phrase-Based & Neural Unsupervised Machine Translation

• The work targets a major limitation of current MT: reliance on large parallel corpora, which are scarce or absent for most language pairs.The introduction argues that learning algorithms must better leverage monolingual data to make MT more widely applicable.Source: §1 · p. 1
• Prior fully unsupervised approaches (Lample et al., 2018; Artetxe et al., 2018) use bilingual-dictionary initialization, denoising language models, back-translation, and shared encoder representations.The paper identifies these as common principles underlying the approaches’ success.Source: §1 · p. 1
• The paper identifies a model as its first contribution.Source: §1 · p. 1

2 Principles of Unsupervised MT

• Three components address the ill-posed task of unsupervised MT: suitable translation-model initialization, language modeling, and iterative back-translation.The section presents these as common principles underlying unsupervised MT.Source: §2 · p. 2

Initialization

• Initialization supplies approximate translations of words, phrases, or subword units as a prior, preserving some semantics even when word-by-word translations are poor.This jump-starts learning by narrowing the space of possible solutions.Source: §2 · p. 2

Language Modeling

• Language models trained on source and target monolingual data provide data-driven sentence priors and improve translation through local substitutions and word reorderings.These priors reflect how sentences should read in each language.Source: §2 · pp. 2–3

Iterative Back-translation

• Iterative back-translation couples forward and backward translation, turning monolingual sentences into noisy supervised examples and updating the coupled models as translation improves.The backward model generates a source sentence for each target sentence in the monolingual corpus.Source: §2 · p. 3

3.1 Unsupervised NMT

• NMT initialization jointly tokenizes both monolingual corpora with BPE and uses embeddings learned on the combined corpus, avoiding a bilingual dictionary for related languages.For unrelated languages, the authors say a dictionary is needed to initialize embeddings.Source: §3.1 · p. 3
• NMT combines denoising autoencoding with back-translation: inferred translations of monolingual sentences form synthetic parallel pairs to train both translation directions.The training objective sums language-modeling and back-translation losses, without back-propagating through the model that generated the synthetic data.Source: §3.1 · pp. 3–4
• NMT shares encoder parameters across languages to create a language-robust latent representation; decoder sharing adds regularization, and the decoder’s first token specifies the target language.The shared representation lets denoising language-modeling benefits transfer to translation from noisy sources.Source: §3.1 · p. 4

3.2 Unsupervised PBSMT

• PBSMT initializes source-to-target and target-to-source phrase tables from a bilingual dictionary inferred from monolingual corpora using the method of Conneau et al. (2018).Translation scores use mapped source and target word embeddings.Source: §3.2 · p. 4
• PBSMT alternates synthetic-data generation and supervised training in both directions, using fixed source and target n-gram language models to support iterative improvement.The language model can correct some noisy phrase-table errors during generation; improved translations then yield better, larger phrase tables that support longer phrase swaps.Source: §3.2 · pp. 4–5

4 Experiments

• The experiments compare the two unsupervised approaches with prior methods, semi-supervised systems, and the same models trained with varying amounts of labeled data.They also include an ablation study of the three principles introduced earlier in the paper.Source: §4 · p. 5

4.1 Datasets and Methodology

• The study tests five language pairs, using English-French and English-German for comparison and three other pairs to evaluate PBSMT on low-resource or unrelated languages.The pairs are English-French, English-German, English-Romanian, English-Russian, and English-Urdu; test sets are WMT newstest 2014 for en-fr and newstest 2016 for en-de, en-ro, and en-ru, with LDC corpora for Urdu.Source: §4.1 · p. 5

4.2 Initialization

• Both proposed approaches use fastText embeddings for initialization; PBSMT aligns 300,000 frequent source phrases to 200 target neighbors each, forming 60 million phrase pairs.NMT obtains cross-lingual BPE embeddings by training on concatenated source and target corpora; PBSMT independently embeds each language’s n-grams and aligns them with MUSE.Source: §4.2 · p. 5
• A phrase table of bigrams improves performance by about 1 BLEU point over unigrams, while longer phrases provide no observed improvement.The French-English phrase-table extract shows unigram-to-bigram and bigram-to-unigram alignments.Source: §4.2 · p. 5

4.3 Training

• The training section introduces the models’ architecture and training procedure.The following subsections provide the NMT and PBSMT implementation details.Source: §4.3 · p. 6

4.3.1 NMT

• NMT uses LSTM or Transformer cells with shared encoder-decoder and source-target lookup tables, 512-dimensional embeddings and hidden layers, and greedy decoding.The Transformer has four encoder and four decoder layers; training uses Adam with learning rate 10^-4, β1 = 0.5, and batch size 32.Source: §4.3.1 · p. 6

4.4 Model selection

• On en →fr, PBSMT tuned on 100 parallel sentences scores 26.42 BLEU, versus 27.09 with defaults and 28.02 when tuned on 3,000 sentences.This overfitting motivates using default PBSMT hyper-parameters, without parallel resources, unless otherwise specified.Source: §4.4 · p. 6
• For NMT, round-trip BLEU selection correlates highly with test performance for Transformers, but not always for LSTMs.The experiments therefore select LSTM models using 100 parallel validation sentences and Transformer models using the unsupervised criterion.Source: §4.4 · p. 6

4.5 Results

• 28.1 BLEU on en →fr let unsupervised PBSMT beat the previous best by more than 11 BLEU points, while both systems exceed the baseline by more than 10 on en →de.The reported systems show large gains across language pairs and directions.Source: §4.5 · p. 6, Table 3 · p. 7
• Before iterative back-translation, PBSMT already significantly outperforms previous approaches and can be trained in a few minutes.Source: §4.5 · p. 7
• 17.5 to 27.2 BLEU across iterations for fr →en highlights back-translation’s importance; PBSMT + NMT reaches 20.2 on en →de and 25.2 on de →en, while NMT + PBSMT adds no gain over PBSMT alone.PBSMT + NMT adds PBSMT-generated data to the NMT model’s back-translated data.Source: §4.5 · p. 7, Table 3 · p. 7
• Unsupervised PBSMT matches the performance of its supervised counterpart trained on more than 100,000 parallel sentences.The comparison uses the same architectures trained with parallel sets of varying sizes.Source: §4.5 · p. 7
• 23.9 BLEU on ro →en exceeds Gu et al. (2018)’s method by 1 point, despite that method using 6,000 parallel sentences, a seed dictionary, and a multi-NMT system.The multi-NMT system combines parallel resources from five languages.Source: §4.5 · p. 7
• 16.6 BLEU on ru →en demonstrates results on a distant language, while unsupervised ur →en reaches 12.3 versus 9.8 for supervised PBSMT trained on 800,000 noisy, out-of-domain parallel sentences.For ur →en, the unsupervised system uses only a validation set of 1800 sentences to tune Moses hyper-parameters.Source: §4.5 · p. 7

4.6 Ablation Study

• More iterations only partially offset lower PBSMT phrase-table initialization quality, less language-model training data, or less monolingual data; language-model quality matters more as iterations increase.The ablation varies alignment quality, language-model training sentences, and back-translation sentences for fr →en.Source: §4.6 · p. 7, Figure 3 · p. 8
• NMT depends critically on back-translation and auto-encoding, and BPE embeddings learned on concatenated monolingual corpora yield 7 BLEU points over prior work (Artetxe et al., 2018; Lample et al., 2018).Embedding initialization is also very important.Source: §4.6 · p. 7

Related work

• Prior work often supplemented monolingual data with limited supervision from small parallel sets, related-language bitext, cross-lingual dictionaries, or comparable corpora.Examples include Sennrich et al. (2015a), Gulcehre et al. (2015), He et al. (2016), Gu et al. (2018), Wang et al. (2018), Firat et al. (2016), Johnson et al. (2016), Chen et al. (2017), Zheng et al. (2017), Klementiev et al. (2012), Irvine and Callison-Burch (2014, 2016), Munteanu et al. (2004), and Irvine and Callison-Burch (2013).Source: §5 · p. 8
• Unsupervised MT antecedents reframed translation as deciphering using linguistic prior knowledge or seeded it with a bilingual dictionary, and used target-side language models to improve fluency.Ravi and Knight demonstrated feasibility on short sentences with limited vocabulary.Source: §5 · p. 8
• Earlier methods used bilingual dictionaries, small parallel corpora, and linguistically motivated features; this work instead initializes expanded phrase tables with an inferred bilingual n-gram dictionary.Irvine and Callison-Burch (2014) used monolingual data to expand phrase tables learned under supervision; neural models also improved PBSMT (Cho et al., 2014).Source: §5 · p. 8
• Unlike dual learning, which backpropagates gradients through the reverse model and uses substantial labeled-data pretraining, this approach is fully unsupervised.Back-translation augments training sets with target-side monolingual data (Sennrich et al., 2015a) and is integrated into dual learning (He et al., 2016; Wang et al., 2018).Source: §5 · p. 8
• Compared with recent fully unsupervised MT studies (Lample et al., 2018; Artetxe et al., 2018; Yang et al., 2018), the work proposes simpler initialization and applies three principles to PBSMT, which outperforms the original NMT.Combining PBSMT and NMT achieves even better performance.Source: §5 · p. 8

6 Conclusions and Future Work

• Combining phrase-based and neural systems can greatly outperform prior approaches; the work applies three principles to both and reports state-of-the-art results on benchmarks and low-resource, under-explored language pairs.The authors also find that phrase-based systems often outperform neural systems in fully unsupervised machine translation.Source: §6 · p. 8
• Whether more effective instantiations or additional principles exist, and when the iterative process is guaranteed to converge, remain open questions.The authors identify these as directions for future work.Source: §6 · p. 8
• Future work may extend the approach to the semi-supervised setting.Source: §6 · p. 8

• The appendix also reports an NMT ablation study and example translations.Source: A Supplemental Material · p. 12
• The PBSMT algorithm learns a bilingual dictionary using Conneau et al. (2018).Source: A Supplemental Material · p. 12
• It populates phrase tables using Eq. 3 and learns a language model to build P (0).Source: A Supplemental Material · p. 12
• The algorithm trains models on monolingual datasets, yielding D(i) datasets.Source: A Supplemental Material · p. 12

A.1 NMT Ablation study

• The ablation table reports results for the en→fr task with the Transformer architecture.The study reports these results in Table 4.Source: §A.1 · p. 12
• Using the word-aligned procedure of Lample et al. (2018), based on words and MUSE, gains about 7 BLEU points over the compared initialization.The study reports ablations for the en→fr task using the Transformer architecture.Source: §A.1 · p. 12
• Switching from BPE tokens to words drops BLEU by 4 points.This is an initialization-method ablation.Source: §A.1 · p. 12
• Training BPE embeddings independently by language and mapping them via MUSE (Conneau et al., 2018) drops BLEU by 3 points.This is compared with the study’s embedding initialization procedure.Source: §A.1 · p. 12
• Randomly initialized BPE embeddings converge more slowly and to lower accuracy, reaching a BLEU score of 10.5.The result is reported to stress the importance of initialization.Source: §A.1 · p. 12
• The ablations show that the auto-encoding and back-translation terms in the loss are critical, and that the approach is robust to architecture choice.Source: §A.1 · p. 12

A.2 Qualitative study

• Both PBSMT and NMT failed to adapt to French “langue” polysemy; with both translations present in the unsupervised phrase table, P(language|langue) was 0.92 versus P(tongue|langue) 0.0005.For comparison, a supervised Moses phrase table had P(language|langue) = 0.633 and P(tongue|langue) = 0.0076; the passage says this underlines the importance of the initial unsupervised phrase alignment procedure.Source: §A.2 · p. 12
• Russian-English PBSMT examples mostly retain semantics but include mistakes in grammar, word choice, and entities.Source: §A.2 · p. 12, Table 6 · p. 13
• German-English examples show PBSMT+NMT performing better than PBSMT or NMT individually.Source: §A.2 · p. 12, Table 7 · p. 14
• PBSMT BLEU rose from 15.4 (iteration 0) to 23.7 (iteration 1) and 24.7 (iteration 4), while NMT rose from 12.3 (epoch 1) to 17.5 (epoch 8) and 24.2 (epoch 42).At the end of training, both models generate very good translations.Source: §A.2 · p. 12, Table 5 · p. 13