Source-linked AI summary
Explicit Cross-lingual Pre-training for Unsupervised Machine Translation
Shuo Ren, Yu Wu, Shujie Liu, Ming Zhou, Shuai Ma
TL;DR
Existing cross-lingual pre-training relies on shared BPE spaces that provide limited, implicit information for unsupervised MT. The paper infers n-gram translation tables and trains CMLM to predict translation candidates for masked source n-grams, yielding stronger representations and significantly better unsupervised MT. The method is evaluated on en-fr, en-de, and en-ro, while broader language-pair application remains future work.
Problem
Shared BPE spaces provide limited and implicit cross-lingual information, particularly for n-gram semantics and language pairs with little alphabet overlap.
Source: §1 · p. 1Method
The method infers n-gram translation pairs from cross-lingual embeddings and trains CMLM to predict translation candidates for randomly masked source n-grams, using IBM models for step-wise loss.
Source: §1 · pp. 1–2, §3.3 · p. 4Results
The method significantly improves unsupervised MT, outperforming previous methods across language pairs by an average BLEU score of 1.7 and improving over Lample and Conneau (2019) at p<0.01.
Source: §4.2 · pp. 5–6Takeaways & Limitations
Explicit cross-lingual signals produce stronger context-aware representations that help word alignment, cross-lingual classification, and unsupervised machine translation.
Source: §1 · p. 2, §4.2 · p. 6The paper identifies applying the method to other language pairs and evaluating pre-trained encoders on additional NLP tasks as future work.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Pre-training has proven to be effective in unsupervised machine translation due to its ability to model deep context information in cross-lingual scenarios. However, the cross-lingual information obtained from shared BPE spaces is inexplicit and limited. In this paper, we propose a novel cross-lingual pre-training method for unsupervised machine translation by incorporating explicit cross-lingual training signals. Specifically, we first calculate cross-lingual n-gram embeddings and infer an n-gram translation table from them. With those n-gram translation pairs, we propose a new pre-training model called Cross-lingual Masked Language Model (CMLM), which randomly chooses source n-grams in the input text stream and predicts their translation candidates at each time step. Experiments show that our method can incorporate beneficial cross-lingual information into pre-trained models. Taking pre-trained CMLM models as the encoder and decoder, we significantly improve the performance of unsupervised machine translation.
1 Introduction
Unsupervised MT depends critically on initialization, but existing cross-lingual pre-training derives limited, implicit information from shared BPE spaces. The paper introduces CMLM to provide explicit n-gram translation signals and reports stronger representations and improved unsupervised MT.
1 Introduction
- Shared BPE spaces provide limited and implicit cross-lingual information, especially when languages do not share much of their alphabet.N-gram- and sentence-level semantics may not be properly shared, despite their importance for initializing unsupervised MT models.Source: §1 · p. 1
- The method infers n-gram translation pairs from cross-lingual n-gram embeddings, then uses them to train a Cross-lingual Masked Language Model.CMLM randomly masks source n-grams and predicts corresponding translation candidates.Source: §1 · pp. 1–2
- The proposed novel cross-lingual pre-training method incorporates explicit cross-lingual information into pre-trained models and significantly improves unsupervised machine translation performance.Source: §1 · p. 2
- IBM models calculate step-wise CMLM losses, allowing masked source n-grams and predicted translation candidates to have different lengths.This removes the same-length constraint of standard BERT masking objectives.Source: §1 · pp. 1–2
- The pre-training method produces context-aware cross-lingual representations that help word alignment and cross-lingual classification.The pre-trained models are used as encoder and decoder components for translation.Source: §1 · p. 2
2 Background
BERT pre-trains a bidirectional Transformer encoder through masked-token prediction and next-sentence prediction. XLM extends this framework with shared BPE vocabularies and Translation Language Modeling, while the proposed overview adds n-gram translation-table inference, CMLM pre-training, and iterative MT fine-tuning.
2.1 BERT
- BERT pre-trains a deep bidirectional Transformer encoder with Masked Language Modeling and Next Sentence Prediction.MLM predicts randomly masked input tokens, while Next Sentence Prediction models whether two sentences are adjacent.Source: §2.1 · p. 2
2.2 XLM
- XLM uses a shared BPE vocabulary for potential cross-lingual information and introduces Translation Language Modeling over concatenated parallel sentences.TLM is unavailable for unsupervised MT because parallel sentences are absent.Source: §2.2 · p. 2
- The method overview has three stages: infer n-gram translation tables, pre-train with CMLM, and iteratively fine-tune an MT model using the pre-trained cross-lingual language models.CMLM predicts translation candidates for randomly masked n-grams.Source: §2.2 · p. 2, Figure 2 · p. 3
3 Method
The method infers explicit n-gram translation signals, trains CMLM models to predict translation candidates, and uses them to initialize and iteratively refine unsupervised MT models.
3.1 Overview
- The pipeline infers source-to-target and target-to-source n-gram translation tables from unsupervised cross-lingual n-gram embeddings.Similarity scores become translation probabilities, and top-k target candidates are retrieved for frequent source n-grams.Source: §3.1 · p. 3, §3.2 · p. 3
3.3 Cross-lingual Masked Language Model
- CMLM randomly masks source n-grams and predicts their translation candidates, providing explicit cross-lingual training signals during BERT pre-training.The objective uses retrieved candidates rather than reconstructing the masked source n-grams themselves.Source: §3.3 · p. 4
- IBM Model 2 supplies step-wise training loss and alignment probabilities, allowing masked source and predicted target n-grams to have different lengths.Candidate cross-entropies are weighted by their translation probabilities in the inferred n-gram table.Source: §3.3 · p. 4
- The pre-training procedure alternates MLM and CMLM objectives, uses shared BPE inputs, and includes code-switching with translation candidates during part of MLM training.Fifteen percent of BPE n-grams are sampled, with masked replacements used 70% of the time.Source: §3.3 · p. 4
3.4 Unsupervised Machine Translation
- Two pre-trained cross-lingual language models initialize the encoder and decoder, which are then optimized with denoising auto-encoding and iterative back-translation.A further iteration extracts n-gram pairs from pseudo-parallel data with GIZA++ and updates the translation table; the authors report additional translation gains.Source: §3.4 · pp. 4–5
- Using a pre-trained decoder as a feature extractor with a new attention-equipped decoder improves some translation directions.The authors motivate this variant by noting that decoder pre-training has a smaller effect than encoder pre-training in prior work.Source: §3.4 · p. 5
4 Experiments
The experiments evaluate the proposed pre-training method across three language pairs using NewsCrawl data and specified WMT test sets.
4 Experiments
- The experimental program includes final unsupervised MT results, decoder-use variants, word alignment, cross-lingual classification, and component ablations.The stated evaluation plan examines both downstream translation and cross-lingual modeling capacity.Source: §4 · p. 5
Data and Preprocess
- Experiments cover English-French, English-German, and English-Romanian using available NewsCrawl sentences through 2018.The monolingual data support both pre-training and subsequent unsupervised NMT iterations.Source: Data and Preprocess · p. 5
- The study uses newstest 2014 for English-French and newstest 2016 for English-German and English-Romanian evaluation.The pre-trained starting models are those released by Lample and Conneau (2019).Source: Data and Preprocess · p. 5
- Preprocessing tokenizes with Moses and applies fastBPE using 60,000 shared BPE codes for each language pair.The released BPE codes are used for the subword segmentation.Source: Data and Preprocess · p. 5
Implementation Details
The experiments use a Transformer-based setup and compare the proposed method with six unsupervised MT baselines. The method significantly outperforms prior systems, including XLM, on the reported translation results.
Implementation Details
- The implementation uses a Transformer with 1024 hidden units, 8 heads, GELU activations, 0.1 dropout, Adam optimization, and 256-token streams.The reported setup also uses mini-batches of 64 for MLM and CMLM, averaged multilingual perplexity for stopping, and six Transformer layers for machine translation.Source: Implementation Details · p. 5
Baselines
- The comparison includes six unsupervised MT baselines spanning NMT, PBSMT, hybrid systems, and XLM.The baselines use denoising auto-encoding, iterative back-translation, cross-lingual embeddings, PBSMT integration, posterior regularization, or cross-lingual pre-training.Source: Baselines · p. 5
4.2 Overall Results
- 1.7 average BLEU points over previous methods, with en2fr and ro2en gains exceeding 2 BLEU points.Table 1 reports the overall final unsupervised MT comparison; the proposed method significantly outperforms previous methods on all language pairs.Source: §4.2 · pp. 5–6, Table 1 · p. 6
- A second iteration with updated n-gram tables further improves most translation directions, with en2fr and ro2en gains exceeding 0.5 BLEU points.The authors interpret this as evidence that fine-tuned translation models provide additional cross-lingual information for iterative pre-training enhancement.Source: §4.2 · p. 6
- The proposed method significantly improves translation over Lample and Conneau (2019), with statistical significance at p<0.01.The authors attribute the improvement to stronger and more explicit cross-lingual training signals than those obtained mainly from shared subword spaces.Source: §4.2 · p. 6
4.3 Usage of Pre-trained Decoder
The decoder experiments test direct use of the pre-trained decoder against adding Transformer layers and freezing the pre-trained component. Direct use is generally best, while freezing the pre-trained decoder causes a drastic performance drop.
Usage of Pre-trained Decoder
- The alternative decoder experiments use the pre-trained decoder as a feature extractor and add Transformer layers with encoder-to-decoder attention.The experiments are conducted from the Iter 1 translation models.Source: §4.3 · p. 6
- Directly using the pre-trained model as decoder performs best for most translation directions, except en2fr and ro2en.Adding six Transformer layers improves those exceptions, but the improvement is not significant.Source: §4.3 · p. 6
- Freezing the pre-trained decoder while tuning only added Transformer layers causes a drastic performance drop.This indicates that the decoder representation space requires substantial adaptation during machine-translation tuning.Source: §4.3 · p. 6
4.4 Evaluation of Cross-lingual Pre-trained Encoder
The encoder evaluation measures cross-lingual word alignment using contextual representations and compares CMLM with context-unaware embeddings and XLM. CMLM significantly outperforms XLM, showing stronger source-target representation connections.
Word Alignment
- The alignment procedure feeds parallel English and French sentences through the pre-trained encoder and aligns target words using maximum output similarities.The evaluation uses the HLT/NAACL 2003 English-French alignment dataset.Source: Word Alignment · p. 6
- Table 3 reports precision, recall, F-measure, and alignment error rate for different cross-lingual word-embedding methods.The experiment excludes out-of-vocabulary words and words split apart by BPE operations.Source: Table 3 · p. 7, Word Alignment · p. 7
- CMLM significantly outperforms XLM on cross-lingual word alignment, demonstrating stronger connections between source and target representation spaces.Both contextual pre-training methods outperform context-unaware word-embedding similarity, indicating that context information enhances cross-lingual mapping.Source: Word Alignment · p. 7, Table 3 · p. 7
Cross-lingual Classification
The paper evaluates cross-lingual classification with zero-shot French and German XNLI data after fine-tuning on English NLI. CMLM reaches a new record on this task.
Cross-lingual Classification
- The evaluation adds a linear classifier to the first hidden state and reports zero-shot results for French and German.Only the English NLI training data is labeled and used for fine-tuning.Source: Cross-lingual Classification · p. 7
- CMLM reaches a new record on zero-shot cross-lingual classification for French and German XNLI test sets.The model is fine-tuned using English NLI data without labeled French or German examples.Source: Cross-lingual Classification · p. 7, Table 4 · p. 7
4.5 Ablation Study
The ablation study shows that combining CMLM with MLM and retaining CMLM’s design choices improves unsupervised translation performance.
Ablation Study
- 0.6 to 0.7 BLEU improvement comes from combining CMLM and MLM compared with using either loss alone.The comparison uses the “Iter 1” setting.Source: §4.5 · p. 7, Table 5 · p. 8
- CMLM’s translation prediction and n-gram masking modifications contribute nearly equally, except that n-gram masking has little influence for en2de.Source: §4.5 · p. 8, Table 5 · p. 8
5 Related Work
Related work develops unsupervised translation through shared latent spaces, back-translation, SMT or hybrid systems, and cross-lingual pre-training. The paper positions its method against BPE-based pre-training because that cross-lingual information is implicit and limited.
5 Related Work
- Earlier NMT systems use shared encoders, denoising or adversarial training, and iterative back-translation, while SMT and hybrid systems initialize from cross-lingual word or n-gram embeddings.Source: §5 · p. 8
- Lample and Conneau (2019) use BERT-based cross-lingual language-model pre-training with two language models as encoder and decoder, achieving state-of-the-art en-fr and en-de performance.Source: §5 · p. 8
- The paper identifies shared BPE spaces as a limitation because they provide mostly implicit and limited cross-lingual information.Source: §5 · p. 8
- Song et al. (2019) jointly pre-train encoder and decoder components, but their cross-lingual information also comes from BPE sharing; the proposed method could be combined with theirs in future multi-task training.Source: §5 · p. 8
6 Conclusion
The paper proposes CMLM-based cross-lingual pre-training with explicit cross-lingual information and reports effectiveness on en-fr, en-de, and en-ro. Future work will extend evaluation to other language pairs and NLP tasks.
6 Conclusion
- Experimental results on en-fr, en-de, and en-ro demonstrate the effectiveness of the proposed CMLM-based cross-lingual pre-training method.Source: §6 · p. 8
- Future work may apply the pre-training method to other language pairs and evaluate pre-trained encoders on tasks such as Named Entity Recognition.Source: §6 · p. 8