PaperlayerView exact revision on arXiv

Source-linked AI summary

Multilingual Denoising Pre-training for Neural Machine Translation

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer

arXiv:2001.08210v2cs.CL

TL;DR

Machine translation had limited self-supervised pre-training of complete multilingual sequence-to-sequence models. This paper introduces mBART, which denoises full multilingual texts and is fine-tuned across MT settings. It reports broad gains, including up to 12 BLEU points in low-resource translation, while enabling transfer across and beyond pre-training languages.

  • Problem

    Existing MT pre-training commonly covers only model components, parts of text, or English, leaving complete multilingual sequence-to-sequence pre-training limited.

  • Method

    mBART pre-trains a multilingual sequence-to-sequence denoising auto-encoder on large-scale monolingual corpora using phrase masking and sentence permutation.

  • Results

    mBART consistently improves supervised, document-level, and unsupervised MT, with gains up to 12 BLEU points in low-resource settings and new transfer across language pairs.

  • Takeaways & Limitations

    A single multilingual pre-trained model can be fine-tuned for supervised and unsupervised sentence- and document-level MT without task-specific modifications.

  • Takeaways & Limitations

    Fine-tuning unseen source languages is more difficult, and the model’s size makes production deployment expensive.

Abstract

from arXiv · show

This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART -- a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective. mBART is one of the first methods for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task-specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show it also enables new types of transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.

1 Introduction

mBART addresses gaps in machine-translation pre-training by denoising complete multilingual texts with a full autoregressive sequence-to-sequence model. Its initialization improves supervised, document-level, and unsupervised translation, while enabling transfer across and beyond pre-training languages.

  • Existing MT pre-training typically targets only the encoder, decoder, parts of text, or English-only corpora.
  • mBART pre-trains a complete autoregressive sequence-to-sequence model by adding noise to and reconstructing full texts across many languages.
  • Up to 12 BLEU points of gains occur across low- and medium-resource supervised sentence-level pairs, while high-resource performance is not sacrificed.
  • Document-level pre-training improves results by up to 5.5, and unsupervised translation gains include 9.5 BLEU on Nepali-English.
  • mBART supports transfer from one trained language pair to other pre-training languages and benefits languages absent from the pre-training corpus.

2 Multilingual Denoising Pre-training

The mBART pre-training setup uses multilingual Common Crawl data, BART-style denoising, and multi-sentence instances to support later sentence- and document-level translation. A family of multilingual, bilingual, monolingual, and random baselines enables analysis of multilinguality and pre-training effects.

  • Data: The pre-training corpus is CC25, a 25-language Common Crawl subset containing languages from different families and varied corpus sizes.
  • Data: Language data are rebalanced using λ_i based on each language’s corpus proportion, with smoothing parameter α = 0.7.
  • Pre-processing: The shared SentencePiece vocabulary contains 250,000 subword tokens and also supports fine-tuning on additional languages.
  • Model: mBART applies phrase masking and sentence permutation, then trains one Transformer to reconstruct the original multilingual text.
  • Model: Multi-sentence instances pack consecutive sentences up to 512 tokens, enabling pre-training for both sentence and document translation.
  • Pre-trained Models: The study compares mBART25, mBART06, mBART02, monolingual BART baselines, and randomly initialized models to assess multilinguality and initialization.

3 Sentence-level Machine Translation

mBART provides consistent gains in low- and medium-resource sentence-level MT, including with back-translation, while multilingual pre-training outperforms alternative pre-training schemes. Its benefits depend on data availability and transfer setting, weakening in high-resource tasks and when both languages are unseen.

  • 3 Sentence-level Machine Translation: mBART improves low- and medium-resource sentence-level MT, with gains of 12+ BLEU on pairs such as En-Vi and En-Tr.Fine-tuning fails in the extremely low-resource En-Gu setting, where unsupervised translation is suggested as more appropriate.
  • 3 Sentence-level Machine Translation: mBART initialization improves BLEU at every back-translation iteration on low-resource FLoRes pairs, producing new state-of-the-art results in all four directions.The experiments use the same monolingual data as prior work to generate back-translated data.
  • 3 Sentence-level Machine Translation: mBART outperforms other pre-trained models on WMT16 Ro-En, with additional gains from back-translation and a new state-of-the-art result.Conventional BART trained only on English and Romanian improves over baselines but performs worse than mBART, indicating the importance of multilingual pre-training in this comparison.
  • 3 Sentence-level Machine Translation: Pre-training on more languages helps most when target-language monolingual data is limited, but can slightly hurt results by less than 1 BLEU when data is plentiful.The authors suggest that additional languages may reduce capacity available for each test language; similar-language pre-training is particularly helpful for Ro-En.
  • 3 Sentence-level Machine Translation: Pre-training continues improving Ro-En performance by over 3 BLEU after the first 25K steps and has not fully converged after 500K steps.After 25K steps, both models already outperform the best baseline, whereas no pre-training causes overfitting and much worse performance.
  • 3 Sentence-level Machine Translation: mBART transfers to unseen languages, including Arabic, German, and Dutch, while performance is worse when both source and target languages are unseen.Similar improvements for Arabic and Dutch-English suggest that pretrained Transformer layers generalize with minimal lexical overlap; unseen source languages remain more difficult.

4 Document-level Machine Translation

mBART pre-training improves document-level machine translation by learning dependencies across sentences, without task-specific architectural techniques. Pre-trained document models outperform random initialization and HAN on the evaluated En-De and Zh-En tasks.

  • 4 Document-level Machine Translation: Pre-training on document fragments up to 512 tokens improves document-level translation by modeling dependencies between sentences.The approach packs consecutive sentences during pre-training and supports document translation directly.
  • 4.1 Experimental Settings: Evaluation covers WMT19 En-De and TED15 Zh-En, using document-level BLEU where translated segments cannot be aligned for randomly initialized Doc-MT.For En-De, aligned outputs allow both sentence-level and document-level BLEU evaluation; only document-level BLEU is reported for the unaligned case.
  • 4.2 Main Results: mBART25 document models outperform randomly initialized models by large margins on both sentence-level and document-level training.
  • 4.2 Main Results: mBART25 models outperform HAN despite using no document-level customization.HAN uses hierarchical attention within and across sentences, whereas the mBART models are not customized for document-level MT.
  • 4.2 Main Results: mBART25 Doc-MT outperforms its sentence-level counterpart on En-De and En-Zh, while randomly initialized Doc-MT models fail to work effectively.The performance gaps indicate that pre-training is critical for document-level performance in these experiments.

5 Unsupervised Machine Translation

The paper evaluates mBART for unsupervised translation without target-pair bitext, using back-translation and language transfer. mBART produces large gains over non-pretrained systems, supports transfer across language pairs, and improves further when transfer and back-translation are combined.

  • 5 Unsupervised Machine Translation: mBART provides a simple and effective initialization for unsupervised translation methods when no bitext is available.The paper evaluates both back-translation and language transfer regimes.
  • 5.2 Language Transfer: mBART enables language transfer when the target language appears in bitext with another source language, including settings where the source language has no bitext.A model fine-tuned on one pair is directly applied to other pairs.
  • 5.1 Back-Translation: mBART models achieve large gains over non-pretrained models in all evaluated back-translation directions.They outperform XLM significantly on dissimilar pairs En-Ne and En-Si, where existing approaches fail, and perform well against XLM and MASS on similar pairs.
  • 5.2 Language Transfer: Randomly initialized transfer models consistently obtain approximately 0 BLEU, unlike mBART-based models.This comparison supports the importance of multilingual pre-training for transfer across languages.
  • 5.2 Language Transfer: Language transfer works better for most pairs within the same language family, but significant vocabulary sharing is not required.Results include effective transfer from Zh-En to Ko-En and It-En to Ar-En.
  • 5.2 Language Transfer: Combining language transfer with one iteration of back-translation improves results for all evaluated pairs.The procedure starts from the best transferred model and applies iterative back-translation on the pre-training monolingual corpus.

6 Related Work

The work builds on self-supervised text generation, multilingual representation learning, multilingual translation, document translation, and unsupervised translation. It differs by applying multilingual denoising pre-training to a complete Seq2Seq model and demonstrating document-level gains without task-specific techniques.

  • Pre-training for Text Generation: mBART extends self-supervised text-generation pre-training from English-focused or partial objectives to multilingual denoising of complete Seq2Seq models.Prior approaches pre-trained encoders or used objectives reconstructing only parts of text.
  • Multilinguality in NLP tasks: The paper focuses on multilingualism during pre-training and fine-tunes models on individual translation pairs, rather than jointly training multilingual translation directions.
  • Document Translation: Prior document-translation work used task-specific techniques or constrained contexts, while this work reports improved document-level translation with standard Seq2Seq learning.
  • Unsupervised Translation: The paper categorizes unsupervised translation into three regimes and focuses on the first and third kinds.These include settings with no parallel corpus and settings where bitext exists for another source language into the target.

7 Conclusion

The paper concludes that multilingual denoising pre-training improves supervised and unsupervised MT at both sentence and document levels, while also enabling transfer across languages. It identifies model size and deployment efficiency as directions for future work.

  • 7 Conclusion: Multilingual denoising pre-training significantly improves supervised and unsupervised machine translation at both sentence and document levels.
  • 7 Conclusion: mBART representations support transfer learning across language pairs and can be combined with approaches such as back-translation.The paper also analyzes when and how pre-training is most effective.
  • 7 Conclusion: The model is expensive to deploy in production because of its size, motivating future work on more efficient models.The authors also propose scaling pre-training to more languages, such as mBART100.

A Evaluation Details

The evaluation uses BLEU scores, with language-specific tokenization and normalization procedures applied to translation outputs before scoring.

  • Evaluation Metric: BLEU scores evaluate translation performance across the paper's tasks.System outputs and references are generally scored over tokenized text.
  • Language-specific Processing: Indic outputs are tokenized with the Indic-NLP Library.
  • Language-specific Processing: Japanese texts are segmented with KyTea.
  • Language-specific Processing: Korean texts are segmented with Mecab-Ko using its default dictionary.
  • Language-specific Processing: Arabic, Burmese, and Romanian outputs use QCRI normalization, an official Burmese segmentation tool, and Moses tokenization with Romanian normalization, respectively.
  • Language-specific Processing: Chinese outputs use the official sacreBLEU Chinese tokenizer, while other languages use sacreBLEU's DEFAULT tokenization.

B Translation Examples

The examples illustrate document-level translation improvements and language transfer in unsupervised machine translation with mBART-based systems.

  • Document-level Translation: Doc-MT produces more fluent and coherent translations than Sent-MT in the illustrated TED15 Zh-En example.The example says Doc-MT is closer to the reference and connects sentences with several “And” tokens, whereas Sent-MT translates sentences independently.
  • Document-level Translation: Sent-MT lacks global knowledge and produces sentences independently in the document-level example.
  • Unsupervised Language Transfer: mBART25 fine-tuned on any one of Ja, Ko, or Zh to En directly translates well in the other two language pairs.The example presents this as unsupervised MT via language transfer.
  • Unsupervised Language Transfer: The language-transfer examples involve languages with different character sets and syntactic structures but cultural and historical correlations.
Loading 2001.08210v2…