PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-lingual Language Model Pretraining

Guillaume Lample, Alexis Conneau

arXiv:1901.07291v1cs.CL

TL;DR

The paper asks whether generative pretraining can provide effective cross-lingual representations beyond English. It introduces unsupervised monolingual-data objectives and a supervised parallel-data objective, achieving state-of-the-art results across classification and machine translation.

  • Problem

    Existing generative pretraining and sentence-representation research was largely monolingual and focused on English benchmarks, motivating cross-lingual pretraining across multiple languages.

  • Method

    The paper trains cross-lingual language models with CLM or MLM from monolingual data and adds TLM, an objective using parallel sentences, when parallel data is available.

  • Results

    The approach significantly outperforms prior state of the art across cross-lingual classification, unsupervised machine translation, and supervised machine translation.

  • Takeaways & Limitations

    Cross-lingual language-model pretraining provides strong cross-lingual features for classification and both supervised and unsupervised machine translation.

Abstract

from arXiv · show

Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we obtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.

1 Introduction

The paper extends generative language-model pretraining beyond English to cross-lingual representations, introducing unsupervised and supervised objectives and evaluating them across major cross-lingual tasks.

  • Motivation: The work addresses research that has been largely monolingual and English-focused by developing general-purpose cross-lingual language-model pretraining.It evaluates cross-lingual understanding and related applications across multiple languages.
  • Contributions: The unsupervised method learns cross-lingual representations with cross-lingual language modeling using monolingual pretraining objectives.The method requires only monolingual data.
  • Contributions: The supervised method introduces a new learning objective that improves cross-lingual pretraining when parallel data is available.The objective complements the monolingual-data approach.
  • Results: The approach significantly outperforms previous state of the art on cross-lingual classification, unsupervised machine translation, and supervised machine translation.These results span both understanding and translation benchmarks.
  • Results: Cross-lingual language models also provide significant perplexity improvements for low-resource languages.This is reported as an additional benefit beyond the main benchmark results.
  • Availability: The authors state that code and pretrained models will be made publicly available.This supports reuse of the released resources.

2 Related Work

Prior work aligned multilingual word or sentence representations and used multilingual translation models, but successful cross-lingual methods often relied on substantial parallel data.

  • Word representations: Earlier research aligned word-embedding distributions across languages using dictionaries, transformations, and multilingual alignment techniques.These approaches established cross-lingual representation learning before language-model pretraining.
  • Sentence representations: Multilingual sentence-representation work used parallel data for zero-shot cross-lingual sentence classification.This line of research targets shared sentence representations across languages.
  • Multilingual translation: A shared multilingual sequence-to-sequence model enabled translation across many language pairs and improved low-resource translation results.The model used a shared LSTM encoder and decoder.
  • Parallel-data methods: Artetxe and Schwenk used more than 200 million parallel sentences and achieved state-of-the-art XNLI classification with fixed sentence representations.Their approach learned a classifier on top of the encoder representations.
  • Language-model alignment: Wada and Iwata trained a shared-parameter LSTM language model with separate lookup tables to align word representations.They evaluated the approach on word translation tasks.

3 Cross-lingual language models

The paper defines monolingual and parallel-data objectives for cross-lingual language modeling, using shared subword processing and sampling choices designed to support low-resource languages.

  • Shared vocabulary: Languages share a BPE vocabulary learned from randomly sampled sentences concatenated across monolingual corpora.Shared vocabularies improve alignment across languages with common alphabets or anchor tokens.
  • Shared vocabulary: Sampling with α = 0.5 increases the representation of low-resource languages and prevents their words from being split at the character level.The distribution reduces bias toward high-resource languages.
  • Unsupervised objectives: Causal language modeling predicts each word from the preceding words in a sentence using a Transformer language model.The objective is trained without parallel data.
  • Unsupervised objectives: Masked language modeling randomly masks BPE tokens in continuous text streams and trains the model to predict them from context.The masking procedure replaces tokens with [MASK], random tokens, or unchanged tokens at specified rates.
  • Supervised objective: Translation language modeling extends masked language modeling to concatenated parallel sentences, letting predictions attend across source and target languages.Reset target-sentence positions further facilitates representation alignment.
  • Objective comparison: CLM and MLM require only monolingual data, whereas TLM leverages parallel data when available.The paper combines MLM and TLM for cross-lingual pretraining.

4 Cross-lingual language model pretraining

The pretrained cross-lingual models are applied to classification, machine translation, low-resource language modeling, and related representation tasks through task-specific initialization and objectives.

  • Cross-lingual classification: The paper uses cross-lingual language models to initialize zero-shot cross-lingual sentence encoders.XNLI evaluation adds a linear classifier to the pretrained Transformer and fine-tunes all parameters.
  • Machine translation: Cross-lingual pretraining initializes supervised and unsupervised neural machine translation systems.The unsupervised setup pretrains the entire encoder and decoder to bootstrap iterative translation.
  • Word representations: The framework additionally supports unsupervised cross-lingual word embeddings.This is listed as a further application of the pretrained models.
  • Pretraining mechanism: The MLM and TLM objectives align representations by allowing masked-token prediction to use both sides of a parallel sentence pair.Figure 1 illustrates this cross-lingual attention mechanism.
  • Machine translation: The paper evaluates CLM and MLM pretraining for supervised multilingual translation on WMT’16 Romanian-English.Both pretraining objectives are compared in this setting.
  • Low-resource language modeling: Cross-lingual language models are also used to improve language modeling for low-resource languages by enriching Nepali training with Hindi and English data.The Nepali example compares language-model perplexity under different data combinations.

5 Experiments and results

Experiments evaluate cross-lingual language-model pretraining across classification, unsupervised and supervised translation, low-resource language modeling, and word similarity. The results show consistent gains from MLM and TLM pretraining, including state-of-the-art performance across several benchmarks.

  • Cross-lingual classification: Cross-lingual classification evaluates unsupervised MLM pretraining and supervised MLM+TLM pretraining on 15 XNLI languages.The supervised model adds parallel data through the TLM objective.
  • Cross-lingual classification: 71.5% average accuracy makes fully unsupervised MLM a new zero-shot cross-lingual classification state of the art.It outperforms Artetxe and Schwenk’s 70.2% result despite their use of 223 million parallel sentences.
  • Unsupervised machine translation: Unsupervised translation improves across English-French, English-German, and English-Romanian when full models are initialized with cross-lingual language pretraining.MLM pretraining outperforms CLM, including increases from 30.4 to 33.4 BLEU on English-French and from 28.0 to 31.8 on Romanian-English.
  • Supervised machine translation: 38.5 BLEU on WMT’16 Romanian-English is obtained by the bidirectional model with back-translation and MLM pretraining.This exceeds the previous SOTA of Sennrich et al. by more than 4 BLEU, while MLM gives the best performance among pretraining objectives.
  • Low-resource language modeling: 109.3 Nepali perplexity results when additional English and Hindi data are combined, versus 157.2 with Nepali-only modeling.Hindi alone reduces perplexity by 41.6 points, while English alone reduces it by 17.1 points.
  • Unsupervised cross-lingual word embeddings: XLM reaches a 0.69 Pearson correlation on SemEval’17 cross-lingual word similarity, outperforming MUSE and Concat.The comparison uses the same word vocabulary and translation pairs from MUSE dictionaries.

6 Conclusion

Cross-lingual language model pretraining provides strong cross-lingual features across understanding and translation tasks, using both monolingual-only and parallel-data objectives. It achieves state-of-the-art machine translation results and improves cross-lingual classification, language-model perplexity, and word embeddings.

  • 34.3 BLEU on WMT’16 German-English establishes a new unsupervised machine translation state of the art, exceeding the previous best approach by more than 9 BLEU.
  • 38.5 BLEU on WMT’16 Romanian-English establishes a new supervised machine translation state of the art, improving on the previous best approach by more than 4 BLEU points.
  • CLM and MLM trained only on monolingual corpora provide strong cross-lingual features for model pretraining.MLM pretraining is especially effective for unsupervised machine translation.
  • Cross-lingual language models also improve Nepali language-model perplexity and provide unsupervised cross-lingual word embeddings.
Loading 1901.07291v1…